As I write this, it seems that my fears on Friday have not come true over the weekend. To date, things have been quiet. Nevertheless, we still expect a storm. We just don’t know when. While we wait, let’s consider the story so far.
The MS12-020 RDP vulnerability
On Tuesday 13 March, Microsoft released updates on their regular monthly cycle, including one addressing a vulnerability in the Windows Remote Desktop Protocol (RDP) service. This is remotely exploitable by an unauthenticated user. Microsoft themselves rate this as a “critical” vulnerability and likely to be exploited. In posting our own bulletin on Tuesday evening (sorry, only available within the University network), we drew particular attention to this, knowing that RDP is widely used within the University.
In the past there have been many notable computer worms – self-propagating malicious software. Examples include Code Red, Blaster, and SQL Slammer – the last one significant enough to paralyse a significant proportion of the global internet. It has been a while since we have seen a major new internet worm. In recent years there have been two major factors at work: firstly, Windows has shipped with a firewall enabled by default since the release of XP Service Pack 2 in 2004; secondly, attacks are increasingly the result of criminal gangs seeking to profit from their work, resulting in a trend towards stealthier malware. The most major worm in recent years has been Conficker, something that we still see on our network from time to time, particularly on systems belonging to visitors to the University.
Developments during the week
After internal discussion on Wednesday, we decided to perform a scan of the University network the following lunchtime for systems listening on the standard Microsoft RDP port, tcp/3389. We knew that RDP was widely used within the University but quite how much, we weren’t sure. We scanned all systems from within the University network initially, with followup probes from an external host as we suspected a common configuration would be for systems to be accessible from within the University but not open to the world at large. Our results showed several thousand systems listening on port 3389 and we proceeded to pass the results to the relevant IT officers on Thursday afternoon and Friday morning. Our scans could not determine whether systems had been patched against MS12-020 or even what software was listening on that port, merely that the port in question was open.
Some may ask why RDP isn’t firewalled within the university. We have always operated an extremely open network, with few centrally-imposed restrictions on what people may run; any attempt to impose restrictions tends to result in howls of protest. Only a few very specific threats are blocked across the site, and experience has shown us that anything for which we grant exceptions will tend to impose massive management overheads. Additionally, a perimeter firewall cannot protect against threats from elsewhere on the University network. It is therefore left to individual colleges and departments to operate their own firewalls according to their local requirements. From the centre, we can recommend particular configurations, but not dictate.
Microsoft themselves announced on Wednesday that they expected exploitation of MS12-020 “within thirty days”. If it took, say, a couple of weeks for the bad guys to develop an exploit then this is clearly of concern but gives IT staff a reasonable amount of time to address the problem, taking into account needs to test patch deployments and schedule necessary downtime.
The threat as seen on Friday
Unfortunately, the situation appeared to have changed by Friday lunchtime: reports stated that “proof of concept” exploit code had already appeared on message boards in China. Moving from proof-of-concept to a fully “weaponised” system may be a matter of days or even hours going by past experience. With hours to go before the weekend, this was cause for concern: under 72 hours had occurred since Microsoft had released patches and published their bulletin, exploit code had seemingly already been produced, and the weekend was fast approaching.
We felt that the chances of significant malicious activity before Monday morning were high enough to warrant a strong warning to IT officers. Something that would get their attention to the seriousness of the situation. As we stated, if our fears came true, there was a high probability of major disruption by Monday, requiring major effort to resolve. If our fears were unfounded, efforts had merely gone into securing systems sooner than might otherwise have been the case.
The important message to get across was to ensure all systems were patched if at all possible; if this wasn’t feasible before the weekend, to restrict access. A vulnerable system poses far less of a risk if it is only accessible from a handful of trusted hosts than if it is accessible from anywhere on the global internet.
Systems running RDP will not be a randomly-selected set of systems across the University, but will be heavily biased towards key assets: servers, desktops used by senior staff members, systems holding valuable or sensitive information. Compromises of such systems would clearly be far more damaging and time-consuming to address than compromises of a typical student-owned laptop.
The situation now
Now, the weekend has gone by without significant incident, as far as we are aware, although there has been a further development. On Friday evening, it became evident that the published exploit code was not the work of the “bad guys”. Instead, the code appeared to be that supplied to Microsoft by the researcher who originally discovered the vulnerability, and this had somehow been leaked. Possible sources for the leak are the discoverer, Microsoft or a partner in the Microsoft Active Protections Program, which shares vulnerability information with (supposedly) trusted industry partners; in time we may perhaps see more details of what actually happened.
What none of us can know is what is yet to come; all we can do is make educated guesses. We must still assume that an RDP worm will appear at any moment, The threat is real, it is still there. It is critical to ensure that appropriate protection is in place as soon as reasonably possible. Security researcher Dan Kaminsky estimates around 5 million globally-accessible systems running RDP; even if many have been patched, a very significant number remain vulnerable.
What lessons can we learn?
Much of what we have been saying to IT officers is not new. For years we have been recommending that services are not made globally-accessible unless absolutely necessary. For instance, a public webserver has to be able to serve pages to the world. It probably doesn’t require RDP to be globally accessible. Those needing RDP access most likely only come from a handful of hosts or networks, perhaps only internal to the University, perhaps their home addresses, perhaps also some external contractor has access. There may be a requirement to support mobile users or users without fixed IP addresses. Nevertheless for such users, perhaps a facility such as VPN would be acceptable, providing a “layered” security model. Successful attack requires compromising the VPN (or at least one VPN account) and the RDP daemon. This is unlikely to be a deterrent to targetted attacks but will guard against the random attacks that would result from worm activity.
Allowing access from within the University network alone is far better than allowing from anywhere, but is still inherently risky and is not an alternative to keeping systems patched. Such systems are likely to be attacked as soon as another host within the University network is infected, or a pre-infected host is brought within the University network. We have seen this in the past and will inevitably see it again.
A flexible approach is required to patch management. Some may be happy for patches to be applied as soon as they are released; others may be more conservative and wish to test things before deployment to production systems. This is perfectly understandable; updates certainly have been known to cause unexpected problems, as indeed happened last week on a major University service when another vendor’s updates were applied. If policy is that patches are not immediately applied but must await testing and/or a scheduled maintenance window, then it is useful to have plans for addressing emergencies. Can procedures be fast-tracked in the event of a “zero day” vulnerability, or a significant probability of exploitation in the very near future, where the risks of inaction outweigh the risks of an untested update or of inconveniently-timed disruption to service?
No doubt there will be further lessons to be learned when (we still consider it a “when”) the attacks come. No doubt some systems will have been missed. With so many threats to University IT systems around we cannot go overboard on one particular risk at the expense of neglecting the others. For now, we feel we’ve done what we can given our resources, and must move on.