Visibility and context are your priorities

The following text is from a entry in David Lacey's Security Blog for Computer Weekly:

"I've just installed the latest critical security patch from Microsoft. Fortunately, I was warned about its release by good friends in Seattle. In fact, it's unusual these days for Microsoft to release out-of-band updates. One would hope that most of these could safely await the regular 2nd Tuesday update cycle. That's an easy date for London based security managers to remember, as it's exactly a week after the regular City booze up.  

"But clearly there's something urgent about this patch. Either it's really damaging, or there's an exploit already circulating. Whatever the reason, the implication is that, these days, you have to be on your guard 24 by 7 to maintain security. That means you have to establish really good intelligence feeds. I've long said that visibility and context are the cornerstones of good security. You must be equipped to see new threats, exposures and incidents. And you must be able to assess their significance in real time.

"In fact, this is the basis of professional security. Regardless of what the textbooks tell you, the first thing you must do is set up an effective intelligence system: one that reports new threats, existing vulnerabilities and current incidents. And one that can assess the significance of everything reported. All of this is possible and achievable, within reasonable cost and budget, by sensibly exploiting today's technology and services. So, if you, or your staff, didn't immediately pick up and respond to this latest scare, then you should aim to raise your game right now."

In the SWWARP, a High rated advisory was sent to all members as soon as the WARP operators heard of the new update. (This was around 18:30 on a Thursday evening). The email was waiting in members' inboxes when they arrive at work the following day. By 10am, one of the SWWARP members started a discussion on the advice brokering forum, informing other members that the SANS institute had inscreased their ThretCon level in light of the patch. This is a rare occurance, and varified the importantance of the initial critical rated advisory that was sent initially. One of the members who had the chance to listen to the Microsoft Broadcast also posted its key messages: allowing WARP members who did not listen to the broadcast to benefit from the information. Coupling this information with a discussion about the practicalities of applying the patch shows how useful the WARP can be.

We Did it Right, but it Went Wrong

Case Study of a patching exercise that didn't go according to plan

The following event demonstrates the risks in supporting just part of a secure IT environment for approximately three thousand email users.

The scenario: We run a server that provides mail scanning (in and outbound), this filters out “Spam” and checks the content for suitability for forward processing (delivery) or should a message fail this check holds it for inspection and subsequent processing. In addition this process also checks for email messages that contain viral infections or worms and deals with them accordingly.

The technology used: Microsoft Windows 2000 Advanced Server, Clear Swift Mail Sweeper, Sophos Anti Virus. In addition at the time of the incident there was a run time copy of Microsoft SQL 2000 database supplied as part of the Clear Swift product. This is housed on a Compaq (HP) DL380 server (rack mounted), this server sites in our DMZ.

It should be noted that we keep the Anti Virus signatures up to date, keep up with service and security packs from Clear Swift and Microsoft after testing on a “like” machine.

What happened: On the day in question at around 7:30am we responded to a security alert from Microsoft with a security update, we tested update this per our procedure and found it worked and installed okay, we then installed it to our “live” server and the server ceased to work following a reboot, being unable to reload its operating system fully and going in to a re-boot cycle.

What happened next: The senior technician did some initial diagnostic work and sought the support of a Microsoft Server specialist from the main IT support team, between them they worked imperially though a number of ideas to resolve the problem, including things like looking at reversal of the update, running the system repair option, reinstallation / reconfiguration of some of the software on the server, this however did not restore the machine to its correct operation.

At around lunch time, it was decided cease the patch / test / adjust approach and go for a complete rebuild of the server, it should be noted that this server had originally been configured by a contractor, however the two technicians where familiar with the software and the configuration and so with supporting documentation commenced the rebuild.

It was agreed that the re-build would be slightly different from the original specification and that a full version of Microsoft SQL 2000 would be used rather than the run-time as this run time component had been disappointing in terms of reporting on the work of the mail sweeper product.

By mid afternoon things where looking better, the operating system was installed and fully patched, MS SQL2000 was on and patched, Sophos Anti Virus software was on and fully up to date with the necessary signature files and the Mail Sweeper software was loaded, the routes for the movement of email between this server and the almost redundant MS Exchange System and the new GroupWise SMTP server where proven as was the route to the Internet.

Some complexity in the relationship between the Mail Sweeper data and the Anti Virus product where overcome with conversations with the Vendors and the contractors who originally set the server up.

The server was made live for a short period to test all was working with “real” data, this processing was carefully studied and after agreement between the two technicians and I we decided to go back to “normal operation” at around tea time.

Outcomes:

• We have now formulated a fall back plan if this happens again, this would involve quickly setting up a fall back server with just anti virus scanning on it, while this system was rebuilt, we also resolved to review further resilience in this area, and other methods of achieving content and virus scanning.

• We had updated from a run time to a full version of MS2000 which helps with reports from the Mail Sweeper product.

• We had tolerance tested the lack of email! We found (with no surprise) that where as the lost of inbound and out bound email for a day a year ago was “not good, but not the end of the world” that in fact it was now “the end of the world”....

• Due to time restraints and the need to get systems back on line, we never did get to the bottom of what went wrong, although we have many ideas.

• Clearer focus about when to feed information in to the London Connects WARP, this data would have been useful at the point the problem occurred, in this instance around lunch time and finally when all was up and running again.