An attack downloaded pornographic images to a hospital's computers.
In late July Kern Medical Center's information system came to a grinding halt.
The hospital believed it had the standard security systems in place to protect its medical records. But for 16 long days that stretched into August, the hospital struggled to get its systems operational and isolate the problem from its patient care.
Forbes caught up with Bill Fawns, CIO of Kern Medical Center in Bakersfield, Calif., to talk about what happened and what to do when a cyber attack does occur.
Forbes: When did you first become aware of this attack?
Bill Fawns: Microsoft had issued a warning about a security threat through .lmk files, the ones that are on your desktop that are links to other executables. That was the start of it.
How bad did it get?
All in all we had 13 different pieces of malware that were able to get in. We don't know how many pieces came in as part of that initial attack, but it did impact us for 16 days.
It brought us to our knees initially. The first thing that happened was people called us saying their printers were printing long jobs gibberish until it would run out of paper. When we asked what they were doing about it, they told us they were adding more paper to the printers. That was the first indication we had a problem. Then it got much uglier. Systems began to get slower and slower to the point where they were crawling along. Then it began to download large quantities of pornographic images.
Inside the hospital?
How much of your system is electronic?
We're both online and paper. We're in the process of moving to electronic medical records. [Medsphere is currently implementing the OpenVista electronic health record at Kern.] Patient registration and information are all electronic. There's also a lot of electronic support in the emergency department and in labor and delivery. It's piecemeal across the hospital. But there's also a paper record at the bedside.
Did you believe your security was sufficient?
Yes. The hospital is one department in the county. We have about 35 departments in the county and the other 34 departments were not hit. We later traced it to why. It was a very small mistake by an employee. But we did have antivirus software for the entire county. We also had an e-mail appliance and all the county departments were behind that appliance except the hospital. After the situation was corrected, we've put everything behind the appliance.
Still, you thought it was secure?
Yes. We thought there was no reason to worry. That wasn't the case.
So what did you do?
At a hospital, you always hear about a code blue or a code red for a fire or some other emergency. We did a code triage where we brought together the managers across the hospital and asked each of them how this was impacting their organization and their ability to provide patient care. After that, we quarantined everything. We took the hospital out of the county network so it didn't spread to other departments. Then we assessed how much of our hospital was affected.
Which systems were hit?
Those that were Microsoft-based. Those systems that weren't hit were Linux-based. We had migrated our application servers where we could over to Linux. Those were in good shape. So after about six hours of down time on the second day, we determined that patient records were in good shape because they were on the Linux servers. The Microsoft desktops and servers were a mess, but there were only a handful of servers running Microsoft applications. What we learned there was that diversity of technology is probably the best thing you can put into an organization. If you have the ability to run Linux servers and Microsoft desktops, you're probably not going to get hit at the same time and you break the ability for the malware to pass itself back and forth.
So it's no better if it's all Linux?
Exactly. My recommendation is a Microsoft desktop environment because that is the 800-pound gorilla. The majority of desktop applications are available in the Windows environment. You can get the best of both worlds this way.
How did the virus come into your organization in the first place?
It came in as an e-mail attachment.
Someone clicked on it and it spread?
Yes. It's an exponential spread. It starts slowly. One passes it to two, two pass it to four, and so on.
Did you find out when it came in?
We had begun so much of the cleanup process that we lost the specifics of where it came in. There is so much of this stuff, and there are a few places around that are malware capitals such as Thailand or Estonia.
What did you lose?
We were lucky in that regard. All of our patient records and all the financial information were on the Linux servers. What we ended up with was a serious operational hit, but on the information leakage side we were lucky.
How much impact did it have on patient care?
The very first thing you have to do with one of these breaches is remember your business. We immediately focused on anything that would affect patient care. We pushed the administrative side of the house--the ability to collect money--to a second-tier position. While people had to wait two or three minutes for a response, we had almost no impact on patient care. On the financial side, we probably lost $1 million a day in billing.
Do you think this was preventable with technology or education?
It's a combination of things. I serve as CIO of the hospital and the head of IT for the county. I spoke to the board of supervisors and said we need to embrace a security function within the county--someone who could do an audit and recommend policy to the board. We need to take a hard look at whether we're short of technology or training. It's probably a combination of both.
Are the regulations for patient information sufficient?
Yes, I think they're more than adequate. Everyone in the medical industry understands that patient information is a priority. We called in the FBI and they recommended what they refer to as protection through depth, which is multiple levels of protection. In addition to our antivirus software and our e-mail appliance we are looking at new products that approach the detection process differently. The traditional antivirus looks for a signature of malware and takes action. HBGary has a product we're evaluating that looks for the activities of software. Rather than looking for a signature, which is after the fact, the HBGary software sits in RAM. It looks at whether any program has a characteristic or trait that may be malware. We're looking at adding another layer of security.
Is this your only breach?
We haven't had any others here. At another hospital, someone broke into the system, then walked down the hallway and slipped a note under the CEO's door. He said, "Here's my Cayman Island bank account and put money in or I'll release your records." At another hospital, the FBI had a video the hacker had made of himself and in 4 minutes and 26 seconds he broke into the hospital. He put it on YouTube afterward. The potential lawsuits that come from the release of patient information are huge.
There's a litigation issue, too, isn't there?
Yes. Not only do you deal with potential litigation from the patients. The government is also going to slap fines on you.
What's your advice to other CIOs?
Remember the business you're in. You need to look at prevention, but if you are hit you need to be able to take care of the patient. The second thing is, don't expect your antivirus software company to help you if you are hit. Their business is protection, so you'd better have a plan in place. And then there's diversity. We recommend Linux and Microsoft, but for those folks in an all-Microsoft shop you can diversify the antivirus software. You can put one antivirus software package on the desktops and a different brand on the servers. If each of them only catch 80%, then hopefully one or the other will be able to stop what comes in. If you do get hit you need to work from the inside out--from the servers to the thin clients to the desktop PC, which will be the most labor-intensive and provide the least return. You also want to be aware of distractions. When we were hit, at first we were distracted by those things that were almost a red herring. You have to find the real culprit and focus on that.
How about the network?
Absolutely segment your LAN. If you're a trauma center you may want to get the emergency department up and running first and keep it off the rest of the network. If you have a segmented LAN you can bring up your organization on a priority basis. And then you should be backing up and backing up, both on-site and remotely.
Do you assume you can really stop breaches?
You have to try. But you also need to be prepared for the worst. It's hope for the best and prepare for the worst.