In the world of IT, we often stress the importance of robust backups and Business Continuity and Disaster Recovery (BCDR) plans. Today, I offer a real-world example that underscores this emphasis: a school that became the unfortunate target of a ransomware attack. The aftermath? Hundreds of their computers and servers rendered useless.

The story begins with myself and a good friend, Michael Paul, hearing from one of our account managers who was seeking help for a new customer affected by a ransomware attack. Despite the urgency of the situation, no one in the office seemed willing to get involved, likely due to the school’s reliance on Microsoft Data Protection Manager (DPM) for network backup. To our surprise, we discovered that the school had already reached out to five other service providers, all of whom had declined to help. That’s when I turned to Michael and asked, “Are you up for it?” We joked that we couldn’t make things any worse.

Assessing the situation

Upon arriving at the school, Michael and I were met with a handwritten note that read “To all staff, Due to computer problems the system will be down until further notice. Apologies”. Beyond this, the atmosphere was palpably tense. We could see the strain on the IT team; the weight of the situation evident. It was a gloomy and daunting situation, but we knew we had to help.

We approached the IT manager, who was clearly overwhelmed and exhausted from the ordeal. He had flown back from a family holiday just to attend to the emergency situation, leaving his loved ones behind. We could see the worry and fear etched on his face as he greeted us with a nod and a quick hello before beginning to explain the situation they found themselves in.

In these situations it is wise to take a moment to understand the whole picture rather than just diving in and starting to try and fix things. We initially sat down and took them through a series of questions to help us understand their environment and help plan how to recover from this situation as quickly as possible. Upon assessing the damage and trying to determine the extent of the ransomware attack, it was quickly apparent that the attack had been incredibly devastating, with almost all of the school’s computers and servers affected. We began by checking the backups, but it soon became clear that the school’s backup system was inadequate, and they were not going to be able to recover any data quickly.

Breaking difficult news to a customer about the extent of their issues is always challenging. However, I believe an empathetic and reassuring manner offers them a glimmer of hope. It’s similar to a surgeon discussing a procedure with a child’s parents; they desire a surgeon who exudes confidence, professionalism, and an unwavering demeanour, as if they handle such situations routinely without any doubts. Similarly, IT Managers and Business Leaders seek the comfort of entrusting their worries to a competent third party for some time. You need to tread the line between reassuring them that they will be okay and being honest about the situation they’re in – it’s a delicate balance.

Rebuilding and restoring data

The backup server they had been backing up to was not only running horrifically slow, it was also, seemingly, unable to produce any restores. The backup strategy had further issues when it transpired that the school had been backing up to tape, but the tape drive had failed at some point and it hadn’t been noticed. Fortunately, the incident coincided with the school summer holidays, ensuring that the school premises were vacant which provided us with some valuable time, something that is not always on our side in these situations. Hastened to say that time was still a pressure and, as such, our only option was to construct an entirely new domain for them, followed by an effort to retrieve as much data as we could from the backups and restore the files into this fresh setup. Sometimes you have to cut the foot off to save the leg and, in this instance, we had to cut out all of the old infrastructure in order to get the school up and running in time for GCSE results day.

Michael and I set about documenting their existing server setup and beginning to plan a new infrastructure for them, using best practice methodology and, of course, implementing a Veeam backup solution using the 32110 backup model. We worked with the IT team to plan a whole new network from scratch, including stripping down and rebuilding networking setups with their switches and firewalls. We got the domain setup as quickly as possible to enable the IT staff to begin their arduous task of imaging over 600 computers across the campus. We also ensured we documented every part of their new setup as we went along; documenting as you go ensures accuracy and also helps you notice anything that’s been missed.

As the team focused on constructing a new environment, I shifted my efforts to the DPM server, aiming to restore their lost data. It soon became evident that about two weeks’ worth of data was lost due to consistent backup failures, compounded by a malfunctioning tape drive that couldn’t read any tapes. In our quest to find a replacement LTO drive, we reached out to peers and even made heartfelt appeals on social media platforms like LinkedIn, to no avail. This challenge, however, gave me a unique opportunity to draw upon my prior experience as a micro electrician. Employing the skills from my pre-IT days, I stripped down the tape drive and fixed the issues that were preventing it from reading tapes. I rebuilt the tape library and powered it all back up with nervous anticipation. To much excitement, the tapes loaded and started reading data! Over the subsequent days, I embarked on the arduous journey of retrieving as much data as possible, much to the client’s relief, as they witnessed their new domain gradually repopulated with their data.

After several days, we began transitioning the remaining rebuilding tasks to the school’s IT department. Recognizing that our services come at a premium, it was most economical for the school if their IT team took over tasks such as installations, configurations, and data restoration. However, we remained available via phone or Microsoft Teams to provide assistance and address any queries they encountered. Once the school’s systems were fully operational again, we convened at our office to conduct a thorough root cause analysis. Our aim was to devise a strategy to bolster their defences against potential attacks and to initiate the development of a comprehensive BCDR plan.

Conclusions

This story underscores the crucial need for a robust BCDR plan, which should be routinely tested to identify any vulnerabilities. It’s essential to maintain verified backups of your systems and have a contingency plan to restore operations in case of disruptions, whether caused by cyberattacks or other unforeseen disasters. While safeguarding your infrastructure with antivirus solutions and other protective measures is fundamental, we cannot ignore the escalating threat of ransomware attacks. The frequency of these attacks has made it more a question of “when” rather than “if”. Hence, a well-prepared and tested response is your strongest shield against such threats. A key takeaway from this incident was the practice of IT staff using Domain Admin credentials for everyday tasks. The fastest way to bolster security is to eliminate such blanket trust. Admin accounts should be reserved strictly for administrative duties, while regular tasks should be conducted using accounts with only the necessary permissions.

I would also highly recommend taking your business on a journey to become Cyber Essentials certified. Cyber Essentials is a UK government-endorsed program guarding against common cyber threats. It offers two certifications:

• Cyber Essentials – A self-assessment protecting against typical cyber attacks.
• Cyber Essentials Plus – Similar to Cyber Essentials but includes hands-on technical verification.

The scheme also educates on cyber security terms, empowering users to enhance their IT security. You can find out more about this programs here: About Cyber Essentials – NCSC.GOV.UK.

Another invaluable lesson is the importance of having allies, like Michael Paul, whose expertise, guidance, and generosity are unparalleled in my world of IT. For consultants stepping into such scenarios, a strong partnership and mutual respect for each others expertise provides added assurance to clients that they’ve enlisted the right professionals.