I recently encountered a frustrating situation involving a rushed server update that lacked proper planning and collaboration within the team. This incident not only caused significant problems for the company but also turned out to be an unnecessary update altogether.

Unfortunately, the update went awry. The expected behaviour didn’t materialise, and upon rebooting the server, the company discovered that the software had been completely uninstalled, along with all the crucial settings. Consequently, the customer was left without a vital line of business application. To mitigate the situation, I inquired about the availability of a server snapshot, only to find out that none had been taken. Additionally, when I checked their backups in the hopes of restoring them to a previous point, I was disheartened to learn that their last successful backup was over 168 days old.

This incident serves as a timeless reminder of the importance of executing planned, significant changes under controlled and well-thought-out conditions. It also highlights the criticality of maintaining reliable and regularly tested backups. In a previous post I shared, “Happy World Backup Day!” I outlined the golden rules of backups.

The whole situation could have been avoided had a clear change plan been implemented, complete with a rollback strategy. Such a plan would likely have prompted the realisation of the need for a server snapshot and exposed the backup failures. Moreover, it is crucial to adopt a collaborative mindset, valuing the expertise of colleagues and avoiding the assumption that one knows more than the rest. The presence of an individual on your team who makes unilateral decisions without considering the potential outcomes or seeking input from colleagues can lead to profoundly detrimental consequences for the company and can also pose a major security risk.

Another essential lesson from this incident is the significance of having robust, tested backups. Relying solely on hope that backups will be sufficient when the need arises is an unreliable approach. Instead, following the 32110 backup methodology and working with the company to establish effective BCDR (Business Continuity and Disaster Recovery) policies is imperative.