A mysterious invitation had appeared in my inbox, titled “Review” and from Andre’s account. The only other invitee was Phil, the Head of Infrastructure. No agenda or context – and the meeting was happening in less than an hour from when the invitation arrived. I arrived at the meeting room, half expecting to be ambushed by some heartless HR entity with my notice papers; so was relieved to find just Andre and Phil taking their seats. I’d survived another day…

An infrastructure project – outside of my portfolio – had gone badly wrong at go-live. Andre briefed me on the situation. He explained that a data centre migration had taken place over a bank holiday weekend, but somehow the new server, following going active, had repeatedly erased the latest data and replaced it with a copy from the previous server. Work across multiple teams had been lost, and now nobody trusted the systems.

My task was to review the incident, interview everyone involved and report on the causes of the failure. Andre left, but Phil gestured for me to stay. As the door closed, he leaned forward.

‘I’ve told Andre I don’t want this to be a scapegoat exercise. We still need the project manager to complete the remaining work.’ As he spoke, I could see the difficult position Phil was in. Several days of work had been lost, confidence in the systems had been shattered and now people wanted answers. That often means finding someone to blame.

But Phil had a more immediate problem. He still needed to get the project finished, and losing his project manager now could make an already difficult situation much worse. And, there was a human side to this, too. A project manager was potentially facing the consequences of a very public failure, and Phil was trying to ensure that whatever happened next was fair.

I respected him for that. But I also knew that fairness didn't mean protecting someone from the consequences of their mistakes. We needed to establish what had actually happened.

‘Phil, I’ve been a project manager when things went wrong,’ I replied, ‘and I’ve been treated unfairly in the past.’ Damaged, I thought. I could see Phil’s posture beginning to relax. ‘I’m not looking for someone to blame, I’m looking for the cause so we can prevent it happening again.’

It wasn’t the entire truth. I knew that I’d be called to share my judgement of the situation and performance of individuals as part of my feedback, and that may influence whatever happened next to the project and its manager. I think Phil knew that too, but the key message had been received – I’ll be fair, because I know what it’s like on the other side.

I’m sure you’d love to hear all about my investigation, the interviews with participants, the reports I read, the email chains I followed, etc. But I’m afraid if I did so, the next several pages of the book would resemble an Agatha Christie novel, so instead I’ll will skip to the end of the review…

A week later I was back in the meeting room with Andre and Phil, with my report on the meeting room screen. A simple sequence of events were shown:

1. Pre-migration: An automated daily copy was configured to transfer data from the old server to the new server.
2. Saturday and Sunday: The migration work was carried out over the weekend. The new server went live on Sunday, but the automated copy process was not disabled as required.
3. Sunday midnight / Monday morning: The first scheduled copy ran after go-live, overwriting data on the new server with data from the old server.

4. Monday: Bank holiday testing took place on the new system.
5. Tuesday: Normal business resumed. Work completed on the new server was subsequently overwritten overnight with data from the old server.
6. Wednesday: Further data loss was discovered and the cause was identified, but too late to prevent another overnight overwrite.
7. Thursday: The automated copy process was disabled, preventing further data loss.
8. Friday morning: The servers were operating normally, with the previous day's data available.

‘Before I continue,’ I checked, ‘Are you comfortable that this description aligns with your understanding of what happened?’

They both nodded.

‘Okay, cool. Here, we can see what went wrong: the overwrite wasn’t switched off. But something else went wrong, the problem wasn’t detected quickly enough and recurred over multiple days. This didn’t happen because of one failure – it happened because of many small failures that compounded on each other.’ I leaned back in my seat, and noticed both Andre and Phil leaning forwards.

 

The sequence of events that caused the failure were explained:

1. An earlier delay in the project, several months earlier, had forced the project to adjust its go-live date. It had moved to a UK Bank Holiday weekend. Key UK personnel had already booked annual leave for that week and would not be present to support US migration specialists who were travelling to the UK to carry out the migration. This had not been communicated to the US team.

2. The US Team travelled overnight on Friday and reported to work on Saturday morning when they arrived in London. They hadn’t slept on the journey and were, in their own words, exhausted. They arrived to find no local support, and their workload increased beyond their expectations, whilst still facing time constraints.

3. Having carried out dozens of migrations as part of the project already, the migration team no longer followed documentation when carrying out the work. They felt that this enabled them to move quickly. A simple checklist could have slowed them, however it may have reminded the team to switch off the copy function.

4. As the migration team had no checklists or documentation to ensure all necessary steps were taken, the project manager had to rely on their verbal assurances that all work had been completed. I pointed out that the organisations own governance did not call for written documentation, checklists, etc but that the project manager could have included this as a requirement for the migration team.

 

As for the delay in recognising there was a problem, the events that took place were:

1. The project manager had requested of operations to have testers provided from various teams to work on Sunday and the bank holiday Monday to test the new system. Whilst the request was made, the project manager hadn’t identified it had been largely ignored with only two testers from the required thirty available.

3. Due to the shortage of testers, an issue with drive names went undetected until work resumed as normal on Tuesday. A number of broken links started to be reported, along with other difficulties in finding files and folders. On Wednesday, when work had disappeared, the drive name issue was initially believed to be the cause again, delaying further investigation.
4. It was not until Wednesday that enough lost work was reported for the team to realise there was another problem. The automated copy process was identified as the cause, but too late to prevent it from running again that night. The process was disabled on Thursday, and by Friday morning the servers were operating normally.

If the point of the review was to find culpability, there now seemed a lot to share. After a moment of silence, I moved onto my judgements and recommendations.

1. I felt it would be unfair to blame an exhausted and overwork migration team for the error, but recommended that checklists be brought into use for future migrations which should be signed off and handed over to the project manager upon completion of the task.
2. Likewise, I couldn’t blame the project manager for the individual error on the copy function being left active. I felt that his most notable failing was in his communication style with the operations team. Instead of having a two-way dialogue where he could ask for testers, then seek named individuals and verify their participation, he had instead broadcasted his requirements and assumed they would be acted on.
3. Escalations to be made in the event of tester shortages during testing.
4. As having a bank holiday weekend to make the migration and carry out testing was seen as desirable, I could not blame the use of these dates. However, I did point out that allowing all the senior members of the UK technical team to book holiday over the same week contributed to the challenges faced by the US team.
5. Additionally, I stated that when complex work of this nature is taking place, in future any significant journeys should allow a period of rest on arrival before work takes place. A rested US team may have avoided this mistake.
6. Improve project guidance materials to include clear rules around process and ownership.

I wondered whether Andre and Phil would be disappointed with my conclusions. They'd asked me to find out what had gone wrong, and I’d explained that several people had made mistakes, but that I didn't consider any one of them deserving of the blame for the overall failure.

Would they see that as a fair assessment, or would they think I'd been too soft? I wasn't entirely sure myself. But I was confident in the evidence, and I wasn't going to recommend consequences that the evidence didn't justify.

But both Andre and Phil seemed relieved when the presentation ended. Perhaps it was because there was no single villain in the story, or perhaps because the findings gave them something more useful than someone to blame. There were things they could actually do to prevent it happening again.

The project manager continued the project which turned out to be a great success and even landed a promotion at the end of it, which I was delighted to hear about.

As for the recommendations, well, they were wrote down in a report and shared upwards. So I’m sure they were all implemented immediately. Right? Right..?

 

 

This article is adapted from my upcoming second book, Becoming the PMO, which shares the experiences, challenges and lessons that shaped my journey into PMO leadership.

By Stuart Taylor

Keywords: Leadership, Project Management

Follow Us On

Become a Contributor Newsletter Signup