This month I have the honor of hosting T-SQL Tuesday #107 about Death March Projects. The topic for this months T-SQL Tuesday #107 (hosted by yours truly Blog/Twitter) is:

“Tell me your project horror stories – the worse the better”

Through some redaction, slight change of facts, and creative control with some details I will protect the innocent; and in this case, the guilty as well.

If you don’t have time to read my lengthy exposition then please skip down to the section titled “The Making of a Death March Project”.

What is a Death March Project?

In my invitation post, I introduce us to the concept of a death march project in software engineering. Quoting Ed Yourdon from his book: “Simply put, a death march project is one whose “project parameters” exceed the norm by at least 50 percent”.

How does this happen? Typically one or more of the following leads to a death march project:

• Schedule compression – estimates are so far off and unreasonable. When a project that, by all rational appraisal, plans a project that would take 12 months into 6 months – you have the recipe for a death march project.
• Staff reductions – there are some places that experience a lot of turnaround in workers. This can be voluntary or involuntary. Business pressures compel firms to compete hard.
• Budget has been cut – especially when the budget is low there will be a long project ahead. Do you see a recession? Time to tighten that belt.
• Wild scope creep – the number of features and complexity of establishing a good proof of concept – let alone a solid skeleton of code – will increase the changes of sad times ahead.

How could estimations be so far off? I think it involves human behavior and the nature of our work. Dr. Fred Brooks talks about this in his seminal work “The Mythical Man-Month“.

If you have not yet read The Mythical Man Month – please procure and read at least the first few chapters. It is a goldmine that has conceptually stood the test of time.

We see some of the same elements presented there that we do with project pains. Let’s discuss some key points and then come back to tell the story.

The Programming Project and the Mythical Man Month

Developers love to hack; and by that, I mean the traditional definition of coding an elegant solution in short order. You won’t be taught this stuff in school. You only find out when you are employed as a developer on a project team.

Yet there is so much more to consider. We are no longer hacking a solution to a fun self-directed hobby but rather forced to take on more overhead than actual development. It is not a simple program anymore – it is a programming systems product. The amount of effort will be orders of magnitude greater than just writing a program.

This is why it takes so much longer to produce a production ready application. The overhead take way more than the development. New programmers won’t realize this but experienced veterans will be nodding their head.

You come to find that there is so much more to writing good software than the coding piece. As we move from a simple program to a programming system there is usually some integration with existing applications. As we move from a simple program to a programming product now we have to do testing, documentation, and maintain it through the defined life-cycle. The book describes development as being in a tar pit where struggling only makes you more stuck.

The Joy of Being a Developer

In the Mythical Man Month, the pros of software development are:

• The sheer joy of making things – building things is a rush to some people.
• The pleasure of making things that are useful to other people.
• The fascination of fashioning complex puzzles of interlocking moving parts and watching them work. We’re all enraptured by the blinking server lights in a data center. Developers get this feeling with building complex code.
• The joy of always learning – you will never climb Mt. Development.
• The delight of working in such a tractable medium – we are constructing “castles in the air” with our imagination.

The Frustration of Being a Developer

It is not always a fun thing to be a developer:

• Perfect performance – there will be high quality demands that won’t get checked into production unless it actually works…every demonstrable time. People aren’t that great at being perfect so you can see how this is a recipe for bugs and long hours.
• Other people set the objective. Whether this is the project manager, an architect, a manager, or a senior developer. This is a hard one for newbie code cowboys to wrangle.
• Dependence on others – if you are integrating your code into an existing application or codebase then you are at the mercy of that developer (if they are even still around). Many hours will be spent staging things so they are complete, available, and usable.
• Testing drags on – welcome to debugging hell. If you estimate 40 hours to develop then you need to estimate 80+ hours to test. Even that is a difficult target to hit. You’re likely not considering distractions, interruptions, other work, fires and emergencies that you must tend to, and personal issues. Life happens.
• Obsolete! The project you have poured blood, sweat, and tears into has been axed or dead on arrival. This will not be a resume item.

Partition-able Tasks

Brooks illustrates graphs explaining project staff against time. We have tasks that we can not have everyone contribute to at the same time. If it were perfectly partitionable then we could taking it more efficiently – but sadly that is not the state of projects.

It is like saying we have 4 women – can we have them all work together on having a baby in 2.25 months instead of 9 months.

The above is an analogy in the book apt for describing how we cannot partition all tasks.

The salient point made: Adding manpower to a late software project makes it later!

Here are some graphs that illustrate how tasks can be broken up among workers:

Types of Death March Projects

This typically happens less with small or medium projects but rather with huge projects that require a ton of work.

• Small scale projects – not many people working against nearly impossible odds to finish the project in 3-6 months
• Medium size projects – bigger team (maybe 20-30) involved in a project that will take 1-2 years
• Large scale projects – a few hundred people working on the project slated for 3-5 years
• Mind boggling projects – an army of staff working furiously on a project. This will last > 5 years

We see less and less mind boggling project because the risk of failure or obsolescence upon completion is just too high. Nonetheless, these death march projects still happen.

Why Do They Keep Happening?

All programmers are optimists. This modern sorcery skews the mind with poor estimates. Development is building abstract things instead of physical things. We have been building bridges, for example, for thousands of years. We know how to build a bridge. However, we have only been writing bad code for decades and since it is not physical we are not optimized to think about it the same way that our senses perceive things.

Every time you proclaim “I’ve found the last bug!” a developer gets locked in a supply closet… 🙂

The best reason to explain why estimating a software project I found on Quora – Why are software development task estimations regularly off by a factor of 2-3? In particular, read the answer by Michael Wolfe midway through the page. I could not put it any better.

In the book, brooks lays out the salient point:

Adding manpower to a late software project makes it later!

Yourdon spells out some reasons why this keeps happening – see which ones you foresee disappearing in the future:

• Politics, politics, politics
• Naive promises made to key stakeholders e.g. an executive
• Naive optimism of youth – you don’t see this as much with 40 year old developers
• The “startup mentality”
• The “Marine Corps mentality” – this is gonna hurt. Programmers don’t need sleep, right? Well real programmers don’t need sleep…
• Intense global competition

Why Does Anyone Want to be a Developer?

What reasons would anyone have for participating as a developer on a sinking ship?

• High risks but high rewards
• The “Mt. Everest” syndrome – there is something about thinking you are going to run up Mt. Everest barefoot. Sounds stupid but that is the equivalent of what it is like.
• The buzz of working with equally committed people
• Naive youth and optimism
• Not participating means unemployment or other similar woe

For some reason people are attracted to this madness. It is something you have to experience to fully understand.

The Making of a Death March Project

Finally we get to my story! The preamble was necessary to set the tone. Gather around and get the popcorn!

I started a new job as the main developer on a project team. Our goal was to produce some software to accomplish something set out by a government regulation in 3 months. The backstory was that a developer had left the company and that’s why I was hired. I later found there was a long succession of developers who came in, worked, then stormed out in failure.

The project team looked like this:

I left some people out but these were the main players. There were bad managers, a project manager who didn’t know how to turn the ship from hitting the iceberg, constantly changing requirements, outrageous schedule, low morale, and otherwise difficult people to deal with. All of this combined to form a perfect storm.

Welcome Aboard!

When I was on-boarded, I could immediately tell that the existing code didn’t work and likely would be hard to correct. Since the previous developer was no longer around all I had was the source code, some fuzzy requirements, and an analyst to try and guide me through. Eliciting requirements is the first step in a typical software development life-cycle. They could not quite pin down the requirements and the person we had to deal with (our government client) was worse than useless. Details were either missing or misconstrued and had to be redone. 

Additionally, the environment was toxic. There was an undue sense of urgency with absolutely everything. People would curse at each other on a regular basis. Flipping out on a conference call was not uncommon. I mean a real meltdown too. The kind where you might want to call a hotline to make sure a colleague is ok. Management was super involved and tried to micromanage it from their position without having any idea that their efforts hurt us and slowed progress.

GOTO, Cursors, and Spaghetti

There were cursors everywhere. Cursors have their place but are often misused. The people notorious for this are non-database programmers. They are used to handling data their way and not in a set oriented way that a RDBMS will like. It was debugging spaghetti code. I actually came across GOTO statements!? It was very difficult to unwind what was happening and even harder to change it to conform to expectations.

The company had a military mindset. By that I mean the disposition was a lot of top level management oversight and micromanagement coupled with a hurry up style at all times. Rice will cook well in a pressure cooker but a developer will not write good code like that.

This Will Just Take a Second

I remember working a ton of hours. When you are young you may not see the problem but as you age you find how it can affect your health and relationships. I thought working 60 hours a week was no problem. During the project I averaged between 70 and 80 hours per week! My highest week was 96 hours! I am not proud of that. I vividly recall waking up at 6am to join a stand-up call and working until 11pm with little or no breaks. There were no weekends. I had to fight with the project manager to have my anniversary and an already planned and paid for weekend getaway. Getting called and text and emailed was a non-stop event. I have many memories of working from home in my boxers frustrated because there was too much work and pressure to even show up to the office.

Personal life and mental health took a hit. I didn’t exercise for months. One time I stealthily escaped work to get a haircut because I couldn’t find the time and was fed up. My weight gained too with all the stress eating. 

When it became apparent even to the most stubborn stakeholder that failure was imminent, management tried to add more developers to help me. I had asked for that up front with my estimates but they ignored it. Gotta tough it out! I had to explain that adding more developers to this project at this time would simply make the project later. The overhead of bringing someone up to speed with such a complex project was too time consuming. With the deadline approaching it was too late – that ship had sailed. As a former Defense Secretary famously said:

“You go to war with the army you have, not the army you might want or wish to have at a later time.

How I Overcame

Ultimately I threw out all of the previous development and rewrote thousands of lines of code to replace it. The project had 3 weeks until the deadline or we would get fined. I fought analysts with test cases so I knew that my code worked to specification. It was hard to spell out and not easily understandable but I made it an ultimatum – you either provide me some test cases so I can evaluate against requirements and code or I’m through.

I threatened the project manager to back off and let me work. My manager helped some but was powerless. Fights were a daily occurrence. Since I always felt under attack – and defending wasn’t working – I had to be aggressive and go on the offensive. Sounds crazy but it kept my sanity and got it done.

The project was estimated to take 3 months – we somehow got it done in 9 months with a lot of fallout. Honestly – they were lucky it got done and was the least bit useful.

Post Death March Project Analysis

Every project should have some post game analysis. What went right and wrong? Are there things we can improve next time? What pitfalls must we avoid? Who is invaluable and who is dead weight?

The Concept of Triage

The same concept applied to causalities by battlefield medics can be applied here. You need to sort through what is absolutely necessary, what cannot be saved / fixed, etc. Working on 1 thing at a time gets things done. I had to forcibly demand one task at a time and work it to completion – kicking and screaming. 

My Resolution?

After the project was complete I remember feeling more burnt out that ever before. It was performance review season and I asked for a raise. You would think that after successfully completing such a mammoth task that a 3-5% increase in salary would be no problem. You would be wrong. I remember being told the decision by my manager who could not persuade the director (who said I was already overpaid?!).

That day I decided I would leave. I had some time to conduct a job search and found a much better opportunity with a lot more money. I remember calling my manager from the parking lot giving my 2 week notice. He was sad but totally understood. The day you find out that your efforts and rewards are not related is a good day. You find it frees up a lot of time. I no longer felt obligated to start early, work late, or work weekends. There was finally time to update my resume and start networking for a new job.

I spent the next 2 weeks training a contractor / consultant to maintain this monster of a project. Once I was checked out mentally it really didn’t matter what they said or did. I didn’t want to be there and they were lucky I did not just rage out and leave one day to never return.

Later on I found out they somehow got a previous developer to rejoin after I had left. I guess some people never learn. Handing in my laptop was such a great sensation to cap off my tenure. I regret very little.

 

---

Congratulations! You made it to the bottom of this post! If you liked this then you might like to read about My Blogiversary to cheer you up.