Software processes has always been a popular topic of discussion in our industry. Those can get quite heated, with advocates of the "stable / stale" Waterfall method pointing fingers toward "rapid / rabid" Agile methods, with the CMMI people throwing documents around and Lean people standing on the sidelines muttering about Waste.
This isn't a post about a specific software process, I'll defer that to another day. Instead, I want to focus on a flaw in the basic building blocks in many* software building processes.
They ignore the actual building the software.
That may sound ridiculous on the face of it, after all, how can a software process ignore the act of building software. But take a look at the following diagrams:
If you'll pay attention, you'll notice that those processes talk about everything except how to actually build software. They talk about people, about requirements, about managing customers, about a whole lot of things, but not about the part where you have people sitting down and writing code. In most of those, in fact, that part is usually defined as one of those:
Why is that a problem? After all, isn't there a big distinction between software engineering (we know what to do, now let us do it) and project management (getting to know what we need to do, and verifying that we did it right). Those processes deal primarily with project management and leave the engineering part to be defined in a way that fit that particular project. Surely that is better, right? In theory, it might be. But there is a big problem when you have a software process that ignore the software engineering aspects of building software.
The problem is that that in many cases, there are hidden assumptions that are going to hammer you down the road if you use a certain process with engineering practices that doesn't fit it. Take a look at the following chart, showing a team velocity over time, does this look familiar?
The term I heard used for this is Scrum Wall, but I have seen similar results in other processes as well. The best description for that is Allan Kelly's:
You hit the Scrum wall when you adopt Scrum and everything goes well, then, after a few Sprints things don't work any more - to use an English expression, they go pear shaped. You can't keep your commitments, you can't release software, your customers get annoyed and angry, it looks like Scrum is broken.
This is what happens when you adopt Scrum without technical practices such as Test Driven Development, continuous integration and refectoring. When teams adopt the Scrum process, they go faster, show progress, things look good... and then the quality becomes a problem. Now the team are fighting through quick sand.
The code quality is poor and developers are expected to continue to make progress. Maybe the Scrum Master/Project Manager reverts to past behavior and demands overtime and weekend working. Maybe the team start busting a gut to keep their commitments. Either way the team is heading for burn-out.
The major issue is in focusing so much effort and time on project management with what amounts to willful ignorance of the technical and engineering practices will inevitably leads to disaster. The process of building software is intractably linked to the engineering practices involved in building the software. Moreover, some technical practices are actively harmful in some scenarios and life savers in others.
Many Agile and post-Agile processes focus on short cycles, each of them producing something with a distinct value to the customer. That may be an iteration, a commit or a feature, where the goal is to increase the velocity over time so we can provide as much value to the customer in as short a time as possible. What those processes ignore are things like technical debt, large scale refactoring and non functional efforts. Oh, you see those things mentioned on the edge, but they aren't something that is dealt with heads on, as a core issue to consider.
There is a bit more to that, actually. The software engineering practices and the project management strategies are linked and of paramount importance when the time comes to decide how the software should actually be built. No, this is not tautology. We just need to take into account Conway's law and expand on it a bit.
Any organization that designs a system will inevitably produce a design whose structure is a copy of the organization's communication structure.
Part of the design process of a project should include design the team(s) structure, the project management strategy and the software engineering practices in order to align the end result with what is desired. Ignoring this leads to imbalance in the project, and if that imbalance is big enough, and goes on for long enough, the project is going to rip itself apart.
* Nitpicker corner: I said many, not all. Don't bother to list me software process that deals with it. I had a reason to explicitly list the processes that I did.
