It was a dark, and stormy night…
okay it’s not that kind of story. But, many agile transformations happen because something went dark or involved some amount of screaming. This is the story of how I came to embrace agile software development over other traditional methods, like “waterfall”.
“Failure is success deferred. The only person that can declare your project a failure is you.” — Ray Kurzweil
In the beginning…
The greatest thing I learned from studying music in college was how to react to change. The music director or conductor would give a command to the group to play louder, or faster, or to even switch instruments and we had to respond in an instant (see my post “Agile Like Jazz“). Likewise, the best thing I learned from studying computer science in college was Waterfall software development. Yes, this is a blog about being agile, and we’ll get to that, but first things first.
My classic waterfall experience
After college I was eager for a chance to apply all I learned, so I found work with a small mortgage brokerage company. They wanted to transform their industry and increase profitability. It took several months of meetings to determine a project plan and to generate reams of requirements and design documentation. Even at the end of that process, there were many unknowns and the client was getting impatient that we had not started producing any code. So with most of the designs worked out, we started coding. Many months later, we came back to the client and gave them a report of our progress, including a demo. “This isn’t what we wanted” was the common theme, even though we followed the incomplete requirements closely. Even worse, we completely missed capturing an entire second half of their business processes due to being totally focused on the first half in the up-front design. Talk about a mess. Ultimately we failed to deliver before the client ran out of money. All of this just happened to coincide with the housing market crash of late 2004. That client went out of business and we went….
Back to the drawing board
I went back to TCU, my alma mater, to the library to try to determine what went wrong. After a great deal of soul searching and research, I found evidence of a group that was trying to change software development. First I found XP (extreme programming), then agile and Scrum. At first I thought they were just trying to cut out all of the documentation. Setting up a big project plan up front would usually impede the one thing that most of my clients desired most — having the flexibility to change. Whether it be the plan, the look and experience we were try to create with the software or what we were going to release into the wild as we were in progress on its development.
They wanted to be more like a jazz composer and band director, creating a (flexible) plan, the musical score, but tweaking it upon execution, rather than being a passive audience member just listening to the performance.
My agile transformation experience
The first chance we got, we put together a small team and started adopting Scrum practices. We did so while working for an educational software company. Almost immediately we started seeing benefits from the changes we were implementing. Our client (the Product Owner):
- Was more engaged in designing their product.
- Participated in all our Scrum rituals (aka meetings).
- Saw value every few weeks and felt they could ship our product quickly.
However, Scrum is more of a project management tool than a set of technical practices. So we began to see the effects of technical debt and a loss of some quality in the code base. And just like Martin Fowler wrote, we felt the need to re-visit our technical practices. We beefed up our quality practices, hired a new QA person and tackled the debt. Several agile conferences and many retrospective meetings later (that first project is now coming up on its one-hundredth sprint), we also decided a management transformation was necessary. We re-wrote our hiring processes, eliminated administrative wastes across the board and even re-branded our company, including a new mission and vision statement to align with our new found focus on agility.
I heard Steven Borg say that you will fail more with agile. With waterfall, you fail once and the project is over. With agile, you fail often and early, then learn each time how to improve the process and the product as you go. It took more than a year to realize our first failures with the waterfall project, but it only took two weeks to uncover problems in agile. Software development is not a predictable process, but traditional methods would like to assume that it is. For me, being agile is freedom to explore the best possible outcome for a software project as it takes shape, curbing risks and converting failures into great successes.