Agile or Lean, which is the bastard child?

I'm kind of tired of these debates since they don't really do more than the Mac vs. PC debates. Who cares which came first? Does coming first confirm value? superiority? inferiority?

Anyway... Mike Alber posted to the Agile Alliance LinkedIn group the following question:

Agile and Lean Software Development - an Oxymoron?
What do you believe - is Agile is an instance of Lean, or together are they are an oxymoron?

My response:

They are both journeys towards the same end goal with slightly different paths. To me it is like asking if driving from Florida to Maine on I-95 vs. Rt 1 is the same or different.

Yes they are different. But they are both going in parallel and towards the same ultimate goal. They even cross and overlap each other quite a bit. It also seems their start was about the same also.

I really like what Matthew Chave had to say:

1 of 2 ways to go here.....

1. Agile - throw your processes away and start again....take Scrum and XP and implement them....

2. Lean - use lean techniques to remove the waste from your process......

What happens if you do 1......well you probably fail as its too much to do in 1 go......

What happens if you do 2.....you gradually improve your process and in the pursuit of "perfection" gradually refine your heavy-wright process to become something that looks a lot like Agile......

Moral of the story (and to quote Martin Fowler on the topic):

"You can't really talk about them being alternatives, if you are doing agile you are doing lean and vice-versa. Agile was always meant as a very broad concept, a core set of values and principles that was shared by processes that look superficially different. You don't do agile or lean you do agile and lean. The only question is how explicitly you use ideas that draw directly from lean manufacturing."

Are Deadlines Important?

Olga Kouzina posted about deadlines on her blog here. She then posted a related discussion thread on LinkedIn's Agile and Agile Alliance groups. Next thing you know, there are 26 combined comments between the two threads!

She is basically questioning the core concept of deadlines. Do we need them? What value to they add? What negative impact do they carry? Unfortunately, some people mistakenly took this to mean that we should never work with goals, delivery dates, or time based measurements. I had to clarify my stance later in the thread with the following:

I believe in and agree with groups that leverage timeboxes, iterations, and release dates. I see these as good metrics and motivating goals.

I believe that Olga and I were talking about Deadlines with a capital "D". These are the dates put in the ground before the team has committed to scope, effort, etc. We know the requirements will change as we work the project, why won't the date change also?

I've seen too many projects where you realize pretty quickly that the "Deadline" was picked due to a desire but no real business need or planning validation. These dates are a great way to de-motivate a team... after all, why work hard if you already know you can't make it?

When Olga asked why I might think we put up with deadlines, I responded with:

We stick to them because people with higher authority don't trust a better solution and force them on us.

So... it's our job to start educating the decision makers around or above us. This is a topic I recently blogged about.

Ben Linders summarized the agile point of view well with the following:

If there really has to be a deadline, then it's up to the PO, project managers and stakeholders to do whatever they can to ensure that the teams know what to deliver, and have an enviroment and the right conditions in which they can be most effective and efficient. IMHO the deadline should never be passed on to the teams! The teams commits to iterative deliveries of highest priority user stories. That automatically leads to maximum result in the shortest time, which increases the change that deadlines are met.

And finally, I like Vinay Krishna's points:

Can we achieve anything without deadline? It looks funny to know that one starts a project and says that he doesn’t know when it would get finished.

I want to quote here the famous Parkinson Law which states that “Work expands to fill (and often exceed) the time allowed.”

This is important to recognize in software development because when we are not restricted by time and/or money, it’s easy to create an endless amount of features or strive for what we perceive as perfection.

Of course we can all agree there are sometimes market reasons for deadlines. After all some dates can't be moved (Y2K, elections, tax dates, etc).

Is the PO part of the team?

Jack Milunsky posted this question on his blog here. He then posted this question to the Agile Project Managers group here.

16 comments later, this is quite the discussion. Many of us agreed that PO's should be part of the team and here are a few snippets:

Jack Milunsky-

I just can't see how a team can think about moving forward without the PO being part of the team. I guess time will lead the way with positive stories being told.

Prasanna Prabhu-

Prime reason why waterfall project fails is b'cos PO's are not involved & when they get involved its too late in the game, any change request can cost ton of money due to which you may loose market share.

Kevin E Schlabach (me)-

The PO is either part of the team, or they need to delegate their authority to people on the team (I've seen this work where the PO delegated to the lead product analyst).

The PO has to participate in sprint planning and review, otherwise you didn't get customer acceptance! The PO should participate in the retrospective. He's part of how well (or not well) the team is doing.

My experience has shown me that teams doing well in agile do want this from their PO, and teams not doing well in agile don't. Also, some teams do well in agile transitions without the PO at this interaction level. For the most part this implies that either the group doesn't truly get agile (holistic self-empowered team), or that they do and the PO is an impediment to this (so they move forward without them).

It wasn't until we trained the PO's in agile that this started to positively shift at Siemens Medical (many teams, many products, many PO's). A lot of companies make the mistake of pushing agile as a tech team solution and the PO and analysis roles (and UI/usability) are left standing alone on the dock. From a Lean viewpoint, they have optimized a small part at a cost of de-optimizing the whole.

Otis D-

I agree with those of you that believe that the Product Owner should be part of the team. As the saying goes in Scrum, the “PO is the single wringable neck”.
This is, amongst others, the person who is responsible for maintaining and prioritizing of the product backlog. The person who makes initial suggestions regarding what stories to work on ( in the sprints), the one who maintains the release plan, and a plethora of other crucial tasks.

I'm happy with the Toyota that doesn't work!

I have to expand on yesterday's post. The LinkedIn discussion has had a few more comments added to it and there are some valid viewpoints. But, because of the discussion, I found myself coming up with this metaphor to further answer the original question-

I ask Toyota to build me a cool new car.

They deliver to me a shiny new one-of-a-kind car. I love it. I check it over and every part is there and connected properly... looks great!

... it doesn't start.

If they forgot to put gas into it... that's just bad account management.

If they never started it themselves to prove it worked before delivery, the questions are-
Is the story DONE?
Do I give them credit for their work and pay the bill?
Do I create a new story to "fix" the problem (additional cost)?

Fundamentally, do I get value out of the work they delivered?

I think the metaphor leads to obvious answers, so now I'll play devil's advocate. The car I ordered is going in the showroom of my dealership. It will never get driven anywhere. It's a halo prototype to get people to come to my dealership. In this light... it is DONE. Why pay (in time or money) for the "fix"?

This counter-point is that you can't answer the question unless you understand the value trying to be achieved.

Testing, Estimates and Burndown...

Adam Feldman posted the following question to the Agile Alliance forum on LinkedIn:

In your experience, what do you all do in estimating for testing? Do you include the time expected to test the story in the points allocated for the actual story, or is this normally not included?

The second part of my question regards burn down charts. If we are not allocating points to the actual testing of the story, is the burn down chart really only telling us what is built - not what is actually complete?

The first three responses answered that testing should be part of the team effort and not an after-thought. To help Adam find a way to explain this for the team and management, I focused on the concept of DONE criteria:

The fundamental question here is "what is your DONE criteria?".

If the team's DONE criteria is built (but not tested), then the burndown and estimates do not include testing. This will lead to an optimized development team, but potentially create problems for testing and ultimate delivery. Things will get built faster, but ultimately may be delivered slower (which is not upper management's goal).

If the team's DONE criteria is delivered business value to the user/customer (the true agile definition), then the estimate and burndown includes testing effort. Testers should be part of the team, tests need to be run within the sprint, stories should be completed within the sprint including testing.

This isn't easy, but the second option is the mature one.

There are two pieces to overcoming this. You (the testing group), need to learn to work quicker and more closely with the people building. They (the developers) need to learn that DONE is defined by business value and not code complete. If "you" and "they" both overcome this, then everyone will become a "we" or "us". Testers become involved in sprint planning and review also.

I've blogged about this several times if you are curious:
Downstream testing
Tests after story closure
What to do with found bugs
How do testers fit in agile

I'm curious where the discussion thread will go over the next few days on LinkedIn. Anyone have additional thoughts?

Pot of Gold at the end of the rainbow...

Many of us get our agile information from many sources. InfoQ tops most of our lists, conferences and specific blogs have their followings, and specific groups have their own blogs or websites. I tend to uncover lots of odd nuggets by setting RSS keyword scrapes on Wordpress.

Today I learned about a whole new area I've been out of touch with.... email groups in Yahoo. Historically I've avoided Yahoo groups because my Yahoo experiences have been problematic, but I also didn't know what I was missing.

If you are like me, you want to see this. Mark Levison invested some time for our benefit to document the best Yahoo group agile lists out there including a description of each. It's like I found the pot of gold at the end of the rainbow! Hopefully this will help you as much as it might help me!

Good Enough Metrics...

Another post triggered by my interaction with other customers using VersionOne. We were discussing story points, velocity, capacity, and other metrics. Here's what Tom said when he was trying to figure out team capacity each sprint:

As an example some of our teams state the base capacity of every team member (assuming a 2 week sprint) is 10 days * 6 hours a day. The 6 hours rather than 8 is to account for emails, minor interrupts like unplanned meetings etc. Then they reduce capacity based on company mandated holidays (we are global so ever office is different), then they reduce hours for the sprint meetings. this becomes the max anyone on a team can contribute to a sprint. The team members may additionally subtract off vacation time, or other meetings they are required to attend, trainings, etc. it would be nice if a team can define a default formula based on known factors, and team memebrs have an additional ability to have thier own formula that can further adjust thier own availabilites beyond that, or even just override it with a typed in value.

Wow! This sounds like a lot of work! Does this additional planning accuracy actually provide a similar level of accuracy predicting the outcome? I'm guessing it doesn't. When we transition to agile, over time we slowly learn to trust what our agile coaches / education told us up front. Here was my response:

This is great and I remember going through this myself. I had excel spreadsheets in the back of my sprint backlog where I could enter holiday and other data to get this value.

What we found over time was that it became somewhat unnecessary as your velocity and team matures. We slowly transitioned to story points and relying on them to do sprint planning. We picked our stories and blew out the tasks for them only when the story was picked up (mid-sprint we swarmed on open stories and closed them every day instead of running them all in parallel). We did just in time task planning and estimation (more efficient).

This led to us no longer needing all those calculations. Holidays and vacations were a simple head calculation (example: typically we do 20pts per 10 day sprint with 10 people = ~.2pts/person/day, we have 5 people taking 1 day of vacation = ~1pt... so plan 19 pts this sprint.)

It's an interesting problem to solve in the short term, but I'm guessing every team will have a slightly unique view on the calculation... and I'm guessing most people will outgrow this need over 6 months to 1 year.

I'm a strong believer that conversations of planning and capacity should focus on days and story points, not hours and tasks. Without getting into the debate that estimation isn't needed at all (that's another discussion), what are your thoughts?

Agile and Usability, shake and stir...

I've talked in the past about Agile and how usability practices or resources fit in because I spent some time working in that field and I find it is an often overlooked and misunderstood part of software development in general.

Here's an interesting article in MSDN magazine that delves into the topic further.

Estimation in Kanban?

Before I start, I should note that there is a lot going on in the Agile community related to Lean and Kanban. In case you've been in a cave, you might want to check out this great slide deck by Henrik Kniberg comparing and contrasting Kanban vs. Scrum, or check out Karl Scotland's blog on the topic.

That being said, Anna Forss blogged today about how estimates do or don't fit into Kanban. An excerpt:

The main reason for estimating tasks is to be able to decide on an appropriate work load during the sprint. During the sprint planning meeting, the team estimate how big tasks are and when the work load (estimated task sizes) matches the available resources, the team commits to the sprint backlog.

So, how about kanban?

Jump to her full post to see how she answers this question.

Here's my comment on the topic, even if it is a little open-ended:

I was under the impression that estimates aren’t needed as much in Kanban because items are supposed to be closer to being “of similar size”? Looking back to the original Toyota manufacturing process application, this would have been true.

Otherwise, a large item might hang in your work queue for weeks (because it is too large) while small items fly through. If several large items “get stuck” then it might clog the whole kanban system.

So… do you need to estimate to insure that items are small and closely equivalent? (maybe)

This also make the concept of MMF (minimal marketable feature) an important goal when writing stories!

Automated Acceptance Tests...

There's an interesting article on InfoQ about automating acceptance tests. It is hinting that this practice hasn't had the successful following that some of the other XP practices have had. The article appropriately ends with this conclusion:

Now, consider if the tests written by the QA department are written before the development begins. The information provided by these scenarios now occurs at the beginning of the iteration in a predictable fashion. Therefore the uncertainties are reduced, velocity becomes more stable (fewer random interruptions), and that means more predictability.

So, are automated acceptance tests jus something the elite (or lucky) have been able to make work? Is there an internal flaw that is unseen that has caused it's less-than-stellar adoption? Or is it just difficult with proven benefits and a practice that every software development team should aspire to adopt?

Having attempted to do automated acceptance testing with my last team in a healthcare setting using FitNesse, I left the following comment:

... The value of automation is the repeated run.

Automated acceptance tests can have a high value for high data exchange (as opposed to screen manipulation). For example, test a signup or registration form for all of the required fields, field level logic, etc. With regression testing, this insures data integrity. They are also a good fit for testing REST interfaces or other public accessible API's.

Don't use them for drag and drop UI testing or color/stylesheet testing.

Do any of you have experiences in this area?

Scrum weaknesses and Potentially Shippable...

Jack Milunsky posted another great article on the Agile Software Development blog about the areas where scrum falls short. It's an interesting read and I encourage you to consume it. I'm a strong believer that Scrum is a great starting point, but an agile transition shouldn't end with it.

One point I didn't agree with though was this:

Potential Shippable code - what does this really mean?
Most Agile thought leaders define the output of an iteration as an increment of potentially shippable code. And more often than not, it appears to be acceptable (please tell me if I perceive this incorrectly) that this potentially shippable increment of code is not actually live production code. And therein lies my beef with Scrum.

Here's the response I left:

We have to remember that when "potentially shippable" was created as a concept, we were combating waiting months or years to ship a product. In large enterprise environments, this was an even larger/longer problem. This new concept embodied the idea that we could stop at the end of any 1 month iteration and have something working (and probably of value).

Since then, the industry at large (due to the internet and companies like Google) has embraced eternal betas, imperfect software, and constantly evolving software focused on constant value. Within agile, we've seen a migration to iterations under a month. I believe it is time to rename this to "proven shippable". This is the compromise between Jack's comments and the enterprise's needs. Enterprises build large solutions to large problems and they understand the needs of the market. When a complex domain (like banking or healthcare) already has mature software solutions, you know your product won't displace the competitor's software after the 2nd or 3rd iteration. There is effort to marketing and advertising new products. It is important to have a release cycle and the power that comes with it. You can't just dribble out small changes every week and expect to catch the market's attention.

This is why I think that in small continuous flow shops, I agree with Jack's done = production use. But in large shops with large customer bases, you might need to allow the business and market determine when to take on the new release and not force it on them. If your releases are compelling enough, they will migrate (VersionOne has mastered this!) In this case I don't think your done criteria can be production.

Thus, I propose -> "proven shippable". This means you have shipped it to somewhere external to your development group that is a proxy for the customer base.

Agile Maturity Model & Scrum Success Story...

I wanted to make sure my readers had the chance to review these two links, so I'll share them here:

• Ryan Martens over at Rally talks about IBM's (Scott Ambler's) proposed Agile Maturity Model. It's a good read, I agree almost perfectly with his statements. My take on these topics is this: once you box and define agile, you lose the magic. Agile is about inspect and adapt. Definition for the mainstream removes the adapt part (it just sets a straight goal) and then inspection is unnecessary because the definition allows you to falsely believe you've attained your goal. (Hint: The goal keeps moving as you get better!)
• Samir Bellouti posted a great experience report about his team's successful first year of Scrum implementation. I'm no longer a strict follower of Scrum, I tend to blend various agile approaches, but I believe that if you don't have a coach or resource to help you through implementation, then Scrum is a great recipe to start with. It's most documented and supported and will allow you to figure it out on your own with the published works available. This story is an example of how it SHOULD turn out and notes the value provided to the business.

Folks, agile is not easy! Just like dieting to lose weight, going agile is hard work for a more efficient and valuable outcome.