A short story about fish



About 6 months ago my team embarked on an initiative to upgrade our Continuous Integration (CI) & Automation pipeline. As an iOS team, the fact we already had a comprehensive suite of automated functional tests hooked up to Jenkins was impressive and gave us a solid foundation. However, our setup had started to creak at the seams. Inexplicable test failures, brittle tests, failing tests that miraculously worked if we gave them a swift kick in the nether regions and ran them again! We decided to evolve to a CI 2.0, which would be a lot more stable, massively reduce the amount of time we spent nursing our tests and ultimately give us better confidence in our system. We are not done yet but we have re-learned a number of basics along the way

Lesson 1: with CI & automation, fast feedback is king
Our monolithic test suites took hours to run. This discouraged developers from running them on every check-in, which is counter productive. Not running them means you increase the chances of breaking functionality or introducing defects with each commit.

To achieve faster feedback we have split our tests into smaller test suites each with its own Jenkins job which means we now start to get feedback in tens of minutes rather than hours.

Next steps :
We want to set up a grid system, to run the tests in parallel to further reduce the time to run

Lesson 2: Responsibility

When you don't have fast feedback You also get in a situation where breakages can't be easily traced back to a single code commit, as the 'window of culpability  is open for a few hours/ half a day. Anyone who commits in that period could be at fault. Who does the investigation to see what caused the breakage? Without trace-ability  it's often a case of thinking 'someone else will look into the failing tests' which means, probably, no one will look into it!

We introduced the concept of a 'Weatherman'. Someone who is responsible for investigating any breakages and getting them fixed as priority. This role is rotated on a daily basis so it doesn't become too much of a chore for any one individual.

This brought more visibility of breakages and helped change the culture of the team to have quality and visibility of quality as a priority on a daily basis.

Lesson 3: Granular feedback

I mentioned already that we broke our tests into smaller suites. Great for fast feedback, but also granular feedback which helps when trying to pinpoint the breakages. Also, if you have one failing suite you still have confidence in all the other passing suites. With the monolithic test suite we didn't have that confidence.

Lesson 4 : Keep your tools sharp & get new ones when you have to

We were using Apple's UI automation tool, but it started to show signs of flakiness and incompatibility with what we wanted to do. You would expect apple to produce a solid tool but further investigation showed that they don't really invest too much in UI automation which meant we had to start considering other tools.

We have now started to use FRANK for automated acceptance testing, which has led us down a behaviour driven development (bdd) path. (Bonus!)

Lesson 5: Appreciate legacy code

Appreciate when your legacy code is giving you value and not costing you much - you don't always have to migrate everything.

We continue to run our stable legacy tests in parallel with our shiny new frank automation tests. They are stable and don't cost us much time in maintenance and still give us good value. It would take months to fully migrate them so we won't bother unless we have a compelling case for making that investment.

Lesson 6: Adopt a culture of quality

As mentioned previously, the concept of a weatherman, who gives the team it's daily weather report, has massively helped change the culture of the team. Quality is everyone's responsibility but on a daily basis it's even more so the weatherman's responsibility!

Having experienced the pain of brittle tests packs, slow feedback, poor tools etc the team now knows the value of staying on top of our CI and automation packs and ensuring we have the speed and feedback that we need. .

Conclusion 

That's our story, we are more than halfway to having a slick CI pipeline. The improvement we have made is fantastic bearing in mind that all this was done on top of regular sprint work and deliveries. Great commitment from the team to improve the culture, environment and toolset.
Previously we were getting some of the value of CI & automation with a considerable maintenance cost. Now we are getting a lot more benefit with a lot less maintenance. Happy days.

Finally, To make sure nobody leaves disappointed ..... Here is a picture of some lovely fish!


Comments

Post a Comment

Popular posts from this blog

Expanded EPIC boards

Image
What's the problem? As scrum teams, we always look to the Product Owner as the fountain of all knowledge and requirements with respect to our supply of work. Yet, I've found that often the big ideas or concepts for projects are so light on detail that POs are desperately in need of a way to translate the high level idea into a backlog of stories to get the dev team up and running We've all been part of projects where the BAs and POs lock themselves away for a few weeks with a high level understanding of a project and come back with a list of user stories for the dev team to work on. By the time the devs see the requirements, its hard to see the wood for the trees. We get bogged down in the details, the stories are sliced poorly & are not delivering vertical slices of functionality. Often there's no wiggle room. Its all MMF. We cant cut anything out. Additionally, defining an entire release worth of stories in advance which, ultimately, require change or may
Read more

Things to consider when you want to change from Scrum to Kanban

Image
Scrum or Kanban? A common problem with agile teams is whether they should choose Scrum or Kanban as their primary methodology. Scrum is the usual starting point for most teams and the current market forerunner when it comes to selecting your agile weapon of choice. According to the 13th State of  Agile report 54% or teams surveyed claim they are using Scrum, with a further 10% claiming they do a Scrum/ XP hybrid. Kanban trails behind at 5% with Scrum-ban hybrid adding a further 8% "Scrum doesn't work for us!" I often hear teams saying that scrum doesn't work for them. When I ask "why?" I usually hear the following: We can't finish anything in a 2 week sprint There are too many meetings Our list of priorities keeps changing We can't plan for 2 weeks.  Our product owner keeps changing his/her mind They often feel like Kanban is a silver bullet because: It doesn't ask you to finish anything in 2 weeks It doesn't im
Read more

Moving beyond tasks

Image
One of the most common agile anti-patterns I witness is a compulsion towards breaking user stories into tasks. Tasks take focus away from stories The problem with tasks is that they take the focus away from user stories. Teams that focus on completing tasks risk losing sight of the real value which lies in the stories. They are often unable to see the wood for the trees. How often have you seen sprints that are? •             On day 9 of 10, but <30% of story points are complete •             On day 5 of 10, but <10% of story points are complete •             On day 9 of 10, with 90% of tasks complete, but still <30% of story points complete I firmly believe this is a by-product of too much focus on tasks and not enough focus on stories. Tasks themselves add little or no value on their own. I often see tasks such as “Manual QA”, “Code Review”, “Write test cases”, “Database work”, “Front end work”, “Back end work”. People often argue to me that the task breakd
Read more