Improve Test Automation visibility to succeed in Agile Quality transformation

One of the main success factors of Agile software development is, it makes everything terribly visible. If you walk into the area of a development team that’s practicing Agile, its highly likely to see some burn down charts, visible dashboards, monitors showing backlog, sprint progress, blockers and of course list of Tech Debts that they currently work or want to knock down soon.

For Quality practices to be successful and to benefit the overall agility, visibility is one of the key success factors too. In another blog post, I would have mentioned about preparing your QA’s for Agile practices. Knocking down separate QA\Perf\Security teams is the first step towards building self-sufficient product team to deliver concept to cash. However, once the team is formed, it’s absolutely critical manage visibility of the backlog.

For example, below is the task out of a story including automation tasks and keep it very visible part of the backlog.

• Visibility is critical to make fellow team members aware of the tasks
• Visibility will encourage them to pull the test automation tasks even though it might not be their speciality
• Visibility will improve collaboration between developers and those who are trying to make transition from conventional QA role.
  • For example, one of the thumb rules in test automation is “DRY” tests. Don’t Repeat Yourself tests. It’s crucial to analyze each and every test that we plan automate and answer this one question. “Are we automating the right thing and by applying the right technique?” ..If something can be automated at the unit test level, that’ll offer much faster feedback, love and care from everyone. If something is already unit tested, should we repeat that again in another form of test such as Integration or GUI level surface tests?

Further more, if you practice Kanban, WIP limits could come handy when team attempts to commit more development tasks before completing a story meeting DoD.

Once everything becomes part of the same backlog (remember no separate QA\Automation backlog), collaboration kicks in, its vital to elevate the visibility around the state of the union. Automated Tests should be part of the delivery pipeline and highly visible to each and every team member across the globe. When tests run part of pipeline and failure stops the build from being promoted that’s going to gain more attention.

For example, below is snapshot of test execution dashboard.

Little note about the tooling here, regardless of the frameworks in use, ie. Specflow + xunit or Jasmine tests or JMeter tests or Selenium tests, what not.. everything would offer a way to gather the test run logs and plot some sort of nice chart showing the results.

More interesting  visual could be a histogram showing the success\failure of tests over time so that the team gains confidence by seeing more success

Time taken for each test run so that the team can analyze and go smart to shorten the execution time

Flakiness index (sort of failure rate over time) that might help connecting some global event that affected my app even though there is no change on my side

If we provide a way for easier test failure analysis without having to wait for the expert to come in from another side of the planet, something like test execution screen cast or log link from Saucelabs, Blazemeter, Visual studio online etc..

(more detailed logs in each line in the highlighted area below)

If we start practicing some of these techniques, it’ll make the Agile Quality transition more visible and next step is to identify obstacles and make them visible and keep pushing forward.