After reaching certain level of maturity in Continuous Integration (CI), teams would want to push forward and get to Continuous Delivery as quick as possible. Possibly, this is the first place where dots have to be connected together. I’m positive that deployments would have been automated while CI was trending but mostly those automated deployments would have been scheduled nightly.
Meaning, if you have Dev, Integration, Perf, Stage and Prod environments, every check-in gets compiled, unit tested and runs code analysis and may be deploys to Dev. Nightly deployment repeats this same process and deploys the package to Integration. All other environments gets new build in some schedule.
Also, teams would have been automating the test case execution and might be running them on night schedule.
Similarly, nightly deployment and test execution feedback are not connected to each other.
If this is your current state of the union, this exposes some deficiencies to be fixed before connecting them together.
- who owns the deployment automation scripts?
- why deployments are running nightly?
- how long does it take to complete the deployment?
- Does it run deployment verification test post deployment?
- Do we care, if nightly deployments fail?
- What is the success rate of deployments in last 30 days?
Nightly test execution
- who owns those tests?
- what kind of tests are they? API, GUI?
- what is the success rate of those tests in last 30 days?
- what happens if those nightly tests are failing? does the team care and fix it immediately?
Answers to these questions will help determine the current state, culture and next steps..
If deployment scripts are owned by another team (build team/deployment team/ALM/Ops/what not..) – this capability needs to be brought into the team in order to be able to self service. Otherwise, you will be in the Queue with the other team when your deployment fails.
If deployments take too long (may be >30 mins) – better to optimize the deployment before integrating with the CD pipeline. If we want to instantaneously promote every single check-in, longer deployment would delay the feedback cycle. Developer time is precious, not reasonable to wait for more than 30 mins to get feedback on every check-in.
If deployment success ratio is poor (may be <95%) and team is happy about it– better to fix that cultural problem first. In CD journey, team commits to keep the software production ready over working on new features, can’t afford to ignore failures in pipeline.
If there are no deployment verification tests – first lets create automated verification tests to gain confidence that deployment actually works.
If automated test cases are owned by another team (QA team) – first step is to break that culture and build cross functional Agile team and nurture the culture of “whole team owns quality”
If automated test success rate is pretty poor (45 of 250 test cases fails randomly every night) and team doesn’t care about it – immediate next step is to alter the strategy to build confidence and ownership around test automation. Unless team is committed to react and fix the failure asap, CD journey won’t make progress. Deployment will pass –>automated tests will fail –> team will get used to the failing tests –> teams will ignore legit failures –> bad code gets promoted –> defeats the purpose of Continuous Delivery (always be production ready)
Overall, part of CI to CD cultural shift journey will look like
We know its a long journey, but, did we gain anything by making these changes? some obvious benefits..
- Improved feedback cycle. Developer highly likely to get Integration feedback within few mins of check-in not after 24 hours. Helps to keep up clean code base.
- You are building highly confident continuous delivery culture, extended feedback cycle becomes routine for team members.
- Step towards true cross functional Agile teams and DevOps culture, no more QA or build/deployment team dependency.
- Improved confidence on automated testing, you likely have fewer meanigful tests with high success rate.