While you might not be able to get your head around how to measure DevOps when it is defined as a cultural and professional movement, there are some key factors to consider that will help us to measure the success of shifting to a collaborative and integrated team throughout the value stream that includes Dev and Ops.
It is clear that DevOps goes beyond the integration of the development and operation staff but includes breaking down those silos between all parties and stakeholders including the Business, external partners, and 3rd party suppliers and IT. If you think it is difficult to get your own internal teams to play in the same sandbox consider the challenges when 3rd party vendors and suppliers are then thrown into the mix. Being able to demonstrate proof that DevOps practices benefit the organization requires examining factors that influence overall performance. Practices that enable your organization to improve the flow of work between the Business and Operations enables improved IT performance.
IT Performance in measured in terms of throughput and stability.
- Throughput is measured by deployment frequency and lead time for change
- Stability is measured by mean time to recover and the ability to preemptively detect and mitigate problems
Simply put, you cannot measure Development outcomes separately from Operational outcomes.
Showing proof that DevOps practices benefit the organization also requires baselining key metrics before and after making improvements and can include:
- Deployment Frequency
- Change Lead Time – Time from request to delivery
- Cycle Time – Time from start of work to ready for delivery to customer
- Change Failure Rate – Backed out, Rescheduled or unforeseen impact/incidents
- MTTD – Mean time to detect incidents
- MTTR - Mean Time to Recover or repair - Component
- MTRS - Mean Time to Restore - Service
Change lead time is one of the most important metrics as it represents what the customer sees. Cycle time is a more mechanical measure of process capability. Lead time depends on cycle time, but also depends on your willingness to keep a backlog, the customer’s patience, and the customer’s readiness for delivery. MTTR is measured from when a component fails until it is repaired. MTTR does not include the time required to recover or restore service. e.g., data may need to be recovered before service is fully restored and delivering its normal functionality.