Skip to main content

Measuring Goals for DevOps Test Succcess

Each organization will have to define what their quality goals are for integrated testing based on the business and customer requirements for speed and outcomes.  The fact is that these goals must be quantified or in other words measurable. If you are not measuring your testing activities in alignment with the strategic goals then success becomes subjective and it will be very difficult to show value for your effort.

Some examples of measurements might be in the form of Run-To-Plan and Pass-To-Plan.
  • Run-To-Plan (RTP) is the number of the total planned tests that have completed and typical goals are to have 95% RTP
  • Pass-TopPlan is the number of tests that have passed and a typical goal for this metric is to have a 90% PTP
The criteria for determining when testing is complete is agreed by all stakeholders.   It will be impossible to have all stakeholders on the sprint team, but certainly input and validation from key stakeholders will have to be included before acceptance of the test results.

When considering goals for reliability an S-curve diagram for pass rates could be used to show when the pass rate stabilizes to meet your goal. 
  • Example:  95% pass rate for 3 consecutive test cycle
The recommendation is to consider your strategic and customer goals.   Determine the metrics and the measurements like those examples above for testing and how measuring these things can help you to meet those higher-level goals.  Other goals and metrics will need to be aligned and none more so than the release criteria which will include metrics to determine whether the Entity Under Test (EUT) is ready for release.  What are your “goals”?  What metrics and measurements from integrated testing will be required for you to meet them?  Share with us what you are measuring and what goals these metrics might help others to meet.

For DevOps education and certification:
…educate and inspire!


Comments

Popular posts from this blog

What is the difference between Process Owner, Process Manager and Process Practitioner?

I was recently asked to clarify the roles of the Process Owner, Process Manager and Process Practitioner and wanted to share this with you. Roles and Responsibilities: Process Owner – this individual is “Accountable” for the process. They are the goto person and represent this process across the entire organization. They will ensure that the process is clearly defined, designed and documented. They will ensure that the process has a set of Policies for governance. Example: The process owner for Incident management will ensure that all of the activities to Identify, Record, Categorize, Investigate, … all the way to closing the incident are defined and documented with clearly defined roles, responsibilities, handoffs, and deliverables.  An example of a policy in could be… “All Incidents must be logged”. Policies are rules that govern the process. Process Owner ensures that all Process activities, (what to do), Procedures (details on how to perform the activity) and th

The Four Ps of Service Design - It’s not all about Technology

People ask me why I think that many designs and projects often fail. The most common answer is from a lack of preparation and management. Many IT organizations just think about the technology (product) implementation and fail to understand the risks of not planning for the effective and efficient use of the four Ps: People, Process, Products (services, technology and tools) and Partners (suppliers, manufacturers and vendors). A holistic approach should be adopted for all Service Design aspects and areas to ensure consistency and integration within all activities and processes across the entire IT environment, providing end to end business-related functionality and quality. (SD 2.4.2) People:   Have to have proper skills and possess the necessary competencies in order to get involved in the provision of IT services. The right skills, the right knowledge, the right level of experience must be kept current and aligned to the business needs. Products:   These are the technology managem

The ITIL Maturity Model

Most organizations, especially service management organizations, strive to improve themselves. For those of us leveraging the ITIL® best practices, continual improvement is part of our DNA. We are constantly evaluating our organizations and looking for ways to improve. To aid in our improvement goals and underscore one of the major components of the ITIL Service Value System , Continual Improvement .   AXELOS has updated the ITIL Maturity Model and is offering new ITIL Assessment services. This will enable organizations to conduct evaluations and establish baselines to facilitate a continual improvement program. A while back I wrote an article on the importance of conducting an assessment . I explained the need to understand where you are before you can achieve your improvement goals. Understanding where you are deficient, how significant gaps are from your maturity objectives, and prioritizing which areas to focus on first are key to successfully improving. One method many organi