Skip to main content

DevOps Test Monitoring Strategy

The combination of continuous monitoring with continuous testing and analytic tools can provide a broader strategic view of test results.  This view is necessary to collect, aggregate and organize test data that enables a gain in confidence for each release. 

Key Concepts for Realizing Your Test Monitoring Strategy:

Determine continuous test monitoring priorities: Some examples of problems that continuous test monitoring can help with include intermittent failures caused by marginal designs, marginal test designs, environmental condition changes not detected by individual tests, memory leaks, varying starting conditions, interactions with other systems, system topologies and performance degradation within the margin of a test. These can and will accumulate over time. The best practice for continuous monitoring indicates that the problems of most concern to a specific product or DevOps environment will be monitored.

Regression test product areas even though there were no expected changes:One not so good practice is to skip tests for areas that are “known“ or those that have not changed during individual test cycles. Unintended consequences of indirect changes may impact performance, so a regression suite should audit these areas occasionally just to be sure. Typical examples that are often caught by this are features important audit functions, system backwards compatibility and those that could disrupt critical business operation.

Select continuous test monitoring tools that collect and report trends:Tools that can correlate and report test results across multiple dimensions such as test cases, product features, system performance categories, build versions, releases and functional tags can find intermittent bugs or problem trends. Thresholds, email alerts and dashboards that highlight short-term results views from long-term results views are especially valuable.

Use continuous test monitoring results to diagnose and resolve problems:Once a diagnosis is determined, the root cause can be verified through targeted retest cases that set all the conditions in accordance with the diagnosis. Once confirmed then the offending design can be refactored to handle or avoid the failure condition.

A Test Monitoring Strategy with a holistic view is critical for test results evaluation over multiple releases. Intermittent failures and problems only become apparent when test results are presented as trends over a long-term trend. Failures are easier to diagnose when a large data set is accumulated and filtered. Take action. Get a test strategy that includes a Test Monitoring Strategy that works.

Inspire & educate…

Comments

Popular posts from this blog

What is the difference between Process Owner, Process Manager and Process Practitioner?

I was recently asked to clarify the roles of the Process Owner, Process Manager and Process Practitioner and wanted to share this with you.

Roles and Responsibilities:
Process Owner – this individual is “Accountable” for the process. They are the goto person and represent this process across the entire organization. They will ensure that the process is clearly defined, designed and documented. They will ensure that the process has a set of Policies for governance.Example: The process owner for Incident management will ensure that all of the activities to Identify, Record, Categorize, Investigate, … all the way to closing the incident are defined and documented with clearly defined roles, responsibilities, handoffs, and deliverables. An example of a policy in could be… “All Incidents must be logged”. Policies are rules that govern the process. Process Owner ensures that all Process activities, (what to do), Procedures (details on how to perform the activity) and the policies (r…

How Does ITIL Help in the Management of the SDLC?

I was recently asked how ITIL helps in the management of the SDLC (Software Development Lifecycle).  Simply put... SDLC is a Lifecycle approach to produce the software or the "product".  ITIL is a Lifecycle approach that focuses on the "service".
I’ll start by reviewing both SDLC and ITIL Lifecycles and then summarize:
SDLC  -  The intent of an SDLC process is to help produce a product that is cost-efficient, effective and of high quality. Once an application is created, the SDLC maps the proper deployment of the software into the live environment. The SDLC methodology usually contains the following stages: Analysis (requirements and design), construction, testing, release and maintenance.  The focus here is on the Software.  Most organizations will use an Agile or Waterfall approach to implement the software through the Software Development Lifecycle.
ITIL  -  is a best practice for IT service management (ITSM) that focuses on aligning IT services with the needs …

Incidents when a Defect is Involved

Question: We currently track defects in a separate system than our ticket management system. With that said, my question is does anyone have suggestions and/or best practices on how to handle incidents when a defect is involved? Should the incident be closed since the defect is being worked on in another defect tracking system if it is noted in the incident ticket? I am considering creating an incident statuses of 'closed-unresolved' so the incident can still be reported on in our ticket management system but know it is being worked on/tracked in the defect system. With defects, it is possible that we may never work on them because they are very low priority and the impact is low to the user. However, in some cases a defect is being worked on. Should we create a problem ticket instead?
Thanks, René W.

Answer: RenĂ©. In ITIL, the activity you are describing is handled by the Problem Management process. ITIL does not use the term “defect” but it does use the term “known error” to…