Skip to main content

DevOps Test Monitoring Strategy

The combination of continuous monitoring with continuous testing and analytic tools can provide a broader strategic view of test results.  This view is necessary to collect, aggregate and organize test data that enables a gain in confidence for each release. 

Key Concepts for Realizing Your Test Monitoring Strategy:

Determine continuous test monitoring priorities: Some examples of problems that continuous test monitoring can help with include intermittent failures caused by marginal designs, marginal test designs, environmental condition changes not detected by individual tests, memory leaks, varying starting conditions, interactions with other systems, system topologies and performance degradation within the margin of a test. These can and will accumulate over time. The best practice for continuous monitoring indicates that the problems of most concern to a specific product or DevOps environment will be monitored.

Regression test product areas even though there were no expected changes:One not so good practice is to skip tests for areas that are “known“ or those that have not changed during individual test cycles. Unintended consequences of indirect changes may impact performance, so a regression suite should audit these areas occasionally just to be sure. Typical examples that are often caught by this are features important audit functions, system backwards compatibility and those that could disrupt critical business operation.

Select continuous test monitoring tools that collect and report trends:Tools that can correlate and report test results across multiple dimensions such as test cases, product features, system performance categories, build versions, releases and functional tags can find intermittent bugs or problem trends. Thresholds, email alerts and dashboards that highlight short-term results views from long-term results views are especially valuable.

Use continuous test monitoring results to diagnose and resolve problems:Once a diagnosis is determined, the root cause can be verified through targeted retest cases that set all the conditions in accordance with the diagnosis. Once confirmed then the offending design can be refactored to handle or avoid the failure condition.

A Test Monitoring Strategy with a holistic view is critical for test results evaluation over multiple releases. Intermittent failures and problems only become apparent when test results are presented as trends over a long-term trend. Failures are easier to diagnose when a large data set is accumulated and filtered. Take action. Get a test strategy that includes a Test Monitoring Strategy that works.

Inspire & educate…

Comments

Popular posts from this blog

What is the difference between Process Owner, Process Manager and Process Practitioner?

I was recently asked to clarify the roles of the Process Owner, Process Manager and Process Practitioner and wanted to share this with you.

Roles and Responsibilities:
Process Owner – this individual is “Accountable” for the process. They are the goto person and represent this process across the entire organization. They will ensure that the process is clearly defined, designed and documented. They will ensure that the process has a set of Policies for governance.Example: The process owner for Incident management will ensure that all of the activities to Identify, Record, Categorize, Investigate, … all the way to closing the incident are defined and documented with clearly defined roles, responsibilities, handoffs, and deliverables. An example of a policy in could be… “All Incidents must be logged”. Policies are rules that govern the process. Process Owner ensures that all Process activities, (what to do), Procedures (details on how to perform the activity) and the policies (r…

How Does ITIL Help in the Management of the SDLC?

I was recently asked how ITIL helps in the management of the SDLC (Software Development Lifecycle).  Simply put... SDLC is a Lifecycle approach to produce the software or the "product".  ITIL is a Lifecycle approach that focuses on the "service".
I’ll start by reviewing both SDLC and ITIL Lifecycles and then summarize:
SDLC  -  The intent of an SDLC process is to help produce a product that is cost-efficient, effective and of high quality. Once an application is created, the SDLC maps the proper deployment of the software into the live environment. The SDLC methodology usually contains the following stages: Analysis (requirements and design), construction, testing, release and maintenance.  The focus here is on the Software.  Most organizations will use an Agile or Waterfall approach to implement the software through the Software Development Lifecycle.
ITIL  -  is a best practice for IT service management (ITSM) that focuses on aligning IT services with the needs …

ITIL 4 – Mapping the Customer Journey

All service providers are in the business of customer and user experience. It is not enough to compete on products and services, how services are delivered is as important as what is delivered.

The customer journey is the complete end-to-end experience customers have with one or more service providers and/or their products through the touchpoints and service interactions with those providers. In order to focus on the outcomes and on the customer/user experience, service providers are seeking to master the art of mapping their customer journey. Doing so allows them to maximize stakeholder value through co-creation of value throughout the entire value chain.

The customer journey begins by understanding the overall macro-level of steps or groups of activities that generate the need for interaction between the customer and the service provider. These activities begin at “Explore” and end with “Realize” where the value is actually being consumed by the end-users.
The Band of Visibility