Skip to main content

DevOps Test Monitoring Strategy

Image
By
The combination of continuous monitoring with continuous testing and analytic tools can provide a broader strategic view of test results.  This view is necessary to collect, aggregate and organize test data that enables a gain in confidence for each release. 

Key Concepts for Realizing Your Test Monitoring Strategy:

Determine continuous test monitoring priorities: Some examples of problems that continuous test monitoring can help with include intermittent failures caused by marginal designs, marginal test designs, environmental condition changes not detected by individual tests, memory leaks, varying starting conditions, interactions with other systems, system topologies and performance degradation within the margin of a test. These can and will accumulate over time. The best practice for continuous monitoring indicates that the problems of most concern to a specific product or DevOps environment will be monitored.

Regression test product areas even though there were no expected changes:One not so good practice is to skip tests for areas that are “known“ or those that have not changed during individual test cycles. Unintended consequences of indirect changes may impact performance, so a regression suite should audit these areas occasionally just to be sure. Typical examples that are often caught by this are features important audit functions, system backwards compatibility and those that could disrupt critical business operation.

Select continuous test monitoring tools that collect and report trends:Tools that can correlate and report test results across multiple dimensions such as test cases, product features, system performance categories, build versions, releases and functional tags can find intermittent bugs or problem trends. Thresholds, email alerts and dashboards that highlight short-term results views from long-term results views are especially valuable.

Use continuous test monitoring results to diagnose and resolve problems:Once a diagnosis is determined, the root cause can be verified through targeted retest cases that set all the conditions in accordance with the diagnosis. Once confirmed then the offending design can be refactored to handle or avoid the failure condition.

A Test Monitoring Strategy with a holistic view is critical for test results evaluation over multiple releases. Intermittent failures and problems only become apparent when test results are presented as trends over a long-term trend. Failures are easier to diagnose when a large data set is accumulated and filtered. Take action. Get a test strategy that includes a Test Monitoring Strategy that works.

Inspire & educate…

DevOps Test Engineering Certification Available!
Labels:

Comments

Post a Comment

Popular posts from this blog

Four Service Characteristics

By
Recently I came across several articles by researchers and experts that laid out definitions and characteristics of services. ITIL provides us with a definition that can help drive the creation of value-laden services: A means of delivering value to customers by facilitating outcomes customers want to achieve without the ownership of specific costs and risks. An area that ITIL is not so clear is in terms of service characteristics. Several researchers and experts put forth that services have four basic characteristics (IHIP): Intangibility—Services are the results of actions not things. They have no physical presence and represent a logical set of elements. One way to think of service is “work done for others.”  Heterogeneity—Also known as “variability”; services are unique items because of the mechanisms used to deliver services, which is people. Because the people element adds variability, the service is variable. This holds true, especially for the value proposition—not eve...
Post a Comment
Read more

What Is A Service Offering?

By
The ITIL 4 Best Practice Guidance defines a “Service Offering” as a description of one or more services designed to address the needs of a target customer or group.   As a service provider, we can’t stop there!   We must know what the contracts of our service offering are and be able to put them into context as required by the customer.     Let’s explore the three elements that comprise a Service Offering. A “Service Offering” may include:     Goods, Access to Resources, and Service Actions 1. Goods – When we think of “Goods” within a service offering these are the items where ownership is transferred to the consumer and the consumer takes responsibility for the future use of these goods.   Example of goods that are being provided in the offering – If this is a hotel service then toiletries or chocolates are yours to take with you.   You the consumer own these and they are yours to take with you.      ...
Post a Comment
Read more

The New Four Ps of Service Management

By
By Donna Knapp For years, people , process , and technology (PPT) was a widely recognized framework for balancing and integrating the components needed to achieve optimal performance and outcomes. In the ITIL v3 Service Design publication, this framework was expanded to the four Ps: people , processes , products , and partners . ITIL 4 has further expanded and evolved this framework to the four dimensions of service management. These four dimensions are collectively critical to the effective and efficient facilitation of value for customers and other stakeholders in the form of products and services. The four dimensions of service management are: Organizations and people Information and technology Partners and suppliers Value streams and processes. These four dimensions represent perspectives which are relevant to the whole service value system (SVS), including the entirety of the service value chain and all ITIL practices. Each ITIL practice is a set of organizational resources base...
Post a Comment
Read more