DevOps Test Monitoring Strategy
The combination of continuous monitoring with continuous testing and analytic tools can provide a broader strategic view of test results. This view is necessary to collect, aggregate and organize test data that enables a gain in confidence for each release.
Key Concepts for Realizing Your Test Monitoring Strategy:
Regression test product areas even though there were no expected changes:One not so good practice is to skip tests for areas that are “known“ or those that have not changed during individual test cycles. Unintended consequences of indirect changes may impact performance, so a regression suite should audit these areas occasionally just to be sure. Typical examples that are often caught by this are features important audit functions, system backwards compatibility and those that could disrupt critical business operation.
Select continuous test monitoring tools that collect and report trends:Tools that can correlate and report test results across multiple dimensions such as test cases, product features, system performance categories, build versions, releases and functional tags can find intermittent bugs or problem trends. Thresholds, email alerts and dashboards that highlight short-term results views from long-term results views are especially valuable.
Use continuous test monitoring results to diagnose and resolve problems:Once a diagnosis is determined, the root cause can be verified through targeted retest cases that set all the conditions in accordance with the diagnosis. Once confirmed then the offending design can be refactored to handle or avoid the failure condition.
A Test Monitoring Strategy with a holistic view is critical for test results evaluation over multiple releases. Intermittent failures and problems only become apparent when test results are presented as trends over a long-term trend. Failures are easier to diagnose when a large data set is accumulated and filtered. Take action. Get a test strategy that includes a Test Monitoring Strategy that works.
Inspire & educate…