Skip to main content

DevOps Test Monitoring Strategy

The combination of continuous monitoring with continuous testing and analytic tools can provide a broader strategic view of test results.  This view is necessary to collect, aggregate and organize test data that enables a gain in confidence for each release. 

Key Concepts for Realizing Your Test Monitoring Strategy:

Determine continuous test monitoring priorities: Some examples of problems that continuous test monitoring can help with include intermittent failures caused by marginal designs, marginal test designs, environmental condition changes not detected by individual tests, memory leaks, varying starting conditions, interactions with other systems, system topologies and performance degradation within the margin of a test. These can and will accumulate over time. The best practice for continuous monitoring indicates that the problems of most concern to a specific product or DevOps environment will be monitored.

Regression test product areas even though there were no expected changes:One not so good practice is to skip tests for areas that are “known“ or those that have not changed during individual test cycles. Unintended consequences of indirect changes may impact performance, so a regression suite should audit these areas occasionally just to be sure. Typical examples that are often caught by this are features important audit functions, system backwards compatibility and those that could disrupt critical business operation.

Select continuous test monitoring tools that collect and report trends:Tools that can correlate and report test results across multiple dimensions such as test cases, product features, system performance categories, build versions, releases and functional tags can find intermittent bugs or problem trends. Thresholds, email alerts and dashboards that highlight short-term results views from long-term results views are especially valuable.

Use continuous test monitoring results to diagnose and resolve problems:Once a diagnosis is determined, the root cause can be verified through targeted retest cases that set all the conditions in accordance with the diagnosis. Once confirmed then the offending design can be refactored to handle or avoid the failure condition.

A Test Monitoring Strategy with a holistic view is critical for test results evaluation over multiple releases. Intermittent failures and problems only become apparent when test results are presented as trends over a long-term trend. Failures are easier to diagnose when a large data set is accumulated and filtered. Take action. Get a test strategy that includes a Test Monitoring Strategy that works.

Inspire & educate…

Comments

Popular posts from this blog

What is the difference between Process Owner, Process Manager and Process Practitioner?

I was recently asked to clarify the roles of the Process Owner, Process Manager and Process Practitioner and wanted to share this with you. Roles and Responsibilities: Process Owner – this individual is “Accountable” for the process. They are the goto person and represent this process across the entire organization. They will ensure that the process is clearly defined, designed and documented. They will ensure that the process has a set of Policies for governance. Example: The process owner for Incident management will ensure that all of the activities to Identify, Record, Categorize, Investigate, … all the way to closing the incident are defined and documented with clearly defined roles, responsibilities, handoffs, and deliverables.  An example of a policy in could be… “All Incidents must be logged”. Policies are rules that govern the process. Process Owner ensures that all Process activities, (what to do), Procedures (details on how to perform the activity) and the

How Does ITIL Help in the Management of the SDLC?

I was recently asked how ITIL helps in the management of the SDLC (Software Development Lifecycle).  Simply put... SDLC is a Lifecycle approach to produce the software or the "product".  ITIL is a Lifecycle approach that focuses on the "service". I’ll start by reviewing both SDLC and ITIL Lifecycles and then summarize: SDLC  -  The intent of an SDLC process is to help produce a product that is cost-efficient, effective and of high quality. Once an application is created, the SDLC maps the proper deployment of the software into the live environment. The SDLC methodology usually contains the following stages: Analysis (requirements and design), construction, testing, release and maintenance.  The focus here is on the Software.  Most organizations will use an Agile or Waterfall approach to implement the software through the Software Development Lifecycle. ITIL  -  is a best practice for IT service management (ITSM) that focuses on aligning IT services with

Four Service Characteristics

Recently I came across several articles by researchers and experts that laid out definitions and characteristics of services. ITIL provides us with a definition that can help drive the creation of value-laden services: A means of delivering value to customers by facilitating outcomes customers want to achieve without the ownership of specific costs and risks. An area that ITIL is not so clear is in terms of service characteristics. Several researchers and experts put forth that services have four basic characteristics (IHIP): ·          Intangibility—Services are the results of actions not things. They have no physical presence and represent a logical set of elements. One way to think of service is “work done for others.” ·          Heterogeneity—Also known as “variability”; services are unique items because of the mechanisms used to deliver services-that is people. Because the people element adds variability, the service is variable. This holds true especially for th