three greatest roadblocks to steady testing

Let’s face it: Companies don’t need—or want—excellent software program. They do wish to ship new, business-differentiating software program as quickly as attainable. To allow this, you want quick suggestions on whether or not the newest improvements will work as anticipated or crash and burn in manufacturing. You additionally have to know if these adjustments one way or the other broke the core performance that the client base—and thus the enterprise—relies upon upon.

That is the place steady testing is available in.

Steady testing is the method of executing automated exams as a part of the software program supply pipeline to acquire suggestions on the enterprise dangers related to a software program launch candidate as quickly as attainable.

Take a look at automation is crucial for steady testing, however it’s not adequate. Take a look at automation is designed to supply a set of cross/fail information factors correlated to consumer tales or software necessities. Steady testing, then again, focuses on enterprise threat and offering perception on whether or not the software program could be launched. Past take a look at automation, steady testing additionally includes practices resembling aligning testing with your online business threat, making use of service virtualization and stateful take a look at information administration to stabilize testing for steady integration, and performing exploratory testing to show “huge block” points early in every iteration. It’s not merely a matter of extra instruments or completely different instruments. It requires a deeper transformation throughout individuals and processes in addition to applied sciences.

Steady testing has undeniably change into crucial—particularly now that 97 p.c of organizations have adopted agile and 71 p.c are training or adopting devops (based on Sauce Labs). New Forrester research confirms that continuous testing is one of the key factors separating agile/devops leaders from agile/devops laggards. Nevertheless, most enterprises still don’t have a mature, sustainable continuous testing process in place. Forrester found that even the organizations actively practicing agile and devops have a relatively low continuous testing adoption rate: 26 percent. 

Many organizations have experimented with test automation; typically, automating some UI tests and integrating their execution into the continuous integration process. They achieve and celebrate small victories, but the process doesn’t expand. In fact, it decays. Why? Typically, it boils down to roadblocks that fall into the following three categories:

Time and resources
Complexity
Results
Time and resources

Teams severely underestimate the time and resources required for sustainable test automation. Yes, getting some basic UI tests to run automatically is a great start. However, you also need to plan for the time and resources required to:

Keep notoriously brittle tests scripts from overwhelming the team with false positives.
Create tests for every new or modified requirement (or determine where to focus your efforts and what you can skip).
Establish a test framework that supports reuse and data-driven testing—both of which are essential for making automation sustainable over the long term.
Keep individual tests and the broader test framework in sync with the constantly evolving application.
Execute the test suite—especially if you’re trying to frequently run a large, UI-heavy test suite.
Determine how to automate more advanced use cases and keep them running consistently in a continuous testing environment (see the next section for more on this).
Review and interpret test the mounting volume of test results (more on this later too).

With agile and devops, time for test creation, maintenance, execution, and analysis is extremely limited—but fast feedback is essential. How can you ensure that the most important things are sufficiently tested without delaying time to market?

Complexity

It’s one thing to automate a test for a simple “create” action in a web application (e.g., create a new account and complete a simple transaction from scratch). It’s another to automate the most business-critical transactions, which typically pass through multiple technologies (SAP, APIs, mobile interfaces, and even mainframes) and require sophisticated set up and orchestration.  You need to ensure that:

Your testing resources understand how to automate tests across all the different technologies and connect data and results from one technology to another.
You have the stateful, secure, and compliant test data required to set up a realistic test as well as drive the test through a complex series of steps—each and every time the test is executed.
You have reliable, continuous, and cost-effective access to all the dependent systems that are required for your tests—including APIs and third-party applications that may be unstable, evolving, or accessible only at limited times.

Moreover, you also need a systematic way to flush out the critical defects that can only be found with a person evaluating application from an user perspective. Automation is great at rapidly and repeatedly checking whether certain actions continue to produce the expected results, but it can’t uncover the complex usability issues that significantly impact the user experience. 

Without fast, reliable feedback on how application changes impact the core user experience, how do you know if a release will help the business or harm it?

Results

The most commonly cited complaint with test results is the overwhelming number of false positives that need to be reviewed and addressed. When you’re just starting with test automation, it might be feasible to handle the false positives. However, as your test suite grows and your test frequency increases, addressing false positives quickly becomes an insurmountable task. Ultimately, many teams either start ignoring the false positives (which erodes trust in the test results and continuous testing initiative) or giving up on test automation altogether.

When devops and continuous delivery initiatives come into play, another critical issue with results emerges: they don’t provide the risk-based insight needed to make a fast go/go-no decision. If you’ve ever looked at test results, you’ve probably seen something like this:

42,278 tests passed.
10,086 tests failed.
910 tests did not execute.

What does this really tell you? You can see that:

There’s a total of 53,274 tests cases.
Almost 80 percent of those tests passed.
Over 19 percent of them failed.
About 1 percent did not execute.

But, would you be willing to make a release decision based on these results? Maybe the test failures are related to some trivial functionality. Maybe they are the most critical functionality: the engine of your system. Or, maybe your most critical functionality was not even thoroughly tested. Trying to track down this information would require tons of manual investigative work that yields delayed, often-inaccurate answers.

In the era of agile and devops, release decisions need to be made rapidly—even automatically and instantaneously. Test results that focus on the number of test cases leave you with a huge blind spot that becomes absolutely critical, and incredibly dangerous, when you’re moving at the speed of agile and devops.

If your results don’t indicate how much of your business-critical functionality is tested and working, you can’t rely on them to drive automated release gates. Manual review and assessment will be required, and that’s inevitably going to delay each and every delivery.  

This article is published as part of the IDG Contributor Network. Want to Join?

Source link

Read the full article