DevOps Agile Test

Agile and DevOps

Agile and DevOps transformation is driven by business goals and customer needs. As market competition becomes increasingly fierce, the time window for innovation and monetization of new business models becomes shorter and shorter. As a result, more enterprises adopt the lean entrepreneurship mode. After capturing market requirements, enterprises have to shorten the time to market (TTM) of products and launch minimum viable products (MVPs) that satisfy customer requirements as quickly as possible.

Take Huawei as an example. Before 2008, Huawei still adopted the traditional delivery method for its projects. For example, after a project is initiated at the beginning of the year, all customer requirements, including user feedback, will be collected and ranked throughout the year. In the middle of the year, the product will be released to the customer. A patch will be released two months later, and a formal version will be released at the end of the year. Traditional projects have a low speed of version delivery but high quality requirements. If the customer finds a problem in the patch, the customer can only wait for another two months. If the customer does not accept the product in this period, project efforts are wasted. Therefore, the product quality must be strictly controlled.

Today, with products gradually developing towards agility, some R&D tool platforms have been migrated to the cloud, so test tools need to be transformed correspondingly. In the past, product delivery was conducted every half a year or every two months. After transformation, delivery was conducted every one month or even every two weeks. However, the transformation was not thorough, and there were still some problems with customers' delivery processes. Transformation to platform-based and service-oriented tools has changed business models fundamentally. After requirements are migrated to the cloud, customers can quickly get involved. Functions can be developed efficiently on cloud platforms. Products are frequently iterated according to customer requirements. This delivery mode shortens the delivery period from half a year to a couple of weeks, or even to a day or two. From the perspective of requirements, great changes have taken place. Basically, we have achieved step-by-step advancing and fast trial and error.

Test Debt

From waterfall to agile development and then to DevOps, the productivity of development, testing, and delivery is increasing, leaving some issues unresolved, which affects the ongoing improvement of test capabilities and value.

• Some companies attach more importance to development than testing, and limit the career of testers. In addition, manual testers are not familiar with programming, and developers do not pay enough attention to testing. Test teams often have heavy workloads, but are always understaffed.
• Testers sometimes do not fully understand customer requirements. Besides, the department silo between testing and development leads to information transparency issues, as well as insufficient communication and collaboration. Moreover, some companies overcompromise on quality for higher efficiency, and ignore the cultivation of agile culture and values.
• Some products are highly coupled and have poor testability. Testers rely too much on black box testing, and use inappropriate test policy and methods. The test environment deployment takes a long time and is frequently updated.

Focus of Testing: Quality of Service Value

Testing is a quality activity, which means its top priority is quality. Testing is also an engineering activity to obtain the maximum value with limited time, manpower, and resources. Although quality has multiple dimensions, it should have a focus—quality of service value, that is, quality of product value presented to customers. Testers should focus on the service value and determine the weights and priorities of quality in multiple dimensions, such as function, security, performance, usability, and compatibility. Testers should not test every aspect and related points for every testing project.

For example, for online payment functions, the focus of testing should be security; for online shopping functions, the focus should be usability; for large-scale flash sales and promotion activities, the focus should be performance. Therefore, testing must aim at the service value of products, determine product objectives, formulate key quality points and related test policy, and implement the policy in practice. Then, testers need to provide feedback on poor functions and test the improved functions again to check whether the overall quality meets the expected results.

Conventional Security and Elastic Security

Conventionally, we would try to find out and remove all insecure factors, which is an ideal way of testing. In actual work, however, it is impossible to identify all insecure factors in the entire system, which involves various aspects such as capabilities and architectures.

Therefore, elastic security is developed based on this. That is, the insecure factors are displayed as much as possible through scenario simulation. Based on this insecure scenario, a quick repair solution is provided to compensate for the insecure factors, which is not perceived by users. From the perspective of a product, both its commercial and quality goals can be achieved. This is called elastic security. Even if an error occurs, vulnerabilities can be quickly fixed or self-repaired in time to achieve the normal working purpose.

Shift-Left Testing and Shift-Right Testing

Shift-left testing is to push test actions toward the early stages of a project. For example, during behavior-driven development (BDD), test cases are designed based on scenario requirements to match the design. During consumer-driven contract (CDC) testing, services are coupled with each other. CDC can be used to decouple services from each other to prevent problems.

Shift-right testing is to push test actions toward the later stages of a project. Generally, tests are performed only before the software package release. Shift-right testing requires continuously testing from version release to production and online operation.

There are also some practices in these two aspects. For example, online dialing tests are performed to proactively monitor user behavior, quickly capture problems from the behavior track, and proactively push the problems to related owners for them to pay attention to and solve the problems. Therefore, the online process can be fed back to developers through some test methods to let them know the overall performance of the current product. Then the developers can quickly respond to the product.

Test Strategies in Different Stages of Product Development

Is it necessary to build the all automated testing capabilities as soon as a team is built? The following describes how to build automated testing capabilities from the perspective of the software maturity period.

In the initial stage of software exploration, the product is in an uncertain state. The front-end style and overall layout, as well as the back-end APIs, change frequently. Because the life cycle of automated testing cases is short, it is not cost-effective to create some automated testing cases. In this period, the product features can be controlled and only a few tests are performed. Therefore, manual tests can be performed instead of automated testing. In this way, the product can quickly identify errors and users can use the product.

In the product expansion stage when users recognize the product, the number of users and requirements will increase. In this case, automation must be considered because the full verification cost of each iteration in this stage increases and the delivery speed also increases. It is impossible to perform all manual tests during each round of go-live. In this case, automated testing cases are required for old modules.

At the product extraction stage, product requirements and benefits have reached the saturation stage. In this case, product requirements must be strictly controlled, and the responsibilities of automated testing cases must be guarded. No change is allowed to introduce extra risks or major feature changes, which may cause attacks on mature users.

Impact of Team Scale on Test Construction

If the team has fewer than five members and the team is in the exploration stage, the quality activities can be limited to the self-organization stage of the test. In this case, only some basic test management activities are performed, defects are managed, and regression tests are performed. In this stage, a test management process and a mechanism are established, and automated testing are not involved.

With the further expansion of the project, the number of team members gradually increases to 5 to 10. At this time, the test workload suddenly increases, and dedicated testers may be involved. The testers will talk with developers, convert the requirements into automated testing cases, establish continuous integration, and gradually evolve some test methods. At this stage, some automation attempts have been made.

As the team size increases, if one person cannot handle the workload, more testers will be recruited and a dedicated test team will be set up. This team will shift from automated testing to test automation and involve more management work. During the management process, we will interconnect some products, including developing dedicated tools to implement the overall automation capability, not only automation execution.

After the preceding evolution cycles, the test team has a lot of test automation experience. In this case, the cloud-oriented transformation can be performed. Currently, many teams are performing DevOps transformation. The most concerned aspect is to set up a DevOps full-function team. What are these people doing before transformation? What is the original test team with 10 to 15 members? In this stage, the team needs to transform special test capabilities into service-oriented capabilities. Test specialists will provide training at the early team stage, including test engineering construction, early test case formulation, standard template development, and special capability training for non-functional tests. All teams review the test process, including the test policy, test plans, and test cases. Then, they check the improvements in the process of the entire team. Last but not least, the special test teams are transformed from all aspects to servitization to achieve automation transformation.

Automated Testing and Test Automation

This part introduces the concept of test automation.

Test automation aims to reduce manual tests and operations. Test automation includes not only automated testing, but also all other activities that can reduce manpower input, such as automated test environment creation, deployment of tested systems, monitoring, and data analysis. In many cases, automated testing is only the execution part of the test. For example, some manual test methods during test execution are changed to automated testing. However, test automation is not only an execution part, but also includes obtaining and generating test data from the environment, executing automated testing, and generating results. If there is any problem, it will be automatically pushed to related personnel and the corresponding organization will solve the problem. Test reports are automatically generated so that testers can directly obtain the test results.