Managing JMeter Test Reports

Test Report Description

Real-time and offline JMeter test reports allow you to view and analyze test data at any time.

For details about the JMeter test report, see Table 1.

This report shows the response performance of the tested system in a scenario with a large number of concurrent users. To understand the report, see the following information:

• Statistical dimension: In this report, RPS, response time, and concurrency are measured in a single thread group. If a request has multiple packets, the request is considered successful only when all the packets are responded to. The response time of the request is the sum of the response time of the packets.
• Response timeout: If the corresponding TCP connection does not return the response data within the set response timeout (customized in *.jmx files), the thread group request is counted as a response timeout. Possible causes include: the tested server is busy or crashes, or the network bandwidth is fully occupied.
• Verification failure: The response packet content and response code returned from the server do not meet the expectation (the default expected response code of HTTP/HTTPS is 200), such as code 404 or 502. A possible cause is that the tested service cannot be processed normally in scenarios with a large number of concurrent users. For example, a database bottleneck exists in the distributed system or the backend application returns an error.
• Parsing failure: All response packets are received, but some packets are lost. As a result, the entire case response is incomplete. This may be caused by network packet loss.
• Bandwidth: This report collects statistics on the bandwidth of the execution end of CodeArts PerfTest. The uplink indicates the traffic sent from CodeArts PerfTest, and the downlink indicates the received traffic. In the external pressure test scenario, you need to check whether the EIP bandwidth of the executor meets the uplink bandwidth requirement and whether the bandwidth exceeds the downlink bandwidth of 1 GB.
• Requests per second (RPS): indicates the number of requests per second. Average RPS = Total number of requests in a statistical period/Statistical period.
• How to determine the quality of tested applications: According to the service quality definition of an application, the optimal status is that there is no response or verification failure. If there is any response or verification failure, it must be within the defined service quality range. Generally, the value does not exceed 1%. The shorter the response time is, the better. User experience is considered good if the response time is within 2s, acceptable if it is within 5s and needs optimization when it is over 5s. TP90 and TP99 can objectively reflect the response time experienced by 90% to 99% of users.

Viewing a Real-Time Test Report

View the monitoring data of each metric during a pressure test through the real-time test report.

Prerequisites

The test task is being executed.

Procedure

1. Log in to the CodeArts PerfTest console, choose JMeter Projects in the left navigation pane, and choose in the row containing the desired test project.
2. On the Performance Reports tab page, click in the row containing the test plan whose test report you want to view. For details about the parameters, see Table 1. Click Stop Task to stop the current task.
3. On the Overview tab page, you can view the number of exception/total requests, average RT, maximum concurrency, success rate, bandwidth, dynamic trend, and response codes.
4. On the Detail tab, you can view the logs, common test metrics, and request details of the test plan.
   1. Click View Log. In the View Logs dialog box, view request logs, event logs, and pod information. Ten request logs are displayed based on the name, return code, and result.

      Request logs can be filtered by return code, result, or request name.

   2. The sampling modes displayed in the report details are equidistant sampling value and equidistant average value.
      • Equidistant sampling value: Based on the case execution duration, the trend chart of cases whose execution duration is longer than 45 minutes is displayed with sampling points at a fixed interval.
      • Equidistant average value: Based on the case execution duration, the trend chart of cases whose execution duration is longer than 45 minutes is displayed with average values at a fixed interval.
   3. Click the Data View drop-down list box and enter a keyword to search for the required thread group or request data. You can also click the thread group or request data to be displayed in the directory in the Data View drop-down list box.
   4. On the Detail tab page, you can also click List to go to the report metric summary page.
      • Click Customized Column. In the displayed dialog box, select the items to be displayed, and drag the selected items in the Selected list on the right to change the item sequence.
        Figure 1 Table settings
        

      • Click in the Operation column to view logs.
      • Click in the Operation column to edit a thread group.

5. If the test plan has been associated with Application Monitor (APM) in the intelligent analysis object, the Invoke Analysis tab is displayed in the test report. On the Invoke Analysis tab, you can set search criteria to view the APM invoking details.

   If the test plan has been associated with Node Monitor (AOM) in the intelligent analysis object, the Monitored Indicator tab is displayed in the test report. On the Monitored Indicator tab, you can view information such as CPU usage, memory usage, disk read speed, and disk write speed.

Viewing an Offline Test Report

After a pressure test is complete, the system generates an offline test result report.

Prerequisites

The test task is complete.

Procedure

1. Log in to the CodeArts PerfTest console, choose JMeter Projects in the left navigation pane, and choose in the row containing the desired test project.
2. On the Performance Reports tab page, click in the row containing the test plan whose test report you want to view. For details about the parameters, see Table 1.
   

   • You can download an offline PDF report to a local PC and export the raw data (in CSV) for further processing.
   • CodeArts PerfTest retains offline report data for one year.

3. On the Overview tab page, you can view the number of exception/total requests, average RT, maximum concurrency, success rate, bandwidth, dynamic trend, and response codes.
4. On the Detail tab, you can view the logs, common test metrics, and request details of the test plan.
   1. Click View Log. In the View Logs dialog box, view request logs, event logs, and pod information. Ten request logs are displayed based on the name, return code, and result.

      Request logs can be filtered by return code, result, or request name.

   2. The sampling modes displayed in the report details are equidistant sampling value and equidistant average value.
      • Equidistant sampling value: Based on the case execution duration, the trend chart of cases whose execution duration is longer than 45 minutes is displayed with sampling points at a fixed interval.
      • Equidistant average value: Based on the case execution duration, the trend chart of cases whose execution duration is longer than 45 minutes is displayed with average values at a fixed interval.
   3. Click the Data View drop-down list box and enter a keyword to search for the required thread group or request data. You can also click the thread group or request data to be displayed in the directory in the Data View drop-down list box.
   4. On the Detail tab page, you can also click List to go to the report metric summary page.
      • Click Customized Column. In the displayed dialog box, select the items to be displayed, and drag the selected items in the Selected list on the right to change the item sequence.
        Figure 2 Table settings
        

      • Click in the Operation column to view logs.
      • Click in the Operation column to edit a thread group.

5. If the test plan has been associated with Application Monitor (APM) in the intelligent analysis object, the Invoke Analysis tab is displayed in the test report. On the Invoke Analysis tab, you can set search criteria to view the APM invoking details.

   If the test plan has been associated with Node Monitor (AOM) in the intelligent analysis object, the Monitored Indicator tab is displayed in the test report. On the Monitored Indicator tab, you can view information such as CPU usage, memory usage, disk read speed, and disk write speed.

Report Comparison

For the test reports generated at different time or under different conditions, you can view the test result by comparing the reports.

Procedure

1. Log in to the CodeArts PerfTest console, choose JMeter Projects in the left navigation pane, and choose in the row containing the desired test project.
2. On the Performance Reports tab page, click the name of the desired plan or click in the Operation column.
3. On the Report Comparison tab page, select the desired test reports.

   

   

   You can select a maximum of three offline reports for comparison. The first selected report is used as the baseline report.

4. Select a thread group from the Thread Group Comparison box to compare the metrics of this thread group in different reports.
5. Thread group metric comparison supports metric filtering.
   1. Click Custom Comparison. In the displayed dialog box, select the metrics to be displayed.
   2. (Optional) Drag the selected items in the Selected list on the right of the dialog box to change the item sequence.
   3. Click OK.