Troubleshooting Traffic Interruptions
Symptom
During enterprise cloud operations, data plane faults are among the most common network issues. When a service system is protected by CFW, access traffic must pass through the security policy checks of the firewall. If the firewall is incorrectly configured or the security policy incorrectly blocks normal service traffic, end-to-end services will be interrupted. As a result, the client fails to request the server, the request times out, or services are interrupted after a session is established.
Intended for O&M engineers and security operations personnel, this section describes how to troubleshoot and resolve network interruptions. It helps you quickly determine whether traffic is blocked by CFW and take corresponding recovery measures based on the cause, minimizing service recovery time.
How It Works
In the CFW protection architecture, the service traffic path is as follows:
Client → CFW → Server
CFW performs traffic security checks on multiple layers, including:
- Access control policies: The firewall allows or blocks the matched traffic based on the preconfigured access control rules and blacklist.
- Intrusion prevention system (IPS): The firewall performs in-depth checks on traffic and blocks the traffic that matches the attack characteristics. IPS rules cover basic protection, virtual patching, reverse shell detection, and sensitive directory scan detection.
- Traffic filtering: The firewall filters traffic based on the IP address blacklist.
If service traffic is blocked in any of the preceding checks, services may be interrupted. The core idea of troubleshooting is to check the logs of each firewall module to determine whether the traffic is blocked, identify the responsible module, and apply the appropriate remediation.
Troubleshooting Procedure
- Check whether there are blocking logs.
- Log in to the CFW console.
- Click
in the upper left corner of the management console and select a region or project. - (Optional) Switch to another firewall instance. If there are multiple firewall instances, you can select a desired instance from the drop-down list in the upper left corner of the page.
- (Optional) Switch to another firewall instance. If there are multiple firewall instances, you can select a desired instance from the drop-down list in the upper left corner of the page.
- In the navigation pane on the left, choose . The tab page is displayed by default.
- Click the Access Control Logs tab page. Click a tab based on the traffic exception scenario.
- If the issue occurs during inter-EIP access at the Internet border, click the Internet Border Firewall tab.
- If the issue occurs during service access between VPCs, click the VPC Border Firewall tab.
- Search for logs based on the source and destination IP addresses of the abnormal service and check whether there are blocking logs (whose Action is Blocked). Figure 1 Blocking log
- Determine the reason why the blocking log is generated and rectify the fault.
The blocking log may be generated because traffic matches the access control rule or blacklist. Check the cause of the blocking log:
- On the Access Control Logs tab page, click the rule name or blacklist in the Rule/Rule ID column to go to the details page.
- View the matching information in the blocking log and check the rule parameter settings.
- If an access control rule is matched:
- Check whether the source IP addresses or source address group in the blocking rule contains the IP address that you want to allow.
- Check whether the destination IP addresses or destination address group in the blocking rule contains the IP address that you want to allow.
- Check whether the destination domain names or domain name group in the blocking rule contains the domain name that you want to allow.
- If a blacklist is matched:
Check whether the source or destination IP addresses in the blacklist contain the IP addresses that you want to allow.
- If an access control rule is matched:
- If the blocking is caused by an access control rule or blacklist, modify the rule or blacklist. After the modification, services are expected to recover. If no access control blocking logs are found, go to 3.
For details about how to modify a protection rule, see Modifying a Protection Rule. For details about how to modify a blacklist, see Modifying a Blacklist.
- Modifying an access control rule may affect the protection policies for other service traffic. Before the modification, confirm the impact scope. You are advised to perform verification in the test environment before applying the modification in the production environment.
- Removing an IP address from the blacklist will unblock it. Ensure that the IP address does not pose any security threats before performing this operation.
- Check whether packets are blocked by the intrusion prevention module.
- On the Log Query page, click the Attack Event Logs tab page. Click a tab based on the traffic exception scenario.
- If the issue occurs during inter-EIP access at the Internet border, click the Internet Border Firewall tab.
- If the issue occurs during service access between VPCs, click the VPC Border Firewall tab.
- Search for logs based on the source and destination IP addresses of the abnormal service. Check whether there are blocking logs. Check whether the service packets are blocked by the intrusion prevention module.
Note: If no results are found when you search for attack logs from the source IP address to the destination IP address (A→B), search for attack logs from the destination IP address to the source IP address (B→A). This is because attack logs record source and destination IP addresses based on the direction of the attack, rather than the direction of the session.
Figure 2 Attack event logs
- On the Log Query page, click the Attack Event Logs tab page. Click a tab based on the traffic exception scenario.
- Handle the packets blocked by the intrusion prevention module.
When packets are blocked by the intrusion prevention module, scenarios fall into the following categories based on the rule ID or rule name in the attack log:
- Scenario 1: The rule ID in the attack log is 760001 (reverse shell blocking).
The packets match a reverse shell detection rule. The IP address or session recorded in the log is blocked based on the Reverse Shell Defense settings on the Intrusion Prevention page.
Solution: If the packets are normal service requests initiated by the client, you can add the IP address that initiates the requests to the whitelist to allow its packets to pass through.
For details about how to add a whitelist, see Adding Whitelist Items to Allow Internet Border Traffic and Adding a Whitelist Item to Allow VPC Border Traffic.
- Scenario 2: The rule ID in the attack log is 360021 (sensitive directory scan blocking).
The packets match the sensitive directory scan detection rule. The IP address or session recorded in the log is blocked according to the settings of Sensitive Directory Scan Defense on the Intrusion Prevention page.
Solution: If the packets are normal service requests initiated by the client, you can add the IP address that initiates the requests to the whitelist to allow its packets to pass through.
For details about how to add a whitelist, see Adding Whitelist Items to Allow Internet Border Traffic and Adding a Whitelist Item to Allow VPC Border Traffic.
- Scenario 3: The rule ID in the attack log is an ID of other rules (basic protection or virtual patching).
The packets match the IPS rule of basic protection or virtual patching.
Solution: If the packets are normal service requests initiated by the client, you can use either of the following methods:
- Add a whitelist rule for the source IP address to allow the packets.
For details about how to add a whitelist, see Adding Whitelist Items to Allow Internet Border Traffic and Adding a Whitelist Item to Allow VPC Border Traffic.
- Change the action of the rule to allow packets to pass through.
- In the navigation pane on the left, choose .
- Ensure Basic Protection is enabled.
- Click View Effective Rule under Basic Protection.
- Search for the rule by rule ID, set the current action of the rule to Observe or Disable, and click OK in the displayed dialog box. Figure 3 Changing the action
Changing the action of an IPS rule from Intercept to Observe or Disable reduces protection capabilities. Perform this operation only after confirming that the packets are legitimate service traffic. Restore the original action immediately after the issue is resolved.
- Add a whitelist rule for the source IP address to allow the packets.
- Scenario 4: The rule name in the attack log is Blacklist-IP (traffic filtering and blocking).
This indicates that packets are blocked by the traffic filtering function.
Solution: Remove the blocked IP addresses from the file and import the file again in full overwrite mode.
- In the navigation pane on the left, choose . The Traffic Blocking tab is displayed.
- In the blocked object list, click Download in the Operation column of the target IP address file.
- Delete the blocked IP addresses from the downloaded file.
- On the Traffic Blocking tab page, click Add Object and configure parameters.
- Mode: Select Overwrite.
- Effective Scope: Select EIP.
- Content Type: Select File.
- File: Click Add and select the modified file.
- Click OK. In the displayed dialog box, confirm the information and click OK.
After the check is complete, perform 5 to further check the traffic forwarding status.
- Scenario 1: The rule ID in the attack log is 760001 (reverse shell blocking).
- Check traffic logs.
- On the Log Query page, click the Traffic Logs tab page. Click a tab based on the traffic exception scenario.
- If the issue occurs during inter-EIP access at the Internet border, click the Internet Border Firewall tab.
- If the issue occurs during service access between VPCs, click the VPC Border Firewall tab.
- Search for logs based on the source and destination IP addresses of the abnormal service to check whether there are traffic logs.
- If traffic logs are found, the source and destination IP addresses have created sessions on the firewall. Check the Flow Packets in the logs. Figure 4 Flow packets
- If the number of response packets is 0, check whether the server receives the request packet and sends the response packet.
- If the response packet is not 0, CFW is properly forwarding traffic in both directions. In this case, check whether the return path between the client and the firewall is operating normally.
- If no traffic logs are found, the traffic may not reach the firewall. In this case, go to 6 to further check the issue using the packet capture function.
- If traffic logs are found, the source and destination IP addresses have created sessions on the firewall. Check the Flow Packets in the logs.
- On the Log Query page, click the Traffic Logs tab page. Click a tab based on the traffic exception scenario.
- Capture packets on the network to check whether packets reach the firewall.
- In the navigation tree on the left, choose System Management > Packet Capture.
- Click Create Capture Task. On the displayed page, configure the allowed source and destination IP addresses and ports based on the requests, and click OK. The task starts to be executed.
- After the task is complete, download the packet capture file and check whether packets are captured.
- If yes, further locate the cause on the CFW service side.
The packets have reached the firewall. Further locate the cause on the CFW service side. Submit a service ticket to contact Huawei Cloud technical support.
- If no, the packets do not reach the firewall. In this case, check the link before the firewall.
This indicates that the packets do not reach the firewall. In this case, check the link before the firewall, such as the route configuration and security group rules.
- If yes, further locate the cause on the CFW service side.
References
- For details about how to configure access control rules, see Access Control Policy Overview.
- For details about how to configure the traffic filtering function, see Quickly Blocking Malicious Traffic Through Traffic Blocking.
- For details about the basic concepts and configuration of EIP, see Elastic IP (EIP)
- For details about the basic concepts and network configuration of VPC, see Virtual Private Cloud (VPC).
What is your overall rating for this page?
Thank you very much for your feedback. We will continue working to improve the documentation.See the reply and handling status in My Cloud VOC.
For any further questions, feel free to contact us through the chatbot.
Chatbot