How Are Collection Node or Collection Channel Faults Handled?

Symptom

The component controller isap-agent periodically reports the collection node status and collection channel health status. Despite a delay of about one minute, the Health Status of a collection node or collection channel was still displayed as Faulty 3 minutes after the collection channel is delivered, and the CPU usage or memory usage of the server is about to reached 100%.

Figure 1 Collection node fault

Figure 2 Collection channel fault

Possible Causes

The configured connector or parser has syntax or semantic errors. As a result, the collector cannot run properly and restarts over and over again. The CPU and memory are exhausted.

Fault Location

1. Remotely log in to the ECS where the collection node resides.
   • Log in to the ECS console, locate the target server, and click Remote Login in the Operation column to log in to the server. For details, see Login Using VNC.
   • If your server has an EIP bound, you can also use a remote management tool, such as PuTTY or Xshell, to log in to the server and install the component controller on the server as user root.
2. Run the following command to check the OS running status:

   top

   If the following information is displayed, the Java process in the ECS uses a large number of CPU resources.

   Figure 3 Status
   
3. Run the following command to view the collector run logs:

   docker logs isap-logstash -f

   According to the logs, the filter (parser) configuration of the current collection channel is incorrect, as shown in the following figure.

   Figure 4 Collector run log
   
4. Run the following command to switch to the directory where the collection channel configuration file is stored:

   cd /opt/cloud/logstash/config/files

5. Run the following command to check whether the filter part is abnormal:

   cat Configuration file name

   If the information shown in the following figure is displayed, the current filter is abnormal.

   Figure 5 Filter exceptions
   

Solution

1. Log in to the SecMaster console and access the target workspace.
2. In the navigation pane on the left, choose Settings > Collections. Then, select the Parsers tab.
3. Click Edit in the Operation column of the row containing the target parser. On the edit page, delete the incorrect configuration and configure it again.
   Figure 6 Configurations of an abnormal parser
   
   Figure 7 Modifying the parser configuration
   

4. Click OK.
5. Click the Collection Channels tab, locate the target connection channel, and click Restart in the Operation column.
   Figure 8 Restarting a collection channel
   

6. Check the status of the collection channel and collection node.
   • After the restart is complete, go to the Collection Channels tab and check the health status of the target collection channel.
     Figure 9 Health status of a collection channel
     
   • Select the Collection Nodes tab. On the page displayed, check the health status of the target collection node.
     Figure 10 Health status of a collection node
     

   If the Health Status of the collection channel and collection node is Normal, the fault has been rectified.