Troubleshooting Notebook Instance CLOSE_WAIT Issue

Symptom

• A notebook instance occasionally restarts automatically, or the page freezes or cannot be opened.
• A large number of network connections in the CLOSE_WAIT state are stacked on the terminal.

Fault Locating

1. Open the notebook instance and choose Files > New > Terminal in the upper left corner of the JupyterLab page to open the CLI. For details, see Creating a Terminal in JupyterLab.
2. Check the number of CLOSE_WAIT connections:

   ss -tan | grep CLOSE_WAIT | wc -l

   A number is returned, for example, 0, 16, or 180.

   If the system displays "ss: command not found", you can use netstat instead:

   netstat -tan | grep CLOSE_WAIT | wc -l

3. Wait for 1 to 2 minutes and run the preceding command again to check whether the number of connections keeps increasing.
   • If the number is 0 or remains unchanged, the instance is normal and no action is required.
   • If the number keeps increasing (for example, more than 10 connections are added per minute), the connections are leaking. For details about the causes and solutions, see the following sections.
4. (Optional) To identify the program that generates the connections, run the following command:

   ss -tanp | grep CLOSE_WAIT

   The process name (such as python and node) is displayed at the end of each line, which helps you locate the code or service that does not close the connection.

Possible Causes

If the number of CLOSE_WAIT connections keeps increasing, the source of the leak is the program running in the instance.

The program running in the instance (such as Python scripts, web services, and database connections) establishes a network connection but does not correctly close it. After the peer end closes the connection, the local connection remains in the CLOSE_WAIT state and cannot be reclaimed. As a result, the connection resources are exhausted, causing instance freezing or even automatic restart.

This type of leak is caused by the failure to release connections in the program code. Therefore, the code needs to be fixed.

Solution

• Solution 1: Restarting the Notebook Instance (Temporary Measure)
  On the ModelArts console, find the target notebook instance and click Restart. For details, see Starting, Stopping, or Deleting a Notebook Instance.

  • After the instance is restarted, the connection count is cleared, and the instance can be temporarily restored.
  • Before restarting the instance, save the code to prevent the loss of unsaved content.
  • Restarting the instance is only a temporary measure. If the code is not fixed, the leak will occur again.
• Solution 2: Clearing Connections on the Terminal (Temporary Measure, Applicable When Permissions Are Granted and ss Is Available)

  Run the following command in the JupyterLab terminal:

  ss -K state close-wait

  This command forcibly closes the connections that are in the CLOSE_WAIT state and are not closed in time. Running this command does not affect running tasks.

  If the message "permission denied" is displayed or the ss command is unavailable, use solution 1 to restart the instance.

• Solution 3: Completely Fixing the Connection Leak from the Code

  The root cause of the CLOSE_WAIT accumulation is that the program does not close the connection after it is established. Check the code as follows:

  • When using the HTTP request library (such as requests and urllib), ensure that the response or session is correctly closed after the request is complete.
  • When using the socket or database connection, ensure that the close() method is called to release the connection after the connection is used. You are advised to place the code in the finally block to ensure that the connection can be closed even in the case of an exception.
  • After locating the source of the leak, restart the instance to clear existing connections and verify the fixing effect.

  Operations to Avoid

  Do not kill service processes. CLOSE_WAIT connections belong to running program processes. Killing a process will interrupt tasks and cause loss of the content being edited.

FAQs

• Is It Normal to See Many TIME_WAIT Connections?

  Yes. TIME_WAIT is a temporary state after the TCP connection is closed. It will be automatically reclaimed after about 60 seconds (2MSL). You only need to pay attention to CLOSE_WAIT.

• Do I Need to Handle a Few CLOSE_WAIT Connections That Occur Occasionally?

  No. You only need to handle this issue when the number of connections keeps increasing and affects the instance stability.

• Will Clearing Connections Interrupt Running Tasks?

  No. The ss -K state close-wait command only clears connections that are already in a suspended state and does not affect running tasks.

• Why Does the Instance Automatically Restart?

  The running program does not correctly close the network connection. As a result, CLOSE_WAIT connections are continuously stacked. When the number of CLOSE_WAIT connections reaches a certain threshold, the instance resources are exhausted or the health check is abnormal, triggering the instance to automatically restart. In this case, check the code and release the connections that are not closed.