ALM-27007 Database Enters the Read-Only Mode

Alarm Description

The system checks the disk space usage of the data directory on the active DBServer node every 30 seconds. This alarm is generated when the disk space usage of the data directory exceeds 90%.

This alarm is cleared when the disk space usage of the data directory falls below 80%.

Alarm Attributes

Alarm Parameters

Impact on the System

• Service data is lost.
• Data cannot be written for upper-layer services and the data is lost.

Possible Causes

The disk configuration cannot meet service requirements. The disk usage reaches the upper limit.

Handling Procedure

Check whether the disk space usage reaches the upper limit.

1. On FusionInsight Manager, choose Cluster, click the name of the desired cluster, and choose Services > DBService.
2. On the Dashboard page, view the Disk Space Usage of the Data Directory chart to check whether the disk usage of the data directory exceeds 90%.
   • If yes, go to 3.
   • If no, go to 13.

3. Log in to the active management node of the DBServer as user omm and run the following commands to check whether the database enters the read-only mode:

   source $DBSERVER_HOME/.dbservice_profile

   gsql -U omm -W password -d postgres -p 20051

   show default_transaction_read_only;

   

   In the preceding command, password indicates the password of user omm of the DBService database. You can run the \q command to exit the database page.

   Check whether the value of default_transaction_read_only is on.

   POSTGRES=# show default_transaction_read_only;
    default_transaction_read_only
   -------------------------------
    on
   (1 row)

   • If yes, go to 4.
   • If no, go to 13.

4. Run the following commands to open the dbservice.properties file:

   source $DBSERVER_HOME/.dbservice_profile

   vi ${DBSERVICE_SOFTWARE_DIR}/tools/dbservice.properties

5. Change the value of gaussdb_readonly_auto to OFF. (The default is ON.)

6. Run the following command to open the postgresql.conf file:

   vi ${DBSERVICE_DATA_DIR}/postgresql.conf

7. Delete default_transaction_read_only = on.
8. Run the following command for the configuration to take effect:

   gs_ctl reload -D ${DBSERVICE_DATA_DIR}

9. Log in to FusionInsight Manager, and Choose O&M > Alarm > Alarms. On the right of the alarm "Database Enters the Read-Only Mode", click Clear in the Operation column. In the dialog box that is displayed, click OK Manually clear the alarm.

10. Log in to the active DBServer node as user omm. Run the following commands to check whether files with more than 500 MB are incorrectly written into the disk space of the data directory:

    source $DBSERVER_HOME/.dbservice_profile

    find "$DBSERVICE_DATA_DIR"/../ -type f -size +500M

    • If yes, go to 11.
    • If no, go to 13.

11. Delete the incorrectly written files based on site requirements.
12. Log in to FusionInsight Manager. Click Cluster and choose the name of the desired cluster > Service > DBService. On the Dashboard page, view the Data Directory Disk Usage chart to check whether the disk usage is less than 80% of the threshold.
    • If yes, no further action is required.
    • If no, go to 13.

Collect fault information.

13. On FusionInsight Manager, choose O&M. In the navigation pane on the left, choose Log > Download.
14. Expand the Service drop-down list, and select DBService for the target cluster.
15. Specify Hosts for collecting logs, which is optional. By default, all hosts are selected.
16. Click in the upper right corner, and set Start Date and End Date for log collection to 10 minutes ahead of and after the alarm generation time, respectively. Then, click Download.

17. Contact O&M engineers and provide the collected logs.

Alarm Clearance

This alarm is automatically cleared after the fault is rectified.

Related Information

None.