Replacing the NTP Server for an MRS Cluster

Notes and Constraints

This section applies only to MRS 3.x or later.

Prerequisites

• You have prepared a new NTP server, and the NTP service on the server is running properly.
• The MRS cluster can communicate with the new NTP server over a normal network connection.

Impact on the System

• Replacing the NTP server is a high-risk operation and may result in time change in the cluster.
• If the time difference between the NTP server and the cluster is greater than 150s before the NTP server replacement, you need to stop the cluster first to prevent data loss. Services are unavailable when the cluster is stopped.
• If the time difference between the NTP server and the cluster exceeds 15 minutes, the cluster will be unable to access OBS.
• If your clusters use Kerberos authentication and the time difference between the NTP server and the cluster exceeds 5 minutes, authentication will not work.

Replacing the NTP Server for the Cluster

1. Log in to MRS Manager and check whether there are uncleared alarms.
   For details about how to log in to MRS Manager, see Accessing MRS Manager.

   • If yes, rectify the fault according to the alarm handling procedure. After the alarm is cleared, go to Step 2.
   • If no, go to Step 2.

2. Log in to the active and standby management nodes of the cluster as user root.

   For details about how to obtain information about the active and standby management nodes of the MRS cluster, see Checking MRS Active/Standby Management Nodes.

   The password of user root is the custom password set during MRS cluster creation.

   Run the following command to switch to user omm:

   su - omm

3. Run the following command on the active management node to check the management plane gateway:

   cat ${BIGDATA_HOME}/om-server/OMS/workspace/conf/oms-config.ini | grep om_gateway

4. On both the active and standby management nodes, run the following command to check connectivity with the management plane gateway:

   ping Management plane gateway

   • If yes, go to Step 5.
   • If no, contact the network administrator to rectify the network fault. After the fault is rectified, go to Step 5.

5. Run the following command on the active management node to obtain the domain name of the NTP server in the current environment:

   This section uses ntp.myhuaweicloud.com as an example.

   cat /opt/Bigdata_func/cloudinit/cloudinit_params | grep ntpserver

6. On the active management node, check the time difference between the new NTP server and the cluster.

   For example, to check the time difference with the NTP server ntp.myhuaweicloud.com, run the following command:

   ntpdate -d ntp.myhuaweicloud.com

   Command output:

    6 Dec 15:16:10 ntpdate[2861453]: step time server 10.79.3.251 offset +2.118107 sec

   In the command output, +2.118107 sec indicates the time offset in seconds. A positive value indicates that the NTP server time is earlier than the current cluster time. A negative value indicates the opposite.

   • You can run the ntpq -v or ntpq --version command to query the NTP version. The command output may vary with the actual service environment.

     • Output of the ntpq -v command:

       10.1.1.112: ~# ntpq -v
       ntpq - standard NTP query program - Ver. 4.2.4p8

     • Output of the ntpq --version command:

       10.1.1.112: ~# ntpq --version
       ntpq 4.2.8p10@1.3728-o Mon Jun  6 08:01:59 UTC 2016 (1)

7. Check whether the absolute value of the time difference exceeds 150.
   • If yes, go to Step 8.
   • If no, perform Step 10 as user omm.

8. Check whether the cluster can be stopped.
   • If yes, stop upper-layer services and the cluster, and go to Step 9.
   • If no, no further action is required.

9. Check whether the time of the NTP server is slower than the time of the cluster.
   • If yes, wait a period of the time difference obtained in Step 6 after message Operation successful is displayed on the UI, perform Step 11 as user omm.
   • If no, after message Operation successful is displayed on the UI, perform Step 11 as user omm.

10. Run the following command on the active management node to replace the NTP server:

    sh ${BIGDATA_HOME}/om-server/om/bin/tools/modifyntp.sh --ntp_server_ip ntp.myhuaweicloud.com

    

    The IP address of the NTP server cannot be set to the IP address of a node in the cluster. Otherwise, the service network between the node and the active/standby management node may be disconnected.

11. Run the following command on the active management node to forcibly synchronize time from the NTP server at ntp.myhuaweicloud.com immediately and replace the NTP server:

    sh ${BIGDATA_HOME}/om-server/om/bin/tools/modifyntp.sh --ntp_server_ip ntp.myhuaweicloud.com --force_sync_time

    • If the cluster is stopped, start the cluster after the NTP server is replaced.
    • After the command for forcible time synchronization is executed, it takes about five minutes for time synchronization to complete on cluster nodes.