OBS Certificate in a Cluster Expired

Issue

The certificate has expired when a user attempts to access OBS from an MRS cluster.

Symptom

ALM-12054 Invalid Certificate File or ALM-12055 Certificate File About to Expire is generated for the MRS cluster, and the certificate that triggers the alarm in the alarm details is the OBS certificate.

Figure 1 Alarm Certificate File About to Expire

Figure 2 Alarm Invalid Certificate File

Cause Analysis

The certificate generated by OBS has a validity period. After the validity period expires, the certificate file becomes invalid, and an alarm is generated.

Procedure

1. Query the OBS certificate information of the MRS cluster.

   Log in to the active OMS node of the MRS cluster as user root and run the following command to check whether an OBS certificate exists:

   keytool -v -list -keystore ${JAVA_HOME}/jre/lib/security/cacerts -protected 2> /dev/null|grep -E "Alias name*|Valid from*" | grep obs

   An OBS certificate exists if information similar to the following is returned:

   Alias name: obs.example.com

   • If no certificate exists, no further action is required. Wait until the alarm is cleared.
   • If a certificate exists, go to 2.
   

   • ${java_home} indicates the JDK directory of the cluster. In MRS 3.x, replace it with /opt/Bigdata/common/runtime0/jdk1.8*. In versions earlier than MRS 3.x, replace it with /opt/Bigdata/jdk.
   • In MRS 3.x, if the certificate expiration alarm persists even after you perform the operations provided in this section, replace ${JAVA_HOME} with Client installation directory/JDK/jdk and perform the operations again.

2. Delete the OBS certificate.

   On the active OMS node, run the following commands to delete the OBS certificate queried in 1:

   obs_url=$(keytool -v -list -keystore ${JAVA_HOME}/jre/lib/security/cacerts -protected 2> /dev/null|grep -E "Alias name*|Valid from*" | grep obs | cut -d ':' -f 2 | awk '$1=$1')

   jdk_cacert="${JAVA_HOME}/jre/lib/security/cacerts"

   keytool -delete -alias ${obs_url} -keystore ${jdk_cacert} -storepass changeit

3. Run the following command to check that the OBS certificate does not exist. If the certificate still exists, go to 2.

   keytool -v -list -keystore ${JAVA_HOME}/jre/lib/security/cacerts -protected 2> /dev/null|grep -E "Alias name*|Valid from*" | grep obs