Submitting a Topology When No Client Is Installed on a Linux OS

Scenario

Storm supports topology running in a Linux environment where no Storm client is installed.

Prerequisites

• Ensure that the time difference between the client and the MRS cluster is less than 5 minutes.
• If the host where the client is installed is not a node in the cluster, set the mapping between the host name and the IP address in the hosts file on the node where the client locates. The host name and IP address must be in one-to-one mapping.

Procedure

1. Prepare for dependency JAR packages and configuration files.

   In the Linux environment, create a directory, such as /opt/test, and create subdirectories lib and src/main/resources/. Upload the JAR packages in the lib folder of the example project to the lib directory in the Linux environment. Upload the configuration files in the src/main/resources folder of the example project to the src/main/resources directory in the Linux environment.

2. Modify the WordCountTopology.java class in the IntelliJ IDEA project and change the application submission mode to the remoteSubmit mode. Replace the user keytab file name, user principal name, and the JAR file address.
   • Submit the application in remoteSubmit mode.

     public static void main(String[] args) 
              throws Exception 
          { 
              TopologyBuilder builder = buildTopology(); 
               
              /* 
               * The task submitting modes are as follows: 
               * 1. Command submitting mode: In this mode, you need to copy an application JAR package to the client machine and run related commands for submitting. 
               * 2. Remote submitting mode: In this mode, you need to package related content into an application JAR package and run the main method in IntelliJ IDEA for submitting. 
               * 3. Local submitting mode: In this mode, you need to run an application locally. This mode is usually used for tests. 
               * The command submitting mode and remote submitting mode are supported in both the security and normal modes. 
               * The local submitting mode is supported only in the normal mode. 
               *  
               * The user can only select one mode for task submitting. The command submitting mode is used by default. If another mode is used, delete the code comments. 
               */ 
               
              submitTopology(builder, SubmitType.REMOTE); 
          }

   • Change the value of userJarFilePath to the specified directory /opt/test/lib/example.jar in the Linux environment.

     private static void remoteSubmit(TopologyBuilder builder) 
              throws AlreadyAliveException, InvalidTopologyException, NotALeaderException, AuthorizationException, 
              IOException 
          { 
              Config config = createConf(); 
               
              String userJarFilePath = "/opt/test/lib/example.jar "; 
              System.setProperty(STORM_SUBMIT_JAR_PROPERTY, userJarFilePath); 
               
              //Preparations in security mode 
              if (isSecurityModel()) 
              { 
                  securityPrepare(config); 
              } 
              config.setNumWorkers(1); 
              StormSubmitter.submitTopologyWithProgressBar(TOPOLOGY_NAME, config, builder.createTopology()); 
          }

   • In security mode, make security authentication preparations, and change the values of userKeyTablePath and userPrincipal to the user keytab file name and principal obtained in step 2 of Configuring and Importing Sample Projects based on the actual situation.

     private static void securityPrepare(Config config) 
              throws IOException 
          { 
              String userKeyTablePath = 
                  System.getProperty("user.dir") + File.separator + "src" + File.separator + "main" + File.separator + "resources" + File.separator + "user.keytab"; 
              String userPrincipal = "StreamingDeveloper"; 
              String krbFilePath = System.getProperty("user.dir") + File.separator + "src" + File.separator + "main" + File.separator + "resources" + File.separator +"krb5.conf"; 
               
              //Separators need to be replaced in Windows paths. 
              userKeyTablePath = userKeyTablePath.replace("\\", "\\\\"); 
              krbFilePath = krbFilePath.replace("\\", "\\\\"); 
               
              String principalInstance = String.valueOf(config.get(Config.STORM_SECURITY_PRINCIPAL_INSTANCE)); 
              LoginUtil.setKrb5Config(krbFilePath); 
              LoginUtil.setZookeeperServerPrincipal("zookeeper/" + principalInstance); 
              LoginUtil.setJaasFile(userPrincipal, userKeyTablePath); 
          }

3. Export the JAR package to the Linux environment.
   • Package the IntelliJ IDEA code by referring to Packaging IntelliJ IDEA Code and name the JAR package example.jar.
   • Copy the exported JAR package to the /opt/test/lib directory in the Linux environment.

4. Go to the /opt/test directory, and run the following command to run the JAR package:

   java -classpath /opt/test/lib/*:/opt/test/src/main/resources com.huawei.storm.example.wordcount.WordCountTopology