HetuEngine UDF Development and Application

Developing HetuEngine UDFs

This sample implements one HetuEngine UDF described in the following table.

1. Create a Maven project. Set groupId to com.test.udf and artifactId to udf-test. The two values can be customized based on the site requirements.
2. Modify the pom.xml file as follows:

   <project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
          <modelVersion>4.0.0</modelVersion>
          <groupId>com.test.udf</groupId>
          <artifactId>udf-test</artifactId>
          <version>0.0.1-SNAPSHOT</version>
   
          <build>
            <plugins>
               <plugin>
                   <artifactId>maven-shade-plugin</artifactId>
                   <executions>
                       <execution>
                           <phase>package</phase>
                           <goals>
                               <goal>shade</goal>
                           </goals>
                       </execution>
                   </executions>
               </plugin>
               <plugin>
                   <artifactId>maven-resources-plugin</artifactId>
                   <executions>
                       <execution>
                           <id>copy-resources</id>
                           <phase>package</phase>
                           <goals>
                               <goal>copy-resources</goal>
                           </goals>
                           <configuration>
                               <outputDirectory>${project.build.directory}/</outputDirectory>
                               <resources>
                                   <resource>
                                       <directory>src/main/resources/</directory>
                                       <filtering>false</filtering>
                                   </resource>
                               </resources>
                           </configuration>
                       </execution>
                   </executions>
               </plugin>
            </plugins>
          </build>
   </project>

3. Create the implementation class of the HetuEngine UDF.

   package com.xxx.bigdata.hetuengine.functions;
   
   public class AddTwo {
       public Integer evaluate(Integer num) {
           return num + 2;
       }
   }

4. Package the Maven project. The udf-test-0.0.1-SNAPSHOT.jar file in the target directory is the HetuEngine UDF function package.

• A common HetuEngine UDF must implement at least one evaluate(). The evaluate function supports overloading.
• Currently, HetuEngine UDFs supports only less than or equal to five input parameters. HetuEngine UDFs with more than five input parameters will fail to be registered.
• (Optional) If the HetuEngine UDF depends on a configuration file, you are advised to save the configuration file as a resource file in the resources directory so that it can be packed into the HetuEngine UDF function package.

Deploying HetuEngine UDFs

To use the HetuEngine UDF in HetuEngine, you need to upload the corresponding UDF function package to a specified HDFS directory, for example, /udf/hetuserver. The directory can be customized based on the site requirements.

Create the /udf/hetuserver directory and save the UDF function package to it.

• Upload the files on the HDFS page:
  1. Log in to FusionInsight Manager using the HetuEngine username and choose Cluster > Services > HDFS.
  2. In the Basic Information area on the Dashboard page, click the link next to NameNode WebUI.
  3. Choose Utilities > Browse the file system and click to create the /udf/hetuserver directory.
  4. Go to the /udf/hetuserver directory and click to upload UDF function package.

• Use the HDFS CLI to upload the files.
  1. Log in to the node where the HDFS service client is located and switch to the client installation directory, for example, /opt/client.

     cd /opt/client

  2. Run the following command to configure environment variables:

     source bigdata_env

  3. If the cluster is in security mode, run the following command to authenticate the user. In normal mode, skip user authentication.

     kinitHetuEngineusername

     Enter the password as prompted.

  4. Run the following commands to create a directory and upload the prepared UDF function package to the target directory:

     hdfs dfs -mkdir /udf/hetuserver

     hdfs dfs -put ./UDF function package /udf/hetuserver.

  5. Run the following command to change the permission of the UDF function package:

     hdfs dfs -chmod 644 /udf/hetuserver/UDF function package

• When uploading a UDF JAR file to a user-defined HDFS directory, ensure that the user has the read permission on the JAR file. You are advised to use chmod 644 to set the permission. In addition, if you want the UDF JAR file to be deleted during the HetuEngine service uninstallation, you can create a user-defined directory in the /user/hetuserver/ directory.
• Currently, HetuEngine supports the UDF JAR file to be stored only in hdfs://Resource URI in HDFS.
• If the JAR file is re-uploaded due to function modification or addition, HetuEngine caches the classloader for 5 minutes by default. The JAR file does not take effect immediately, instead, it is updated and reloaded 5 minutes later.

Using HetuEngine UDFs

Use a client to access a HetuEngine UDF.

1. Log in to the HetuEngine client. For details, see Using the HetuEngine Client.
2. Create a HetuEngine UDF.

   CREATE FUNCTION example.namespace01.add_two (
    num integer
   )
   RETURNS integer
   LANGUAGE JAVA
   DETERMINISTIC
   SYMBOL "com.xxx.bigdata.hetuengine.functions.AddTwo"
   URI "hdfs://hacluster/udf/hetuserver/udf-test-0.0.1-SNAPSHOT.jar";

3. Use the HetuEngine UDF.

   select example.namespace01.add_two(2);
   _col0
   -------
        4
   (1 row)

Overloading is used to distinguish functions with the same name in the implementation classes. Therefore, you need to specify different function names when creating a HetuEngine UDF.