Hive UDFs for Interconnecting with HetuEngine

Developing Hive UDFs

This sample implements one Hive UDF described in the following table.

• A common Hive UDF must be inherited from org.apache.hadoop.hive.ql.exec.UDF.
• A common Hive UDF must implement at least one evaluate(). The evaluate function supports overloading.
• Currently, only the following data types are supported:
  • boolean, byte, short, int, long, float, and double
  • Boolean, Byte, Short, Int, Long, Float, and Double
  • List and Map

  UDFs, UDAFs, and UDTFs currently do not support complex data types other than the preceding ones.

• Currently, Hive UDFs support only five or fewer input parameters. UDFs with more than five input parameters will fail during registration.
• If a Hive UDF input parameter is null, the call returns null immediately without executing the UDF logic. Consequently, the UDF result may be inconsistent with the standard Hive execution result.
• To add the hive-exec-3.1.1 dependency package to your Maven project, you can obtain the package from the Hive installation directory.
• (Optional) If the Hive UDF depends on a configuration file, you are advised to save the configuration file as a resource file in the resources directory so that it can be included in the Hive UDF function package.

1. Create a Maven project. Set groupId to com.test.udf and artifactId to udf-test. The two values can be customized based on the site requirements.
2. Modify the pom.xml file as follows:

   <project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
          <modelVersion>4.0.0</modelVersion>
          <groupId>com.test.udf</groupId>
          <artifactId>udf-test</artifactId>
          <version>0.0.1-SNAPSHOT</version>
   
          <dependencies>
            <dependency>
               <groupId>org.apache.hive</groupId>
               <artifactId>hive-exec</artifactId>
               <version>3.1.1</version>
            </dependency>
          </dependencies>
   
          <build>
            <plugins>
               <plugin>
                   <artifactId>maven-shade-plugin</artifactId>
                   <executions>
                       <execution>
                           <phase>package</phase>
                           <goals>
                               <goal>shade</goal>
                           </goals>
                       </execution>
                   </executions>
               </plugin>
               <plugin>
                   <artifactId>maven-resources-plugin</artifactId>
                   <executions>
                       <execution>
                           <id>copy-resources</id>
                           <phase>package</phase>
                           <goals>
                               <goal>copy-resources</goal>
                           </goals>
                           <configuration>
                               <outputDirectory>${project.build.directory}/</outputDirectory>
                               <resources>
                                   <resource>
                                       <directory>src/main/resources/</directory>
                                       <filtering>false</filtering>
                                   </resource>
                               </resources>
                           </configuration>
                       </execution>
                   </executions>
               </plugin>
            </plugins>
          </build>
   </project>

3. Create the implementation class for the Hive UDF.

   import org.apache.hadoop.hive.ql.exec.UDF; 
   
   /** 
    * AutoAddOne 
    * 
    * @since 2020-08-24 
    */ 
   public class AutoAddOne extends UDF { 
       public int evaluate(int data) { 
           return data + 1; 
       } 
   }

4. Package the Maven project. The udf-test-0.0.1-SNAPSHOT.jar file in the target directory is the Hive UDF function package.
   

   You need to pack all dependencies into a JAR package.

Configuring Hive UDFs

In configuration file udf.properties, add registration information in the "Function_name Class_path" format to each line to register the Hive UDF.

The following is an example of registering four Hive UDFs in the udf.properties configuration file:

booleanudf io.hetu.core.hive.dynamicfunctions.examples.udf.BooleanUDF
shortudf io.hetu.core.hive.dynamicfunctions.examples.udf.ShortUDF
byteudf io.hetu.core.hive.dynamicfunctions.examples.udf.ByteUDF
intudf io.hetu.core.hive.dynamicfunctions.examples.udf.IntUDF

• If the added Hive UDF registration information is incorrect, for example, an invalid format or a non-existent class path, the system ignores the information and records a corresponding log.
• If duplicate Hive UDFs are registered, the system performs only one registration and ignores all subsequent duplicates.
• If a Hive UDF being registered matches one already registered in the system, the system throws an exception and fails to start. To resolve this issue, you need to delete the Hive UDF registration information.

Deploying Hive UDFs

To use an existing Hive UDF in HetuEngine, you need to upload the UDF function package, udf.properties file, and any required UDF configuration files to the specified HDFS directory, for example, /user/hetuserver/udf/, and then restart the HetuEngine compute instance.

Using Hive UDFs

Use a client to access a UDF.

1. Log in to the HetuEngine client.
2. Run the following command to execute the Hive UDF:

   select AutoAddOne(1);

   select AutoAddOne(1);
   _col0
   -------
        2
   (1 row)