Updated on 2022-11-18 GMT+08:00

HetuEngine UDF Development and Application

You can customize functions to extend SQL statements to meet personalized requirements. These functions are called UDFs.

This section describes how to develop and apply HetuEngine UDFs.

Developing HetuEngine UDFs

This sample implements one HetuEngine UDF described in the following table.

Table 1 HetuEngine UDF

Parameter

Description

AddTwo

Adds 2 to the input value and returns the result.

  1. Create a Maven project. Set groupId to com.test.udf and artifactId to udf-test. The two values can be customized based on the site requirements.
  2. Modify the pom.xml file as follows:

    <project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
           <modelVersion>4.0.0</modelVersion>
           <groupId>com.test.udf</groupId>
           <artifactId>udf-test</artifactId>
           <version>0.0.1-SNAPSHOT</version>
    
           <build>
             <plugins>
                <plugin>
                    <artifactId>maven-shade-plugin</artifactId>
                    <executions>
                        <execution>
                            <phase>package</phase>
                            <goals>
                                <goal>shade</goal>
                            </goals>
                        </execution>
                    </executions>
                </plugin>
                <plugin>
                    <artifactId>maven-resources-plugin</artifactId>
                    <executions>
                        <execution>
                            <id>copy-resources</id>
                            <phase>package</phase>
                            <goals>
                                <goal>copy-resources</goal>
                            </goals>
                            <configuration>
                                <outputDirectory>${project.build.directory}/</outputDirectory>
                                <resources>
                                    <resource>
                                        <directory>src/main/resources/</directory>
                                        <filtering>false</filtering>
                                    </resource>
                                </resources>
                            </configuration>
                        </execution>
                    </executions>
                </plugin>
             </plugins>
           </build>
    </project>

  3. Create the implementation class of the HetuEngine UDF.

    package com.xxx.bigdata.hetuengine.functions;
    
    public class AddTwo {
        public Integer evaluate(Integer num) {
            return num + 2;
        }
    }

  4. Package the Maven project. The udf-test-0.0.1-SNAPSHOT.jar file in the target directory is the HetuEngine UDF function package.
  • A common HetuEngine UDF must implement at least one evaluate(). The evaluate function supports overloading.
  • Currently, HetuEngine UDFs supports only less than or equal to five input parameters. HetuEngine UDFs with more than five input parameters will fail to be registered.
  • (Optional) If the HetuEngine UDF depends on a configuration file, you are advised to save the configuration file as a resource file in the resources directory so that it can be packed into the HetuEngine UDF function package.

Deploying HetuEngine UDFs

To use the HetuEngine UDF in HetuEngine, you need to upload the corresponding UDF function package to a specified HDFS directory, for example, /udf/hetuserver. The directory can be customized based on the site requirements.

Create the /udf/hetuserver directory and save the UDF function package to it.
  • Upload the files on the HDFS page:
    1. Log in to FusionInsight Manager using the HetuEngine username and choose Cluster > Services > HDFS.
    2. In the Basic Information area on the Dashboard page, click the link next to NameNode WebUI.
    3. Choose Utilities > Browse the file system and click to create the /udf/hetuserver directory.
    4. Go to the /udf/hetuserver directory and click to upload UDF function package.
  • Use the HDFS CLI to upload the files.
    1. Log in to the node where the HDFS service client is located and switch to the client installation directory, for example, /opt/client.

      cd /opt/client

    2. Run the following command to configure environment variables:

      source bigdata_env

    3. If the cluster is in security mode, run the following command to authenticate the user. In normal mode, skip user authentication.

      kinitHetuEngineusername

      Enter the password as prompted.

    4. Run the following commands to create a directory and upload the prepared UDF function package to the target directory:

      hdfs dfs -mkdir /udf/hetuserver

      hdfs dfs -put ./UDF function package /udf/hetuserver.

    5. Run the following command to change the permission of the UDF function package:

      hdfs dfs -chmod 644 /udf/hetuserver/UDF function package

  • When uploading a UDF JAR file to a user-defined HDFS directory, ensure that the user has the read permission on the JAR file. You are advised to use chmod 644 to set the permission. In addition, if you want the UDF JAR file to be deleted during the HetuEngine service uninstallation, you can create a user-defined directory in the /user/hetuserver/ directory.
  • Currently, HetuEngine supports the UDF JAR file to be stored only in hdfs://Resource URI in HDFS.
  • If the JAR file is re-uploaded due to function modification or addition, HetuEngine caches the classloader for 5 minutes by default. The JAR file does not take effect immediately, instead, it is updated and reloaded 5 minutes later.

Using HetuEngine UDFs

Use a client to access a HetuEngine UDF.

  1. Log in to the HetuEngine client. For details, see Using the HetuEngine Client.
  2. Create a HetuEngine UDF.
    CREATE FUNCTION example.namespace01.add_two (
     num integer
    )
    RETURNS integer
    LANGUAGE JAVA
    DETERMINISTIC
    SYMBOL "com.xxx.bigdata.hetuengine.functions.AddTwo"
    URI "hdfs://hacluster/udf/hetuserver/udf-test-0.0.1-SNAPSHOT.jar";
  3. Use the HetuEngine UDF.
    select example.namespace01.add_two(2);
    _col0
    -------
         4
    (1 row)

Overloading is used to distinguish functions with the same name in the implementation classes. Therefore, you need to specify different function names when creating a HetuEngine UDF.