On this page

Show all

Help Center/ MapReduce Service/ Component Operation Guide (LTS) (Paris Region)/ Using HetuEngine/ Function & UDF Development and Application/ HetuEngine UDF Development and Application

HetuEngine UDF Development and Application

Updated on 2022-12-14 GMT+08:00

View PDF

You can customize functions to extend SQL statements to meet personalized requirements. These functions are called UDFs.

This section describes how to develop and apply HetuEngine UDFs.

Developing HetuEngine UDFs

This sample implements one HetuEngine UDF described in the following table.

**Table 1** HetuEngine UDF
Parameter	Description
AddTwo	Adds 2 to the input value and returns the result.

Create a Maven project. Set groupId to com.test.udf and artifactId to udf-test. The two values can be customized based on the site requirements.

Modify the pom.xml file as follows:

<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
       <modelVersion>4.0.0</modelVersion>
       <groupId>com.test.udf</groupId>
       <artifactId>udf-test</artifactId>
       <version>0.0.1-SNAPSHOT</version>

       <build>
         <plugins>
            <plugin>
                <artifactId>maven-shade-plugin</artifactId>
                <executions>
                    <execution>
                        <phase>package</phase>
                        <goals>
                            <goal>shade</goal>
                        </goals>
                    </execution>
                </executions>
            </plugin>
            <plugin>
                <artifactId>maven-resources-plugin</artifactId>
                <executions>
                    <execution>
                        <id>copy-resources</id>
                        <phase>package</phase>
                        <goals>
                            <goal>copy-resources</goal>
                        </goals>
                        <configuration>
                            <outputDirectory>${project.build.directory}/</outputDirectory>
                            <resources>
                                <resource>
                                    <directory>src/main/resources/</directory>
                                    <filtering>false</filtering>
                                </resource>
                            </resources>
                        </configuration>
                    </execution>
                </executions>
            </plugin>
         </plugins>
       </build>
</project>

Create the implementation class of the HetuEngine UDF.

package com.xxx.bigdata.hetuengine.functions;

public class AddTwo {
    public Integer evaluate(Integer num) {
        return num + 2;
    }
}

Package the Maven project. The udf-test-0.0.1-SNAPSHOT.jar file in the target directory is the HetuEngine UDF function package.

NOTE:

A common HetuEngine UDF must implement at least one evaluate(). The evaluate function supports overloading.
Currently, HetuEngine UDFs supports only less than or equal to five input parameters. HetuEngine UDFs with more than five input parameters will fail to be registered.
(Optional) If the HetuEngine UDF depends on a configuration file, you are advised to save the configuration file as a resource file in the resources directory so that it can be packed into the HetuEngine UDF function package.

Deploying HetuEngine UDFs

To use the HetuEngine UDF in HetuEngine, you need to upload the corresponding UDF function package to a specified HDFS directory, for example, /udf/hetuserver. The directory can be customized based on the site requirements.

Create the /udf/hetuserver directory and save the UDF function package to it.

Upload the files on the HDFS page:
1. Log in to FusionInsight Manager using the HetuEngine username and choose Cluster > Services > HDFS.
2. In the Basic Information area on the Dashboard page, click the link next to NameNode WebUI.
3. Choose Utilities > Browse the file system and click to create the /udf/hetuserver directory.
4. Go to the /udf/hetuserver directory and click to upload UDF function package.

Use the HDFS CLI to upload the files.
1. Log in to the node where the HDFS service client is located and switch to the client installation directory, for example, /opt/client.
  cd /opt/client
2. Run the following command to configure environment variables:
  source bigdata_env
3. If the cluster is in security mode, run the following command to authenticate the user. In normal mode, skip user authentication.
  kinitHetuEngineusername
  
  Enter the password as prompted.
4. Run the following commands to create a directory and upload the prepared UDF function package to the target directory:
  hdfs dfs -mkdir /udf/hetuserver
  
  hdfs dfs -put ./UDF function package /udf/hetuserver.
5. Run the following command to change the permission of the UDF function package:
  hdfs dfs -chmod 644 /udf/hetuserver/UDF function package

NOTICE:

When uploading a UDF JAR file to a user-defined HDFS directory, ensure that the user has the read permission on the JAR file. You are advised to use chmod 644 to set the permission. In addition, if you want the UDF JAR file to be deleted during the HetuEngine service uninstallation, you can create a user-defined directory in the /user/hetuserver/ directory.
Currently, HetuEngine supports the UDF JAR file to be stored only in hdfs://Resource URI in HDFS.
If the JAR file is re-uploaded due to function modification or addition, HetuEngine caches the classloader for 5 minutes by default. The JAR file does not take effect immediately, instead, it is updated and reloaded 5 minutes later.

Using HetuEngine UDFs

Use a client to access a HetuEngine UDF.

Create a HetuEngine UDF.

CREATE FUNCTION example.namespace01.add_two (
 num integer
)
RETURNS integer
LANGUAGE JAVA
DETERMINISTIC
SYMBOL "com.xxx.bigdata.hetuengine.functions.AddTwo"
URI "hdfs://hacluster/udf/hetuserver/udf-test-0.0.1-SNAPSHOT.jar";

Use the HetuEngine UDF.

select example.namespace01.add_two(2);
_col0
-------
     4
(1 row)

NOTE:

Overloading is used to distinguish functions with the same name in the implementation classes. Therefore, you need to specify different function names when creating a HetuEngine UDF.

Parent topic: Function & UDF Development and Application

Previous topic: Hive UDF Development and Application

Next topic: Introduction to HetuEngine Logs

Feedback

Was this page helpful?

Helpful Not helpful

Provide feedback

Thank you very much for your feedback. We will continue working to improve the documentation.See the reply and handling status in My Cloud VOC.

The system is busy. Please try again later.

Which of the following issues have you encountered?

Content is inconsistent with the product UI

Unclear descriptions

Lack of examples or code

Incorrect steps

Can't find what I need

Lack of best practices

Feedback (optional)

0/500

Select at least one type of issue, and enter your comments or suggestions.

Enter a maximum of 500 characters.

Submit Cancel

For any further questions, feel free to contact us through the chatbot.

Chatbot