UDF

Updated on 2022-07-11 GMT+08:00

View PDF

When internal functions of Hive cannot meet requirements, you can compile user-defined functions (UDFs) and use them for query.

According to implementation methods, UDFs are classified as follows:

Common UDFs: used to perform operations on a single data row and export a single data row.
User-defined aggregating functions (UDAFs): used to input multiple data rows and export a single data row.
User-defined table-generating functions (UDTFs): used to perform operations on a single data row and export multiple data rows.

According to usage methods, UDFs are classified as follows:

Temporary functions: used only in the current session and must be recreated after a session restarts.
Permanent function: used in multiple sessions and do not have to be created every time a session restarts.

The following uses AddDoublesUDF as an example to describe how to compile and use UDFs.

Function

AddDoublesUDF is used to add two or multiple floating point values. The following example describes how to compile and use UDFs.

NOTE:

The normal UDF must be originated from org.apache.hadoop.hive.ql.exec.UDF.
The normal UDF must implement at least one evaluate(). The evaluate function supports overloading.
To develop a customized function, you need to add the hive-exec-3.1.0.jar dependency package to the project. The package can be obtained from the Hive installation directory.

Example Codes

The following is a UDF example:

package com.huawei.bigdata.hive.example.udf;
import org.apache.hadoop.hive.ql.exec.UDF;

public class AddDoublesUDF extends UDF { 
  public Double evaluate(Double... a) { 
     Double total = 0.0; 
     // Processing logic 
     for (int i = 0; i < a.length; i++) 
       if (a[i] != null) 
         total += a[i]; 
     return total; 
   } 
 }

How to Use

Package the preceding program into AddDoublesUDF.jar, and upload it to a directory on the HDFS (such as /user/hive_examples_jars/). The user who creates the UDF and the user who uses the UDF function must have read right on this JAR file. Example statements:

hdfs dfs -put ./hive_examples_jars /user/hive_examples_jars

hdfs dfs -chmod 777 /user/hive_examples_jars
Use a user with admin rights to log in to the Beeline client and run the following command:

kinit Hive service user

beeline

set role admin;
Define the function in Hive Server. Run the following SQL statement to create a permanent function:

CREATE FUNCTION addDoubles AS 'com.huawei.bigdata.hive.example.udf.AddDoublesUDF' using jar 'hdfs://hacluster/user/hive_examples_jars/AddDoublesUDF.jar';

addDoubles indicates the function alias that is used for SELECT query.

Run the following statement to create a temporary function:

CREATE TEMPORARY FUNCTION addDoubles AS 'com.huawei.bigdata.hive.example.udf.AddDoublesUDF' using jar 'hdfs://hacluster/user/hive_examples_jars/AddDoublesUDF.jar';
- addDoubles indicates the function alias that is used for SELECT query.
- TEMPORARY indicates that the function is used only in the current session with the Hive server.
Run the following SQL statement to use the function in the Hive server:

SELECT addDoubles(1,2,3);

NOTE:

If an [Error 10011] error is displayed when you log in to the client again, run the reload function; command and then use this function.
Run the following SQL statement to delete the function from the Hive server:

DROP FUNCTION addDoubles;

Extensions

None

Parent topic: Example Codes

Previous topic: Querying Data

Next topic: Example Program Guide

Feedback

Was this page helpful?

Helpful Not helpful

Provide feedback

Thank you very much for your feedback. We will continue working to improve the documentation.See the reply and handling status in My Cloud VOC.

The system is busy. Please try again later.

Which of the following issues have you encountered?

Content is inconsistent with the product UI

Unclear descriptions

Lack of examples or code

Incorrect steps

Can't find what I need

Lack of best practices

Feedback (optional)

0/500

Select at least one type of issue, and enter your comments or suggestions.

Enter a maximum of 500 characters.

Submit Cancel

For any further questions, feel free to contact us through the chatbot.

Chatbot