Updated on 2025-09-18 GMT+08:00

Python UDF Performance Tuning

DataArts Fabric SQL allows you to configure resource specifications and concurrency for Python UDF runtime, and collect critical performance metrics during execution.

UDF Performance Monitoring

You can run the explain performance command to print the performance metrics of the UDF Actor. As illustrated in Figure 1, it gathers statistics on the performance of the UDF Actor associated with the function calculate_0311 across each DN. The following uses three lines of performance data as an example.

Figure 1 Performance metrics of Python UDF actors
  • In executor0000_es_group (UdfName:calculate_0311 time=10907.3 rows=152 loops=1 cpu=1.0U memory=4.0GB timeout=1000s):
    • executor0_es_group indicates the DN ID.
    • UdfName indicates the name of the UDF to be executed.
    • time indicates the total runtime of the UDF.
    • rows indicates the amount of data processed on this DN.
    • loops indicates the number of times that the UDF was called.
    • cpu indicates the CPU size requested by the actor during UDF execution, measured in U.
    • memory indicates the memory size requested by the actor during UDF execution, measured in GB.
    • timeout indicates the maximum execution duration for a single batch of the actor during UDF execution, measured in seconds. Exceeding this limit will result in an error.
  • executor0000_es_group [actor 0](Summary: rows=152 bytes=42.11MB rtt=10804.3 execute=2041.0 setup=4003.0 send=4686.0 wait=1725.9 get=15.8 invoke=4748.0 close=1.8 terminate=1.9 serialize=63.1 deserialize=60.4) provides detailed information about the 0th UDF actor, as outlined below:
    • rows indicates the amount of data processed on this actor.
    • bytes indicates the size of data handled by this actor.
    • rtt indicates the end-to-end time from when the kernel sends data to the actor until the actor receives and processes it.
    • execute indicates the effective computation time of the actor.
    • setup indicates the time spent preparing the environment and loading the UDF code post-initialization.
    • send indicates the time taken to transmit data from the kernel to the actor.
    • wait indicates the delay while the kernel awaits data returns from the actor.
    • get indicates the successful retrieval time of actor data by the kernel.
    • invoke indicates the time required for starting an actor.
    • close indicates the time when the actor closes the computation.
    • terminate indicates the time required for the data system to bring the actor offline.
    • serialize indicates the time when data undergoes serialization.
    • deserialize indicates the time taken for data deserialization.
  • In executor0000_es_group [actor 0](Details: ip=10.42.0.103 rows=[152, 152] bytes=[42.11MB, 42.11MB] rtt=[10804.3, 10804.3] send=[4686.0, 4686.0] recv=[49.0, 49.0]):
    • ip indicates the IP address of the actor.
    • rows indicates the minimum and maximum numbers of rows in all batches executed by the actor.
    • bytes indicates the minimum and maximum sizes of data in all batches executed by the actor.
    • rtt indicates the minimum and maximum end-to-end execution times of all batches performed by the actor.
    • send indicates the minimum and maximum values of data sent from the kernel to the actor in all batches executed by the actor.
    • recv indicates the minimum and maximum values of data received by the kernel from Python across all batches executed by the actor.

As illustrated in the figure below, Query Summary displays the summary information about all UDF Actors in the query.

Actor total count indicates the total number of actors invoked by the current query.

Figure 2 Overview of data stream transmission information of Python UDF actors