Help Center/ MapReduce Service/ Developer Guide (LTS)/ Flink Development Guide (Normal Mode)/ Developing an Application/ PyFlink Sample Program/ Using Python to Submit a Common Flink Job
Updated on 2026-08-27 GMT+08:00
View PDF
Share

Using Python to Submit a Common Flink Job

  1. Obtain pyflink-kafk a.py and insertData2kafka.sql from the sample project flink-examples/pyflink-example/pyflink-kafka.
  2. Package the prepared Python virtual environment by referring to Preparing a Local Application Development Environment and obtain the venv.zip file. 

    zip -q -r venv.zip venv/

  3. Log in to the active management node as the root user and upload venv.zip, pyflink-kafka.py, and insertData2kafka.sql files obtained in 1and 2 to the client environment.

    yarn-application: Upload the preceding files and the flink-connector-kafka-Actual version number.jar package to Client installation directory/Flink/flink/yarnship.

  4. Change the specific_jars path in pyflink-kafka.py.

    yarn-application: Change to file://"+os.getcwd()+"/../../../../yarnship/flink-connector-kafka-Actual version number.jar.

  5. Change file_path in pyflink-kafka.py.

    yarn-application: Change the path to os.getcwd () + "/../../../../yarnship/insertData2kafka.sql"

  6. Run the following command to specify the running environment: 

    export PYFLINK_CLIENT_EXECUTABLE=venv.zip/venv/bin/python3

  7. Run the following command to run the program:

    yarn-application 
    ./bin/flink run-application --detached -t yarn-application -Dyarn.application.name=py_kafka -Dyarn.ship-files=/opt/client/Flink/flink/yarnship/ -pyarch yarnship/venv.zip -pyexec venv.zip/venv/bin/python3 -pyclientexec venv.zip/venv/bin/python3 -pyfs yarnship -pym pyflink-kafka

    Execution result:

Parent topic: PyFlink Sample Program