内置依赖包

Spark 2.3.2 依赖包

• accessors-smart-1.2.jar
• activation-1.1.1.jar
• aircompressor-0.8.jar
• alluxio-2.3.1-luxor-SNAPSHOT-client.jar
• antlr-2.7.7.jar
• antlr4-runtime-4.8-1.jar
• antlr-runtime-3.4.jar
• aopalliance-1.0.jar
• aopalliance-repackaged-2.4.0-b34.jar
• apache-log4j-extras-1.2.17.jar
• arpack_combined_all-0.1.jar
• arrow-format-0.8.0.jar
• arrow-memory-0.8.0.jar
• arrow-vector-0.8.0.jar
• asm-5.0.4.jar
• audience-annotations-0.5.0.jar
• automaton-1.11-8.jar
• avro-1.7.7.jar
• avro-ipc-1.7.7.jar
• avro-ipc-1.7.7-tests.jar
• avro-mapred-1.7.7-hadoop2.jar
• aws-java-sdk-bundle-1.11.271.jar
• base64-2.3.8.jar
• bcpkix-jdk15on-1.66.jar
• bcprov-jdk15on-1.66.jar
• bonecp-0.8.0.RELEASE.jar
• breeze_2.11-0.13.2.jar
• breeze-macros_2.11-0.13.2.jar
• calcite-avatica-1.2.0-incubating.jar
• calcite-core-1.2.0-incubating.jar
• calcite-linq4j-1.2.0-incubating.jar
• carbondata-bloom-1.6.1.1032-2.0.0.dli-20201012.110925-6.jar
• carbondata-cli-1.6.1.1032-2.0.0.dli-20201012.110947-6.jar
• carbondata-common-1.6.1.1032-2.0.0.dli-20201012.110840-6.jar
• carbondata-common-plugin-1.6.1.1032-2.0.0.dli-20201012.110945-6.jar
• carbondata-core-1.6.1.1032-2.0.0.dli-20201012.110843-6.jar
• carbondata-format-1.6.1.1032-2.0.0.dli-20201012.110953-12.jar
• carbondata-hadoop-1.6.1.1032-2.0.0.dli-20201012.110849-6.jar
• carbondata-hive-1.6.1.1032-2.0.0.dli-20201012.110911-6.jar
• carbondata-mv-core-1.6.1.1032-2.0.0.dli-20201012.110936-6.jar
• carbondata-mv-plan-1.6.1.1032-2.0.0.dli-20201012.110933-6.jar
• carbondata-processing-1.6.1.1032-2.0.0.dli-20201012.110846-6.jar
• carbondata-secondary_index-1.6.1.1032-2.0.0.dli-20201012.110942-6.jar
• carbondata-spark2-1.6.1.1032-2.0.0.dli-20201012.110857-6.jar
• carbondata-spark-common-1.6.1.1032-2.0.0.dli-20201012.110903-6.jar
• carbondata-spark-datasource-1.6.1.1032-2.0.0.dli-20201012.110900-6.jar
• carbondata-store-sdk-1.6.1.1032-2.0.0.dli-20201012.110852-6.jar
• carbondata-streaming-1.6.1.1032-2.0.0.dli-20201012.110922-6.jar
• carbondata-vector-1.6.1.1032-2.0.0.dli-20201012.110956-6.jar
• checker-qual-2.11.1.jar
• chill_2.11-0.8.4.jar
• chill-java-0.8.4.jar
• commons-beanutils-1.9.4.jar
• commons-cli-1.2.jar
• commons-codec-2.0-20130428.202122-59.jar
• commons-collections-3.2.2.jar
• commons-collections4-4.2.jar
• commons-compiler-3.0.8.jar
• commons-compress-1.4.1.jar
• commons-configuration2-2.1.1.jar
• commons-crypto-1.0.0-20191105.jar
• commons-daemon-1.0.13.jar
• commons-dbcp-1.4.jar
• commons-dbcp2-2.7.0.jar
• commons-httpclient-3.1.jar
• commons-io-2.5.jar
• commons-lang-2.6.jar
• commons-lang3-3.5.jar
• commons-logging-1.2.jar
• commons-math3-3.4.1.jar
• commons-net-2.2.jar
• commons-pool-1.5.4.jar
• commons-pool2-2.8.0.jar
• commons-text-1.3.jar
• compress-lzf-1.0.3.jar
• core-1.1.2.jar
• curator-client-4.2.0.jar
• curator-framework-4.2.0.jar
• curator-recipes-2.7.1.jar
• datanucleus-api-jdo-3.2.6.jar
• datanucleus-core-3.2.10.jar
• datanucleus-rdbms-3.2.9.jar
• derby-10.12.1.1.jar
• dnsjava-2.1.7.jar
• ehcache-3.3.1.jar
• eigenbase-properties-1.1.5.jar
• error_prone_annotations-2.3.4.jar
• failureaccess-1.0.1.jar
• fastutil-8.2.3.jar
• ffmpeg-4.3.1-1.5.4.jar
• ffmpeg-4.3.1-1.5.4-linux-x86_64.jar
• flatbuffers-1.2.0-3f79e055.jar
• generex-1.0.2.jar
• geronimo-jcache_1.0_spec-1.0-alpha-1.jar
• gson-2.2.4.jar
• guava-29.0-jre.jar
• guice-4.0.jar
• guice-servlet-4.0.jar
• hadoop-annotations-3.1.1-ei-302002.jar
• hadoop-auth-3.1.1-ei-302002.jar
• hadoop-aws-3.1.1-ei-302002.jar
• hadoop-client-3.1.1-ei-302002.jar
• hadoop-common-3.1.1-ei-302002.jar
• hadoop-hdfs-3.1.1-ei-302002.jar
• hadoop-hdfs-client-3.1.1-ei-302002.jar
• hadoop-mapreduce-client-common-3.1.1-ei-302002.jar
• hadoop-mapreduce-client-core-3.1.1-ei-302002.jar
• hadoop-mapreduce-client-jobclient-3.1.1-ei-302002.jar
• hadoop-minikdc-3.1.1-hw-ei-302002.jar
• hadoop-yarn-api-3.1.1-ei-302002.jar
• hadoop-yarn-client-3.1.1-ei-302002.jar
• hadoop-yarn-common-3.1.1-ei-302002.jar
• hadoop-yarn-registry-3.1.1-ei-302002.jar
• hadoop-yarn-server-common-3.1.1-ei-302002.jar
• hadoop-yarn-server-web-proxy-3.1.1-ei-302002.jar
• hamcrest-core-1.3.jar
• HikariCP-java7-2.4.12.jar
• hive-common-1.2.1-2.1.0.dli-20201111.064115-91.jar
• hive-exec-1.2.1-2.1.0.dli-20201111.064444-91.jar
• hive-metastore-1.2.1-2.1.0.dli-20201111.064230-91.jar
• hk2-api-2.4.0-b34.jar
• hk2-locator-2.4.0-b34.jar
• hk2-utils-2.4.0-b34.jar
• hppc-0.7.2.jar
• htrace-core4-4.2.0-incubating-hw-1.0.0.jar
• httpclient-4.5.4.jar
• httpcore-4.4.7.jar
• ivy-2.4.0.jar
• j2objc-annotations-1.3.jar
• jackson-annotations-2.10.0.jar
• jackson-core-2.10.0.jar
• jackson-core-asl-1.9.13-atlassian-4.jar
• jackson-databind-2.10.0.jar
• jackson-dataformat-yaml-2.10.0.jar
• jackson-datatype-jsr310-2.10.3.jar
• jackson-jaxrs-base-2.10.3.jar
• jackson-jaxrs-json-provider-2.10.3.jar
• jackson-mapper-asl-1.9.13-atlassian-4.jar
• jackson-module-jaxb-annotations-2.10.3.jar
• jackson-module-paranamer-2.10.0.jar
• jackson-module-scala_2.11-2.10.0.jar
• jakarta.activation-api-1.2.1.jar
• jakarta.xml.bind-api-2.3.2.jar
• janino-3.0.8.jar
• javacpp-1.5.4.jar
• javacpp-1.5.4-linux-x86_64.jar
• javacv-1.5.4.jar
• JavaEWAH-1.1.7.jar
• javassist-3.18.1-GA.jar
• javax.annotation-api-1.2.jar
• javax.inject-1.jar
• javax.inject-2.4.0-b34.jar
• javax.servlet-api-3.1.0.jar
• javax.ws.rs-api-2.0.1.jar
• java-xmlbuilder-1.1.jar
• javolution-5.3.1.jar
• jaxb-api-2.2.11.jar
• jcip-annotations-1.0-1.jar
• jcl-over-slf4j-1.7.26.jar
• jdo-api-3.0.1.jar
• jersey-client-2.23.1.jar
• jersey-common-2.23.1.jar
• jersey-container-servlet-2.23.1.jar
• jersey-container-servlet-core-2.23.1.jar
• jersey-guava-2.23.1.jar
• jersey-media-jaxb-2.23.1.jar
• jersey-server-2.23.1.jar
• jets3t-0.9.4.jar
• jetty-http-9.4.31.v20200723.jar
• jetty-io-9.4.31.v20200723.jar
• jetty-security-9.4.31.v20200723.jar
• jetty-server-9.4.31.v20200723.jar
• jetty-servlet-9.4.31.v20200723.jar
• jetty-util-9.4.31.v20200723.jar
• jetty-util-ajax-9.4.31.v20200723.jar
• jetty-webapp-9.4.31.v20200723.jar
• jetty-xml-9.4.31.v20200723.jar
• joda-time-2.9.3.jar
• jodd-core-4.2.0.jar
• json-20200518.jar
• json4s-ast_2.11-3.2.11.jar
• json4s-core_2.11-3.2.11.jar
• json4s-jackson_2.11-3.2.11.jar
• json-sanitizer-1.2.1.jar
• json-smart-2.3.jar
• jsp-api-2.1.jar
• jsr305-3.0.2.jar
• jta-1.1.jar
• jtransforms-2.4.0.jar
• jul-to-slf4j-1.7.26.jar
• junit-4.11.jar
• kerb-admin-1.0.1.jar
• kerb-client-1.0.1.jar
• kerb-common-1.0.1.jar
• kerb-core-1.0.1.jar
• kerb-crypto-1.0.1.jar
• kerb-identity-1.0.1.jar
• kerb-server-1.0.1.jar
• kerb-simplekdc-1.0.1.jar
• kerb-util-1.0.1.jar
• kerby-asn1-1.0.1.jar
• kerby-config-1.0.1.jar
• kerby-pkix-1.0.1.jar
• kerby-util-1.0.1.jar
• kerby-xdr-1.0.1.jar
• kryo-shaded-3.0.3.jar
• kubernetes-client-4.9.2-20200804.jar
• kubernetes-model-4.9.2-20200804.jar
• kubernetes-model-common-4.9.2-20200804.jar
• leveldbjni-all-1.8-20191105.jar
• libfb303-0.9.3.jar
• libthrift-0.12.0.jar
• listenablefuture-9999.0-empty-to-avoid-conflict-with-guava.jar
• log4j-1.2.17-cloudera1.jar
• log4j-rolling-appender-20131024-2017.jar
• logging-interceptor-3.14.4.jar
• luxor-encrypt-2.1.0-20201106.065437-53.jar
• luxor-fs3-2.1.0-20201106.065612-53.jar
• luxor-obs-fs3-2.1.0-20201106.065616-53.jar
• luxor-rpc_2.11-2.1.0-20201106.065541-53.jar
• luxor-rpc-protobuf2-2.1.0-20201106.065551-53.jar
• lz4-java-1.7.1.jar
• machinist_2.11-0.6.1.jar
• macro-compat_2.11-1.1.1.jar
• metrics-core-3.1.5.jar
• metrics-graphite-3.1.5.jar
• metrics-jmx-4.1.12.1.jar
• metrics-json-3.1.5.jar
• metrics-jvm-3.1.5.jar
• minlog-1.3.0.jar
• mssql-jdbc-6.2.1.jre7.jar
• netty-3.10.6.Final.jar
• netty-all-4.1.51.Final.jar
• nimbus-jose-jwt-8.19.jar
• objenesis-2.1.jar
• okhttp-3.14.4.jar
• okio-1.17.2.jar
• opencsv-2.3.jar
• opencsv-4.6.jar
• opencv-4.3.0-2.jar
• orc-core-1.4.4-nohive.jar
• orc-mapreduce-1.4.4-nohive.jar
• oro-2.0.8.jar
• osgi-resource-locator-1.0.1.jar
• paranamer-2.8.jar
• parquet-column-1.8.3.jar
• parquet-common-1.8.3.jar
• parquet-encoding-1.8.3.jar
• parquet-format-2.3.1.jar
• parquet-hadoop-1.8.3.jar
• parquet-hadoop-bundle-1.6.0.jar
• parquet-jackson-1.8.3.jar
• parquet-format-2.3.1.jar
• parquet-hadoop-1.8.3.jar
• parquet-hadoop-bundle-1.6.0.jar
• parquet-jackson-1.8.3.jar
• postgresql-42.2.14.jar
• protobuf-java-2.5.0.jar
• py4j-0.10.7.jar
• pyrolite-4.13.jar
• re2j-1.1.jar
• RoaringBitmap-0.5.11.jar
• scala-compiler-2.11.12.jar
• scala-library-2.11.12.jar
• scalap-2.11.0.jar
• scala-parser-combinators_2.11-1.1.0.jar
• scala-reflect-2.11.12.jar
• scala-xml_2.11-1.0.5.jar
• secComponentApi-1.0.5c.jar
• shapeless_2.11-2.3.2.jar
• slf4j-api-1.7.30.jar
• slf4j-log4j12-1.7.30.jar
• snakeyaml-1.24.jar
• snappy-java-1.1.7.5.jar
• spark-catalyst_2.11-2.3.2.0101-2.1.0.dli-20201111.073826-143.jar
• spark-core_2.11-2.3.2.0101-2.1.0.dli-20201111.073836-134.jar
• spark-graphx_2.11-2.3.2.0101-2.1.0.dli-20201111.073847-129.jar
• spark-hive_2.11-2.3.2.0101-2.1.0.dli-20201111.073854-132.jar
• spark-kubernetes_2.11-2.3.2.0101-2.1.0.dli-20201111.073916-85.jar
• spark-kvstore_2.11-2.3.2.0101-2.1.0.dli-20201111.073933-127.jar
• spark-launcher_2.11-2.3.2.0101-2.1.0.dli-20201111.073940-127.jar
• spark-mllib_2.11-2.3.2.0101-2.1.0.dli-20201111.073946-127.jar
• spark-mllib-local_2.11-2.3.2.0101-2.1.0.dli-20201111.073953-127.jar
• spark-network-common_2.11-2.3.2.0101-2.1.0.dli-20201111.073959-127.jar
• spark-network-shuffle_2.11-2.3.2.0101-2.1.0.dli-20201111.074007-127.jar
• spark-om_2.11-2.3.2.0101-2.1.0.dli-20201111.074019-125.jar
• spark-repl_2.11-2.3.2.0101-2.1.0.dli-20201111.074028-125.jar
• spark-sketch_2.11-2.3.2.01012.1.0.dli-20201111.074035-125.jar
• spark-sql_2.11-2.3.2.0101-2.1.0.dli-20201111.074041-126.jar
• spark-streaming_2.11-2.3.2.0101-2.1.0.dli-20201111.074100-123.jar
• spark-tags_2.11-2.3.2.0101-2.1.0.dli-20201111.074136-123.jar
• spark-tags_2.11-2.3.2.0101-2.1.0.dli-20201111.074141-124-tests.jar
• spark-unsafe_2.11-2.3.2.0101-2.1.0.dli-20201111.074144-123.jar
• spark-uquery_2.11-2.3.2.0101-2.1.0.dli-20201111.074906-210.jar
• spark-yarn_2.11-2.3.2.0101-2.1.0.dli-20201111.074151-123.jar
• spire_2.11-0.13.0.jar
• spire-macros_2.11-0.13.0.jar
• ST4-4.3.1.jar
• stax2-api-3.1.4.jar
• stax-api-1.0-2.jar
• stream-2.7.0.jar
• stringtemplate-3.2.1.jar
• token-provider-1.0.1.jar
• univocity-parsers-2.5.9.jar
• validation-api-1.1.0.Final.jar
• woodstox-core-5.0.3.jar
• xbean-asm5-shaded-4.4.jar
• xercesImpl-2.12.0.jar
• xml-apis-1.4.01.jar
• xz-1.0.jar
• zjsonpatch-0.3.0.jar
• zookeeper-3.5.6-ei-302002.jar
• zookeeper-jute-3.5.6-ei-302002.jar
• zstd-jni-1.4.4-11.jar

Flink 1.7.2 依赖包

• bcpkix-jdk15on-1.60.jar
• bcprov-jdk15on-1.60.jar
• commons-codec-1.9.jar
• commons-configuration-1.7.jar
• deeplearning4j-core-0.9.1.jar
• deeplearning4j-nlp-0.9.1.jar
• deeplearning4j-nn-0.9.1.jar
• ejml-cdense-0.33.jar
• ejml-core-0.33.jar
• ejml-ddense-0.33.jar
• ejml-dsparse-0.33.jar
• ejml-experimental-0.33.jar
• ejml-fdense-0.33.jar
• ejml-simple-0.33.jar
• ejml-zdense-0.33.jar
• elsa-3.0.0-M7.jar
• esdk-obs-java-3.1.3.jar
• flink-cep_2.11-1.7.0.jar
• flink-cep-scala_2.11-1.7.0.jar
• flink-dist_2.11-1.7.0.jar
• flink-gelly_2.11-1.7.0.jar
• flink-gelly-scala_2.11-1.7.0.jar
• flink-ml_2.11-1.7.0.jar
• flink-python_2.11-1.7.0.jar
• flink-queryable-state-runtime_2.11-1.7.0.jar
• flink-shaded-curator-1.7.0.jar
• flink-shaded-hadoop2-uber-1.7.0.jar
• flink-table_2.11-1.7.0.jar
• guava-26.0-jre.jar
• hadoop-cloud-3.1.1-41-20201014.085840-4.jar
• httpasyncclient-4.1.2.jar
• httpclient-4.5.12.jar
• httpcore-4.4.4.jar
• httpcore-nio-4.4.4.jar
• java-xmlbuilder-1.1.jar
• jna-4.1.0.jar
• libtensorflow-1.12.0.jar
• log4j-api-2.8.2.jar
• log4j-core-2.8.2.jar
• log4j-over-slf4j-1.7.21.jar
• logback-classic-1.2.3.jar
• logback-core-1.2.3.jar
• nd4j-api-0.9.1.jar
• nd4j-native-0.9.1.jar
• nd4j-native-api-0.9.1.jar
• nd4j-native-platform-0.9.1.jar
• okhttp-3.14.8.jar
• okio-1.14.0.jar
• slf4j-api-1.7.21.jar
• tensorflow-1.12.0.jar

Flink 1.10依赖包

2020年12月之后创建的新队列才能使用Flink 1.10依赖包。

• bcpkix-jdk15on-1.60.jar
• bcprov-jdk15on-1.60.jar
• commons-codec-1.9.jar
• commons-configuration-1.7.jar
• deeplearning4j-core-0.9.1.jar
• deeplearning4j-nlp-0.9.1.jar
• deeplearning4j-nn-0.9.1.jar
• ejml-cdense-0.33.jar
• ejml-core-0.33.jar
• ejml-ddense-0.33.jar
• ejml-dsparse-0.33.jar
• ejml-experimental-0.33.jar
• ejml-fdense-0.33.jar
• ejml-simple-0.33.jar
• ejml-zdense-0.33.jar
• elsa-3.0.0-M7.jar
• esdk-obs-java-3.20.6.1.jar
• flink-cep_2.11-1.10.0.jar
• flink-cep-scala_2.11-1.10.0.jar
• flink-dist_2.11-1.10.0.jar
• flink-python_2.11-1.10.0.jar
• flink-queryable-state-runtime_2.11-1.10.0.jar
• flink-sql-client_2.11-1.10.0.jar
• flink-state-processor-api_2.11-1.10.0.jar
• flink-table_2.11-1.10.0.jar
• flink-table-blink_2.11-1.10.0.jar
• guava-26.0-jre.jar
• hadoop-3.1.1-41.jar
• httpasyncclient-4.1.2.jar
• httpclient-4.5.3.jar
• httpcore-4.4.4.jar
• httpcore-nio-4.4.4.jar
• java-xmlbuilder-1.1.jar
• jna-4.1.0.jar
• libtensorflow-1.12.0.jar
• log4j-over-slf4j-1.7.26.jar
• logback-classic-1.2.3.jar
• logback-core-1.2.3.jar
• nd4j-api-0.9.1.jar
• nd4j-native-0.9.1.jar
• nd4j-native-api-0.9.1.jar
• nd4j-native-platform-0.9.1.jar
• okhttp-3.14.8.jar
• okio-1.14.0.jar
• secComponentApi-1.0.5.jar
• slf4j-api-1.7.26.jar
• tensorflow-1.12.0.jar

Python 3.7.5 依赖包

• absl-py-0.8.1
• astor-0.8.0
• backports.functools_lru_cache-1.5
• backports.weakref-1.0.post1
• blaze-0.10.1
• Click-7.0
• contextlib2-0.6.0.post1
• cycler-0.10.0
• dataclasses-0.6
• datashape-0.5.2
• decorator-4.4.0
• Flask-1.1.1
• Flask_Cors-3.0.8
• Flask-1.16.0
• funcsigs-1.0.2
• gast-0.2.2
• google_pasta-0.1.7
• grpcio-1.24.1
• h5py-2.10.0
• itsdangerous-1.1.0
• Jinja2-2.10.3
• joblib-0.14.0
• Keras-2.3.1
• Keras_Applications-1.0.8
• Keras_Preprocessing-1.1.0
• kiwisolver-1.1.0
• Markdown-3.1.1
• MarkupSafe-1.1.1
• matplotlib-3.1.1
• mock-3.0.5
• multipledispatch-0.6.0
• multipledispatch2-0.4.0
• networkx-2.2
• numpy-1.16.1
• odo-0.5.0
• opt_einsum-3.1.0
• pandas-0.24.1
• pip-19.2.3
• protobuf-3.10.0
• psutil-5.6.3
• pyarrow-0.11.1
• pyparsing-2.4.2
• python-dateutil-2.8.0
• pytz-2019.3
• PyYAML-5.1.2
• scikit-learn-0.21.3
• scipy-1.1.0
• setuptools-41.2.0
• six-1.12.0
• SQLAlchemy-1.3.10
• subprocess32-3.5.4
• tensorboard-1.13.1
• tensorflow-1.13.1
• tensorflow-estimator-1.13.0
• termcolor-1.1.0
• toolz-0.10.0
• Werkzeug-0.16.0
• wheel-0.33.6
• wrapt-1.11.2

AI镜像安装依赖包

AI镜像在原有的Spark镜像基础上，集成了一系列通用的AI相关算法包，将Spark与AI能力相结合，方便用户运行机器学习相关的作业。

• absl-py-0.11.0
• astor-0.8.1
• cached-property-1.5.2
• easydict-1.9
• enum34-1.1.10
• gast-0.4.0
• gpg-1.13.1
• grpcio-1.33.2
• h5py-3.1.0
• importlib-metadata-2.0.0
• joblib-0.17.0
• Keras-Applications-1.0.8
• Keras-Preprocessing-1.1.2
• lxml-4.6.1
• Markdown-3.3.3
• mock-4.0.2
• numpy-1.19.4
• opencv-python-4.4.0.46
• pandas-0.24.2
• pathlib2-2.3.5
• Pillow-8.0.1
• pip-20.1.1
• protobuf-3.13.0
• psutil-5.7.0
• pyarrow-0.11.1
• python-dateutil-2.8.1
• pytz-2020.4
• PyYAML-3.12
• rpm-4.15.1
• scikit-learn-0.22.1
• scipy-1.5.4
• setuptools-47.1.1
• six-1.15.0
• tensorboard-1.13.1
• tensorflow-1.13.1
• tensorflow-estimator-1.13.0
• termcolor-1.1.0
• Werkzeug-1.0.1
• wheel-0.34.2
• zipp-3.4.0