Built-in Dependency Packages

DLI built-in dependencies are provided by the platform by default. In case of conflicts, you do not need to upload them when packaging JAR packages of user-defined Spark or Flink jobs.

Spark 2.3.2 Dependency Packages

accessors-smart-1.2.jar
activation-1.1.1.jar
aircompressor-0.8.jar
alluxio-2.3.1-hw-luxor-SNAPSHOT-client.jar
antlr-2.7.7.jar
antlr4-runtime-4.8-1.jar
antlr-runtime-3.4.jar
aopalliance-1.0.jar
aopalliance-repackaged-2.4.0-b34.jar
apache-log4j-extras-1.2.17.jar
arpack_combined_all-0.1.jar
arrow-format-0.8.0.jar
arrow-memory-0.8.0.jar
arrow-vector-0.8.0.jar
asm-5.0.4.jar
audience-annotations-0.5.0.jar
automaton-1.11-8.jar
avro-1.7.7.jar
avro-ipc-1.7.7.jar
avro-ipc-1.7.7-tests.jar
avro-mapred-1.7.7-hadoop2.jar
aws-java-sdk-bundle-1.11.271.jar
base64-2.3.8.jar
bcpkix-jdk15on-1.66.jar
bcprov-jdk15on-1.66.jar
bonecp-0.8.0.RELEASE.jar
breeze_2.11-0.13.2.jar
breeze-macros_2.11-0.13.2.jar
calcite-avatica-1.2.0-incubating.jar
calcite-core-1.2.0-incubating.jar
calcite-linq4j-1.2.0-incubating.jar
carbondata-bloom-1.6.1.1032-2.0.0.dli-20201012.110925-6.jar
carbondata-cli-1.6.1.1032-hw-2.0.0.dli-20201012.110947-6.jar
carbondata-common-1.6.1.1032-hw-2.0.0.dli-20201012.110840-6.jar
carbondata-common-plugin-1.6.1.1032-hw-2.0.0.dli-20201012.110945-6.jar
carbondata-core-1.6.1.1032-hw-2.0.0.dli-20201012.110843-6.jar
carbondata-format-1.6.1.1032-hw-2.0.0.dli-20201012.110953-12.jar
carbondata-hadoop-1.6.1.1032-hw-2.0.0.dli-20201012.110849-6.jar
carbondata-hive-1.6.1.1032-hw-2.0.0.dli-20201012.110911-6.jar
carbondata-mv-core-1.6.1.1032-hw-2.0.0.dli-20201012.110936-6.jar
carbondata-mv-plan-1.6.1.1032-hw-2.0.0.dli-20201012.110933-6.jar
carbondata-processing-1.6.1.1032-hw-2.0.0.dli-20201012.110846-6.jar
carbondata-secondary_index-1.6.1.1032-hw-2.0.0.dli-20201012.110942-6.jar
carbondata-spark2-1.6.1.1032-hw-2.0.0.dli-20201012.110857-6.jar
carbondata-spark-common-1.6.1.1032-hw-2.0.0.dli-20201012.110903-6.jar
carbondata-spark-datasource-1.6.1.1032-hw-2.0.0.dli-20201012.110900-6.jar
carbondata-store-sdk-1.6.1.1032-hw-2.0.0.dli-20201012.110852-6.jar
carbondata-streaming-1.6.1.1032-hw-2.0.0.dli-20201012.110922-6.jar
carbondata-vector-1.6.1.1032-hw-2.0.0.dli-20201012.110956-6.jar
checker-qual-2.11.1.jar
chill_2.11-0.8.4.jar
chill-java-0.8.4.jar
commons-beanutils-1.9.4.jar
commons-cli-1.2.jar
commons-codec-2.0-20130428.202122-59.jar
commons-collections-3.2.2.jar
commons-collections4-4.2.jar
commons-compiler-3.0.8.jar
commons-compress-1.4.1.jar
commons-configuration2-2.1.1.jar
commons-crypto-1.0.0-hw-20191105.jar
commons-daemon-1.0.13.jar
commons-dbcp-1.4.jar
commons-dbcp2-2.7.0.jar
commons-httpclient-3.1.jar
commons-io-2.5.jar
commons-lang-2.6.jar
commons-lang3-3.5.jar
commons-logging-1.2.jar
commons-math3-3.4.1.jar
commons-net-2.2.jar
commons-pool-1.5.4.jar
commons-pool2-2.8.0.jar
commons-text-1.3.jar
compress-lzf-1.0.3.jar
core-1.1.2.jar
curator-client-4.2.0.jar
curator-framework-4.2.0.jar
curator-recipes-2.7.1.jar
datanucleus-api-jdo-3.2.6.jar
datanucleus-core-3.2.10.jar
datanucleus-rdbms-3.2.9.jar
derby-10.12.1.1.jar
dnsjava-2.1.7.jar
ehcache-3.3.1.jar
eigenbase-properties-1.1.5.jar
error_prone_annotations-2.3.4.jar
failureaccess-1.0.1.jar
fastutil-8.2.3.jar
ffmpeg-4.3.1-1.5.4.jar
ffmpeg-4.3.1-1.5.4-linux-x86_64.jar
flatbuffers-1.2.0-3f79e055.jar
generex-1.0.2.jar
geronimo-jcache_1.0_spec-1.0-alpha-1.jar
gson-2.2.4.jar
guava-29.0-jre.jar
guice-4.0.jar
guice-servlet-4.0.jar
hadoop-annotations-3.1.1-hw-ei-302002.jar
hadoop-auth-3.1.1-hw-ei-302002.jar
hadoop-aws-3.1.1-hw-ei-302002.jar
hadoop-client-3.1.1-hw-ei-302002.jar
hadoop-common-3.1.1-hw-ei-302002.jar
hadoop-hdfs-3.1.1-hw-ei-302002.jar
hadoop-hdfs-client-3.1.1-hw-ei-302002.jar
hadoop-mapreduce-client-common-3.1.1-hw-ei-302002.jar
hadoop-mapreduce-client-core-3.1.1-hw-ei-302002.jar
hadoop-mapreduce-client-jobclient-3.1.1-hw-ei-302002.jar
hadoop-minikdc-3.1.1-hw-ei-302002.jar
hadoop-yarn-api-3.1.1-hw-ei-302002.jar
hadoop-yarn-client-3.1.1-hw-ei-302002.jar
hadoop-yarn-common-3.1.1-hw-ei-302002.jar
hadoop-yarn-registry-3.1.1-hw-ei-302002.jar
hadoop-yarn-server-common-3.1.1-hw-ei-302002.jar
hadoop-yarn-server-web-proxy-3.1.1-hw-ei-302002.jar
hamcrest-core-1.3.jar
HikariCP-java7-2.4.12.jar
hive-common-1.2.1-hw-2.1.0.dli-20201111.064115-91.jar
hive-exec-1.2.1-hw-2.1.0.dli-20201111.064444-91.jar
hive-metastore-1.2.1-hw-2.1.0.dli-20201111.064230-91.jar
hk2-api-2.4.0-b34.jar
hk2-locator-2.4.0-b34.jar
hk2-utils-2.4.0-b34.jar
hppc-0.7.2.jar
htrace-core4-4.2.0-incubating-hw-1.0.0.jar
httpclient-4.5.4.jar
httpcore-4.4.7.jar
ivy-2.4.0.jar
j2objc-annotations-1.3.jar
jackson-annotations-2.10.0.jar
jackson-core-2.10.0.jar
jackson-core-asl-1.9.13-atlassian-4.jar
jackson-databind-2.10.0.jar
jackson-dataformat-yaml-2.10.0.jar
jackson-datatype-jsr310-2.10.3.jar
jackson-jaxrs-base-2.10.3.jar
jackson-jaxrs-json-provider-2.10.3.jar
jackson-mapper-asl-1.9.13-atlassian-4.jar
jackson-module-jaxb-annotations-2.10.3.jar
jackson-module-paranamer-2.10.0.jar
jackson-module-scala_2.11-2.10.0.jar
jakarta.activation-api-1.2.1.jar
jakarta.xml.bind-api-2.3.2.jar
janino-3.0.8.jar
javacpp-1.5.4.jar
javacpp-1.5.4-linux-x86_64.jar
javacv-1.5.4.jar
JavaEWAH-1.1.7.jar
javassist-3.18.1-GA.jar
javax.annotation-api-1.2.jar
javax.inject-1.jar
javax.inject-2.4.0-b34.jar
javax.servlet-api-3.1.0.jar
javax.ws.rs-api-2.0.1.jar
java-xmlbuilder-1.1.jar
javolution-5.3.1.jar
jaxb-api-2.2.11.jar
jcip-annotations-1.0-1.jar
jcl-over-slf4j-1.7.26.jar
jdo-api-3.0.1.jar
jersey-client-2.23.1.jar
jersey-common-2.23.1.jar
jersey-container-servlet-2.23.1.jar
jersey-container-servlet-core-2.23.1.jar
jersey-guava-2.23.1.jar
jersey-media-jaxb-2.23.1.jar
jersey-server-2.23.1.jar
jets3t-0.9.4.jar
jetty-http-9.4.31.v20200723.jar
jetty-io-9.4.31.v20200723.jar
jetty-security-9.4.31.v20200723.jar
jetty-server-9.4.31.v20200723.jar
jetty-servlet-9.4.31.v20200723.jar
jetty-util-9.4.31.v20200723.jar
jetty-util-ajax-9.4.31.v20200723.jar
jetty-webapp-9.4.31.v20200723.jar
jetty-xml-9.4.31.v20200723.jar
joda-time-2.9.3.jar
jodd-core-4.2.0.jar
json-20200518.jar
json4s-ast_2.11-3.2.11.jar
json4s-core_2.11-3.2.11.jar
json4s-jackson_2.11-3.2.11.jar
json-sanitizer-1.2.1.jar
json-smart-2.3.jar
jsp-api-2.1.jar
jsr305-3.0.2.jar
jta-1.1.jar
jtransforms-2.4.0.jar
jul-to-slf4j-1.7.26.jar
junit-4.11.jar
kerb-admin-1.0.1.jar
kerb-client-1.0.1.jar
kerb-common-1.0.1.jar
kerb-core-1.0.1.jar
kerb-crypto-1.0.1.jar
kerb-identity-1.0.1.jar
kerb-server-1.0.1.jar
kerb-simplekdc-1.0.1.jar
kerb-util-1.0.1.jar
kerby-asn1-1.0.1.jar
kerby-config-1.0.1.jar
kerby-pkix-1.0.1.jar
kerby-util-1.0.1.jar
kerby-xdr-1.0.1.jar
kryo-shaded-3.0.3.jar
kubernetes-client-4.9.2-hw-20200804.jar
kubernetes-model-4.9.2-hw-20200804.jar
kubernetes-model-common-4.9.2-hw-20200804.jar
leveldbjni-all-1.8-hw-20191105.jar
libfb303-0.9.3.jar
libthrift-0.12.0.jar
listenablefuture-9999.0-empty-to-avoid-conflict-with-guava.jar
log4j-1.2.17-cloudera1.jar
log4j-rolling-appender-20131024-2017.jar
logging-interceptor-3.14.4.jar
luxor-encrypt-2.1.0-20201106.065437-53.jar
luxor-fs3-2.1.0-20201106.065612-53.jar
luxor-obs-fs3-2.1.0-20201106.065616-53.jar
luxor-rpc_2.11-2.1.0-20201106.065541-53.jar
luxor-rpc-protobuf2-2.1.0-20201106.065551-53.jar
lz4-java-1.7.1.jar
machinist_2.11-0.6.1.jar
macro-compat_2.11-1.1.1.jar
metrics-core-3.1.5.jar
metrics-graphite-3.1.5.jar
metrics-jmx-4.1.12.1.jar
metrics-json-3.1.5.jar
metrics-jvm-3.1.5.jar
minlog-1.3.0.jar
mssql-jdbc-6.2.1.jre7.jar
netty-3.10.6.Final.jar
netty-all-4.1.51.Final.jar
nimbus-jose-jwt-8.19.jar
objenesis-2.1.jar
okhttp-3.14.4.jar
okio-1.17.2.jar
opencsv-2.3.jar
opencsv-4.6.jar
opencv-4.3.0-2.jar
orc-core-1.4.4-nohive.jar
orc-mapreduce-1.4.4-nohive.jar
oro-2.0.8.jar
osgi-resource-locator-1.0.1.jar
paranamer-2.8.jar
parquet-column-1.8.3.jar
parquet-common-1.8.3.jar
parquet-encoding-1.8.3.jar
parquet-format-2.3.1.jar
parquet-hadoop-1.8.3.jar
parquet-hadoop-bundle-1.6.0.jar
parquet-jackson-1.8.3.jar
parquet-format-2.3.1.jar
parquet-hadoop-1.8.3.jar
parquet-hadoop-bundle-1.6.0.jar
parquet-jackson-1.8.3.jar
postgresql-42.2.14.jar
protobuf-java-2.5.0.jar
py4j-0.10.7.jar
pyrolite-4.13.jar
re2j-1.1.jar
RoaringBitmap-0.5.11.jar
scala-compiler-2.11.12.jar
scala-library-2.11.12.jar
scalap-2.11.0.jar
scala-parser-combinators_2.11-1.1.0.jar
scala-reflect-2.11.12.jar
scala-xml_2.11-1.0.5.jar
secComponentApi-1.0.5c.jar
shapeless_2.11-2.3.2.jar
slf4j-api-1.7.30.jar
slf4j-log4j12-1.7.30.jar
snakeyaml-1.24.jar
snappy-java-1.1.7.5.jar
spark-catalyst_2.11-2.3.2.0101-hw-2.1.0.dli-20201111.073826-143.jar
spark-core_2.11-2.3.2.0101-hw-2.1.0.dli-20201111.073836-134.jar
spark-graphx_2.11-2.3.2.0101-hw-2.1.0.dli-20201111.073847-129.jar
spark-hive_2.11-2.3.2.0101-hw-2.1.0.dli-20201111.073854-132.jar
spark-kubernetes_2.11-2.3.2.0101-hw-2.1.0.dli-20201111.073916-85.jar
spark-kvstore_2.11-2.3.2.0101-hw-2.1.0.dli-20201111.073933-127.jar
spark-launcher_2.11-2.3.2.0101-hw-2.1.0.dli-20201111.073940-127.jar
spark-mllib_2.11-2.3.2.0101-hw-2.1.0.dli-20201111.073946-127.jar
spark-mllib-local_2.11-2.3.2.0101-hw-2.1.0.dli-20201111.073953-127.jar
spark-network-common_2.11-2.3.2.0101-hw-2.1.0.dli-20201111.073959-127.jar
spark-network-shuffle_2.11-2.3.2.0101-hw-2.1.0.dli-20201111.074007-127.jar
spark-om_2.11-2.3.2.0101-hw-2.1.0.dli-20201111.074019-125.jar
spark-repl_2.11-2.3.2.0101-hw-2.1.0.dli-20201111.074028-125.jar
spark-sketch_2.11-2.3.2.0101-hw-2.1.0.dli-20201111.074035-125.jar
spark-sql_2.11-2.3.2.0101-hw-2.1.0.dli-20201111.074041-126.jar
spark-streaming_2.11-2.3.2.0101-hw-2.1.0.dli-20201111.074100-123.jar
spark-tags_2.11-2.3.2.0101-hw-2.1.0.dli-20201111.074136-123.jar
spark-tags_2.11-2.3.2.0101-hw-2.1.0.dli-20201111.074141-124-tests.jar
spark-unsafe_2.11-2.3.2.0101-hw-2.1.0.dli-20201111.074144-123.jar
spark-uquery_2.11-2.3.2.0101-hw-2.1.0.dli-20201111.074906-210.jar
spark-yarn_2.11-2.3.2.0101-hw-2.1.0.dli-20201111.074151-123.jar
spire_2.11-0.13.0.jar
spire-macros_2.11-0.13.0.jar
ST4-4.3.1.jar
stax2-api-3.1.4.jar
stax-api-1.0-2.jar
stream-2.7.0.jar
stringtemplate-3.2.1.jar
token-provider-1.0.1.jar
univocity-parsers-2.5.9.jar
validation-api-1.1.0.Final.jar
woodstox-core-5.0.3.jar
xbean-asm5-shaded-4.4.jar
xercesImpl-2.12.0.jar
xml-apis-1.4.01.jar
xz-1.0.jar
zjsonpatch-0.3.0.jar
zookeeper-3.5.6-hw-ei-302002.jar
zookeeper-jute-3.5.6-hw-ei-302002.jar
zstd-jni-1.4.4-11.jar

Flink 1.7.2 Dependency Packages

bcpkix-jdk15on-1.60.jar
bcprov-jdk15on-1.60.jar
commons-codec-1.9.jar
commons-configuration-1.7.jar
deeplearning4j-core-0.9.1.jar
deeplearning4j-nlp-0.9.1.jar
deeplearning4j-nn-0.9.1.jar
ejml-cdense-0.33.jar
ejml-core-0.33.jar
ejml-ddense-0.33.jar
ejml-dsparse-0.33.jar
ejml-experimental-0.33.jar
ejml-fdense-0.33.jar
ejml-simple-0.33.jar
ejml-zdense-0.33.jar
elsa-3.0.0-M7.jar
esdk-obs-java-3.1.3.jar
flink-cep_2.11-1.7.0.jar
flink-cep-scala_2.11-1.7.0.jar
flink-dist_2.11-1.7.0.jar
flink-gelly_2.11-1.7.0.jar
flink-gelly-scala_2.11-1.7.0.jar
flink-ml_2.11-1.7.0.jar
flink-python_2.11-1.7.0.jar
flink-queryable-state-runtime_2.11-1.7.0.jar
flink-shaded-curator-1.7.0.jar
flink-shaded-hadoop2-uber-1.7.0.jar
flink-table_2.11-1.7.0.jar
guava-26.0-jre.jar
hadoop-cloud-3.1.1-41-20201014.085840-4.jar
httpasyncclient-4.1.2.jar
httpclient-4.5.12.jar
httpcore-4.4.4.jar
httpcore-nio-4.4.4.jar
java-xmlbuilder-1.1.jar
jna-4.1.0.jar
libtensorflow-1.12.0.jar
log4j-api-2.8.2.jar
log4j-core-2.8.2.jar
log4j-over-slf4j-1.7.21.jar
logback-classic-1.2.3.jar
logback-core-1.2.3.jar
nd4j-api-0.9.1.jar
nd4j-native-0.9.1.jar
nd4j-native-api-0.9.1.jar
nd4j-native-platform-0.9.1.jar
okhttp-3.14.8.jar
okio-1.14.0.jar
slf4j-api-1.7.21.jar
tensorflow-1.12.0.jar

Python 3.7.5 Dependency Packages

absl-py-0.8.1
astor-0.8.0
backports.functools_lru_cache-1.5
backports.weakref-1.0.post1
blaze-0.10.1
Click-7.0
contextlib2-0.6.0.post1
cycler-0.10.0
dataclasses-0.6
datashape-0.5.2
decorator-4.4.0
Flask-1.1.1
Flask_Cors-3.0.8
Flask-1.16.0
funcsigs-1.0.2
gast-0.2.2
google_pasta-0.1.7
grpcio-1.24.1
h5py-2.10.0
itsdangerous-1.1.0
Jinja2-2.10.3
joblib-0.14.0
Keras-2.3.1
Keras_Applications-1.0.8
Keras_Preprocessing-1.1.0
kiwisolver-1.1.0
Markdown-3.1.1
MarkupSafe-1.1.1
matplotlib-3.1.1
mock-3.0.5
multipledispatch-0.6.0
multipledispatch2-0.4.0
networkx-2.2
numpy-1.16.1
odo-0.5.0
opt_einsum-3.1.0
pandas-0.24.1
pip-19.2.3
protobuf-3.10.0
psutil-5.6.3
pyarrow-0.11.1
pyparsing-2.4.2
python-dateutil-2.8.0
pytz-2019.3
PyYAML-5.1.2
scikit-learn-0.21.3
scipy-1.1.0
setuptools-41.2.0
six-1.12.0
SQLAlchemy-1.3.10
subprocess32-3.5.4
tensorboard-1.13.1
tensorflow-1.13.1
tensorflow-estimator-1.13.0
termcolor-1.1.0
toolz-0.10.0
Werkzeug-0.16.0
wheel-0.33.6
wrapt-1.11.2

Dependency Packages for AI Image Installation

DLI uses AI images to integrate a series of common AI algorithm packages based on the original Spark images. Spark and AI capabilities can be combined, helping you run machine learning jobs.                 &