Help Center/ Data Lake Insight/ Service Overview/ Security/ Data Protection Technologies
Updated on 2025-01-09 GMT+08:00
View PDF
Share

Data Protection Technologies

Data Storage Security

To prevent your personal data (such as username, password, and mobile number) from being obtained by unauthorized or unauthenticated entities or individuals, DLI encrypts your data during storage and transmission to prevent data leakage.

Data Destruction Mechanism

When you delete a DLI cluster, your personal data stored in the cluster will also be deleted.

When you delete your mobile number and email address on the console and disable message notifications, the mobile number and email address will also be deleted from the database.

Data Transmission Security

Your personal data is encrypted using TLS 1.2 or TLS 1.3 during transmission. All the calls made to DLI APIs use HTTPS to encrypt the transmitted data.

Communication Encryption During Spark Job Transmission

For Spark jobs, you can enable communication encryption by setting the parameters listed in Table 1.

Make sure that you have uploaded the keys and certificates to the specified OBS path and included them in the other dependencies in the job configuration.

Table 1 Parameters used to configure communication encryption for Spark job transmission

Parameter

Description

Example Value

spark.network.crypto.enabled

This parameter lets you enable or disable communication encryption for data transmission between nodes. When set to true, Spark encrypts all communications between Executors and Drivers and between Executors. This configuration is critical to ensure data transmission security.

true

spark.network.sasl.serverAlwaysEncrypt

This parameter is used to configure whether the server uses encryption to communicate with clients. When set to true, the server requires all clients to use encrypted connections, which improves communication security.

true

spark.authenticate

This parameter is used to configure whether to authenticate the accesses to the components of a Spark application. Enabling authentication blocks unauthorized accesses. Setting to true means enabling authentication.

true

Communication Encryption During Flink Job Transmission

For Flink jobs, you can enable SSL transmission by setting the parameters listed in Table 2.

  • Enabling SSL for the data transmission streams between Task Managers significantly affects system performance. Thoroughly evaluate both security and performance before you enable it.
  • Configure certificate file information in Other Dependencies of the job editing page.
  • The default path for storing dependency files is /opt/flink/usrlib/userData/ in OBS.
  • Make sure that you have uploaded the keys and certificates to the specified OBS path and included them in the other dependencies in the job configuration.
Table 2 Parameters used to configure communication encryption for Flink job transmission

Parameter

Description

Mandatory

Example Value

security.ssl.enabled

Whether to enable SSL.

Yes

true

akka.ssl.enabled

Whether to enable Akka SSL.

No

true

blob.service.ssl.enabled

Whether to enable SSL for the Blob channel.

No

true

taskmanager.data.ssl.enable

Whether to enable SSL for communications between TaskManagers.

No

true

security.ssl.algorithms

Whether to configure SSL encryption algorithms.

No

TLS_DHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_DHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384

security.ssl.keystore

Path used to store keystore files. flink.keystore indicates the keystore file generated by generate_keystore.sh*.

Yes

/opt/flink/usrlib/userData/flink.keystore

security.ssl.keystore-password

Password for the keystore file, which can be customized

Yes

-

security.ssl.key-password

Password for the SSL key, which can be customized

Yes

-

security.ssl.truststore

Path used to store truststore files. flink.truststore indicates the truststore file generated by generate_keystore.sh*.

Yes

/opt/flink/usrlib/userData/flink.truststore

security.ssl.truststore-password

Password for the truststore file, which can be customized

Yes

-

security.ssl.rest.enabled

Whether to enable SSL/TLS encryption for REST APIs

Yes

false

security.ssl.verify-hostname

Controls whether to check if the host name (hostname) of the peer matches the information in the certificate when establishing an SSL/TLS connection.

No

false

security.ssl.protocol

Protocol version used when SSL/TLS connections are specified

No

TLS 1.2, TLS 1.3

security.ssl.encrypt.enabled

Whether to enable data encryption for communications within a Flink cluster and between Flink and other components

No

false

Example of enabling communication encryption for Flink job transmission:

security.ssl.enabled: true
security.ssl.encrypt.enabled: false
security.ssl.key-password: ***
security.ssl.keystore-password: Admin12!
security.ssl.keystore:  /opt/flink/usrlib/userData/*.keystore
security.ssl.protocol: TLSv1.2
security.ssl.rest.enabled: false
security.ssl.truststore-password: ***
security.ssl.truststore:  /opt/flink/usrlib/userData/*.truststore
security.ssl.verify-hostname: false
Parent topic: Security