Obtaining Temporary Credentials from a Spark Job's Agency for Accessing Other Cloud Services

Function

DLI provides a common interface to obtain temporary credentials from Spark job's agencies set by users during job launch. The interface encapsulates temporary credentials obtained from the job's agency in the com.huaweicloud.sdk.core.auth.BasicCredentials class.

• Encapsulate temporary credentials obtained from the agency in the return value of getCredentials() of the com.huaweicloud.sdk.core.auth.ICredentialProvider interface.
• The return type is com.huaweicloud.sdk.core.auth.BasicCredentials.
• Only AKs, SKs, and security tokens can be obtained.
• After obtaining the AK, SK, and security token, query temporary credentials by referring to Using CSMS to Change Hard-coded Database Account Passwords.

Notes and Constraints

• An agency can be authorized to access temporary credentials only in Spark 3.3.1 (Spark general queue scenario).
  • When creating a Spark job, select version 3.3.1.
  • The information of the agency that allows DLI to access DEW has been configured for the job. spark.dli.job.agency.name indicates the custom agency name.

    For details about how to create a custom agency, see Customizing DLI Agency Permissions.

    Note that double quotes ("") or single quotes ('') are not required when configuring parameters.

• The Spark 3.3.1 basic image has built-in huaweicloud-sdk-core 3.1.62.

Preparing the Environment

The information of the agency that allows DLI to access DEW has been configured for the job.

Dependency package in POM file configurations

<dependency>
    <groupId>com.huaweicloud.sdk</groupId>
    <artifactId>huaweicloud-sdk-core</artifactId>
    <version>3.1.62</version>
    <scope>provided</scope>
</dependency>

Sample Code

This section's Java sample code demonstrates how to obtain BasicCredentials and retrieve a temporary agency's AK, SK, and security token.

• Obtaining job agency credentials for Spark Jar jobs

  import org.apache.spark.sql.SparkSession;
  import org.slf4j.Logger;
  import org.slf4j.LoggerFactory;
  import com.huaweicloud.sdk.core.auth.BasicCredentials;
  import com.huaweicloud.sdk.core.auth.ICredentialProvider;
  import static com.huawei.dli.demo.DLIJobAgencyCredentialUtils.getICredentialProvider;
  public class GetUserCredentialsSparkJar {
  private static final Logger LOG = LoggerFactory.getLogger(GetUserCredentialsSparkJar.class);
  public static void main(String[] args) throws Exception {
            SparkSession spark = SparkSession
                                 .builder()
                                 .appName("test_spark")
                                 .getOrCreate();
    ICredentialProvider credentialProvider = getICredentialProvider();
    BasicCredentials basicCredentials = (BasicCredentials) credentialProvider.getCredentials();
    String ak = basicCredentials.getAk();
    String sk = basicCredentials.getSk();
    String securityToken = basicCredentials.getSecurityToken();
    LOG.info(">>" + " ak " + ak + " sk " + sk.length() + " token " + securityToken.length());
    spark.stop();
   }
  }

• Tool class for obtaining job agencies

  import com.huaweicloud.sdk.core.auth.ICredentialProvider;
  import org.apache.spark.sql.SparkSession;
  import java.util.ArrayList;
  import java.util.List;
  import java.util.ServiceLoader;
  public class DLIJobAgencyCredentialUtils {
  public static ICredentialProvider getICredentialProvider() {
           List<ICredentialProvider> credentialProviders = new ArrayList<>();
           ServiceLoader.load(ICredentialProvider.class, SparkSession.class.getClassLoader())
                         .iterator()
                         .forEachRemaining(credentialProviders::add);
           if (credentialProviders.size() != 1) {
              throw new RuntimeException("Failed to obtain temporary user credential");
           }
           return credentialProviders.get(0);
      }
  }