Obtaining Temporary Credentials from a Spark Job's Agency for Accessing Other Cloud Services

Function

DLI provides a common interface to obtain temporary credentials from Spark job's agencies set by users during job launch. The interface encapsulates temporary credentials obtained from the job's agency in the com.huaweicloud.sdk.core.auth.BasicCredentials class.

• Encapsulate temporary credentials obtained from the agency in the return value of getCredentials() of the com.huaweicloud.sdk.core.auth.ICredentialProvider interface.
• The return type is com.huaweicloud.sdk.core.auth.BasicCredentials.
• Only AKs, SKs, and security tokens can be obtained.

Notes and Constraints

• An agency can be authorized to access temporary credentials only in Spark 3.3.1 (Spark general queue scenario).
  • When creating a Spark job, select version 3.3.1.
  • The information of the agency that allows DLI to access DEW has been configured for the job. spark.dli.job.agency.name indicates the custom agency name.

    Note that double quotes ("") or single quotes ('') are not required when configuring parameters.

• The Spark 3.3.1 basic image has built-in huaweicloud-sdk-core 3.1.62.

Preparing the Environment

The information of the agency that allows DLI to access DEW has been configured for the job.

Dependency package in POM file configurations

<dependency>
    <groupId>com.huaweicloud.sdk</groupId>
    <artifactId>huaweicloud-sdk-core</artifactId>
    <version>3.1.62</version>
    <scope>provided</scope>
</dependency>

Sample Code

This section's Java sample code demonstrates how to obtain BasicCredentials and retrieve a temporary agency's AK, SK, and security token.

• Obtaining job agency credentials for Spark Jar jobs

  import org.apache.spark.sql.SparkSession;
  import org.slf4j.Logger;
  import org.slf4j.LoggerFactory;
  import com.huaweicloud.sdk.core.auth.BasicCredentials;
  import com.huaweicloud.sdk.core.auth.ICredentialProvider;
  import static com.huawei.dli.demo.DLIJobAgencyCredentialUtils.getICredentialProvider;
  public class GetUserCredentialsSparkJar {
  private static final Logger LOG = LoggerFactory.getLogger(GetUserCredentialsSparkJar.class);
  public static void main(String[] args) throws Exception {
            SparkSession spark = SparkSession
                                 .builder()
                                 .appName("test_spark")
                                 .getOrCreate();
    ICredentialProvider credentialProvider = getICredentialProvider();
    BasicCredentials basicCredentials = (BasicCredentials) credentialProvider.getCredentials();
    String ak = basicCredentials.getAk();
    String sk = basicCredentials.getSk();
    String securityToken = basicCredentials.getSecurityToken();
    LOG.info(">>" + " ak " + ak + " sk " + sk.length() + " token " + securityToken.length());
    spark.stop();
   }
  }

• Tool class for obtaining job agencies

  import com.huaweicloud.sdk.core.auth.ICredentialProvider;
  import org.apache.spark.sql.SparkSession;
  import java.util.ArrayList;
  import java.util.List;
  import java.util.ServiceLoader;
  public class DLIJobAgencyCredentialUtils {
  public static ICredentialProvider getICredentialProvider() {
           List<ICredentialProvider> credentialProviders = new ArrayList<>();
           ServiceLoader.load(ICredentialProvider.class, SparkSession.class.getClassLoader())
                         .iterator()
                         .forEachRemaining(credentialProviders::add);
           if (credentialProviders.size() != 1) {
              throw new RuntimeException("Failed to obtain temporary user credential");
           }
           return credentialProviders.get(0);
      }
  }