Obtaining Temporary Credentials for Spark Job Agencies

Function

DLI provides a common interface to obtain temporary credentials for Spark job agencies set by users during job launch. The interface encapsulates the obtained temporary credentials for the job agency in the com.huaweicloud.sdk.core.auth.BasicCredentials class.

• Encapsulate the obtained temporary credential for the agency in the return value of getCredentials() of the com.huaweicloud.sdk.core.auth.ICredentialProvider interface.
• The return type is com.huaweicloud.sdk.core.auth.BasicCredentials.
• Only AKs, SKs, and security tokens can be obtained.

Constraints and Limitations

• An agency can be authorized to access temporary credentials only in Spark 3.3.1 (Spark general queue scenario).
  • When creating a Spark job, select version 3.3.1.
  • The information of the agency that allows DLI to access DEW has been configured for the job. spark.dli.job.agency.name indicates the custom agency name.

    For details about how to create a custom agency, see Customizing DLI Agency Permissions.

    Note that double quotes ("") or single quotes ('') are not required when configuring parameters.

• The Spark 3.3.1 basic image has built-in huaweicloud-sdk-core 3.1.62.

Preparing the Environment

The information of the agency that allows DLI to access DEW has been configured for the job.

Dependency package in POM file configurations

<dependency>
    <groupId>com.huaweicloud.sdk</groupId>
    <artifactId>huaweicloud-sdk-core</artifactId>
    <version>3.1.62</version>
    <scope>provided</scope>
</dependency>

Sample Code

This section's Java sample code demonstrates how to obtain BasicCredentials and retrieve a temporary agency's AK, SK, and security token.

• Obtaining job agency credentials for Spark Jar jobs

  import org.apache.spark.sql.SparkSession;
  import org.slf4j.Logger;
  import org.slf4j.LoggerFactory;
  import com.huaweicloud.sdk.core.auth.BasicCredentials;
  import com.huaweicloud.sdk.core.auth.ICredentialProvider;
  import static com.huawei.dli.demo.DLIJobAgencyCredentialUtils.getICredentialProvider;
  public class GetUserCredentialsSparkJar {
  private static final Logger LOG = LoggerFactory.getLogger(GetUserCredentialsSparkJar.class);
  public static void main(String[] args) throws Exception {
            SparkSession spark = SparkSession
                                 .builder()
                                 .appName("test_spark")
                                 .getOrCreate();
    ICredentialProvider credentialProvider = getICredentialProvider();
    BasicCredentials basicCredentials = (BasicCredentials) credentialProvider.getCredentials();
    String ak = basicCredentials.getAk();
    String sk = basicCredentials.getSk();
    String securityToken = basicCredentials.getSecurityToken();
    LOG.info(">>" + " ak " + ak + " sk " + sk.length() + " token " + securityToken.length());
    spark.stop();
   }
  }

• Tool class for obtaining job agencies

  import com.huaweicloud.sdk.core.auth.ICredentialProvider;
  import org.apache.spark.sql.SparkSession;
  import java.util.ArrayList;
  import java.util.List;
  import java.util.ServiceLoader;
  public class DLIJobAgencyCredentialUtils {
  public static ICredentialProvider getICredentialProvider() {
           List<ICredentialProvider> credentialProviders = new ArrayList<>();
           ServiceLoader.load(ICredentialProvider.class, SparkSession.class.getClassLoader())
                         .iterator()
                         .forEachRemaining(credentialProviders::add);
           if (credentialProviders.size() != 1) {
              throw new RuntimeException("Failed to obtain temporary user credential");
           }
           return credentialProviders.get(0);
      }
  }