Mounting Object Storage Across Accounts

Application Scenarios

• Cross-account data sharing. For example, multiple teams within a company need to share data, but each team uses a different account.
• Cross-account data migration and backup. For example, if account A is about to be disabled, all data stored in the account will need to be migrated to a new account (account B).
• Data processing and analysis. For example, account B is an external data processor and needs to access raw data from account A to perform tasks such as big data analysis and machine learning.

By linking object storage across accounts, you can share data, lower storage and transmission expenses, and guarantee data security and consistency. This enables various teams or organizations to securely and conveniently access each other's data, eliminating the need for repeated storage and redundant transmission. Additionally, data is kept current and compliant, enhancing overall service efficiency and security.

Procedure

Assume that account B needs to access and use an OBS bucket of account A. For details, see Figure 1 and Table 1.

Figure 1 Mounting an OBS bucket across accounts

Prerequisites

• The involved accounts are in the same region.
• You have created a cluster where the CCE Container Storage (Everest) add-on is installed. The add-on version must be 1.1.11 or later, and the cluster version must be v1.15 or later. If no cluster is available, create one by referring to Buying a CCE Standard/Turbo Cluster.
• An ECS with an EIP bound has been created in the same VPC as the cluster, and the ECS can access the cluster using kubectl. For details about how to access a cluster from an ECS, see Accessing a Cluster Using kubectl.

Step 1: Create an OBS Bucket Policy and ACL

Configure an OBS bucket policy and ACL using account A and grant account B the required permissions like the read and write permissions.

1. Log in to the OBS console. In the navigation pane, choose Object Storage.
2. Click the name of the target bucket to go to the Objects page.
3. In the navigation pane, choose Permissions > Bucket Policies. On the page displayed, click Create.
4. Configure the parameters. In this example, only some mandatory parameters are described. You can keep the default values for other parameters. For details about the parameters, see Bucket Policy Overview.

   Table 2 Bucket policy parameters

   Parameter	Description	Example Value
   Policy Name	Enter a name.	example01
   Effect	Specify the behavior of a policy. Allow: The actions defined in the policy are allowed. Deny: The actions defined in the policy are denied.	Allow
   Principal	Specify authorized accounts. Multiple accounts can be selected. For different types of authorized accounts, the OBS console provides different templates for authorizations. For details, see Creating a Bucket Policy with a Template. All accounts: Any account can execute the current bucket policy without identity authentication, which may pose data security risks. Current account: Grant permissions to a specific IAM user under the current account. Other accounts: Grant permissions to a specific IAM user under another account.	Other accounts XXX(account ID)/XXX (IAM ID)
   Resources	Specify the authorized resources. Entire bucket (including the objects in it): Allow authorized accounts to perform certain actions on a bucket and the objects in it. Current bucket: Allow authorized accounts to perform certain actions on the current bucket. Specific objects: Allow authorized accounts to perform certain actions on the specified objects in a bucket.	Entire bucket (including the objects in it)
   Actions	Specify actions. Use a template: Use a permission template preset on the OBS console. If you selected Bucket Read/Write, the Specified actions option will be selected by default in the Advanced Settings area. Customize: Customize the actions.	Use a template > Bucket Read/Write

   Figure 2 Creating a bucket policy
   

5. In the navigation pane, choose Permissions > Bucket ACL. In the right pane, click Add under User Access. Enter the account ID of the authorized user, select Read and Write for Access to Bucket, select Object read for Access to Objects, select Read and Write for Access to ACL, and click OK.

Step 2: Create a Workload with an OBS Volume Mounted

Create a PV and a PVC based on the OBS bucket of account A using account B and mount the PVC to the required workload.

1. Create a ConfigMap named paas-obs-endpoint and configure the region and endpoint of OBS.

   vim config.yaml

   The content is shown below. For details about the parameters, see Table 3.

   apiVersion: v1
   kind: ConfigMap
   metadata:
     name: paas-obs-endpoint   # The value must be paas-obs-endpoint.
     namespace: kube-system    # The value must be kube-system.
   data:
     obs-endpoint: |
       {"<region_name>": "<endpoint_address>"}

   Create the ConfigMap using config.yaml.

   kubectl create -f config.yaml

   Table 3 ConfigMap parameters

   ConfigMap	Description	Example Value
   metadata.name	ConfigMap name, which is fixed at paas-obs-endpoint and cannot be changed	paas-obs-endpoint
   metadata.namespace	Namespace, which is fixed at kube-system and cannot be changed	kube-system
   data.obs-endpoint	Region names and endpoints are in key-value pairs. Replace <region_name> and <endpoint_address> with specific values. If multiple values are needed, use commas (,) to separate them. For details about its value, see Regions and Endpoints.	{"ap-southeast-1": "https://obs.ap-southeast-1.myhuaweicloud.com:443", "ap-southeast-3": "https://obs.ap-southeast-3.myhuaweicloud.com:443"}

2. Create a secret named test-user. This secret is used to provide access credentials when volumes are mounted to CSI, and its name can be customized.
   1. Obtain the AK. Go back to the management console, hover the cursor over the username in the upper right corner and choose My Credentials from the drop-down list.

      In the navigation pane, choose Access Keys. On the page displayed, click Create Access Key.

      Click OK and download the AK.

   2. Encode the AK using Base64 and save the encoded AK and SK. If the AK obtained is xxx and the SK is yyy, run the following commands:

      echo -n xxx|base64
      echo -n yyy|base64

   3. Create a secret YAML file, for example, test_user.yaml.

      vim test_user.yaml

      The content is shown below. For details about the parameters, see Table 4.

      apiVersion: v1
      data:
        access.key: QUxPQUlJU******
        secret.key: aVMwZkduQ******
      kind: Secret
      metadata:
        name: test-user
        namespace: default
      type: cfe/secure-opaque

      Create a secret using test_user.yaml.

      kubectl create -f test_user.yaml

      Table 4 Secret parameters

      Parameter	Description	Example Value
      access.key	A Base64-encoded AK	QUxPQUlJU******
      secret.key	A Base64-encoded SK	aVMwZkduQ******
      type	Key type, which is fixed at cfe/secure-opaque and cannot be changed When this type is used, the data entered by users will be automatically encrypted.	cfe/secure-opaque

3. Create a PV named testing_abc and mount the secret named test_user to the PV.

   vim testing_abc.yaml

   The content is shown below. For details about the parameters, see Table 5.

   kind: PersistentVolume
   apiVersion: v1
   metadata:
     name: testing-abc
     annotations:
       pv.kubernetes.io/bound-by-controller: 'yes'
       pv.kubernetes.io/provisioned-by: everest-csi-provisioner
   spec:
     capacity:
       storage: 1Gi
     mountOptions:
     - default_acl=bucket-owner-full-control      #New OBS mounting parameters
     csi:
       driver: obs.csi.everest.io
       volumeHandle: obs-cce-test                 # Name of the OBS bucket to be mounted
       fsType: s3fs                               # obsfs indicates a parallel file system, and s3fs indicates an OBS bucket.
       volumeAttributes:
         everest.io/obs-volume-type: STANDARD     # Bucket type, which can be STANDARD or WARM when an OBS bucket is used
         everest.io/region: <region_name>         # Region where the OBS bucket is located (Replace it with the actual value.)
         storage.kubernetes.io/csiProvisionerIdentity: everest-csi-provisioner
       nodePublishSecretRef:                      # AK/SK used for mounting the OBS bucket
         name: test-user
         namespace: default
     accessModes:
       - ReadWriteMany
     persistentVolumeReclaimPolicy: Retain        # PV reclaim policy
     storageClassName: csi-obs                    # csi-obs specifies an OBS storage class that is automatically created. You can customize it as required.
     volumeMode: Filesystem

   Create the PV using testing_abc.yaml.

   kubectl create -f testing_abc.yaml

   Table 5 PV parameters

   Parameter	Description	Example Value
   mountOptions.default_acl	Specify access control policies for a bucket and objects in the bucket. In this example, account A owns the bucket, and both account A and account B have the ability to upload data to it. private: The bucket or objects can only be fully accessed by the owner of the bucket. public-read: The owner of the bucket has complete control over both the bucket and its objects. While other users can read data from the bucket, they are unable to modify, delete, or upload any data within it. public-read-write: The owner of the bucket has complete control over the bucket and its objects. Other users can read and write data from and to the bucket. bucket-owner-read: Users who have uploaded objects to the bucket have complete control over the objects, while the bucket owner has only been granted the read permissions for said objects. This mode is usually used in cross-account sharing scenarios. bucket-owner-full-control: Users who have uploaded objects to the bucket are granted write permissions for those specific objects, but not read permissions by default. The bucket owner has complete control over all objects in the bucket. This mode is usually used in cross-account sharing scenarios.	bucket-owner-full-control NOTE: Because account A's bucket policy grants account B read and write permissions to the entire bucket, account B has both read and write permissions for all objects in that bucket, including objects it uploads itself and those uploaded by account A.
   csi.nodePublishSecretRef	Specify the secret to be mounted. name: name of a secret namespace: namespace where the secret is in	test-user default
   csi.volumeHandle	Specify the name of the OBS bucket to be mounted.	obs-cce-test (OBS bucket name authorized by account A)
   csi.fsType	Specify the file type. obsfs: Create an OBS parallel file system. s3fs: Create an OBS bucket.	s3fs NOTICE: The file type specified by the PV and the PVC must match in order for the PV to be bound to the corresponding PVC. If they do not match, the binding cannot occur.
   accessModes	Specify the access mode of the storage volume. OBS supports only ReadWriteMany. ReadWriteOnce: A storage volume can be mounted to a single node in read-write mode. ReadWriteMany: A storage volume can be mounted to multiple nodes in read-write mode.	ReadWriteMany
   persistentVolumeReclaimPolicy	Specify the reclaim policy for the PV. Delete: When a PVC is deleted, the PV and underlying storage resource will be deleted. If the everest.io/reclaim-policy: retain-volume-only annotation is added to the YAML file, the underlying storage resources will be retained. Retain: When a PVC is deleted, both the PV and underlying storage resources will be retained. You need to manually delete them. After the PVC is deleted, the PV is in the Released state and cannot be bound to a PVC again.	Retain NOTE: If multiple PVs use the same OBS volume, use Retain to prevent the underlying volume from being deleted with one of the PVs.

4. Create a PVC named pvc-test-abc and bind the new PV testing_abc to it.

   vim pvc_test_abc.yaml

   The file content is as follows:

   apiVersion: v1
   kind: PersistentVolumeClaim
   metadata:
     name: pvc-test-abc
     namespace: default
     annotations:
       csi.storage.k8s.io/node-publish-secret-name: test-user      # Mount a secret.
       csi.storage.k8s.io/node-publish-secret-namespace: default   # Namespace of the secret
         everest.io/obs-volume-type: STANDARD     # Bucket type, which can be STANDARD or WARM when an OBS bucket is used
       csi.storage.k8s.io/fstype: s3fs            # File type. obsfs indicates a parallel file system, and s3fs indicates an OBS bucket.
       volume.beta.kubernetes.io/storage-provisioner: everest-csi-provisioner
   spec:
     accessModes:
     - ReadWriteMany             # The value must be ReadWriteMany for object storage.
     resources:
       requests:
         storage: 1Gi            # Storage capacity of a PVC. This parameter is valid only for verification (fixed to 1, cannot be empty or 0). The value setting does not take effect for OBS buckets.
     storageClassName: csi-obs   # csi-obs specifies an OBS storage class that is automatically created. You can customize it as required.
     volumeName: testing-abc     # PV name

   Create the PVC using pvc_test_abc.yaml.

   kubectl create -f pvc_test_abc.yaml

5. Create a workload and mount the PVC to it. An Nginx Deployment is used as an example.

   vim obs_deployment_example.yaml

   The file content is as follows:

   apiVersion: apps/v1 
   kind: Deployment 
   metadata: 
     name: obs-deployment-example                  # Workload name, which can be customized
     namespace: default 
   spec: 
     replicas: 1 
     selector: 
       matchLabels: 
         app: obs-deployment-example              # Label, which can be customized
     template: 
       metadata: 
         labels: 
           app: obs-deployment-example 
       spec: 
         containers: 
         - image: nginx
           name: container-0 
           volumeMounts: 
           - mountPath: /tmp                       # PVC mount path, which can be customized as required
             name: pvc-obs-example 
         restartPolicy: Always
         imagePullSecrets:
           - name: default-secret
         volumes: 
         - name: pvc-obs-example  
           persistentVolumeClaim: 
             claimName: pvc-test-abc               # PVC name

   Create the workload named obs-deployment-example using obs_deployment_example.yaml.

   kubectl create -f obs_deployment_example.yaml

   Check whether the workload has been created.

   kubectl get pod

   If information similar to the following is displayed and the workload is in the Running state, the workload has been created.

   NAME                                      READY   STATUS              RESTARTS        AGE
   obs-deployment-example-6b4dfd7b57-frfxv   1/1     Running             0               22h

Step 3: Check the Pod Actions on the OBS Bucket

Check whether the pod created by account B has the required permissions based on the bucket policy.

1. Check whether the pod can read and write objects in the OBS bucket created by account A and assume that a test.txt file is present in the OBS bucket.
   Access the created workload. You can press Ctrl+D to exit the current workload.

   kubectl -n default exec -it obs-deployment-example-6b4dfd7b57-frfxv -c container-0 -- bash

   Check the pod actions on test.txt. /tmp specifies the PVC mount path.

   ls -l /tmp/test.txt

   If information similar to the following is displayed, the pod has the read and write permissions on the test.txt file, which is related to the bucket policy set by account A.

   -rwxrwxrwx 1 root root 4 Sep  5 09:09 /tmp/test.txt

2. Check whether the pod can read and write data from and to the objects uploaded by itself in the OBS bucket.
   Create a test01.txt file in /tmp and write test\n into the file.

   echo -e "test\n" > /tmp/test01.txt

   Check the test01.txt content and check whether the pod can read and write new objects uploaded by itself. Account A can check the new objects in the OBS bucket.

   cat /tmp/test01.txt

   If information similar to the following is displayed, the pod has the read and write permissions on the objects uploaded by itself.

   test

Step 4: Clear Resources

After you finish working through the example, delete any associated resources so you do not get billed. If you plan to learn other examples, wait until they are finished before doing any clean-up.

1. Delete the workload.

   kubectl delete -f obs_deployment_example.yaml  

   Information similar to the following is displayed:

   deployment.apps "obs-deployment-example" deleted

2. Delete the PVC.

   kubectl delete -f pvc_test_abc.yaml  

   Information similar to the following is displayed:

   persistentvolumeclaim "pvc-test-abc" deleted

3. Delete the PV.

   kubectl delete -f testing_abc.yaml  

   Information similar to the following is displayed:

   persistentvolume "testing-abc" deleted

4. Delete the secret.

   kubectl delete -f test_user.yaml  

   Information similar to the following is displayed:

   secret "test-user" deleted

5. Delete the ConfigMap.

   kubectl delete -f config.yaml  

   Information similar to the following is displayed:

   configmap "paas-obs-endpoint" deleted

Common Issues

If a workload fails to be created, locate the fault based on the error information in the pod events. For details, see Table 6. If the problem persists, submit a service ticket to contact customer service for help.