Help Center/ Cloud Container Engine/ Best Practices/ DevOps/ Installing and Deploying Jenkins on CCE/ Procedure/ Interconnecting Jenkins with RBAC of Kubernetes Clusters (Example)
Updated on 2024-12-28 GMT+08:00
View PDF
Share

Interconnecting Jenkins with RBAC of Kubernetes Clusters (Example)

Prerequisites

RBAC must be enabled for the cluster.

Scenario 1: Namespace-based Permissions Control

Create a service account and a role, and add a RoleBinding.

# kubectl create ns dev
# kubectl -n dev create sa dev

# cat <<EOF > dev-user-role.yml
kind: Role
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  namespace: dev
  name: dev-user-pod
rules:
- apiGroups: ["*"]
  resources: ["deployments", "pods", "pods/log"]
  verbs: ["get", "watch", "list", "update", "create", "delete"]
EOF
# kubectl create -f dev-user-role.yml

# kubectl create rolebinding dev-view-pod \
    --role=dev-user-pod \
    --serviceaccount=dev:dev \
    --namespace=dev

Generate the kubeconfig file of a specified service account.

  • In clusters earlier than v1.21, a token is obtained by mounting the secret of the service account to a pod. Tokens obtained this way are permanent. This approach is no longer recommended starting from version 1.21. Service accounts will stop auto creating secrets in clusters from version 1.25.

    In clusters of version 1.21 or later, you can use the TokenRequest API to obtain the token and use the projected volume to mount the token to the pod. Such tokens are valid for a fixed period. When the mounting pod is deleted, the token automatically becomes invalid. For details, see Service Account Token Security Improvement.

  • If you need a token that never expires, you can also manually manage secrets for service accounts. Although a permanent service account token can be manually created, you are advised to use a short-lived token by calling the TokenRequest API for higher security.

Because the cluster version used in this case is v1.25, the ServiceAccount will not have a secret created automatically. This example shows how to manually create a secret named dev-secret and associate it with the ServiceAccount named dev.

The manually created secret must be in the same namespace as the ServiceAccount to be associated with. Otherwise, the creation may fail.

# kubectl apply -f - <<EOF
apiVersion: v1 
kind: Secret 
metadata:
  namespace: dev  # Namespace
  name: dev-secret   
  annotations:     
    kubernetes.io/service-account.name: dev 
type: kubernetes.io/service-account-token 
EOF

Check whether dev-secret has been created. If dev-secret is present in secrets of the dev namespace, then it has been created.

# kubectl get secrets -n dev
NAME               TYPE                                  DATA        AGE
default-secret     kubernetes.io/dockerconfigjson           1      2d22h
dev-secret       kubernetes.io/service-account-token      3      4h14m
paas.elb           cfe/secure-opaque                        1      2d22h

Generate a kubeconfig file using dev-secret.

# API_SERVER="https://172.22.132.51:6443"
# CA_CERT=$(kubectl -n dev get secret dev-secret -o yaml | awk '/ca.crt:/{print $2}')
# cat <<EOF > dev.conf
apiVersion: v1
kind: Config
clusters:
- cluster:
    certificate-authority-data: $CA_CERT
    server: $API_SERVER
  name: cluster
EOF

# TOKEN=$(kubectl -n dev get secret dev-secret -o go-template='{{.data.token}}')
# kubectl config set-credentials dev-user \
    --token=`echo ${TOKEN} | base64 -d` \
    --kubeconfig=dev.conf

# kubectl config set-context default \
    --cluster=cluster \
    --user=dev-user \
    --kubeconfig=dev.conf

# kubectl config use-context default \
    --kubeconfig=dev.conf

Verify the configuration.

# kubectl --kubeconfig=dev.conf get po
Error from server (Forbidden): pods is forbidden: User "system:serviceaccount:dev:dev" cannot list pods in the namespace "default"

# kubectl -n dev --kubeconfig=dev.conf run nginx --image nginx --port 80 --restart=Never
# kubectl -n dev --kubeconfig=dev.conf get po
NAME      READY     STATUS    RESTARTS   AGE
nginx     1/1       Running   0          39s

Verify whether the permissions meet the expectation in Jenkins.

  1. Add the kubeconfig file with permissions control settings to Jenkins.

  2. Start the Jenkins job. In this example, Jenkins fails to be deployed in namespace default but is successfully deployed in namespace dev.

Scenario 2: Resource-based Permissions Control

  1. Generate the service account, role, and binding. 

    # kubectl -n dev create sa sa-test0304

# cat <<EOF > test0304-role.yml
kind: Role
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  namespace: dev
  name: role-test0304
rules:
- apiGroups: ["*"]
  resources: ["deployments"]
  resourceNames: ["tomcat03", "tomcat04"]
  verbs: ["get", "update", "patch"]
EOF
# kubectl create -f test0304-role.yml

# kubectl create rolebinding test0304-bind \
    --role=role-test0304 \
    --serviceaccount=dev:sa-test0304\
    --namespace=dev

  2. Generate a kubeconfig file.

    • In clusters earlier than v1.21, a token is obtained by mounting the secret of the service account to a pod. Tokens obtained this way are permanent. This approach is no longer recommended starting from version 1.21. Service accounts will stop auto creating secrets in clusters from version 1.25.

      In clusters of version 1.21 or later, you can use the TokenRequest API to obtain the token and use the projected volume to mount the token to the pod. Such tokens are valid for a fixed period. When the mounting pod is deleted, the token automatically becomes invalid. For details, see Service Account Token Security Improvement.

    • If you need a token that never expires, you can also manually manage secrets for service accounts. Although a permanent service account token can be manually created, you are advised to use a short-lived token by calling the TokenRequest API for higher security.

    Because the cluster version used in this case is v1.25, the ServiceAccount will not have a secret created automatically. This example shows how to manually create a secret named test-secret and associate it with the ServiceAccount named sa-test0304.

    # kubectl apply -f - <<EOF
apiVersion: v1 
kind: Secret 
metadata:
  namespace: dev   
  name: test-secret   
  annotations:     
    kubernetes.io/service-account.name: sa-test0304 
type: kubernetes.io/service-account-token 
EOF

    Check whether test-secret has been created. If test-secret is present in secrets of the dev namespace, then it has been created.

    # kubectl get secrets -n dev
NAME               TYPE                                  DATA        AGE
default-secret     kubernetes.io/dockerconfigjson           1      2d22h
dev-secret         kubernetes.io/service-account-token      3      4h14m
paas.elb           cfe/secure-opaque                        1      2d22h
test-secret       kubernetes.io/service-account-token      3      25m

    Generate a kubeconfig file using test-secret.

    # API_SERVER=" https://192.168.0.153:5443"
# CA_CERT=$(kubectl -n dev get secret test-secret -o yaml | awk '/ca.crt:/{print $2}')
# cat <<EOF > test0304.conf
apiVersion: v1
kind: Config
clusters:
- cluster:
    certificate-authority-data: $CA_CERT
    server: $API_SERVER
  name: cluster
EOF

# TOKEN=$(kubectl -n dev get secret test-secret -o go-template='{{.data.token}}')
# kubectl config set-credentials test0304-user \
    --token=`echo ${TOKEN} | base64 -d` \
    --kubeconfig=test0304.conf

# kubectl config set-context default \
    --cluster=cluster \
    --user=test0304-user \
    --kubeconfig=test0304.conf

# kubectl config use-context default \
    --kubeconfig=test0304.conf

  3. Verify that Jenkins is running as expected.

    In the pipeline script, update the Deployments of tomcat03, tomcat04, and tomcat05 in sequence.

           try { 
         kubernetesDeploy(
             kubeconfigId: "test0304",
             configs: "test03.yaml")
         println "hooray, success"
        } catch (e) {
            println "oh no! Deployment failed! "
            println e
        }
        echo "test04"
        try { 
         kubernetesDeploy(
             kubeconfigId: "test0304",
             configs: "test04.yaml")
         println "hooray, success"
        } catch (e) {
            println "oh no! Deployment failed! "
            println e
        }
        echo "test05"
        try { 
         kubernetesDeploy(
             kubeconfigId: "test0304",
             configs: "test05.yaml")
         println "hooray, success"
        } catch (e) {
            println "oh no! Deployment failed! "
            println e
        }

    The following shows examples of the running results.

    Figure 1 test03
    Figure 2 test04

Parent Topic: Procedure