Help Center/ Cloud Container Engine/ Best Practices/ DevOps/ Installing and Deploying Jenkins on CCE/ Procedure/ Interconnecting Jenkins with RBAC of Kubernetes Clusters (Example)
Updated on 2024-05-31 GMT+08:00

Interconnecting Jenkins with RBAC of Kubernetes Clusters (Example)

Prerequisites

RBAC must be enabled for the cluster.

Scenario 1: Namespace-based Permissions Control

Create a service account and a role, and add a RoleBinding.

$ kubectl create ns dev
$ kubectl -n dev create sa dev

$ cat <<EOF > dev-user-role.yml
kind: Role
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  namespace: dev
  name: dev-user-pod
rules:
- apiGroups: ["*"]
  resources: ["deployments", "pods", "pods/log"]
  verbs: ["get", "watch", "list", "update", "create", "delete"]
EOF
kubectl create -f dev-user-role.yml

$ kubectl create rolebinding dev-view-pod \
    --role=dev-user-pod \
    --serviceaccount=dev:dev \
    --namespace=dev

Generate the kubeconfig file of a specified service account.

  • In clusters earlier than v1.21, a token is obtained by mounting the secret of the service account to a pod. Tokens obtained this way are permanent. This approach is no longer recommended starting from version 1.21. Service accounts will stop auto creating secrets in clusters from version 1.25.

    In clusters of version 1.21 or later, you can use the TokenRequest API to obtain the token and use the projected volume to mount the token to the pod. Such tokens are valid for a fixed period. When the mounting pod is deleted, the token automatically becomes invalid. For details, see Service Account Token Security Improvement.

  • If you need a token that never expires, you can also manually manage secrets for service accounts. Although a permanent service account token can be manually created, you are advised to use a short-lived token by calling the TokenRequest API for higher security.
$ SECRET=$(kubectl -n dev get sa dev -o go-template='{{range .secrets}}{{.name}}{{end}}')
$ API_SERVER="https://172.22.132.51:6443"
$ CA_CERT=$(kubectl -n dev get secret ${SECRET} -o yaml | awk '/ca.crt:/{print $2}')
$ cat <<EOF > dev.conf
apiVersion: v1
kind: Config
clusters:
- cluster:
    certificate-authority-data: $CA_CERT
    server: $API_SERVER
  name: cluster
EOF

$ TOKEN=$(kubectl -n dev get secret ${SECRET} -o go-template='{{.data.token}}')
$ kubectl config set-credentials dev-user \
    --token=`echo ${TOKEN} | base64 -d` \
    --kubeconfig=dev.conf

$ kubectl config set-context default \
    --cluster=cluster \
    --user=dev-user \
    --kubeconfig=dev.conf

$ kubectl config use-context default \
    --kubeconfig=dev.conf

Verification in the CLI

$ kubectl --kubeconfig=dev.conf get po
Error from server (Forbidden): pods is forbidden: User "system:serviceaccount:dev:dev" cannot list pods in the namespace "default"

$ kubectl -n dev --kubeconfig=dev.conf run nginx --image nginx --port 80 --restart=Never
$ kubectl -n dev --kubeconfig=dev.conf get po
NAME      READY     STATUS    RESTARTS   AGE
nginx     1/1       Running   0          39s

Verify whether the permissions meet the expectation in Jenkins.

  1. Add the kubeconfig file with permissions control settings to Jenkins.

  2. Start the Jenkins job. In this example, Jenkins fails to be deployed in namespace default but is successfully deployed in namespace dev.

Scenario 2: Resource-based Permissions Control

  1. Generate the service account, role, and binding.

     kubectl -n dev create sa sa-test0304
    
     cat <<EOF > test0304-role.yml
    kind: Role
    apiVersion: rbac.authorization.k8s.io/v1
    metadata:
      namespace: dev
      name: role-test0304
    rules:
    - apiGroups: ["*"]
      resources: ["deployments"]
      resourceNames: ["tomcat03", "tomcat04"]
      verbs: ["get", "update", "patch"]
    EOF
    kubectl create -f test0304-role.yml
    
     kubectl create rolebinding test0304-bind \
        --role=role-test0304 \
        --serviceaccount=dev:sa-test0304\
        --namespace=dev

  2. Generate the kubeconfig file.

    • In clusters earlier than v1.21, a token is obtained by mounting the secret of the service account to a pod. Tokens obtained this way are permanent. This approach is no longer recommended starting from version 1.21. Service accounts will stop auto creating secrets in clusters from version 1.25.

      In clusters of version 1.21 or later, you can use the TokenRequest API to obtain the token and use the projected volume to mount the token to the pod. Such tokens are valid for a fixed period. When the mounting pod is deleted, the token automatically becomes invalid. For details, see Service Account Token Security Improvement.

    • If you need a token that never expires, you can also manually manage secrets for service accounts. Although a permanent service account token can be manually created, you are advised to use a short-lived token by calling the TokenRequest API for higher security.
     SECRET=$(kubectl -n dev get sa sa-test0304 -o go-template='{{range .secrets}}{{.name}}{{end}}')
     API_SERVER=" https://192.168.0.153:5443"
     CA_CERT=$(kubectl -n dev get secret ${SECRET} -o yaml | awk '/ca.crt:/{print $2}')
     cat <<EOF > test0304.conf
    apiVersion: v1
    kind: Config
    clusters:
    - cluster:
        certificate-authority-data: $CA_CERT
        server: $API_SERVER
      name: cluster
    EOF
    
    TOKEN=$(kubectl -n dev get secret ${SECRET} -o go-template='{{.data.token}}')
     kubectl config set-credentials test0304-user \
        --token=`echo ${TOKEN} | base64 -d` \
        --kubeconfig=test0304.conf
    
     kubectl config set-context default \
        --cluster=cluster \
        --user=test0304-user \
        --kubeconfig=test0304.conf
    
     kubectl config use-context default \
        --kubeconfig=test0304.conf

  3. Verify that Jenkins is running as expected.

    In the pipeline script, update the Deployments of tomcat03, tomcat04, and tomcat05 in sequence.

           try { 
             kubernetesDeploy(
                 kubeconfigId: "test0304",
                 configs: "test03.yaml")
             println "hooray, success"
            } catch (e) {
                println "oh no! Deployment failed! "
                println e
            }
            echo "test04"
            try { 
             kubernetesDeploy(
                 kubeconfigId: "test0304",
                 configs: "test04.yaml")
             println "hooray, success"
            } catch (e) {
                println "oh no! Deployment failed! "
                println e
            }
            echo "test05"
            try { 
             kubernetesDeploy(
                 kubeconfigId: "test0304",
                 configs: "test05.yaml")
             println "hooray, success"
            } catch (e) {
                println "oh no! Deployment failed! "
                println e
            }

    Viewing the running result:

    Figure 1 test03
    Figure 2 test04