Interconnecting Jenkins with RBAC of Kubernetes Clusters (Example)

Prerequisites

RBAC must be enabled for the cluster.

Scenario 1: Namespace-based Permissions Control

Create a service account and a role, and add a RoleBinding.

$ kubectl create ns dev
$ kubectl -n dev create sa dev

$ cat <<EOF > dev-user-role.yml
kind: Role
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  namespace: dev
  name: dev-user-pod
rules:
- apiGroups: ["*"]
  resources: ["deployments", "pods", "pods/log"]
  verbs: ["get", "watch", "list", "update", "create", "delete"]
EOF
kubectl create -f dev-user-role.yml

$ kubectl create rolebinding dev-view-pod \
    --role=dev-user-pod \
    --serviceaccount=dev:dev \
    --namespace=dev

Generate the kubeconfig file of a specified service account.

• In clusters earlier than v1.21, a token is obtained by mounting the secret of the service account to a pod. Tokens obtained this way are permanent. This approach is no longer recommended starting from version 1.21. Service accounts will stop auto creating secrets in clusters from version 1.25.

  In clusters of version 1.21 or later, you can use the TokenRequest API to obtain the token and use the projected volume to mount the token to the pod. Such tokens are valid for a fixed period. When the mounting pod is deleted, the token automatically becomes invalid. For details, see Service Account Token Security Improvement.

• If you need a token that never expires, you can also manually manage secrets for service accounts. Although a permanent service account token can be manually created, you are advised to use a short-lived token by calling the TokenRequest API for higher security.

$ SECRET=$(kubectl -n dev get sa dev -o go-template='{{range .secrets}}{{.name}}{{end}}')
$ API_SERVER="https://172.22.132.51:6443"
$ CA_CERT=$(kubectl -n dev get secret ${SECRET} -o yaml | awk '/ca.crt:/{print $2}')
$ cat <<EOF > dev.conf
apiVersion: v1
kind: Config
clusters:
- cluster:
    certificate-authority-data: $CA_CERT
    server: $API_SERVER
  name: cluster
EOF

$ TOKEN=$(kubectl -n dev get secret ${SECRET} -o go-template='{{.data.token}}')
$ kubectl config set-credentials dev-user \
    --token=`echo ${TOKEN} | base64 -d` \
    --kubeconfig=dev.conf

$ kubectl config set-context default \
    --cluster=cluster \
    --user=dev-user \
    --kubeconfig=dev.conf

$ kubectl config use-context default \
    --kubeconfig=dev.conf

Verification in the CLI

$ kubectl --kubeconfig=dev.conf get po
Error from server (Forbidden): pods is forbidden: User "system:serviceaccount:dev:dev" cannot list pods in the namespace "default"

$ kubectl -n dev --kubeconfig=dev.conf run nginx --image nginx --port 80 --restart=Never
$ kubectl -n dev --kubeconfig=dev.conf get po
NAME      READY     STATUS    RESTARTS   AGE
nginx     1/1       Running   0          39s

Verify whether the permissions meet the expectation in Jenkins.

1. Add the kubeconfig file with permissions control settings to Jenkins.

   

2. Start the Jenkins job. In this example, Jenkins fails to be deployed in namespace default but is successfully deployed in namespace dev.

   

   

Scenario 2: Resource-based Permissions Control

1. Generate the service account, role, and binding.

    kubectl -n dev create sa sa-test0304
   
    cat <<EOF > test0304-role.yml
   kind: Role
   apiVersion: rbac.authorization.k8s.io/v1
   metadata:
     namespace: dev
     name: role-test0304
   rules:
   - apiGroups: ["*"]
     resources: ["deployments"]
     resourceNames: ["tomcat03", "tomcat04"]
     verbs: ["get", "update", "patch"]
   EOF
   kubectl create -f test0304-role.yml
   
    kubectl create rolebinding test0304-bind \
       --role=role-test0304 \
       --serviceaccount=dev:sa-test0304\
       --namespace=dev

2. Generate the kubeconfig file.
   

   • In clusters earlier than v1.21, a token is obtained by mounting the secret of the service account to a pod. Tokens obtained this way are permanent. This approach is no longer recommended starting from version 1.21. Service accounts will stop auto creating secrets in clusters from version 1.25.

     In clusters of version 1.21 or later, you can use the TokenRequest API to obtain the token and use the projected volume to mount the token to the pod. Such tokens are valid for a fixed period. When the mounting pod is deleted, the token automatically becomes invalid. For details, see Service Account Token Security Improvement.

   • If you need a token that never expires, you can also manually manage secrets for service accounts. Although a permanent service account token can be manually created, you are advised to use a short-lived token by calling the TokenRequest API for higher security.

    SECRET=$(kubectl -n dev get sa sa-test0304 -o go-template='{{range .secrets}}{{.name}}{{end}}')
    API_SERVER=" https://192.168.0.153:5443"
    CA_CERT=$(kubectl -n dev get secret ${SECRET} -o yaml | awk '/ca.crt:/{print $2}')
    cat <<EOF > test0304.conf
   apiVersion: v1
   kind: Config
   clusters:
   - cluster:
       certificate-authority-data: $CA_CERT
       server: $API_SERVER
     name: cluster
   EOF
   
   TOKEN=$(kubectl -n dev get secret ${SECRET} -o go-template='{{.data.token}}')
    kubectl config set-credentials test0304-user \
       --token=`echo ${TOKEN} | base64 -d` \
       --kubeconfig=test0304.conf
   
    kubectl config set-context default \
       --cluster=cluster \
       --user=test0304-user \
       --kubeconfig=test0304.conf
   
    kubectl config use-context default \
       --kubeconfig=test0304.conf

3. Verify that Jenkins is running as expected.

   In the pipeline script, update the Deployments of tomcat03, tomcat04, and tomcat05 in sequence.

          try { 
            kubernetesDeploy(
                kubeconfigId: "test0304",
                configs: "test03.yaml")
            println "hooray, success"
           } catch (e) {
               println "oh no! Deployment failed! "
               println e
           }
           echo "test04"
           try { 
            kubernetesDeploy(
                kubeconfigId: "test0304",
                configs: "test04.yaml")
            println "hooray, success"
           } catch (e) {
               println "oh no! Deployment failed! "
               println e
           }
           echo "test05"
           try { 
            kubernetesDeploy(
                kubeconfigId: "test0304",
                configs: "test05.yaml")
            println "hooray, success"
           } catch (e) {
               println "oh no! Deployment failed! "
               println e
           }

   Viewing the running result:

   Figure 1 test03
   
   Figure 2 test04