How Do I Enable the API Server Audit for an On-Premises Kubernetes Container?

Scenario

On-premises Kubernetes containers are used.

Prerequisites

• Container protection has been enabled. For details, see Enabling Container Node Protection.
• API server audit is disabled. Perform the following steps to check its status:
  1. Log in to the node where kube-apiserver is located.
  2. Check the kube-apiserver.yaml file or the started kube-apiserver process.
     • Go to the /etc/kubernetes/manifest directory and check whether --audit-log-path and --audit-policy-file exist in kube-apiserver.yaml. If they do not exist, API server audit is disabled.
     • Run the ps command to check whether --audit-log-path and --audit-policy-file exist in the command lines of the kube-apiserver process. If they do not exist, the audit function of the kube-apiserver process is disabled.

Enabling API Server Audit

1. Copy the following YAML content, save it to the YAML file, and name the file audit-policy.yaml.
   This YAML file is the configuration file of the Kubernetes audit function. You can directly use the file or compile it as needed.

   apiVersion: audit.k8s.io/v1 # This is required.
   kind: Policy
   # Don't generate audit events for all requests in RequestReceived stage.
   omitStages:
     - "RequestReceived"
   rules:
     # The following requests were manually identified as high-volume and low-risk,
     # so drop them.
     # Kube-Proxy running on each node will watch services and endpoint objects in real time
     - level: None
       users: ["system:kube-proxy"]
       verbs: ["watch"]
       resources:
         - group: "" # core
           resources: ["endpoints", "services"]
     # Some health checks
     - level: None
       users: ["kubelet"] # legacy kubelet identity
       verbs: ["get"]
       resources:
         - group: "" # core
           resources: ["nodes"]
     - level: None
       userGroups: ["system:nodes"]
       verbs: ["get"]
       resources:
         - group: "" # core
           resources: ["nodes"]
     - level: None
       users: ["system:apiserver"]
       verbs: ["get"]
       resources:
         - group: "" # core
           resources: ["namespaces"]
     # Some system component certificates reuse the master user, which cannot be accurately distinguished from user behavior,
     # considering that subsequent new functions may continue to add system operations under kube-system, the cost of targeted configuration is relatively high,
     # in terms of the overall strategy, it is not recommended (allowed) for users to operate under the kube-system,
     # so overall drop has no direct impact on user experience
     - level: None
       verbs: ["get", "update"]
       namespaces: ["kube-system"]
     # Don't log these read-only URLs.
     - level: None
       nonResourceURLs:
         - /healthz*
         - /version
         - /swagger*
     # Don't log events requests.
     - level: None
       resources:
         - group: "" # core
           resources: ["events"]
     # Don't log leases requests
     - level: None
       verbs: [ "get", "update" ]
       resources:
         - group: "coordination.k8s.io"
           resources: ["leases"]
     # Secrets, ConfigMaps, and TokenReviews can contain sensitive & binary data,
     # so only log at the Metadata level.
     - level: Metadata
       resources:
         - group: "" # core
           resources: ["secrets", "configmaps"]
         - group: authentication.k8s.io
           resources: ["tokenreviews"]
     # Get responses can be large; skip them.
     - level: Request
       verbs: ["get", "list", "watch"]
       resources:
         - group: "" # core
         - group: "admissionregistration.k8s.io"
         - group: "apps"
         - group: "authentication.k8s.io"
         - group: "authorization.k8s.io"
         - group: "autoscaling"
         - group: "batch"
         - group: "certificates.k8s.io"
         - group: "extensions"
         - group: "networking.k8s.io"
         - group: "policy"
         - group: "rbac.authorization.k8s.io"
         - group: "settings.k8s.io"
         - group: "storage.k8s.io"
     # Default level for known APIs
     - level: RequestResponse
       resources:
         - group: "" # core
         - group: "admissionregistration.k8s.io"
         - group: "apps"
         - group: "authentication.k8s.io"
         - group: "authorization.k8s.io"
         - group: "autoscaling"
         - group: "batch"
         - group: "certificates.k8s.io"
         - group: "extensions"
         - group: "networking.k8s.io"
         - group: "policy"
         - group: "rbac.authorization.k8s.io"
         - group: "settings.k8s.io"
         - group: "storage.k8s.io"
     # Default level for all other requests.
     - level: Metadata

2. Upload the audit-policy.yaml file to the /etc/kubernetes/ directory.
3. Go to the /etc/kubernetes/manifests directory and add the following content to the kube-apiserver.yaml file to enable API server audit:

   --audit-policy-file=/etc/kubernetes/audit-policy.yaml
   --audit-log-path=/var/log/kubernetes/audit/audit.log
   --audit-log-maxsize=100
   --audit-log-maxage=1
   --audit-log-maxbackup=10

   

   • --audit-policy-file: configuration file used by the audit function.
   • --audit-log-path: path of the log file where audit events are written. If this flag is not specified, the logging backend will be disabled.
   • --audit-log-maxsize: maximum size (in MB) of an audit log file before rotation.
   • --audit-log-maxage: maximum number of days for storing old audit log files.
   • --audit-log-maxbackup: maximum number of retained audit log files.
   • Add the preceding parameters to the kube-apiserver.yaml file, ensure that the format of the parameters is the same as that in the kube-apiserver.yaml file and can not contain tab characters.

4. (Optional) If your kube-apiserver runs as a pod, perform the following steps to persist logs on the server:
   1. Locate the volumeMounts field in kube-apiserver.yaml and configure volume mounting as follows:

      volumeMounts:
        - mountPath: /etc/kubernetes/audit-policy.yaml
          name: audit
          readOnly: true
        - mountPath: /var/log/kubernetes/audit/
          name: audit-log
          readOnly: false

   2. Locate the volumes field in kube-apiserver.yaml and configure it as follows:

      volumes:
      - name: audit
        hostPath:
          path: /etc/kubernetes/audit-policy.yaml
          type: File
      - name: audit-log
        hostPath:
          path: /var/log/kubernetes/audit/
          type: DirectoryOrCreate