Help Center/ Host Security Service/ User Guide (Paris)/ FAQs/ Container Guard Service/ How Do I Enable the API Server Audit for an On-Premises Kubernetes Container?
Updated on 2024-08-14 GMT+08:00

How Do I Enable the API Server Audit for an On-Premises Kubernetes Container?

Scenario

On-premises Kubernetes containers are used.

Prerequisites

  • Container protection has been enabled.
  • API server audit is disabled. Perform the following steps to check its status:
    1. Log in to the node where kube-apiserver is located.
    2. Check the kube-apiserver.yaml file or the started kube-apiserver process.
      • Go to the /etc/kubernetes/manifest directory and check whether --audit-log-path and --audit-policy-file exist in kube-apiserver.yaml. If they do not exist, API server audit is disabled.
      • Run the ps command to check whether --audit-log-path and --audit-policy-file exist in the command lines of the kube-apiserver process. If they do not exist, the audit function of the kube-apiserver process is disabled.

Enabling API Server Audit

  1. Copy the following YAML content, save it to the YAML file, and name the file audit-policy.yaml.

    This YAML file is the configuration file of the Kubernetes audit function. You can directly use the file or compile it as needed.
    apiVersion: audit.k8s.io/v1 # This is required.
    kind: Policy
    # Don't generate audit events for all requests in RequestReceived stage.
    omitStages:
      - "RequestReceived"
    rules:
      # The following requests were manually identified as high-volume and low-risk,
      # so drop them.
      # Kube-Proxy running on each node will watch services and endpoint objects in real time
      - level: None
        users: ["system:kube-proxy"]
        verbs: ["watch"]
        resources:
          - group: "" # core
            resources: ["endpoints", "services"]
      # Some health checks
      - level: None
        users: ["kubelet"] # legacy kubelet identity
        verbs: ["get"]
        resources:
          - group: "" # core
            resources: ["nodes"]
      - level: None
        userGroups: ["system:nodes"]
        verbs: ["get"]
        resources:
          - group: "" # core
            resources: ["nodes"]
      - level: None
        users: ["system:apiserver"]
        verbs: ["get"]
        resources:
          - group: "" # core
            resources: ["namespaces"]
      # Some system component certificates reuse the master user, which cannot be accurately distinguished from user behavior,
      # considering that subsequent new functions may continue to add system operations under kube-system, the cost of targeted configuration is relatively high,
      # in terms of the overall strategy, it is not recommended (allowed) for users to operate under the kube-system,
      # so overall drop has no direct impact on user experience
      - level: None
        verbs: ["get", "update"]
        namespaces: ["kube-system"]
      # Don't log these read-only URLs.
      - level: None
        nonResourceURLs:
          - /healthz*
          - /version
          - /swagger*
      # Don't log events requests.
      - level: None
        resources:
          - group: "" # core
            resources: ["events"]
      # Don't log leases requests
      - level: None
        verbs: [ "get", "update" ]
        resources:
          - group: "coordination.k8s.io"
            resources: ["leases"]
      # Secrets, ConfigMaps, and TokenReviews can contain sensitive & binary data,
      # so only log at the Metadata level.
      - level: Metadata
        resources:
          - group: "" # core
            resources: ["secrets", "configmaps"]
          - group: authentication.k8s.io
            resources: ["tokenreviews"]
      # Get responses can be large; skip them.
      - level: Request
        verbs: ["get", "list", "watch"]
        resources:
          - group: "" # core
          - group: "admissionregistration.k8s.io"
          - group: "apps"
          - group: "authentication.k8s.io"
          - group: "authorization.k8s.io"
          - group: "autoscaling"
          - group: "batch"
          - group: "certificates.k8s.io"
          - group: "extensions"
          - group: "networking.k8s.io"
          - group: "policy"
          - group: "rbac.authorization.k8s.io"
          - group: "settings.k8s.io"
          - group: "storage.k8s.io"
      # Default level for known APIs
      - level: RequestResponse
        resources:
          - group: "" # core
          - group: "admissionregistration.k8s.io"
          - group: "apps"
          - group: "authentication.k8s.io"
          - group: "authorization.k8s.io"
          - group: "autoscaling"
          - group: "batch"
          - group: "certificates.k8s.io"
          - group: "extensions"
          - group: "networking.k8s.io"
          - group: "policy"
          - group: "rbac.authorization.k8s.io"
          - group: "settings.k8s.io"
          - group: "storage.k8s.io"
      # Default level for all other requests.
      - level: Metadata

  1. Upload the audit-policy.yaml file to the /etc/kubernetes/ directory.
  2. Go to the /etc/kubernetes/manifests directory and add the following content to the kube-apiserver.yaml file to enable API server audit:

    --audit-policy-file=/etc/kubernetes/audit-policy.yaml
    --audit-log-path=/var/log/kubernetes/audit/audit.log
    --audit-log-maxsize=100
    --audit-log-maxage=1
    --audit-log-maxbackup=10
    • --audit-policy-file: configuration file used by the audit function.
    • --audit-log-path: path of the log file where audit events are written. If this flag is not specified, the logging backend will be disabled.
    • --audit-log-maxsize: maximum size (in MB) of an audit log file before rotation.
    • --audit-log-maxage: maximum number of days for storing old audit log files.
    • --audit-log-maxbackup: maximum number of retained audit log files.
    • Add the preceding parameters to the kube-apiserver.yaml file, ensure that the format of the parameters is the same as that in the kube-apiserver.yaml file and can not contain tab characters.

  1. (Optional) If your kube-apiserver runs as a pod, perform the following steps to persist logs on the server:

    1. Locate the volumeMounts field in kube-apiserver.yaml and configure volume mounting as follows:
      volumeMounts:
        - mountPath: /etc/kubernetes/audit-policy.yaml
          name: audit
          readOnly: true
        - mountPath: /var/log/kubernetes/audit/
          name: audit-log
          readOnly: false
    2. Locate the volumes field in kube-apiserver.yaml and configure it as follows:
      volumes:
      - name: audit
        hostPath:
          path: /etc/kubernetes/audit-policy.yaml
          type: File
      - name: audit-log
        hostPath:
          path: /var/log/kubernetes/audit/
          type: DirectoryOrCreate