Deploying and Using ClickHouse in a CCE Cluster

ClickHouse is a columnar database management system for online analytical processing (OLAP). It is suitable for real-time query and analysis of large-scale datasets. There are four ways to deploy ClickHouse in containers. For details, see Table 1. ClickHouse Operator is a tool for deploying and managing ClickHouse in Kubernetes clusters. It can replicate clusters and manage users, configuration files, and persistent volumes. These functions simplify application configuration, management, and monitoring.

This section describes how to deploy ClickHouse in a CCE cluster using kubectl and Operator. For details, see https://github.com/Altinity/clickhouse-operator.

Example 1: Creating a ClickHouse Cluster with a PV Provisioned Dynamically

The following example describes how to create a ClickHouse cluster with a PV dynamically provisioned. An EVS volume is used as an example to describe how to dynamically provision a PV for a ClickHouse cluster.

VolumeClaimTemplate can only be used to provision EVS disks and local PVs for StatefulSets.

1. Create a storage class.
   1. Create a YAML file named csi-disk-ssd.yaml.

      vim csi-disk-ssd.yaml

      By default, CCE supports SAS disks. If you want to use another type of disk, you need to create the corresponding storage class. For details about storage class parameters, see Table 3.

      allowVolumeExpansion: true
      apiVersion: storage.k8s.io/v1
      kind: StorageClass
      metadata:
        name: csi-disk-ssd
      provisioner: everest-csi-provisioner
      parameters:
        csi.storage.k8s.io/csi-driver-name: disk.csi.everest.io
        csi.storage.k8s.io/fstype: ext4
        everest.io/disk-volume-type: SSD
        everest.io/passthrough: "true"
      reclaimPolicy: Delete
      volumeBindingMode: Immediate

      Table 3 Storage class parameters

      Parameter	Description
      provisioner	Specifies the storage resource provider, which is the CCE Container Storage (Everest) add-on. Set this parameter to everest-csi-provisioner.
      parameters	Specifies the storage parameters, which vary with storage types. NOTICE: everest.io/disk-volume-type indicates the cloud disk type, which can be any of the following: SAS: high I/O SSD: ultra-high I/O GPSSD: general purpose SSD ESSD: extreme SSD GPSSD2: general purpose SSD v2, which is supported when the CCE Container Storage (Everest) version is 2.4.4 or later and the everest.io/disk-iops and everest.io/disk-throughput annotations are configured. ESSD2: extreme SSD v2, which is supported when the CCE Container Storage (Everest) version is 2.4.4 or later and the everest.io/disk-iops annotation is configured. Default: SAS
      reclaimPolicy	Specifies the value of persistentVolumeReclaimPolicy for creating a PV. The value can be Delete or Retain. If reclaimPolicy is not specified when a storage class object is created, the value defaults to Delete. Delete: indicates that a dynamically provisioned PV will be automatically deleted when the PVC is deleted. Retain: indicates that a dynamically provisioned PV will be retained when the PVC is deleted.
      volumeBindingMode	Specifies when a PV is dynamically provisioned. The value can be Immediate or WaitForFirstConsumer. Immediate: The PV is dynamically provisioned when a PVC is created. WaitForFirstConsumer: The PV is dynamically provisioned when the PVC is used by the workload.

   2. Use csi-disk-ssd.yaml to create a storage class named csi-disk-ssd.

      kubectl create -f csi-disk-ssd.yaml

2. Create a ClickHouse cluster with a PV dynamically provisioned.
   1. Create a YAML file named pv-simple.yaml.

      vim pv-simple.yaml

      For details about the file content, see https://github.com/Altinity/clickhouse-operator/blob/master/docs/chi-examples/03-persistent-volume-01-default-volume.yaml.

      

      EVS volumes can be mounted as read-write to a single node, so accessModes must be set to ReadWriteOnce.

      apiVersion: "clickhouse.altinity.com/v1"
      kind: "ClickHouseInstallation"
      metadata:
        name: "pv-simple"
        namespace: test-clickhouse-operator
      spec:
        defaults:
          templates:
            dataVolumeClaimTemplate: data-volume-template
            logVolumeClaimTemplate: log-volume-template
        configuration:
          clusters:
            - name: "simple"
              layout:
                shardsCount: 1
                replicasCount: 1
        templates:
          volumeClaimTemplates:                   # Dynamic provisioning
            - name: data-volume-template          # Template for defining a data storage volume
              spec:
                accessModes:
                  - ReadWriteOnce                 # EVS disks can be mounted as read-write by a single node, so accessModes must be set to ReadWriteOnce.
                resources:
                  requests:
                    storage: 10Gi
                storageClassName: csi-disk-ssd    # Specify the newly created csi-disk-ssd as the storage class.
            - name: log-volume-template           # Template for defining the log storage volume
              spec:
                accessModes:
                  - ReadWriteOnce                 # EVS disks can be mounted as read-write by a single node, so accessModes must be set to ReadWriteOnce.
                resources:
                  requests:
                    storage: 10Gi
                storageClassName: csi-disk-ssd    # Specify the newly created csi-disk-ssd as the storage class.

   2. Create the ClickHouse cluster.

      kubectl -n test-clickhouse-operator create -f pv-simple.yaml

3. Check whether the ClickHouse cluster has been created and whether the PV has been provisioned.
   1. Check the pods in the test-clickhouse-operator namespace. If all pods are in the Running state, the pods have been created.

      kubectl get pod -n test-clickhouse-operator

      If information similar to the following is displayed, the pods have been created:

      NAME                            READY   STATUS    RESTARTS   AGE
      chi-pv-simple-simple-0-0-0      2/2     Running   0          5m2s
      chi-simple-01-simple-0-0-0      1/1     Running   0          3d7h

   2. Check whether the PVCs named data-volume-template and log-volume-template have been created.

      kubectl get pvc -n test-clickhouse-operator

      If STATUS is Bound, the PVCs have been bound to the PVs.

      NAME                                              STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS   VOLUMEATTRIBUTESCLASS   AGE
      data-volume-template-chi-pv-simple-simple-0-0-0   Bound    pvc-981b1d73-a13e-41d5-aade-ea8c6b1199d7   10Gi       RWO            csi-disk-ssd   <unset>                 28s
      log-volume-template-chi-pv-simple-simple-0-0-0    Bound    pvc-fcf70a2e-131d-4da1-a9c2-eddd89887b45   10Gi       RWO            csi-disk-ssd   <unset>                 28s

   3. Check whether the PV has been mounted to the cluster.

      Go to the CLI of the chi-pv-simple-simple-0-0-0 container.

      kubectl -n test-clickhouse-operator exec -ti chi-pv-simple-simple-0-0-0 -c clickhouse bash

      Check whether the PV has been mounted to the container.

      df -h

      The command output shows that the PV has been mounted to the container. You can press Ctrl+D to exit the CLI.

      Filesystem                    Size    Used   Avail  Use% Mounted on
      overlay                        99G    5.1G     89G    6% /
      tmpfs                          64M       0     64M    0% /dev
      tmpfs                          3.9G      0    3.9G    0% /sys/fs/cgroup
      /dev/mapper/vgpaas-share       99G    5.1G     89G    6% /etc/hosts
      shm                            64M       0     64M    0% /dev/shm
      /dev/sdb                       9.8G    66M    9.8G    1% /var/lib/clickhouse
      /dev/sda                       9.8G    37M    9.8G    1% /var/log/clickhouse-server
      tmpfs                          6.3G     12K    6.3G   1% /run/secrets/kubernetes.io/serviceaccount
      tmpfs                          3.9G       0    3.9G   0% /proc/acpi
      tmpfs                          3.9G       0    3.9G   0% /proc/scsi
      tmpfs                          3.9G       0    3.9G   0% /sys/firmware

4. Access the ClickHouse database.

   kubectl -n test-clickhouse-operator exec -ti chi-pv-simple-simple-0-0-0 -- clickhouse-client

   If information similar to that shown below is displayed, you have accessed the ClickHouse database. Enter exit and press Enter to exit the ClickHouse database.

   Defaulted container "clickhouse" out of: clickhouse, clickhouse-log
   ClickHouse client version 24.8.2.3 (official build).
   Connecting to localhost:9000 as user default.
   Connected to ClickHouse server version 24.8.2.
   
   Warnings:
    * Linux transparent hugepages are set to "always". Check /sys/kernel/mm/transparent_hugepage/enabled
   
   chi-pv-simple-simple-0-0-0.chi-pv-simple-simple-0-0.test-clickhouse-operator.svc.cluster.local :) 

5. Clear the ClickHouse cluster resources.

   Delete the ClickHouse cluster with a PV provisioned dynamically.

   kubectl delete -f pv-simple.yaml -n test-clickhouse-operator

   Information similar to the following is displayed:

   clickhouseinstallation.clickhouse.altinity.com "pv-simple" deleted

1. Create a ClickHouse cluster with a LoadBalancer Service so that you can access the ClickHouse cluster from the Internet.
   1. Create a YAML file named elb.yaml.

      vim elb.yaml

      For details about parameters in kubernetes.io/elb.autocreate, see Table 4.

      apiVersion: "clickhouse.altinity.com/v1"
      kind: "ClickHouseInstallation"
      metadata:
        name: "ck-elb"
        namespace: test-clickhouse-operator
      spec:
        defaults:
          templates:
            dataVolumeClaimTemplate: data-volume-nas                    
            serviceTemplate: chi-service-elb                        
        configuration:
          clusters:
            - name: "ck-elb"
              templates:
                podTemplate: pod-template-with-nas
              layout:
                shardsCount: 1
                replicasCount: 1
        templates:
          podTemplates:
            - name: pod-template-with-nas
              spec:
                containers:
                  - name: clickhouse
                    image: clickhouse/clickhouse-server:23.8
                    volumeMounts:
                      - name: data-volume-nas
                        mountPath: /var/lib/clickhouse
          volumeClaimTemplates:               # Specify the storage access mode, requested storage size, and storage class.
            - name: data-volume-nas
              spec:
                accessModes:
                  - ReadWriteOnce
                resources:
                  requests:
                    storage: 20Gi
                storageClassName: csi-disk-ssd
          serviceTemplates:                  # Service template
            - name: chi-service-elb 
              metadata:
                annotations:
                  # Load balancer type. union (default value) indicates shared load balancers, and performance indicates dedicated load balancers.
                  kubernetes.io/elb.class: union   
                  # Automatically create a load balancer associated with the ingress and define load balancer parameters.
                  kubernetes.io/elb.autocreate: >-                             
                    {"type":"public","bandwidth_name":"cce-bandwidth-ck","bandwidth_chargemode":"bandwidth","bandwidth_size":5,"bandwidth_sharetype":"PER","eip_type":"5_bgp"}
              spec:
                ports:
                  - name: http
                    port: 8123
                  - name: client
                    port: 9000
                type: LoadBalancer             # Set the Service type to LoadBalancer.

      Table 4 Parameters in the kubernetes.io/elb.autocreate file

      Parameter	Mandatory	Type	Description
      type	No	String	Network type of the load balancer. public: public network load balancer inner: private network load balancer Default: inner
      bandwidth_name	Yes for public network load balancers	String	Bandwidth name. The default value is cce-bandwidth-******. The value can contain 1 to 64 characters. Only letters, digits, underscores (_), hyphens (-), and periods (.) are allowed.
      bandwidth_chargemode	No	String	Bandwidth billing mode. bandwidth: billed by bandwidth traffic: billed by traffic Default: bandwidth
      bandwidth_size	Yes for public network load balancers	Integer	Bandwidth size. The default value is 1 to 2,000 Mbit/s. Configure this parameter based on the bandwidth range allowed in your region. The minimum increment for bandwidth adjustment varies depending on the bandwidth range. If the allowed bandwidth does not exceed 300 Mbit/s, the minimum increment is 1 Mbit/s. If the allowed bandwidth is greater than 300 Mbit/s but less than or equal to 1,000 Mbit/s, the minimum increment is 50 Mbit/s. If the allowed bandwidth exceeds 1,000 Mbit/s, the minimum increment is 500 Mbit/s.
      bandwidth_sharetype	Yes for public network load balancers	String	The bandwidth sharing mode. PER indicates that the bandwidth is dedicated.
      eip_type	Yes for public network load balancers	String	EIP type. 5_bgp: Dynamic BGP 5_sbgp: Static BGP The types vary by region. For details, see the EIP console.

   2. Create the ClickHouse cluster.

      kubectl create -f elb.yaml -n test-clickhouse-operator

2. Check whether the ClickHouse cluster has been created and has a LoadBalancer Service in it.
   1. Check the pods in the test-clickhouse-operator namespace. If all pods are in the Running state, the pods have been created.

      kubectl get pod -n test-clickhouse-operator

      If information similar to the following information is displayed, the ClickHouse cluster has been created:

      NAME                            READY   STATUS    RESTARTS   AGE
      chi-ck-elb-ck-elb-0-0-0         1/1     Running   0          3m4s
      chi-pv-simple-simple-0-0-0      2/2     Running   0          33m
      chi-simple-01-simple-0-0-0      1/1     Running   0          3d7h

   2. Check whether the LoadBalancer Service has been created.

      kubectl get svc -n test-clickhouse-operator

      If information similar to the following is displayed, the LoadBalancer Service has been created:

      NAME                          TYPE          CLUSTER-IP     EXTERNAL-IP    PORT(S)                          AGE
      chi-ck-elb-ck-elb-0-0         ClusterIP     None           <none>         9000/TCP,8123/TCP,9009/TCP       2s
      chi-pv-simple-simple-0-0      ClusterIP     None           <none>         9000/TCP,8123/TCP,9009/TCP       35m
      chi-simple-01-simple-0-0      ClusterIP     None           <none>         9000/TCP,8123/TCP,9009/TCP       38m
      clickhouse-pv-simple          ClusterIP     None           <none>         8123/TCP,9000/TCP                35m
      clickhouse-simple-01          ClusterIP     None           <none>         8123/TCP,9000/TCP                3d7h

3. Access the ClickHouse database.

   kubectl -n test-clickhouse-operator exec -ti chi-ck-elb-ck-elb-0-0-0 -- clickhouse-client

   If information similar to that shown below is displayed, you have accessed the ClickHouse database. Enter exit and press Enter to exit the ClickHouse database.

   ClickHouse client version 23.8.16.16 (official build).
   Connecting to localhost:9000 as user default.
   Connected to ClickHouse server version 23.8.16 revision 54465.
   
   Warnings:
    * Linux transparent hugepages are set to "always". Check /sys/kernel/mm/transparent_hugepage/enabled
   
   chi-ck-elb-ck-elb-0-0-0.chi-ck-elb-ck-elb-0-0.test-clickhouse-operator.svc.cluster.local :) 

4. Clear the ClickHouse cluster resources.

   Delete the ClickHouse cluster that contains the LoadBalancer Service.

   kubectl delete -f elb.yaml  -n test-clickhouse-operator

   Information similar to the following is displayed:

   clickhouseinstallation.clickhouse.altinity.com "ck-elb1" deleted