# 部署Hubble实现DataPlane V2网络可观测性 在开启了DataPlane V2的集群中，部署开源的可观测项目Hubble可以将容器网络流量进行可视化展示，实现容器网络可观测性。 #### 前提条件 - CCE Standard集群版本在v1.27.16-r50、v1.28.15-r40、v1.29.15-r0、v1.30.14-r0、v1.31.10-r0、v1.32.6-r0及以上，并开启DataPlane V2时支持部署Hubble。 - CCE Turbo集群版本在v1.34.3-r10及以上，并开启DataPlane V2时支持部署Hubble。 #### 步骤一：配置ConfigMap并重建yangtse-cilium 1. 请参见[通过kubectl连接集群](https://support.huaweicloud.com/usermanual-cce/cce_10_0107.html)，使用kubectl工具连接到您的集群。 2. 创建以下cilium社区原生的ConfigMap配置。 ``` apiVersion: v1 kind: ConfigMap metadata: name: cilium-config namespace: kube-system data: enable-hubble: "true" hubble-disable-tls: "true" hubble-listen-address: :4244 hubble-metrics: dns drop tcp flow port-distribution icmp http hubble-metrics-server: :9965 ``` 表1参数说明 | 参数 | 描述 | 备注 | |:---|:---|:---| | enable-hubble | 启用Hubble网络流量可观测，设置为"true"表示启用，设置为"false"表示不启用。 | 本文设置为"true" | | hubble-disable-tls | 关闭Hubble服务端的tls，设置为"true"表示关闭，设置为"false"表示不关闭。 | 本文设置为"true" | | hubble-listen-address | Hubble服务端监听地址。 | 本文设置为":4244"，冒号不可省略 | | hubble-metrics | Hubble需要采集的Metrics，以空格分隔。 Hubble支持采集的Metrics全量列表为"dns drop tcp flow flows-to-world httpV2 icmp kafka port-distribution" ，详情请参见社区支持的[指标](https://docs.cilium.io/en/stable/observability/metrics/#hubble-exported-metrics)。 | 如果您启用Metrics数量比较多，则会对Hubble性能产生一定影响。 | | hubble-metrics-server | Hubble Metrics暴露的地址。 | 本文设置为":9965"，冒号不可省略 | 3. 完成cilium-config配置后，通过执行以下命令滚动重建yangtse-cilium即可生效。 ``` kubectl rollout restart daemonset/yangtse-cilium -n kube-system ``` #### 步骤二：部署Hubble组件 1. 创建hubble.yaml文件。 内容如下： ![](https://support.huaweicloud.com/usermanual-cce/public_sys-resources/note_3.0-zh-cn.png) - 请修改以下配置文件中valid-cluster字符为您当前的集群名。 - 当前YAML样例里hubble-relay、hubble-ui、hubble-ui-backend等组件的镜像版本为[cilium v1.19.4](https://github.com/cilium/cilium/blob/v1.19.4/install/kubernetes/cilium/values.yaml)对应的镜像版本。您可根据当前cilium版本替换为对应的社区hubble镜像版本。 ``` --- apiVersion: v1 automountServiceAccountToken: false kind: ServiceAccount metadata: name: hubble-relay namespace: kube-system --- apiVersion: v1 kind: ServiceAccount metadata: name: hubble-ui namespace: kube-system --- apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRoleBinding metadata: name: hubble-ui roleRef: apiGroup: rbac.authorization.k8s.io kind: ClusterRole name: hubble-ui subjects: - kind: ServiceAccount name: hubble-ui namespace: kube-system --- apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRole metadata: name: hubble-ui rules: - apiGroups: - networking.k8s.io resources: - networkpolicies verbs: - get - list - watch - apiGroups: - "" resources: - componentstatuses - endpoints - namespaces - nodes - pods - services verbs: - get - list - watch - apiGroups: - apiextensions.k8s.io resources: - customresourcedefinitions verbs: - get - list - watch - apiGroups: - cilium.io resources: - '*' verbs: - get - list - watch --- apiVersion: v1 kind: Service metadata: labels: app: hubble-relay name: hubble-relay namespace: kube-system spec: internalTrafficPolicy: Cluster ipFamilies: - IPv4 ipFamilyPolicy: SingleStack ports: - port: 80 protocol: TCP targetPort: grpc selector: app: hubble-relay sessionAffinity: None type: ClusterIP --- apiVersion: v1 kind: Service metadata: annotations: prometheus.io/port: "9965" prometheus.io/scrape: "true" labels: app: hubble name: hubble-metrics namespace: kube-system spec: clusterIP: None clusterIPs: - None internalTrafficPolicy: Cluster ipFamilies: - IPv4 ipFamilyPolicy: SingleStack ports: - name: hubble-metrics port: 9965 protocol: TCP targetPort: hubble-metrics selector: app: yangtse-cilium sessionAffinity: None type: ClusterIP --- apiVersion: v1 kind: Service metadata: labels: app: cilium name: hubble-peer namespace: kube-system spec: internalTrafficPolicy: Local ipFamilies: - IPv4 ipFamilyPolicy: SingleStack ports: - name: peer-service port: 80 protocol: TCP targetPort: 4244 selector: app: yangtse-cilium sessionAffinity: None type: ClusterIP --- apiVersion: v1 data: nginx.conf: "server {\n listen 8081;\n listen [::]:8081;\n server_name \ localhost;\n root /app;\n index index.html;\n client_max_body_size 1G;\n\n location / {\n proxy_set_header Host $host;\n proxy_set_header X-Real-IP $remote_addr;\n\n location /api {\n proxy_http_version 1.1;\n proxy_pass_request_headers on;\n proxy_pass http://127.0.0.1:8090;\n \ }\n location / {\n # double `/index.html` is required here \n try_files $uri $uri/ /index.html /index.html;\n }\n\n \ # Liveness probe\n location /healthz {\n access_log off;\n add_header Content-Type text/plain;\n return 200 'ok';\n }\n }\n}" kind: ConfigMap metadata: name: hubble-ui-nginx namespace: kube-system --- apiVersion: v1 data: # 请将valid-cluster字符修改为您当前的集群名 config.yaml: "cluster-name: valid-cluster\npeer-service: \"hubble-peer.kube-system.svc.cluster.local.:80\"\nlisten-address: :4245\ngops: true\ngops-port: \"9893\"\nretry-timeout: \nsort-buffer-len-max: \nsort-buffer-drain-timeout: \ndisable-client-tls: true\n\ndisable-server-tls: true\n" kind: ConfigMap metadata: name: hubble-relay-config namespace: kube-system --- apiVersion: apps/v1 kind: Deployment metadata: labels: app: hubble-relay name: hubble-relay namespace: kube-system spec: progressDeadlineSeconds: 600 replicas: 1 revisionHistoryLimit: 10 selector: matchLabels: app: hubble-relay strategy: rollingUpdate: maxSurge: 25% maxUnavailable: 1 type: RollingUpdate template: metadata: creationTimestamp: null labels: app: hubble-relay spec: affinity: podAffinity: requiredDuringSchedulingIgnoredDuringExecution: - labelSelector: matchLabels: app: yangtse-cilium topologyKey: kubernetes.io/hostname automountServiceAccountToken: false containers: - args: - serve command: - hubble-relay image: quay.io/cilium/hubble-relay:v1.19.4 imagePullPolicy: IfNotPresent livenessProbe: failureThreshold: 12 grpc: port: 4222 service: "" initialDelaySeconds: 10 periodSeconds: 10 successThreshold: 1 timeoutSeconds: 10 name: hubble-relay ports: - containerPort: 4245 name: grpc protocol: TCP readinessProbe: failureThreshold: 3 grpc: port: 4222 service: "" periodSeconds: 10 successThreshold: 1 timeoutSeconds: 3 resources: {} securityContext: capabilities: drop: - ALL runAsGroup: 65532 runAsNonRoot: true runAsUser: 65532 startupProbe: failureThreshold: 20 grpc: port: 4222 service: "" initialDelaySeconds: 10 periodSeconds: 3 successThreshold: 1 timeoutSeconds: 1 terminationMessagePath: /dev/termination-log terminationMessagePolicy: FallbackToLogsOnError volumeMounts: - mountPath: /etc/hubble-relay name: config readOnly: true dnsPolicy: ClusterFirst restartPolicy: Always schedulerName: default-scheduler securityContext: fsGroup: 65532 serviceAccount: hubble-relay serviceAccountName: hubble-relay terminationGracePeriodSeconds: 1 volumes: - configMap: defaultMode: 420 items: - key: config.yaml path: config.yaml name: hubble-relay-config name: config --- apiVersion: apps/v1 kind: Deployment metadata: labels: app: hubble-ui name: hubble-ui namespace: kube-system spec: progressDeadlineSeconds: 600 replicas: 1 revisionHistoryLimit: 10 selector: matchLabels: app: hubble-ui strategy: rollingUpdate: maxSurge: 25% maxUnavailable: 1 type: RollingUpdate template: metadata: creationTimestamp: null labels: app: hubble-ui spec: automountServiceAccountToken: true containers: - image: quay.io/cilium/hubble-ui:v0.13.5 imagePullPolicy: IfNotPresent livenessProbe: failureThreshold: 3 httpGet: path: /healthz port: 8081 scheme: HTTP periodSeconds: 10 successThreshold: 1 timeoutSeconds: 1 name: frontend ports: - containerPort: 8081 name: http protocol: TCP readinessProbe: failureThreshold: 3 httpGet: path: / port: 8081 scheme: HTTP periodSeconds: 10 successThreshold: 1 timeoutSeconds: 1 resources: {} terminationMessagePath: /dev/termination-log terminationMessagePolicy: FallbackToLogsOnError volumeMounts: - mountPath: /etc/nginx/conf.d/default.conf name: hubble-ui-nginx-conf subPath: nginx.conf - mountPath: /tmp name: tmp-dir - env: - name: EVENTS_SERVER_PORT value: "8090" - name: FLOWS_API_ADDR value: hubble-relay:80 image: quay.io/cilium/hubble-ui-backend:v0.13.5 imagePullPolicy: IfNotPresent name: backend ports: - containerPort: 8090 name: grpc protocol: TCP resources: {} terminationMessagePath: /dev/termination-log terminationMessagePolicy: FallbackToLogsOnError dnsPolicy: ClusterFirst restartPolicy: Always schedulerName: default-scheduler securityContext: fsGroup: 1001 runAsGroup: 1001 runAsUser: 1001 serviceAccount: hubble-ui serviceAccountName: hubble-ui terminationGracePeriodSeconds: 30 volumes: - configMap: defaultMode: 420 name: hubble-ui-nginx name: hubble-ui-nginx-conf - emptyDir: {} name: tmp-dir ``` 2. 部署组件。 ``` kubectl apply -f hubble.yaml ``` #### 步骤三：部署hubble-ui服务 您可以通过部署[NodePort类型](https://support.huaweicloud.com/usermanual-cce/cce_10_0142.html)或者[LoadBalancer类型](https://support.huaweicloud.com/usermanual-cce/cce_10_0681.html)的Service，来对外提供hubble-ui服务。 本文通过NodePort来对外提供hubble-ui服务 1. 创建hubble-ui.yaml文件。 内容如下： ``` apiVersion: v1 kind: Service metadata: labels: app: hubble-ui name: hubble-ui namespace: kube-system spec: externalTrafficPolicy: Cluster internalTrafficPolicy: Cluster ipFamilies: - IPv4 ipFamilyPolicy: SingleStack ports: - name: http nodePort: 32222 port: 80 protocol: TCP targetPort: 8081 selector: app: hubble-ui sessionAffinity: None type: NodePort ``` 2. 创建NodePort。 ``` kubectl apply -f hubble-ui.yaml ``` #### 步骤四：访问hubble-ui 在浏览器中打开hubble-ui的对外访问地址，例如"http://{节点的EIP}:32222"。 ![](https://support.huaweicloud.com/usermanual-cce/zh-cn_image_0000002390623404.png "点击放大") #### 步骤五：安装相关插件 1. 在集群中安装云原生监控插件和Grafana。 1. 安装云原生监控插件，详情请参见[云原生监控插件](https://support.huaweicloud.com/usermanual-cce/cce_10_0406.html)。 2. 安装Grafana，详情请参见[Grafana](https://support.huaweicloud.com/usermanual-cce/cce_10_0828.html)。 2. 创建hubble-monitor.yaml文件。 内容如下： ``` apiVersion: monitoring.coreos.com/v1 kind: ServiceMonitor metadata: name: hubble namespace: monitoring spec: endpoints: - honorLabels: true path: /metrics port: hubble-metrics relabelings: - action: replace replacement: ${1} sourceLabels: - __meta_kubernetes_pod_node_name targetLabel: node scheme: http jobLabel: hubble namespaceSelector: matchNames: - kube-system selector: matchLabels: app: hubble ``` 3. 创建监控任务。 ``` kubectl create -f hubble-monitor.yaml ``` #### 步骤六：配置Hubble指标看板 Cilium官网提供了Hubble指标的Grafana看板，相比Prometheus单调的指标数据，Grafana看板更加美观。您可以导入相关看板进行查看。 1. 准备Hubble指标看板的配置文件，打开Cilium开源社区的[Prometheus样例文件](https://raw.githubusercontent.com/cilium/cilium/v1.19.1/examples/kubernetes/addons/prometheus/monitoring-example.yaml)，找到文件中的hubble-dashboard.json并复制。 2. 登录Grafana，在Grafana中选择Dashboards，导入获取的hubble-dashboard.json文件。 ![](https://support.huaweicloud.com/usermanual-cce/public_sys-resources/caution_3.0-zh-cn.png) 如果您在[步骤五：安装相关插件]中安装Grafana插件时启用了"数据源对接AOM"，请将hubble-dashboard.json文件中的**datasource**字段修改为prometheus-aom，然后再进行导入。 3. 您可以通过看板查看Hubble指标，Hubble指标的说明详见[hubble-metrics](https://docs.cilium.io/en/v1.19/observability/metrics/#hubble-metrics)。 ![](https://support.huaweicloud.com/usermanual-cce/zh-cn_image_0000002424303849.png "点击放大")