更新时间:2024-06-13 GMT+08:00
分享

打包到package目录

Helm模板或者operator包有两个修改点:

  • 标记服务的来源,即是来源于开源、华为或者生态伙伴;
  • 统一镜像地址,OSC格式包引用lifecycle.yaml及csd.yaml中配置,Helm模板引用values.yaml中的配置,operator引用*.clusterserviceversion.yaml中的配置。

提供商上传服务包后,镜像会保存到OSC的仓库中,提供商在上传前无法获取镜像的实际保存地址,OSC服务会将value.yaml或者*.clusterserviceversion.yaml中默认的镜像地址替换成实际的镜像地址,其它地方的配置不予替换,因此如果其它地方没有引用lifecycle.yaml、csd.yaml、values.yaml或者*.clusterserviceversion.yaml中的配置,在部署服务实例时,拉取镜像会失败。

标识服务来源

OSC提供来自开源、华为自研以及生态伙伴的服务,生态伙伴需要在服务包中固定字段进行标记说明服务来自生态伙伴。

  • operator服务
    以etcd operator为例,etcdoperator.v0.9.4.clusterserviceversion.yaml的内容为:
    apiVersion: operators.coreos.com/v1alpha1
    kind: ClusterServiceVersion
    metadata:
      annotations:
        capabilities: Full Lifecycle
        categories: Database
        containerImage: quay.io/coreos/etcd-operator@sha256:66a37fd61a06a43969854ee6d3e21087a98b93838e284a6086b13917f96b0d9b
        createdAt: 2019-02-28 01:03:00
        description: Create and maintain highly-available etcd clusters on Kubernetes
        repository: https://github.com/coreos/etcd-operator
        tectonic-visibility: ocs
      name: etcdoperator.v0.9.4
      namespace: placeholder
    ...

  1. 编辑文件。

    在metadata/annotations添加source字段,取值为ISV。

    apiVersion: operators.coreos.com/v1alpha1
    kind: ClusterServiceVersion
    metadata:
      annotations:
        source: ISV
        capabilities: Full Lifecycle
        categories: Database
        containerImage: quay.io/coreos/etcd-operator@sha256:66a37fd61a06a43969854ee6d3e21087a98b93838e284a6086b13917f96b0d9b
        createdAt: 2019-02-28 01:03:00
        description: Create and maintain highly-available etcd clusters on Kubernetes
        repository: https://github.com/coreos/etcd-operator
        tectonic-visibility: ocs
      name: etcdoperator.v0.9.4
      namespace: placeholder
    ...

  2. 打包到package目录。

    把operator包压缩成zip格式,放至package目录下。

    package目录下只能包含一个Operator压缩包,确保使用这个包可以将整个服务部署起来。

    etcd/
        | --- package/
            | --- etcd-operator-0.9.4.zip
        | --- images/
            | --- etcd-3.5.0-x86_64.tar
            | --- etcd-3.5.0-aarch64.tar
        | --- extends/

  • Helm服务
    以etcd helm为例,Chart.yaml的内容为:
    annotations:
      category: Database
    apiVersion: v2
    appVersion: 3.4.14
    dependencies:
      - name: common
        repository: https://charts.bitnami.com/bitnami
        tags:
          - bitnami-common
        version: 1.x.x
    description: etcd is a distributed key value store that provides a reliable way to store data across a cluster of machines
    ...
  1. 编辑Chart.yaml文件。

    如果有annotations属性,则添加子属性source,取值为ISV,如果没有annotations属性,则先添加annotations属性,再添加子属性source。

    annotations:
      source: ISV
      category: Database
    apiVersion: v2
    appVersion: 3.4.14
    dependencies:
      - name: common
        repository: https://charts.bitnami.com/bitnami
        tags:
          - bitnami-common
        version: 1.x.x
    description: etcd is a distributed key value store that provides a reliable way to store data across a cluster of machines
    ...

  2. 打包放置到package目录:

    etcd/
        | --- package/
            | --- etcd-helm-6.7.0.tgz
        | --- images/
            | --- etcd-3.5.0-x86_64.tar
            | --- etcd-3.5.0-aarch64.tar
            | --- mapping.yaml

    package目录下只能包含一个Helm模板包,确保使用这个包可以将整个服务部署起来。

排查和编辑镜像地址

提供商上传服务包后,镜像会保存到OSC的仓库中,提供商在上传前无法知道镜像的实际保存地址,OSC在提供商上传服务后只把value.yaml或者*.clusterserviceversion.yaml中配置的镜像地址替换成实际的镜像地址。

提供商需排查Helm模板或者Operator服务包中的其它镜像地址都是引用values.yaml或者*.clusterserviceversion.yaml中的配置。如果满足,可直接查看替换镜像地址配置

以etcd helm为例,Helm模板目录结构如下所示:

etcd/
    | --- templates/
        | --- secrets.yaml
        | --- servicemonitor.yaml
        | --- snapshot-pvc.yaml
        | --- statefulset.yaml
        | --- svc-headless.yaml
        | --- svc.yaml
    | --- Chart.lock
    | --- Chart.yaml
    | --- README.md
    | --- values.yaml

其中values.yaml文件内容是:

image:
  registry: docker.io
  repository: bitnami/etcd
  tag: 3.4.14-debian-10-r44
  debug: false
volumePermissions:
  enabled: false
  image:
    registry: docker.io
    repository: bitnami/minideb
    tag: buster
    pullPolicy: Always
  resources:
    limits: {}
    #   cpu: 100m
    #   memory: 128Mi
    requests: {}
    #   cpu: 100m
    #   memory: 128Mi
...

templates/statefulset.yaml中引用了values.yaml配置镜像地址的变量:

containers:
  - name: etcd-snapshotter
    image: {{ include "etcd.image" . }}
    imagePullPolicy: {{ .Values.image.pullPolicy | quote }}

服务包上传后实际的镜像地址是swr.cn-east-3.myhuaweicloud.com/osc-opensource/etcd:3.4.14-debian-10-r44,OSC会自动替换服务包中values.yaml中的镜像地址,保证服务部署时能从OSC的仓库中拉取镜像,values.yaml会被替换成:

image:
  registry: swr.cn-east-3.myhuaweicloud.com
  repository: osc-opensource/etcd
  tag: 3.4.14-debian-10-r44

因为Helm模板中其它镜像地址是引用values.yaml的配置,因此OSC替换了values.yaml后能保证部署时拉取到正确的镜像。

对于Operator服务,请排查服务包中的镜像地址都是引用*.clusterserviceversion.yaml中的配置。

配置实例版本定义信息(可选)

OSC提供配置实例版本定义信息以支持实例升级的能力。

  • Operator类型实例
    以redis为例, redis的cr内容如下所示:
    apiVersion: redis.osc/v1
    kind: Redis
    metadata:
      annotations:
        osc.huawei.com/package-source: public
        osc.io/occupied-port: 135,139
      creationTimestamp: '2021-09-29T03:21:57Z'
      finalizers:
        - storage.finalizers.redis.cluster
      generation: 2
      name: redis-fwpydh
      namespace: default
      resourceVersion: '91743432'
      selfLink: /apis/redis.osc/v1/namespaces/default/redises/redis-fwpydh
      uid: 96f0203c-0ae0-48bb-b2b8-b08d2055b0e2
    spec:
      config:
        name: default-redis-fwpydh-unvu7g
        properties:
          aof-load-truncated: 'yes'
          aof-use-rdb-preamble: 'no'
          appendfsync: everysec
          appendonly: 'no'
          hash-max-ziplist-entries: 512
          hash-max-ziplist-value: 64
          latency-monitor-threshold: 100
          list-max-ziplist-size: -2
          loglevel: notice
          maxauthfailtimes: 100
          maxclients: 10000
          maxmemory-policy: noeviction
          repl-diskless-sync: 'yes'
          set-max-intset-entries: 512
          slowlog-log-slower-than: 10000
          stop-writes-on-bgsave-error: 'no'
          timeout: 0
          zset-max-ziplist-entries: 128
          zset-max-ziplist-value: 64
      image: swr.cn-east-3.myhuaweicloud.com/osc-official/redis:21.9.18_20210918221431
      masterSize: 1
      mode: RedisHA
      ......
      ......
      此处中间省略
      ......
      ......
      phase: Available
      serviceAddr: redis-ha-redis-fwpydh.default.svc.cluster.local:6379
      serviceAddrReadonly: redis-ha-redis-fwpydh-readonly.default.svc.cluster.local:6379
      version: 21.9.18_20210918221431

  1. 编辑文件

    实例版本定义配置是在csd文件中配置versionDefinition,实例版本定义支持operator类型和Helm类型。

    versionDefinition:
      mode: url
      path: spec.image
      tags:    
        - 0.0.1    
        - 0.0.2    
        - 0.0.3    

  1. 打包到package目录。

    把operator包压缩成zip格式的压缩包,放到package目录下。

  • Helm服务

    以clickhouse helm为例,values.yaml的内容如下:

    ## Timezone
    timezone: "Asia/Shanghai"
    
    ## Cluster domain
    clusterDomain: "cluster.local"
    
    ##
    ## Clickhouse Node selectors and tolerations for pod assignment
    ## ref: https://kubernetes.io/docs/concepts/configuration/assign-pod-node/#nodeselector
    ## ref: https://kubernetes.io/docs/concepts/configuration/assign-pod-node/#taints-and-tolerations-beta-feature
    ##
    # nodeSelector: {"beta.kubernetes.io/arch": "amd64"}
    # tolerations: []
    ## Clickhouse pod/node affinity/anti-affinity
    ## 
    #affinity:
    #  nodeAffinity:
    #    requiredDuringSchedulingIgnoredDuringExecution:
    #      nodeSelectorTerms:
    #      - matchExpressions:
    #        - key: "application/clickhouse"
    #          operator: In
    #          values:
    #          - "true"
    
    clickhouse:
      ## StatefulSet controller supports relax its ordering guarantees while preserving its uniqueness and identity guarantees. There are two valid pod management policies: OrderedReady and Parallel
      ## ref: https://kubernetes.io/docs/tutorials/stateful-application/basic-stateful-set/#pod-management-policy
      ##
      podManagementPolicy: "Parallel"
    
      ## StatefulSet controller supports automated updates. There are two valid update strategies: RollingUpdate and OnDelete
      ## ref: https://kubernetes.io/docs/tutorials/stateful-application/basic-stateful-set/#updating-statefulsets
      ##
      updateStrategy: "RollingUpdate"
    
      ## Partition update strategy
      ## https://kubernetes.io/docs/concepts/workloads/controllers/statefulset/#partitions
      ##
      # rollingUpdatePartition:
    
      ##
      ## The path to the directory containing data.
      ## Default value: /var/lib/clickhouse
      path: "/var/lib/clickhouse"
      ##
      ## The port for connecting to the server over HTTP
      http_port: "8123"
      ##
      ## Port for communicating with clients over the TCP protocol.
      tcp_port: "9000"
      ##
      ## Port for exchanging data between ClickHouse servers.
      interserver_http_port: "9009"
      ## 
      ## The instance number of Clickhouse
      replicas: "3"
      ## Clickhouse image configuration.
      image: "swr.cn-east-3.myhuaweicloud.com/osctest/clickhouse-server"
      imageVersion: "0.0.1"
      imagePullPolicy: "IfNotPresent"
      imageBusybox: "swr.cn-north-7.myhuaweicloud.com/osctest/busybox:1.26.2"
      #imagePullSecrets: 
      ## Periodic probe of container liveness. Container will be restarted if the probe fails. Cannot be updated. 
      ## More info: https://kubernetes.io/docs/concepts/workloads/pods/pod-lifecycle#container-probes
      livenessProbe:
        enabled: true
        initialDelaySeconds: "30"
        periodSeconds: "30"
        timeoutSeconds: "5"
        failureThreshold: "3"
        successThreshold: "1"
      ## Periodic probe of container service readiness. Container will be removed from service endpoints if the probe fails. Cannot be updated. 
      ## More info: https://kubernetes.io/docs/concepts/workloads/pods/pod-lifecycle#container-probes
      readinessProbe:
        enabled: true
        initialDelaySeconds: "30"
        periodSeconds: "30"
        timeoutSeconds: "5"
        failureThreshold: "3"
        successThreshold: "1"
      ## volumeClaimTemplates is a list of claims that pods are allowed to reference. 
      ## The StatefulSet controller is responsible for mapping network identities to claims in a way that maintains the identity of a pod. 
      ## Every claim in this list must have at least one matching (by name) volumeMount in one container in the template. 
      ## A claim in this list takes precedence over any volumes in the template, with the same name.
      persistentVolumeClaim:
        enabled: false
        ## Clickhouse data volume
        dataPersistentVolume: 
          enabled: false
          accessModes: 
          - "ReadWriteOnce"
          storageClassName: "-"
          storage: "500Gi"
        ## Clickhouse logs volume
        logsPersistentVolume:
          enabled: false
          accessModes: 
          - "ReadWriteOnce"
          storageClassName: "csi-disk"
          storage: "50Gi"
      ##
      ## An API object that manages external access to the services in a cluster, typically HTTP.
      ## Ingress can provide load balancing, SSL termination and name-based virtual hosting.
      ingress: 
        enabled: false
      #  host: "clickhouse.domain.com"
      #  path: "/"
      #  tls: 
      #    enabled: false
      #    hosts: 
      #    - "clickhouse.domain.com"
      #    - "clickhouse.domain1.com"
      #    secretName: "clickhouse-secret"
      ## 
      ## Clickhouse config.xml and metrica.xml
      ......
      ......
      后续省略
      ......
      ......
  1. 编辑文件。

    实例版本定义配置是在csd文件中配置versionDefinition,实例版本定义支持operator类型和Helm类型。

    versionDefinition:
      mode: tag
      path: clickhouse.imageVersion
      tags:    
        - 0.0.1    
        - 0.0.2    
        - 0.0.3    

  1. 打包到package目录。

    把Helm包压缩成zip格式的压缩包,放到package目录下。

相关文档