# 应用服务部署及配置 #### 镜像上传 1. 登录华为云[SWR管理控制台](https://console.huaweicloud.com/rds/)，进入组织管理，单击创建组织 图1创建组织 ![](https://support.huaweicloud.com/uomsp-government/zh-cn_image_0000001739845353.png) 组织名称：xxxx 图2组织名称1 ![](https://support.huaweicloud.com/uomsp-government/zh-cn_image_0000001692006416.png) 图3组织名称2 ![](https://support.huaweicloud.com/uomsp-government/zh-cn_image_0000001691846672.png) 2. 登录华为云[SWR管理控制台](https://console.huaweicloud.com/rds/)，进入我的镜像-\>自有镜像，单击客户端上传 图4上传 ![](https://support.huaweicloud.com/uomsp-government/zh-cn_image_0000001739965213.png) 单击"生成临时登录指令" 图5生成临时登录指令 ![](https://support.huaweicloud.com/uomsp-government/zh-cn_image_0000001739845357.png) 复制登录命令 图6复制登录命令 ![](https://support.huaweicloud.com/uomsp-government/zh-cn_image_0000001692006420.png) 3. 在安装容器引擎的机器中执行上一步复制的登录指令。登录成功会显示"login succeeded"。 图7登录 ![](https://support.huaweicloud.com/uomsp-government/zh-cn_image_0000001691846676.png) 4. 在安装容器引擎的机器给镜像打标签 ``` docker pull [{镜像名称}:{版本名称}] docker tag [{镜像名称}:{版本名称}] swr.cn-east-3.myhuaweicloud.com/{组织名称}/{镜像名称}:{版本名称} ``` 样例如下： ``` docker pull hub.egova.com.cn/microservice/eurban-mis-frontend-library:20201201-3.9.12-microservice-t04251036 docker tag hub.egova.com.cn/microservice/eurban-mis-frontend-library:20201201-3.9.12-microservice-t04251036 swr.cn-east-3.myhuaweicloud.com/egova_eurban/eurban-mis-frontend-library:20201201-3.9.12-microservice-t04251036 ``` 图8打标签 ![](https://support.huaweicloud.com/uomsp-government/zh-cn_image_0000001739965217.png) 5. 上传镜像至镜像仓库 ``` docker push [镜像仓库地址]/[组织名称]/[镜像名称:版本名称] ``` 样例如下： ``` docker push swr.cn-east-3.myhuaweicloud.com/egova_eurban/eurban-mis-frontend-library:20201201-3.9.12-microservice-t04251036 ``` 6. 登录华为云[SWR管理控制台](https://console.huaweicloud.com/rds/)，进入我的镜像-\>自有镜像，查看镜像已上传 图9查看镜像 ![](https://support.huaweicloud.com/uomsp-government/zh-cn_image_0000001739845361.png) **安装集群** 以下安装步骤仅供参考。 建议：安装前对默认配置进行确认，配置文件位于/etc/ansible/hosts 主要确认的配置有： ![](https://support.huaweicloud.com/uomsp-government/public_sys-resources/caution_3.0-zh-cn.png) 集群网络参数：如果现场服务器用到了10.88以及172.88这两个网段，请将此处修改为其他不重复的网段 图10确认配置1 ![](https://support.huaweicloud.com/uomsp-government/zh-cn_image_0000001692006424.png) 服务器部署app参数：如果服务器数量较多(10台左右，如下三个参数请使用默认值no) 图11确认配置2 ![](https://support.huaweicloud.com/uomsp-government/zh-cn_image_0000001691846680.png) 日志采集与监控相关参数（请参考后续章节，设置smtp邮件服务参数）。另外，如果服务器资源比较紧张，可以不启用日志采集服务和监控服务（这两个服务耗费内存较多） 图12确认配置3 ![](https://support.huaweicloud.com/uomsp-government/zh-cn_image_0000001739965221.png) 服务器较多(10+)，且资源充足（cpu,内存足够）时，可以设置多主+多etcd方式，保证集群关键服务的稳定性。其中etcd需要为奇数个 图13确认配置4 ![](https://support.huaweicloud.com/uomsp-government/zh-cn_image_0000001739845365.png) 安装集群有两种方式，一种是运行4，直接一键全部安装。另一种是运行41，逐步进行安装。 图14安装方式 ![](https://support.huaweicloud.com/uomsp-government/zh-cn_image_0000001692006428.png) 以41逐步安装进行说明，安装时需要注意哪些细节 图1541 ![](https://support.huaweicloud.com/uomsp-government/zh-cn_image_0000001691846684.png) - 所有分步步骤均可重复运行 - 每步运行后，不允许出现failed的服务器（failed且非ignored的步骤，需要对安装日志进行分析，解决后，再次运行该步骤） 正常结果如下： 图16结果 ![](https://support.huaweicloud.com/uomsp-government/zh-cn_image_0000001739965225.png) 异常结果如下： 图17异常结果 ![](https://support.huaweicloud.com/uomsp-government/zh-cn_image_0000001739845369.png) - 离线镜像约有6GB，导入耗时较多（10分钟级别），请耐心等待 图18离线镜像 ![](https://support.huaweicloud.com/uomsp-government/zh-cn_image_0000001692006432.png) 集群安装成功后： 通过kubectl get node可以查看集群所有节点 图19查看集群所有节点 ![](https://support.huaweicloud.com/uomsp-government/zh-cn_image_0000001691846688.png) 城管服务及集群维护服务都是以pod容器组方式运行在集群中，通过get pod -A可查看所有的pod 图20查看所有的pod ![](https://support.huaweicloud.com/uomsp-government/zh-cn_image_0000001739965229.png) #### 应用服务部署 1. 创建存储卷 进入云容器引擎下的"资源管理 \> 存储管理"，根据应用服务需求创建所需存储卷。 图21创建存储卷 ![](https://support.huaweicloud.com/uomsp-government/zh-cn_image_0000001739845373.png) 单击"创建文件存储卷"，按需申请容量大小 图22创建存储卷 ![](https://support.huaweicloud.com/uomsp-government/zh-cn_image_0000001692006436.png) 需要申请的存储卷如下： - geoserver：用于存储geoserver数据，主要是data目录的数据，多个geoserver服务统一共享使用同一个pv数据目录，容量大小200G即可。 - gis：用于存储eUrbanGIS数据，主要是data/data3d/tilemap三个目录下的数据，多个gis服务统一共享使用同一个pv数据目录，容量大小200G即可。 - media：用于存储MediaRoot数据，可同时给eurban-fileserver服务（多媒体上传接口）和eurban-media服务（多媒体发布服务）使用，容量大小200G即可。 - public：主要用于挂载微信证书文件，约定目录为/etc/public/data，请确保wechat.properties使用此目录，容量大小为10G即可。 - postgres：用于postgrep数据库物理存储，容量大小100G即可。 2. Service定义 拿eurban-mis-backend服务为例，定义如下： ``` Name: eurban-mis-backend Namespace: prod Labels: app=eurban-mis-backend Annotations: asm.huaweicloud.com/put: {"kind":"Service","apiVersion":"v1","metadata":{"name":"eurban-mis-backend","namespace":"prod","selfLink":"/api/v1/namespaces/prod/service... asm.huaweicloud.com/updateTimestamp: 2021-04-25T09:56:38Z kubectl.kubernetes.io/last-applied-configuration: {"apiVersion":"v1","kind":"Service","metadata":{"annotations":{"prometheus.io/jvm-path":"/metrics","prometheus.io/jvm-port":"1234","promet... prometheus.io/jvm-path: /metrics prometheus.io/jvm-port: 1234 prometheus.io/jvm-scrape: true prometheus.io/scrape: true Selector: app=eurban-mis-backend Type: ClusterIP IP: 10.247.194.115 Port: http-tomcat 8080/TCP TargetPort: 8080/TCP Endpoints: 10.0.2.18:8080 Port: http-prometheus 1234/TCP TargetPort: 1234/TCP Endpoints: 10.0.2.18:1234 Session Affinity: None Events: ``` 其他Service定义可根据以下命令进行查看 图23查看 ![](https://support.huaweicloud.com/uomsp-government/zh-cn_image_0000001691846692.png) 3. Pod定义 拿eurban-mis-backend为例，定义如下： ``` apiVersion: apps/v1 kind: Deployment metadata: annotations: deployment.kubernetes.io/revision: "3" kubectl.kubernetes.io/last-applied-configuration: | creationTimestamp: "2021-05-08T06:22:15Z" generation: 3 labels: app: eurban-mis-backend name: eurban-mis-backend namespace: prod resourceVersion: "31294043" selfLink: /apis/apps/v1/namespaces/prod/deployments/eurban-mis-backend uid: 34de909e-1187-4faf-b7cd-5156eb505737 spec: progressDeadlineSeconds: 600 replicas: 1 revisionHistoryLimit: 10 selector: matchLabels: app: eurban-mis-backend version: v1 strategy: rollingUpdate: maxSurge: 25% maxUnavailable: 25% type: RollingUpdate template: metadata: creationTimestamp: null labels: app: eurban-mis-backend app-group: eurban-mis-backend version: v1 spec: affinity: nodeAffinity: requiredDuringSchedulingIgnoredDuringExecution: nodeSelectorTerms: - matchExpressions: - key: app.egova.com.cn/common operator: In values: - "true" - key: namespace.egova.com.cn/prod operator: In values: - "true" - key: app.egova.com.cn/tomcat-large operator: NotIn values: - "false" podAntiAffinity: preferredDuringSchedulingIgnoredDuringExecution: - podAffinityTerm: labelSelector: matchExpressions: - key: app-group operator: In values: - eurban-mis-backend topologyKey: kubernetes.io/hostname weight: 10 containers: - env: - name: __ENV__MYSQL_SERVER_BIZ_HOST valueFrom: configMapKeyRef: key: db_host_biz name: global-configmap - name: __ENV__MYSQL_SERVER_BIZ_PORT valueFrom: configMapKeyRef: key: db_port_biz name: global-configmap - name: __ENV__DB_BIZ_USER valueFrom: secretKeyRef: key: db_user_biz name: global-secret - name: __ENV__DB_BIZ_PASSWORD valueFrom: secretKeyRef: key: db_passwd_biz name: global-secret - name: __ENV__DB_BIZ_NAME valueFrom: configMapKeyRef: key: db_name_biz name: global-configmap - name: __ENV__MYSQL_SERVER_STAT_HOST valueFrom: configMapKeyRef: key: db_host_stat name: global-configmap - name: __ENV__MYSQL_SERVER_STAT_PORT valueFrom: configMapKeyRef: key: db_port_stat name: global-configmap - name: __ENV__DB_STAT_USER valueFrom: secretKeyRef: key: db_user_stat name: global-secret - name: __ENV__DB_STAT_PASSWORD valueFrom: secretKeyRef: key: db_passwd_stat name: global-secret - name: __ENV__DB_STAT_NAME valueFrom: configMapKeyRef: key: db_name_stat name: global-configmap - name: __ENV__ZOOKEEPER_HOST value: zookeeper.prod - name: __ENV__ZOOKEEPER_PORT value: "2181" - name: __ENV__REDIS_SERVER_HOST valueFrom: configMapKeyRef: key: redis_host name: global-configmap - name: __ENV__REDIS_SERVER_PORT valueFrom: configMapKeyRef: key: redis_port name: global-configmap - name: __ENV__REDIS_SERVER_PASSWORD valueFrom: secretKeyRef: key: redis_passwd name: global-secret - name: __ENV__REDIS_NAMESPACE value: egova - name: __ENV__POSTGRES_SERVICE_HOST valueFrom: configMapKeyRef: key: pg_host name: global-configmap - name: __ENV__POSTGRES_SERVICE_PORT valueFrom: configMapKeyRef: key: pg_port name: global-configmap - name: __ENV__POSTGRES_SERVICE_USER valueFrom: secretKeyRef: key: pg_user name: global-secret valueFrom: configMapKeyRef: key: pg_name name: global-configmap - name: USE_MYSQL_CONNECTOR_PATCH value: "1" - name: REDIS_SESSION_ENABLED value: "1" - name: __ENV__WITH_JOB_MANAGER value: "0" - name: __ENV__CONTEXT_PATH value: eUrbanMIS - name: POD_NAME valueFrom: fieldRef: apiVersion: v1 fieldPath: metadata.name - name: JAVA_OPTS image: swr.cn-east-3.myhuaweicloud.com/egova_eurban/tomcat-redis:2.3 imagePullPolicy: IfNotPresent lifecycle: postStart: exec: command: - bash - -c - | set -ex echo $(date +%s) > /etc/inited preStop: exec: command: - bash - -c - | set -ex rm -f /etc/started livenessProbe: failureThreshold: 12 httpGet: path: /eUrbanMIS/main.htm port: 8080 scheme: HTTP initialDelaySeconds: 900 periodSeconds: 10 successThreshold: 1 timeoutSeconds: 30 name: tomcat ports: - containerPort: 8080 protocol: TCP - containerPort: 1234 protocol: TCP readinessProbe: failureThreshold: 12 httpGet: path: /eUrbanMIS/main.htm port: 8080 scheme: HTTP initialDelaySeconds: 100 periodSeconds: 10 successThreshold: 1 timeoutSeconds: 10 resources: limits: cpu: "3" ephemeral-storage: 2Gi memory: 4200Mi requests: cpu: 200m ephemeral-storage: 1Gi memory: 2096Mi terminationMessagePath: /dev/termination-log terminationMessagePolicy: File volumeMounts: - mountPath: /usr/local/tomcat/logs name: egova-log-tomcat subPathExpr: $(POD_NAME) - mountPath: /etc/localtime name: date-config - mountPath: /usr/local/tomcat/webapps/eUrbanMIS/WEB-INF/log name: egova-log-web subPathExpr: $(POD_NAME) - mountPath: /prometheus name: prom - mountPath: /docker-entrypoint-init.d/after-prepare-configmaps/ name: code-init-configmap - mountPath: /usr/local/tomcat/webapps name: code-volume - mountPath: /etc/config name: mis-configmap-volume - mountPath: /etc/sidecar/ name: sidecar-volume - image: swr.cn-east-3.myhuaweicloud.com/egova_eurban/tool-inotify:1.0 imagePullPolicy: IfNotPresent limits: cpu: 100m memory: 50Mi requests: cpu: 10m memory: 10Mi terminationMessagePath: /dev/termination-log terminationMessagePolicy: File volumeMounts: - mountPath: /etc/config name: mis-configmap-volume - mountPath: /etc/sidecar/ name: sidecar-volume - mountPath: /usr/local/tomcat/webapps name: code-volume dnsPolicy: ClusterFirst initContainers: - command: - bash - -c - | set -ex echo "数据库检查通过！" env: - name: DB_HOST_BIZ valueFrom: configMapKeyRef: key: db_host_biz name: global-configmap - name: DB_PORT_BIZ valueFrom: configMapKeyRef: key: db_port_biz name: global-configmap - name: DB_NAME_BIZ valueFrom: configMapKeyRef: key: db_name_biz name: global-configmap - name: DB_USER_BIZ valueFrom: secretKeyRef: key: db_user_biz valueFrom: secretKeyRef: key: db_passwd_biz name: global-secret image: swr.cn-east-3.myhuaweicloud.com/egova_eurban/mysql-client:1.1 imagePullPolicy: IfNotPresent name: check resources: limits: cpu: 100m memory: 50Mi requests: cpu: 10m memory: 10Mi terminationMessagePath: /dev/termination-log terminationMessagePolicy: File - command: - bash - -c - | set -ex imagePullPolicy: IfNotPresent name: init-monitor resources: limits: cpu: 100m memory: 50Mi requests: cpu: 10m memory: 10Mi terminationMessagePath: /dev/termination-log terminationMessagePolicy: File volumeMounts: - mountPath: /etc/prometheus name: prom - command: - bash - -c - | set -ex mkdir -p /code/webapps cp -r /usr/local/tomcat/webapps/* /code/webapps/ imagePullPolicy: IfNotPresent name: main resources: limits: cpu: 100m memory: 50Mi requests: cpu: 10m memory: 10Mi terminationMessagePath: /dev/termination-log terminationMessagePolicy: File volumeMounts: - mountPath: /code/webapps name: code-volume priorityClassName: priority-b restartPolicy: Always schedulerName: default-scheduler securityContext: {} terminationGracePeriodSeconds: 30 volumes: - configMap: defaultMode: 420 name: eurban-mis-init name: code-init-configmap - emptyDir: {} name: code-volume - emptyDir: {} name: sidecar-volume - emptyDir: {} name: prom - hostPath: path: /egova/log/tomcat type: DirectoryOrCreate path: /etc/localtime type: "" name: date-config - hostPath: path: /egova/log/web type: DirectoryOrCreate name: egova-log-web - configMap: defaultMode: 420 name: eurban-mis-config name: mis-configmap-volume status: availableReplicas: 1 conditions: - lastTransitionTime: "2021-05-08T09:25:30Z" lastUpdateTime: "2021-05-08T09:25:30Z" message: Deployment has minimum availability. reason: MinimumReplicasAvailable status: "True" type: Available - lastTransitionTime: "2021-05-08T06:22:15Z" lastUpdateTime: "2021-06-04T06:01:41Z" message: ReplicaSet "eurban-mis-backend-7db7d7574" has successfully progresss ed. reason: NewReplicaSetAvailable status: "True" type: Progressing observedGeneration: 3 readyReplicas: 1 replicas: 1 updatedReplicas: 1 ``` 其他的pod定义可根据以下命令进行查看： 图24查看 ![](https://support.huaweicloud.com/uomsp-government/zh-cn_image_0000001739965237.png) 4. 部署 定义好Service和Pod之后，可根据以下命令进行部署 图25部署 ![](https://support.huaweicloud.com/uomsp-government/zh-cn_image_0000001739845377.png) 根据以下命令查看服务启动情况 图26查看服务启动情况 ![](https://support.huaweicloud.com/uomsp-government/zh-cn_image_0000001692006440.png) #### 网络配置 - 安全组规则配置 1. 登录华为云，进入网控制台 2. 在网络控制台左侧菜单栏选择"访问控制"-"安全组"，单击安全组的配置规则 图27规则配置 ![](https://support.huaweicloud.com/uomsp-government/zh-cn_image_0000001691846696.png) 3. 单击入方向规则，单击添加规则，设置允许内网互通（[3.1.1章节](https://support.huaweicloud.com/uomsp-government/uomsp_05.html)创建的子网），目的是让该子网下的云服务资源能够访问到被该安全组规则保护的资源。 图28添加规则 ![](https://support.huaweicloud.com/uomsp-government/zh-cn_image_0000001739965241.png) ![](https://support.huaweicloud.com/uomsp-government/public_sys-resources/note_3.0-zh-cn.png) 上述源地址10.20.0.0/24用到地址在这里查看（注意看自己在[VPC\&安全组](https://support.huaweicloud.com/uomsp-government/uomsp_05.html)中创建的VPC和子网是什么网段的，要复制自己的子网网段）： 图29添加规则2 ![](https://support.huaweicloud.com/uomsp-government/zh-cn_image_0000001739845381.png) 4. 单击入方向规则，单击添加规则，设置放通100.125.0.0/16网段（用于ELB后端服务器健康检查） 图30进入方向规则 ![](https://support.huaweicloud.com/uomsp-government/zh-cn_image_0000001692006444.png) 5. 检查一下规则是否都配置好了 图31检查 ![](https://support.huaweicloud.com/uomsp-government/zh-cn_image_0000001691846700.png) - ELB配置 监听器配置 本章节的目的是配置静态nginx服务的监听器，实现：通过ELB公网IP+监听的端口能进入静态nginx前端界面。 1. 配置监听器 图32配置监听器 ![](https://support.huaweicloud.com/uomsp-government/zh-cn_image_0000001739965245.png) 2. 配置后端服务器组 图33配置后端服务器组 ![](https://support.huaweicloud.com/uomsp-government/zh-cn_image_0000001739845385.png) #### 应用服务网格配置 进入到应用服务网格"服务列表"下 图34服务列表 ![](https://support.huaweicloud.com/uomsp-government/zh-cn_image_0000001692006448.png) 单击添加服务，选择命名空间，选择要添加的服务 图35添加服务 ![](https://support.huaweicloud.com/uomsp-government/zh-cn_image_0000001691846704.png) 单击确定，添加成功 图36确定 ![](https://support.huaweicloud.com/uomsp-government/zh-cn_image_0000001739965249.png) #### 数据初始化 由eurban-mis-job服务初始化数据。 图37初始化数据 ![](https://support.huaweicloud.com/uomsp-government/zh-cn_image_0000001739845389.png)