本文导读

展开导读

文档首页/ 政府/ 数字政通城市运行管理服务平台解决方案/ 实施步骤/ 应用服务部署及配置

应用服务部署及配置

更新时间：2024-05-06 GMT+08:00

镜像上传

登录华为云SWR管理控制台，进入组织管理，单击创建组织

图1 创建组织

组织名称：xxxx

图2 组织名称1

图3 组织名称2
登录华为云SWR管理控制台，进入我的镜像->自有镜像，单击客户端上传

图4 上传

单击“生成临时登录指令”

图5 生成临时登录指令

复制登录命令

图6 复制登录命令
在安装容器引擎的机器中执行上一步复制的登录指令。登录成功会显示“login succeeded”。

图7 登录

在安装容器引擎的机器给镜像打标签

docker pull [{镜像名称}:{版本名称}]
docker tag [{镜像名称}:{版本名称}] swr.cn-east-3.myhuaweicloud.com/{组织名称}/{镜像名称}:{版本名称}

样例如下：

docker pull hub.egova.com.cn/microservice/eurban-mis-frontend-library:20201201-3.9.12-microservice-t04251036
docker tag hub.egova.com.cn/microservice/eurban-mis-frontend-library:20201201-3.9.12-microservice-t04251036 swr.cn-east-3.myhuaweicloud.com/egova_eurban/eurban-mis-frontend-library:20201201-3.9.12-microservice-t04251036

图8 打标签

上传镜像至镜像仓库

docker push [镜像仓库地址]/[组织名称]/[镜像名称:版本名称]

样例如下：

docker push swr.cn-east-3.myhuaweicloud.com/egova_eurban/eurban-mis-frontend-library:20201201-3.9.12-microservice-t04251036

登录华为云SWR管理控制台，进入我的镜像->自有镜像，查看镜像已上传

图9 查看镜像

安装集群

以下安装步骤仅供参考。

建议：安装前对默认配置进行确认，配置文件位于/etc/ansible/hosts

主要确认的配置有：

注意：

集群网络参数：如果现场服务器用到了10.88以及172.88这两个网段，请将此处修改为其他不重复的网段

图10 确认配置1

服务器部署app参数：如果服务器数量较多(10台左右，如下三个参数请使用默认值no)

图11 确认配置2

日志采集与监控相关参数（请参考后续章节，设置smtp邮件服务参数）。另外，如果服务器资源比较紧张，可以不启用日志采集服务和监控服务（这两个服务耗费内存较多）

图12 确认配置3

服务器较多(10+)，且资源充足（cpu,内存足够）时，可以设置多主+多etcd方式，保证集群关键服务的稳定性。其中etcd需要为奇数个

图13 确认配置4

安装集群有两种方式，一种是运行4，直接一键全部安装。另一种是运行41，逐步进行安装。

图14 安装方式

以41逐步安装进行说明，安装时需要注意哪些细节

图15 41
- 所有分步步骤均可重复运行
- 每步运行后，不允许出现failed的服务器（failed且非ignored的步骤，需要对安装日志进行分析，解决后，再次运行该步骤）
  正常结果如下：
  
  图16 结果
  
  异常结果如下：
  
  图17 异常结果
- 离线镜像约有6GB，导入耗时较多（10分钟级别），请耐心等待
  图18 离线镜像
  
  集群安装成功后：
  
  通过kubectl get node可以查看集群所有节点
  
  图19 查看集群所有节点
  
  城管服务及集群维护服务都是以pod容器组方式运行在集群中，通过get pod -A可查看所有的pod
  
  图20 查看所有的pod

应用服务部署

创建存储卷
 进入云容器引擎下的“资源管理 > 存储管理”，根据应用服务需求创建所需存储卷。
图21 创建存储卷

单击“创建文件存储卷”，按需申请容量大小

图22 创建存储卷

需要申请的存储卷如下：
- geoserver：用于存储geoserver数据，主要是data目录的数据，多个geoserver服务统一共享使用同一个pv数据目录，容量大小200G即可。
- gis：用于存储eUrbanGIS数据，主要是data/data3d/tilemap三个目录下的数据，多个gis服务统一共享使用同一个pv数据目录，容量大小200G即可。
- media：用于存储MediaRoot数据，可同时给eurban-fileserver服务（多媒体上传接口）和eurban-media服务（多媒体发布服务）使用，容量大小200G即可。
- public：主要用于挂载微信证书文件，约定目录为/etc/public/data，请确保wechat.properties使用此目录，容量大小为10G即可。
- postgres：用于postgrep数据库物理存储，容量大小100G即可。

Service定义

拿eurban-mis-backend服务为例，定义如下：

Name:              eurban-mis-backend
Namespace:         prod
Labels:            app=eurban-mis-backend
Annotations:       asm.huaweicloud.com/put:
{"kind":"Service","apiVersion":"v1","metadata":{"name":"eurban-mis-backend","namespace":"prod","selfLink":"/api/v1/namespaces/prod/service...
asm.huaweicloud.com/updateTimestamp: 2021-04-25T09:56:38Z
kubectl.kubernetes.io/last-applied-configuration:
{"apiVersion":"v1","kind":"Service","metadata":{"annotations":{"prometheus.io/jvm-path":"/metrics","prometheus.io/jvm-port":"1234","promet...
prometheus.io/jvm-path: /metrics
prometheus.io/jvm-port: 1234
prometheus.io/jvm-scrape: true
prometheus.io/scrape: true
Selector:          app=eurban-mis-backend
Type:              ClusterIP
IP:                10.247.194.115
Port:              http-tomcat  8080/TCP
TargetPort:        8080/TCP
Endpoints:         10.0.2.18:8080
Port:              http-prometheus  1234/TCP
TargetPort:        1234/TCP
Endpoints:         10.0.2.18:1234
Session Affinity:  None
Events:            <none>

其他Service定义可根据以下命令进行查看

图23 查看

Pod定义

拿eurban-mis-backend为例，定义如下：

apiVersion: apps/v1
kind: Deployment
metadata:
annotations:
deployment.kubernetes.io/revision: "3"
kubectl.kubernetes.io/last-applied-configuration: |
creationTimestamp: "2021-05-08T06:22:15Z"
generation: 3
labels:
app: eurban-mis-backend
name: eurban-mis-backend
namespace: prod
resourceVersion: "31294043"
selfLink: /apis/apps/v1/namespaces/prod/deployments/eurban-mis-backend
uid: 34de909e-1187-4faf-b7cd-5156eb505737
spec:
progressDeadlineSeconds: 600
replicas: 1
revisionHistoryLimit: 10
selector:
matchLabels:
app: eurban-mis-backend
version: v1
strategy:
rollingUpdate:
maxSurge: 25%
maxUnavailable: 25%
type: RollingUpdate
template:
metadata:
creationTimestamp: null
labels:
app: eurban-mis-backend
app-group: eurban-mis-backend
version: v1
spec:
affinity:
nodeAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
nodeSelectorTerms:
- matchExpressions:
- key: app.egova.com.cn/common
operator: In
values:
- "true"
- key: namespace.egova.com.cn/prod
operator: In
values:
- "true"
- key: app.egova.com.cn/tomcat-large
operator: NotIn
values:
- "false"
podAntiAffinity:
preferredDuringSchedulingIgnoredDuringExecution:
- podAffinityTerm:
labelSelector:
matchExpressions:
- key: app-group
operator: In
values:
- eurban-mis-backend
topologyKey: kubernetes.io/hostname
weight: 10
containers:
- env:
- name: __ENV__MYSQL_SERVER_BIZ_HOST
valueFrom:
configMapKeyRef:
key: db_host_biz
name: global-configmap
- name: __ENV__MYSQL_SERVER_BIZ_PORT
valueFrom:
configMapKeyRef:
key: db_port_biz
name: global-configmap
- name: __ENV__DB_BIZ_USER
valueFrom:
secretKeyRef:
key: db_user_biz
name: global-secret
- name: __ENV__DB_BIZ_PASSWORD
valueFrom:
secretKeyRef:
key: db_passwd_biz
name: global-secret
- name: __ENV__DB_BIZ_NAME
valueFrom:
configMapKeyRef:
key: db_name_biz
name: global-configmap
- name: __ENV__MYSQL_SERVER_STAT_HOST
valueFrom:
configMapKeyRef:
key: db_host_stat
name: global-configmap
- name: __ENV__MYSQL_SERVER_STAT_PORT
valueFrom:
configMapKeyRef:
key: db_port_stat
name: global-configmap
- name: __ENV__DB_STAT_USER
valueFrom:
secretKeyRef:
key: db_user_stat
name: global-secret
- name: __ENV__DB_STAT_PASSWORD
valueFrom:
secretKeyRef:
key: db_passwd_stat
name: global-secret
- name: __ENV__DB_STAT_NAME
valueFrom:
configMapKeyRef:
key: db_name_stat
name: global-configmap
- name: __ENV__ZOOKEEPER_HOST
value: zookeeper.prod
- name: __ENV__ZOOKEEPER_PORT
value: "2181"
- name: __ENV__REDIS_SERVER_HOST
valueFrom:
configMapKeyRef:
key: redis_host
name: global-configmap
- name: __ENV__REDIS_SERVER_PORT
valueFrom:
configMapKeyRef:
key: redis_port
name: global-configmap
- name: __ENV__REDIS_SERVER_PASSWORD
valueFrom:
secretKeyRef:
key: redis_passwd
name: global-secret
- name: __ENV__REDIS_NAMESPACE
value: egova
- name: __ENV__POSTGRES_SERVICE_HOST
valueFrom:
configMapKeyRef:
key: pg_host
name: global-configmap
- name: __ENV__POSTGRES_SERVICE_PORT
valueFrom:
configMapKeyRef:
key: pg_port
name: global-configmap
- name: __ENV__POSTGRES_SERVICE_USER
valueFrom:
secretKeyRef:
key: pg_user
name: global-secret
valueFrom:
configMapKeyRef:
key: pg_name
name: global-configmap
- name: USE_MYSQL_CONNECTOR_PATCH
value: "1"
- name: REDIS_SESSION_ENABLED
value: "1"
- name: __ENV__WITH_JOB_MANAGER
value: "0"
- name: __ENV__CONTEXT_PATH
value: eUrbanMIS
- name: POD_NAME
valueFrom:
fieldRef:
apiVersion: v1
fieldPath: metadata.name
- name: JAVA_OPTS
image: swr.cn-east-3.myhuaweicloud.com/egova_eurban/tomcat-redis:2.3
imagePullPolicy: IfNotPresent
lifecycle:
postStart:
exec:
command:
- bash
- -c
- |
set -ex
echo $(date +%s) > /etc/inited
preStop:
exec:
command:
- bash
- -c
- |
set -ex
rm -f /etc/started
livenessProbe:
failureThreshold: 12
httpGet:
path: /eUrbanMIS/main.htm
port: 8080
scheme: HTTP
initialDelaySeconds: 900
periodSeconds: 10
successThreshold: 1
timeoutSeconds: 30
name: tomcat
ports:
- containerPort: 8080
protocol: TCP
- containerPort: 1234
protocol: TCP
readinessProbe:
failureThreshold: 12
httpGet:
path: /eUrbanMIS/main.htm
port: 8080
scheme: HTTP
initialDelaySeconds: 100
periodSeconds: 10
successThreshold: 1
timeoutSeconds: 10
resources:
limits:
cpu: "3"
ephemeral-storage: 2Gi
memory: 4200Mi
requests:
cpu: 200m
ephemeral-storage: 1Gi
memory: 2096Mi
terminationMessagePath: /dev/termination-log
terminationMessagePolicy: File
volumeMounts:
- mountPath: /usr/local/tomcat/logs
name: egova-log-tomcat
subPathExpr: $(POD_NAME)
- mountPath: /etc/localtime
name: date-config
- mountPath: /usr/local/tomcat/webapps/eUrbanMIS/WEB-INF/log
name: egova-log-web
subPathExpr: $(POD_NAME)
- mountPath: /prometheus
name: prom
- mountPath: /docker-entrypoint-init.d/after-prepare-configmaps/
name: code-init-configmap
- mountPath: /usr/local/tomcat/webapps
name: code-volume
- mountPath: /etc/config
name: mis-configmap-volume
- mountPath: /etc/sidecar/
name: sidecar-volume
- image: swr.cn-east-3.myhuaweicloud.com/egova_eurban/tool-inotify:1.0
imagePullPolicy: IfNotPresent
limits:
cpu: 100m
memory: 50Mi
requests:
cpu: 10m
memory: 10Mi
terminationMessagePath: /dev/termination-log
terminationMessagePolicy: File
volumeMounts:
- mountPath: /etc/config
name: mis-configmap-volume
- mountPath: /etc/sidecar/
name: sidecar-volume
- mountPath: /usr/local/tomcat/webapps
name: code-volume
dnsPolicy: ClusterFirst
initContainers:
- command:
- bash
- -c
- |
set -ex
echo "数据库检查通过！"
env:
- name: DB_HOST_BIZ
valueFrom:
configMapKeyRef:
key: db_host_biz
name: global-configmap
- name: DB_PORT_BIZ
valueFrom:
configMapKeyRef:
key: db_port_biz
name: global-configmap
- name: DB_NAME_BIZ
valueFrom:
configMapKeyRef:
key: db_name_biz
name: global-configmap
- name: DB_USER_BIZ
valueFrom:
secretKeyRef:
key: db_user_biz
valueFrom:
secretKeyRef:
key: db_passwd_biz
name: global-secret
image: swr.cn-east-3.myhuaweicloud.com/egova_eurban/mysql-client:1.1
imagePullPolicy: IfNotPresent
name: check
resources:
limits:
cpu: 100m
memory: 50Mi
requests:
cpu: 10m
memory: 10Mi
terminationMessagePath: /dev/termination-log
terminationMessagePolicy: File
- command:
- bash
- -c
- |
set -ex
imagePullPolicy: IfNotPresent
name: init-monitor
resources:
limits:
cpu: 100m
memory: 50Mi
requests:
cpu: 10m
memory: 10Mi
terminationMessagePath: /dev/termination-log
terminationMessagePolicy: File
volumeMounts:
- mountPath: /etc/prometheus
name: prom
- command:
- bash
- -c
- |
set -ex
mkdir -p /code/webapps
cp -r /usr/local/tomcat/webapps/* /code/webapps/
imagePullPolicy: IfNotPresent
name: main
resources:
limits:
cpu: 100m
memory: 50Mi
requests:
cpu: 10m
memory: 10Mi
terminationMessagePath: /dev/termination-log
terminationMessagePolicy: File
volumeMounts:
- mountPath: /code/webapps
name: code-volume
priorityClassName: priority-b
restartPolicy: Always
schedulerName: default-scheduler
securityContext: {}
terminationGracePeriodSeconds: 30
volumes:
- configMap:
defaultMode: 420
name: eurban-mis-init
name: code-init-configmap
- emptyDir: {}
name: code-volume
- emptyDir: {}
name: sidecar-volume
- emptyDir: {}
name: prom
- hostPath:
path: /egova/log/tomcat
type: DirectoryOrCreate
path: /etc/localtime
type: ""
name: date-config
- hostPath:
path: /egova/log/web
type: DirectoryOrCreate
name: egova-log-web
- configMap:
defaultMode: 420
name: eurban-mis-config
name: mis-configmap-volume
status:
availableReplicas: 1
conditions:
- lastTransitionTime: "2021-05-08T09:25:30Z"
lastUpdateTime: "2021-05-08T09:25:30Z"
message: Deployment has minimum availability.
reason: MinimumReplicasAvailable
status: "True"
type: Available
- lastTransitionTime: "2021-05-08T06:22:15Z"
lastUpdateTime: "2021-06-04T06:01:41Z"
message: ReplicaSet "eurban-mis-backend-7db7d7574" has successfully progresss
ed.
reason: NewReplicaSetAvailable
status: "True"
type: Progressing
observedGeneration: 3
readyReplicas: 1
replicas: 1
updatedReplicas: 1

其他的pod定义可根据以下命令进行查看：

图24 查看

部署
 定义好Service和Pod之后，可根据以下命令进行部署
图25 部署

根据以下命令查看服务启动情况

图26 查看服务启动情况

网络配置

安全组规则配置
1. 登录华为云，进入网控制台
2. 在网络控制台左侧菜单栏选择“访问控制”-“安全组”，单击安全组的配置规则
  图27 规则配置
3. 单击入方向规则，单击添加规则，设置允许内网互通（3.1.1章节创建的子网），目的是让该子网下的云服务资源能够访问到被该安全组规则保护的资源。
  图28 添加规则
  
  说明：
  
  上述源地址10.20.0.0/24用到地址在这里查看（注意看自己在VPC&安全组中创建的VPC和子网是什么网段的，要复制自己的子网网段）：
  
  图29 添加规则2
4. 单击入方向规则，单击添加规则，设置放通100.125.0.0/16网段（用于ELB后端服务器健康检查）
  图30 进入方向规则
5. 检查一下规则是否都配置好了
  图31 检查
ELB配置
 监听器配置
本章节的目的是配置静态nginx服务的监听器，实现：通过ELB公网IP+监听的端口能进入静态nginx前端界面。
1. 配置监听器
  图32 配置监听器
2. 配置后端服务器组
  图33 配置后端服务器组