CCE+RDS完成springboot自动化交付应用上架
应用场景
本文介绍基于springboot+软件包方式的CCE应用构建并完成自动化交付的实践。
实践说明
本示例会构建基于springboot+软件包方式的单机License类应用实现一键自动化交付上云的实践,具体的构建流程如下:
- 应用开发:本文介绍基于springboot+软件包方式的CCE应用构建并完成自动化交付的实践。
- 应用上架:将上一步开发的软件包和脚本托管到云商店并完成漏洞扫描。
- 应用购买:作为用户完成该自动部署测试。
应用部署架构
本应用软件主体部署在CCE的实例上,软件运行依赖数据库,在本示例中使用华为云提供的高阶云服务RDS和CCE,依赖的云资源为ECS、RDS、EIP、NAT、ELB,本服务示例通过弹性公网IP的8765端口对外提供服务,该软件云上部署组网结构为:
应用开发
- 根据应用架构梳理资源清单如下 :
序号
区域
云服务类型
云服务规格说明
数量
1
华北-北京四
弹性公网IP
带宽费用:独享 | 全动态BGP | 按带宽计费 | 10Mbit/s (可选)
2
2
华北-北京四
弹性云服务器
规格:X86 | 统一计算增强型 | c7.large.2 | 2核 | 4G
2
3
华北-北京四
云容器引擎
CCE容器集群 | Standard/Turbo | 50节点 | 3实例(高可用)
1
4
华北-北京四
云数据库 RDS for MySQL
MySQL | 8.0 | 单机 | 独享型 | 2核4GB | SSD云盘 | 40GB
1
5
华北-北京四
ELB
共享型负载均衡 全动态BGP | 1Mbit/s
1
6
华北-北京四
NAT
小型
1
- 软件包制作
- jar包构建:本示例使用的软件代码可以从华为云开发体验馆 Codelabs中获取,使用如下命令构建本demo的jar包;
cd code/linux-single-ecs-rds-demo mvn clean install
- 容器镜像制作:通过docker制作您软件包的容器镜像并上传到镜像仓库,下面是一个打包java应用的Dockerfile 示例;
# 使用官方的OpenJDK镜像作为基础镜像 FROM openjdk:17-jdk-slim # 设置工作目录 WORKDIR /app # 将JAR文件复制到镜像中 COPY target/springapp.jar /app/springapp.jar # 暴露应用运行的端口 EXPOSE 8765 # 定义启动命令 CMD ["java", "-jar", "springapp.jar"] # docker build打包镜像 docker build -t springapp .
- 推送到镜像仓库:以docker方式为例,containerd方式请查阅容器镜像服务文档;
- 创建应用目录:jar包构建完成后,使用如下结构创建软件包目录;
software |——manifest #应用的描述文件 |——config #配置文件模板 |——script #应用安装初始化脚本 |——...
- 华为云SWR密钥配置(可选)
如果您使用的是华为云SWR,需要创建子账号并给子账号赋予SWR只读权限建议创建一个具有访问密钥的IAM用户,访问方式仅设置为编程访问,并为该用户授予工作所需的SWR只读权限。
- 登录命令参考
获取密钥并在安装脚本中登录swr,详情可参考下一步部署脚本开发。
# 创建华为云SWR认证kubectl --kubeconfig /root/.kube/config create secret docker-registry harbor-registry-secret --docker-server=${docker_server} --docker-username=${docker_region}@${ak} --docker-password=${sk}
- jar包构建:本示例使用的软件代码可以从华为云开发体验馆 Codelabs中获取,使用如下命令构建本demo的jar包;
部署脚本开发
- 分别完成下列脚本的开发,并将脚本置于script目录下:
目录 |
说明 |
---|---|
scripts |
configure.sh 导入环境变量,替换占位符 init_db.sh 数据库初始化 install.sh 安装应用 |
manifest |
java-application.yaml java应用的k8s描述文件 python-appliaction.yaml python应用的k8s描述文件 ingress.yaml ingress的k8s描述文件 |
config |
init_db.sql #RDS 初始化脚本 .k swr仓库的AKSK |
脚本示例如下:
software ├── config #配置文件模板 │ ├── init_db.sql │ └── .k ├── manifest #应用的描述文件 │ ├── ingress.yaml │ ├── java-application.yaml │ └── python-application.yaml └── script #应用安装初始化脚本 ├── common.sh ├── configure.sh ├── init_db.sh └── install.sh
- 描述文件的编写:描述文件用于给k8s部署工作负载、服务和路由提供输入,您需要完成这些描述文件的编写
- 应用负载描述文件
- 配置项的描述文件
- 用于给CCE绑定ELB的ingress和service的描述文件
- 工作负载描述文件的示例
- 工作负载1的样例
apiVersion: apps/v1 kind: Deployment metadata: name: springapp namespace: default spec: replicas: 1 selector: matchLabels: { app: springapp } template: metadata: labels: { app: springapp } spec: imagePullSecrets: - name: harbor-registry-secret containers: - name: default image: swr.$region_code.myhuaweicloud.com/wangyinxu/springappdemo:1.0 imagePullPolicy: IfNotPresent env: - name: SERVER_PORT value: "8765" - name: DB_HOST value: "$db_host" - name: DB_PORT value: "$db_port" - name: DB_NAME value: "$db_name" - name: DB_USER value: "$db_user" - name: DB_PASS value: "$db_password" ports: [ { containerPort: 8765 } ] readinessProbe: { tcpSocket: { port: 8765 }, initialDelaySeconds: 10, periodSeconds: 5, failureThreshold: 6 } livenessProbe: { tcpSocket: { port: 8765 }, initialDelaySeconds: 20, periodSeconds: 10, failureThreshold: 6 }
- 工作负载2的样例
apiVersion: apps/v1 kind: Deployment metadata: name: flaskapp namespace: default spec: replicas: 1 selector: matchLabels: { app: flaskapp } template: metadata: labels: { app: flaskapp } spec: imagePullSecrets: - name: harbor-registry-secret containers: - name: default image: swr.$region_code.myhuaweicloud.com/wangyinxu/flask-app:1.0 imagePullPolicy: IfNotPresent ports: [ { containerPort: 5000 } ] readinessProbe: { tcpSocket: { port: 5000 }, initialDelaySeconds: 10, periodSeconds: 5, failureThreshold: 6 } livenessProbe: { tcpSocket: { port: 5000 }, initialDelaySeconds: 20, periodSeconds: 10, failureThreshold: 6 }
- 工作负载1的样例
- 配置项描述文件的示例:
apiVersion: v1 kind: ConfigMap metadata: name: demo-config namespace: default data: demo.properties: | demoname=demo1 demopods=2 ui.properties: | color.good=purple color.bad=yellow allow.textmode=true
- 用于ELB绑定CCE的ingress、service描述文件的示例
在此文件中使用elb_id变量绑定ELB,其中elb_id由部署模板传入,并通过service访问到您的工作负载
metadata: name: java-service namespace: default labels: app: springapp spec: ports: - name: cce-service-0 protocol: TCP port: 8765 targetPort: 8765 selector: app: springapp type: NodePort sessionAffinity: None externalTrafficPolicy: Cluster ipFamilies: - IPv4 ipFamilyPolicy: SingleStack internalTrafficPolicy: Cluster apiVersion: v1 kind: Service --- metadata: name: springappingress namespace: default annotations: kubernetes.io/elb.class: union kubernetes.io/elb.id: $elb_id kubernetes.io/elb.listener-master-ingress: default/springappingress kubernetes.io/elb.port: '80' spec: ingressClassName: cce rules: - http: paths: - path: / pathType: ImplementationSpecific backend: service: name: java-service port: number: 8765 property: ingress.beta.kubernetes.io/url-match-mode: STARTS_WITH apiVersion: networking.k8s.io/v1 kind: Ingress
- 安装脚本编写
1. swr登录
2. k8s命令行apply描述文件
3. 初始化数据库等其他初始化动作(可选)
install.sh的示例,该实例实现了swr登录并使用k8s命令apply描述文件
k8slogin(){ # 读取key kv=$(cat ${APP_HOME}/config/.k) docker_region=${REGION_CODE} docker_server="swr.${REGION_CODE}.myhuaweicloud.com" ak=$(echo ${kv}|awk -F ":" '{print $1}') sk=$(echo ${kv}|awk -F ":" '{print $2}') # 创建华为云SWR认证 kubectl --kubeconfig /root/.kube/config create secret docker-registry harbor-registry-secret --docker-server=${docker_server} --docker-username=${docker_region}@${ak} --docker-password=${sk} echo "$(date "${DATE_FMT[@]}") Login docker successful." } k8slogout(){ # 删除k rm -f ${APP_HOME}/config/.k # 退出k8s kubectl --kubeconfig /root/.kube/config delete secret harbor-registry-secret echo "$(date "${DATE_FMT[@]}") Logout docker successful." } # 启动应用 start_app() { kubectl --kubeconfig /root/.kube/config apply -f ${APP_HOME}/manifest/java-application.yaml } start_ingress() { kubectl --kubeconfig /root/.kube/config apply -f ${APP_HOME}/manifest/ingress.yaml } APP_HOME="${APP_HOME:-/opt/springapp}" source ${APP_HOME}/script/configure.sh source ${APP_HOME}/script/init_db.sh k8slogin start_app start_ingress k8slogout
自动化部署模版开发
- 自动部署模板用于购买梳理的 应用部署架构 中的云资源并将其关联起来。
rfstemplate |-- datas.tf |-- locals.tf |-- main.tf |-- modules | |-- ecs| | |-- main.tf | | |-- outputs.tf | | |-- variables.tf | | `-- versions.tf | |-- eip | | |-- main.tf | | |-- outputs.tf | | |-- variables.tf | | `-- versions.tf | |-- rds_mysql_single_expansion | | |-- main.tf | | |-- outputs.tf | | |-- variables.tf | | `-- versions.tf | |-- security-group | | |-- main.tf | | |-- outputs.tf | | |-- variables.tf | | `-- versions.tf | `-- vpc | |-- main.tf | |-- outputs.tf | |-- variables.tf | `-- versions.tf |-- outputs.tf |-- providers.tf |-- variables.tf `-- versions.tf
- main.tf模版,详情内容可从华为云开发体验馆 Codelabs中获取。
# # TODO asset_id,为创建的资产id;asset_version,资产版本 data "huaweicloud_koogallery_assets" "assets" { asset_id = "" //必填 deployed_type = "software_package" asset_version = "v1.0" //必填,区分大小写 } ############################################## # 1) VPC & 子网 ############################################## module "vpc" { source = "./modules/vpc" # config of vpc name_suffix = local.name_suffix vpc_name = format("%s-%s", local.app_id, "vpc") vpc_cidr = var.vpc_cidr vpc_tags = local.tags # config of subnet subnet_name = format("%s-%s", local.app_id, "subnet") vpc_subnet_dns_list = local.subnet_dns_list_maps[data.huaweicloud_availability_zones.az.region] vpc_subnet_cidr = var.vpc_subnet_cidr vpc_subnet_gateway_ip = var.vpc_subnet_gateway_ip subnet_tags = local.tags } ############################################## # 2) 安全组(ECS 与 RDS 分离) ############################################## # ECS 安全组:放通 SSH(22) + APP(8765),出向全放 module "sg_ecs" { source = "./modules/security-group" is_secgroup_create = true name_suffix = local.name_suffix secgroup_name = "ecs-sg" is_delete_default_rules = true secgroup_rules_configuration = [ { description = "Allow SSH from remote" direction = "ingress" ethertype = "IPv4" protocol = "tcp" ports = "22" remote_ip_prefix = var.remote_ip_cidr remote_group_id = null }, { description = "Allow APP 8080 from remote" direction = "ingress" ethertype = "IPv4" protocol = "tcp" ports = "8765" remote_ip_prefix = var.remote_ip_cidr remote_group_id = null }, { description = "Egress all" direction = "egress" ethertype = "IPv4" protocol = null ports = null remote_ip_prefix = "0.0.0.0/0" remote_group_id = null } ] } # RDS 安全组:仅允许来自 ECS 安全组的 3306 module "sg_rds" { source = "./modules/security-group" is_secgroup_create = true name_suffix = local.name_suffix secgroup_name = "rds-sg" is_delete_default_rules = true secgroup_rules_configuration = [ { description = "Allow MySQL from ECS SG" direction = "ingress" ethertype = "IPv4" protocol = "tcp" ports = "3306" remote_ip_prefix = var.remote_ip_cidr remote_group_id = null }, { description = "Egress all" direction = "egress" ethertype = "IPv4" protocol = null ports = null remote_ip_prefix = var.remote_ip_cidr remote_group_id = null } ] } ############################################## # 3) EIP ELB ############################################## //创建弹性公网 module "eip" { source = "./modules/eip" is_eip_create = true count = 1 name_suffix = local.name_suffix eip_name = format("%s-%s", local.app_id, "eip") publicip_type = local.publicip_type bandwidth_charge_mode = local.bandwidth_charge_mode bandwidth_share_type = local.bandwidth_share_type bandwidth_size = local.bandwidth_size charging_mode = local.bandwidth_charge_mode == "traffic" ? "postPaid" : local.charging_mode period_unit = local.period_unit period = local.period eip_tags = local.tags } # 创建弹性负载均衡 ELB-公网 module "elb-outernet-admin" { source = "./modules/elb" is_elb_creat = true name_suffix = local.name_suffix #ELB loadbalancer_name = format("%s-%s", local.app_id, "elb") ipv4_subnet_id = module.vpc.subnet_ipv4_subnet_id ipv4_eip_ip = module.eip[0].ip[0] charging_mode = local.charging_mode period_unit = local.period_unit period = local.period is_auto_renew = false loadbalancer_tags = local.tags } ############################################## # 4) CCE、ECS ############################################## // Create ECS module "cce-node" { source = "./modules/ecs" is_instance_create = true instance_count = local.cce_node_count instance_image_id = local.instance_image_id_maps[data.huaweicloud_availability_zones.az.region] instance_flavor_id = local.ecs_flavor instance_flavor_performance = local.instance_performance_type instance_flavor_cpu = local.instance_flavor_cpu instance_flavor_memory = local.instance_flavor_memory #agent_list = local.ecs_agent_list name_suffix = local.name_suffix instance_name = format("%s-%s", local.app_id, "ecs") security_group_ids = module.ecs-security-group.secgroup_id # Default size and type of the system disk, You need to modify it according to your actual situation. And the size must be greater than the minimum memory size required by the image. system_disk_type = local.ecs_volume_type system_disk_size = local.ecs_volume_size data_disks = local.ecs_data_disks networks_configuration = [ { subnet_id = module.vpc.subnet_id, fixed_ip_v4 = null, ipv6_enable = false, source_dest_check = true, access_network = false }, ] charging_mode = local.charging_mode period_unit = local.period_unit period = local.period admin_pass = var.admin_password instance_tags = local.tags } # 创建云容器引擎 CCE module "cce" { source = "./modules/cce" name_suffix = local.name_suffix cluster_name = format("%s-%s", local.app_id, "cce") cluster_type = local.cce_cluster_type cluster_version = local.cce_cluster_version cluster_flavor = local.cce_cluster_flavor vpc_id = module.vpc.vpc_id subnet_id = module.vpc.subnet_id container_network_type = local.cce_container_network_type eni_subnet_id = module.vpc.subnet_ipv4_subnet_id container_network_cidr = local.cce_container_network_cidr service_network_cidr = local.cce_service_network_cidr kube_proxy_mode = local.cce_kube_proxy_mode #服务转发模式 charging_mode = var.charging_mode period_unit = var.period_unit period = var.period is_delete_all = true cluster_tags = local.tags # 节点配置 node_count = local.cce_node_count node_ids = module.cce-node.ecs_ids node_os = local.cce_os_type node_password = var.admin_password post_install_script = local.postinstall } module "coc_deploy_cce" { source = "./modules/coc" script_name = format("%s-%s", "deploy", module.cce-node.ecs_ids[0]) script_description = "deploy software" risk_level = "LOW" script_version = "1.0.0" script_type = "SHELL" is_sync = true script_content = local.cce_script instance_id = module.cce-node.ecs_ids[0] execute_timeout = 1800 execute_user = "root" depends_on = [data.huaweicloud_cce_cluster.cluster] } data "huaweicloud_cce_cluster" "cluster" { id = module.cce.cluster_id status = "Available" depends_on = [module.cce] } ############################################## # 5) RDS MySQL ############################################## module "rds" { source = "./modules/rds_mysql_single_expansion" is_rds_create = true charging_mode = local.charging_mode period_unit = local.period_unit period = local.period name_suffix = local.name_suffix db_instance_name = "${local.app_prefix}-rds-mysql" time_zone = "UTC" db_availability_zone = slice(local.rds_azs,0 ,1 ) vpc_id = module.vpc.vpc_id subnet_id = module.vpc.subnet_id security_group_id = module.sg_rds.secgroup_id[0] flavor = data.huaweicloud_rds_flavors.rds_flavor.flavors[0].name lower_case_table_names = "1" # db config db_engines = local.db_type db_version = local.db_version db_password = var.db_password db_port = local.db_port # backup_strategy keep_days = local.db_keep_days start_time = local.db_start_time # volume config db_instance_storage_type = local.db_instance_storage_type db_allocated_storage = local.db_allocated_storage #数据库 database_name = var.db_name character_set = "utf8" #账号 account_name = var.db_user account_pwd = var.db_password rds_tags = local.tags } ############################################## # 6) 购买NAT和EIP用于访问外部网络 ############################################## module "eip-nat" { source = "./modules/eip" is_eip_create = true name_suffix = local.name_suffix eip_name = format("%s-%s", local.app_id, "eip-nat") publicip_type = local.publicip_type bandwidth_charge_mode = local.bandwidth_charge_mode bandwidth_share_type = local.bandwidth_share_type bandwidth_size = 1 charging_mode = local.bandwidth_charge_mode == "traffic" ? "postPaid" : local.charging_mode period_unit = local.period_unit period = local.period eip_tags = local.tags } module "nat" { source = "./modules/nat" gateway_name = format("%s-%s", local.app_id, "nat") charging_mode = local.charging_mode period_unit = local.period_unit period = local.period is_nat_creat = true vpc_id = module.vpc.vpc_id spec = 1 subnet_id = module.vpc.subnet_id publicip_id = module.eip-nat.id[0]
上述main.tf模版的作用是:
① 新建VPC。
② 基于常量配置,以固定镜像ID、CPU核数、计费模式拉起ECS实例。
③ 购买CCE、RDS、EIP、ELB、NAT。
④ 将ECS纳管到CCE实例中。
⑤ 配置安全组。
⑥ 关联EIP到ELB。
⑦ 执行软件部署脚本
应用上架
将开发好的软件包托管至云商店,登录“卖家中心-资产中心-添加应用资产”,具体操作流程请参考:应用资产发布软件包操作指南。

商家提交应用资产发布申请后,云商店会对资产中的软件包进行自动化安全扫描。
- 如果扫描时有漏洞,资产安全审核会驳回您的申请,您需要完成漏洞的修复。
- 如安全问题已完成整改,点击“修改”,重新上传已修复的软件包再次提交审核。
- 如安全扫描存在误报,点击“申诉”,进入申诉页面,提交申诉说明。
自动化部署测试
- 测试券申请:模板测试会产生一部分云资源消耗的费用,商家可以申请测试券来抵扣;
- 登录华为云价格计算器:选择所需云资源(计费模式请选择按需,每次申请时长最长24小时)> 加入清单 > 导出云资源清单;
- 发送邮件,提交云资源清单,邮件内容格式如下:
发送邮件至 wangyinxu1@huawei.com
抄送:gengliang@huawei.com,wangzichun6@h-partners.com
邮件标题:202X年云商店中国站联营伙伴自动化部署测试券申请-xx公司-xx商品
邮件正文:202X年云商店中国站联营伙伴自动化部署测试券申请-xx公司-xx商品,所需云资源清单见附件
邮件附件:自动化部署-xx公司-xx商品云资源清单
- 登录卖家中心> 测试券管理,提交“测试券申请”,填写规则如下:
填写测试券方案名称:202X年云商店中国站联营伙伴自动化部署-xx公司-xx商品
选择申请场景:联营商品上架
选择适用商品:选择需要对接的具体商品
填写计划使用期限:不限
- 自动化部署模版在线测试,参考自动部署模板在线测试