Creating VPC Endpoints for Connecting to On-Premises Clusters over Private Networks

Application Scenarios

If you have Kubernetes clusters in your on-premises data center, you can connect your on-premises data center to UCS and enable Container Intelligent Analysis (CIA) to communicate with SWR and OBS. If the public network is unavailable, you can connect your on-premises data center to Huawei Cloud VPC through VPN and then use VPC endpoints to enable VPC to access UCS, SWR, DNS, OBS, and CIA over private networks.

Preparations

The following table lists the domain names of SWR and OBS in other regions.

Procedure

1. Configure a VPN by referring to Connecting an On-Premises Data Center to a VPC Through a VPN..

   If a VPN has been configured, go to 7.

   

   • The private CIDR block of your on-premises data center cannot overlap with the VPC CIDR block used for connecting to the VPN on Huawei Cloud.
   • The subnet CIDR block of the VPC cannot overlap with the subnet CIDR block of your on-premises data center. If the CIDR blocks overlap, the cluster cannot be connected. For example, if the subnet of an on-premises data center is 192.168.1.0/24, the subnet of the Huawei Cloud VPC cannot be 192.168.1.0/24.

2. Create a VPN gateway on Huawei Cloud.

   Log in to the Huawei Cloud console and choose Virtual Private Network. In the navigation pane, choose Enterprise – VPN Gateways. On the displayed page, click the S2C VPN Gateways tab. In the upper right corner, click Buy S2C VPN Gateway.

   

   Table 1 Planned data

   Category	Planned Item	Planned Value
   VPC	Subnets that need to access the VPC	10.188.1.0/24 and 100.64.0.0/10 (the CIDR blocks of SWR and OBS)
   VPN gateway	Interconnection subnet	This subnet is used for communication between the VPN gateway and VPC. The subnet cannot overlap with the existing VPC subnets.
   		10.188.2.0/24
   	EIPs	EIPs are automatically generated when you buy EIPs. By default, a VPN gateway uses two EIPs. In this example, the following EIPs are generated:
   		Active EIP: 11.xx.xx.11
   		Standby EIP: 11.xx.xx.12
   VPN connections	Tunnel interface address	This address is used by a VPN gateway to establish an IPsec tunnel with a customer gateway. At the two ends of the IPsec tunnel, the configured local and remote tunnel interface addresses must be reversed.
   		VPN connection 1: 169.254.70.1/30
   		VPN connection 2: 169.254.71.1/30

3. Create a customer gateway.

   In the navigation pane, choose Enterprise – Customer Gateways. On the displayed page, click Create Customer Gateway.

   Set Identifier to IP Address and enter the public IP address of the on-premises data center.

   

   

4. Create a VPN connection.

   

   

   Table 2 Parameters for creating a VPN connection

   Parameter	Description	Example Value
   Name	Enter a VPN connection name.	vpn-xxx
   VPN Gateway	Select the VPN gateway created in 2.	vpngw-xxx
   Gateway IP Address	Select the active EIP of the VPN gateway.	11.xx.xx.11
   Customer Gateway	Select the customer gateway created in 3.	cgw-xxx
   VPN Type	Select Static routing.	Static routing
   Customer Subnet	Enter the subnet of the on-premises data center that needs to access the VPC. NOTE: The customer subnet can overlap with the local subnet but cannot be the same as the local subnet. A customer subnet cannot be included in the existing subnets of the VPC associated with the VPN gateway. It also cannot be the destination address in the route table of the VPC associated with the VPN gateway. Customer subnets cannot be the reserved CIDR blocks of VPCs, for example, 100.64.0.0/10 or 214.0.0.0/8. If the interconnection subnet is associated with an ACL rule, ensure that the ACL rule permits the TCP port for traffic between all local and customer subnets.	172.16.0.0/16
   Interface IP Address Assignment	The options are Manually specify and Automatically assign.	Manually specify
   Local Tunnel Interface Address	Configure the tunnel IP address of the VPN gateway. NOTE: The local and remote interface addresses configured on the customer gateway device must be the same as the values of Customer Tunnel Interface Address and Local Tunnel Interface Address, respectively.	169.254.70.2/30
   Customer Tunnel Interface Address	Specify the tunnel interface address configured on the customer gateway device.	169.254.70.1/30
   Link Detection	This function is used for route reliability detection in multi-link scenarios. NOTE: When enabling this function, ensure that the customer gateway supports ICMP and is correctly configured with the customer interface IP address of the VPN connection. Otherwise, VPN traffic will fail to be forwarded.	Select NQA.
   PSK/Confirm PSK	Specify the negotiation key of the VPN connection. The PSKs configured on the VPN console and the customer gateway device must be the same.	Test@123
   Policy Settings	There are IKE Policy and IPsec Policy, which specifies the encryption and authentication algorithms of a VPN tunnel. The policy settings on the VPN console and the customer gateway device must be the same.	Default

5. Configure the customer gateway device.
6. Verify the network connectivity.
   1. Log in to the management console.
   2. Click in the upper left corner and select a region and a project.
   3. Click Service List and choose Compute > Elastic Cloud Server.
   4. Log in to the ECS.

      Multiple methods are available for logging in to an ECS. For details, see Logging In to an ECS.

      In this example, use VNC provided on the management console to log in to an ECS.

   5. Run the following command on the ECS console:

      ping 172.16.0.100

      172.16.0.100 is the IP address of a server in the on-premises data center. Replace it with an actual server IP address.

      If information similar to the following is displayed, the client can communicate with the ECS:

      Reply from xx.xx.xx.xx: bytes=32 time=28ms TTL=245
      Reply from xx.xx.xx.xx: bytes=32 time=28ms TTL=245
      Reply from xx.xx.xx.xx: bytes=32 time=28ms TTL=245
      Reply from xx.xx.xx.xx: bytes=32 time=27ms TTL=245

7. Create VPC endpoints on Huawei Cloud.

   To enable an on-premises data center to access DNS, SWR, OBS, and UCS on Huawei Cloud, you need to create their endpoints in the VPC that communicates with the on-premises data center.

   Creating a VPC Endpoint for DNS

   Click Service List and choose Networking > VPC Endpoint.

   1. In the navigation pane, choose VPC Endpoint > VPC Endpoints.
   2. On the displayed page, click Buy VPC Endpoint.
   3. Set Service Category to Cloud services and select com.myhuaweicloud.cn-north-4.dns from Service List.
   4. Select the VPC that has been connected in 2.
   5. Click the generated VPC endpoint name to view the IP address.

   

   Creating a VPC Endpoint for SWR

   1. Click Service List and choose Networking > VPC Endpoint.
   2. In the navigation pane, choose VPC Endpoint > VPC Endpoints.
   3. On the displayed page, click Buy VPC Endpoint.
   4. Set Service Category to Cloud services and select com.myhuaweicloud.cn-north-4.swr from Service List.
   5. Select the VPC that has been connected in 2.

      

   6. Click the generated VPC endpoint name to view the IP address.

   

   Creating a VPC Endpoint for OBS

   1. Click Service List and choose Networking > VPC Endpoint.
   2. In the navigation pane, choose VPC Endpoint > VPC Endpoints.
   3. On the displayed page, click Buy VPC Endpoint.
   4. Set Service Category to Find a service by name and VPC Endpoint Service Name to cn-north-4.com.myhuaweicloud.v4.obsv2. Then, click Verify.
   5. Select the VPC that has been connected in 2.

   

   Creating a VPC Endpoint for UCS

   1. Click Service List and choose Networking > VPC Endpoint.
   2. In the navigation pane, choose VPC Endpoint > VPC Endpoints.
   3. On the displayed page, click Buy VPC Endpoint.
   4. Set Service Category to Find a service by name and VPC Endpoint Service Name to cn-north-4.open-vpcep-svc.29696ab0-1486-4f70-ab35-a3f6b1b37c02. Then, click Verify.
   5. Select the VPC that has been connected in 2.

   

8. Add the Huawei Cloud DNS forwarder to the DNS server in the on-premises data center.
   1. Add DNS records on the DNS server in your on-premises data center to forward requests for resolving the private domain name of Huawei Cloud to the DNS VPC endpoint.

      Take DNS Bind as an example. In /etc/named.conf, add the DNS forwarder configuration and set forwarders to the IP address of the VPC endpoint for accessing DNS. {xx.xx.xx.xx} represents the IP address of the VPC endpoint for accessing DNS in 7.

      options 
      {
        forward only;
        forwarders{ xx.xx.xx.xx;};
      }

   2. Configure static DNS resolution and add the IP addresses of SWR and CIE instances. The IP addresses can be obtained from the CIA instance.

      Take CN North-Beijing4 as an example. If dnsmasq is used, add the following static resolution to /etc/dnsmasq.conf:

      address=/swr.cn-north-4.myhuaweicloud.com/xx.xx.xx.xx

      xx.xx.xx.xx represents the IP address of the VPC endpoint for accessing SWR in 7.

      address=/cie-{First eight digits in the ID of the CIA instance}{First eight digits in the ID of the VPC subnet}.cn-north-4.myhuaweicloud.com

      Obtains the first eight digits in the ID of the CIA instance.

      Obtains the first eight digits in the ID of the VPC subnet.

      

9. Register an on-premises cluster with UCS as follows: Prepare the kubeconfig file of the cluster to be accessed. Ensure that the value of the server field in this file is a private IP address (not a public IP address or domain name). Log in to the UCS console. In the navigation pane, choose Fleets. In the On-premises cluster card, click Register Cluster. Select a cluster service provider and configure cluster parameters as prompted. For details, see Preparing for Installation.

   After a cluster is connected, you need to configure an endpoint for the cluster to access the network so that the cluster can be taken over by UCS. Click Private access and select the VPC that connects to the on-premises data center through the VPN.

   

   The VPC can be selected only when the configuration in 7 is complete.

   Download the configuration file of the cluster agent and upload it to the Kubernetes cluster in the on-premises data center. Run the following command to deploy the agent in the cluster to be connected:

   kubectl apply -f agent.yaml

   Check the deployment of the cluster agent.

   kubectl -n kube-system get pod | grep proxy-agent

   Expected output for successful deployment:

   proxy-agent-5f7d568f6-6fc4k 1/1 Running 0 9s

   Check the status of the cluster agent.

   kubectl -n kube-system logs<Agent Pod Name>| grep "Start serving"

   Expected log output for normal running:

   Start serving

   Go to the UCS console and refresh the cluster status. The cluster is in the Running state.

   

10. Connect the Kubernetes cluster to CIA.
    1. Log in to the UCS console and choose Container Intelligent Analysis in the navigation pane. Select a CIA instance and click Enable Monitoring in the upper right corner. Select a cluster to be connected in the on-premises data center and click Next: Configure Connection.
    2. Set Data Access to Private access. Private access: Select the VPC that has been connected to the on-premises data center through a VPN.

    3. Complete the add-on configuration.

       The system provides default add-on settings, including the add-on specifications, collection period, and storage. If you want to change the default values, click next to the add-on parameters to expand the configuration items.

       Add-on Specifications: There are Demo (≤ 100 containers) and other options. Different specifications have different requirements on cluster resources such as CPU and memory. UCS preliminarily checks whether an add-on can be installed on the cluster node. If no, a message will be displayed.

       • Storage: used to temporarily store Prometheus data.
       • Storage Type: Attached clusters support emptyDir and Local Storage.
       • If emptyDir is used, Prometheus data will be stored in the pod. Ensure that the storage volume mounted to the container on the node scheduled by prometheus-server-0 is no less than the entered capacity.
       • If Local Storage is used, the monitoring namespace (if it does not exist) and PVs and PVCs of the local storage type will be created in your cluster. Ensure that the entered directory exists on the specified node and the path capacity is sufficient.
       • Capacity: capacity specified when a PVC is created or the maximum storage limit when the pod storage is selected.

       Wait till the cluster is connected. After 2 to 3 minutes, the cluster is in the low-risk, medium-risk, or high-risk state, and monitoring data is displayed.