Accessing Kafka Using a VPC Endpoint Across VPCs

Is Plaintext Access or Ciphertext Access Used When a Client Accesses Kafka Across VPCs Using A VPC Endpoint?

It depends on Cross-VPC Access Protocol. The cross-VPC access protocol can be configured when you create a Kafka instance. After an instance is created, the setting cannot be changed.

Options:

• PLAINTEXT: There is no authentication required in such a connection and data is transmitted in plaintext.
• SASL_SSL: Clients can connect to a Kafka instance with SASL and the data will be encrypted using the SSL certificate.
• SASL_PLAINTEXT: Clients can connect to a Kafka instance with SASL and the data will be transmitted in plaintext.

Creating a VPC Endpoint Service

1. Log in to the console.
2. Click in the upper left corner to select a region.
   

   Select the region where your Kafka instance is located.

3. Click in the upper left corner and choose Middleware > Distributed Message Service (for Kafka)Application > to open the console of DMS for Kafka.
4. Click the desired Kafka instance to view its details.
5. In the Advanced Settings section on the Basic Information tab page, obtain the listeners IP addresses and port IDs of the instance for Cross-VPC Access.
   Figure 2 Cross-VPC access–related listeners IP addresses and corresponding port IDs of the Kafka instance
   

6. In the Network section on the Basic Information tab page, view the VPC to which the Kafka instance belongs.
   Figure 3 Viewing the VPC to which the Kafka instance belongs
   

7. Click the VPC to obtain the VPC ID on the VPC console.
   Figure 4 Obtaining the VPC ID
   

8. Call the VPC Endpoint API to create a VPC endpoint service. For details, see Creating a VPC Endpoint Service.

   POST https://{endpoint}/v1/{project_id}/vpc-endpoint-services

   Set the following request parameter to the specified values, and other parameters as required.

   • port_id: one of the port IDs obtained in 5.
   • vpc_id: VPC ID obtained in 7.
   • server_type: VM
   • client_port: 9011
   • server_port: 9011
   • protocol: TCP
   • approval_enabled: false
   • service_type: interface
   • endpoint: VPCEP endpoint obtained from Regions and Endpoints. The region must be the same as that of the Kafka instance.
   • project_id: project ID obtained from Obtaining a Project ID. The region must be the same as that of the Kafka instance.

   Record the value of service_name in the response. This parameter indicates the name of the VPC endpoint service.

9. Repeat 8 to create VPC endpoint services for other port IDs obtained in 5 and record the VPC endpoint service names.

(Optional) Adding a Whitelist

The VPC endpoint service can be used across accounts through a whitelist.

If the Kafka client and Kafka instance belong to different accounts, add the ID of the account to which the Kafka client belongs to the whitelist of the endpoint service. For details, see Add a Whitelist Record.

Buying a VPC Endpoint

1. Click in the upper left corner of the management console. Then choose Network > VPC Endpoint.
2. Click Buy VPC Endpoint.
3. Set the following parameters:
   • Region: Select the region that the Kafka instance is in.
   • Service Category: Select Find a service by name.
   • VPC Endpoint Service Name: Enter the VPC endpoint service name recorded in 8 and click Verify. If Service name found is displayed, proceed with subsequent operations.
   • VPC: Select the VPC that the Kafka client is in.
   • Subnet: Select the subnet that the Kafka client is in.
   • IPv4 Address: Select Automatically assign IP address.

   Retain the default values for other parameters. For details, see Buying a VPC Endpoint.

   Figure 5 VPC endpoint parameters
   

4. Click Next.
5. Confirm the configurations and submit the request.
6. Go back to the VPC endpoint list and check whether the status of the created VPC endpoint has changed to Accepted. The Accepted state means that the VPC endpoint has been connected to the VPC endpoint service.
   Figure 6 Checking the VPC endpoint status
   

7. Click the VPC endpoint ID. On the Summary tab page, obtain the private IP address.

   You can use the private IP address to access the VPC endpoint service.

   Figure 7 Viewing the private IP address
   

8. Repeat 1 to 7 to buy a VPC endpoint for each VPC endpoint service created in 9, and view and record the private IP addresses of the VPC endpoint services.

Modifying Parameter advertised.listeners IP

1. Click in the upper left corner and choose Middleware > Distributed Message Service (for Kafka) to open the console of DMS for Kafka.
2. Click the desired Kafka instance to view its details.
3. On the Advanced Settings section of the Basic Information tab page, click Modify for Cross-VPC Access to change the value of advertised.listeners IP address to the private IP addresses recorded in 7 and 8. Click Save.
   

   Each IP address must match the corresponding port ID. Otherwise, the network will be disconnected.

   Figure 8 Changing the advertised.listeners IP addresses
   

Verifying Connectivity

Check whether messages can be created and retrieved by referring to Connecting to Kafka Using the Client (SASL Disabled) or Connecting to Kafka Using the Client (SASL Enabled).

Notes:

• The address for connecting to a Kafka instance is in the format of "advertised.listeners IP:9011". For example, the addresses for connecting to the Kafka instance shown in Figure 8 are 192.168.0.71:9011,192.168.0.11:9011,192.168.0.21:9011.
• Configure inbound rules for the security group of the Kafka instance to allow access from 198.19.128.0/17 over port 9011.
• If a network access control list (ACL) has been configured for the subnet of this instance, configure inbound rules for the network ACL to allow access from 198.19.128.0/17 and from the subnet used by the VPC endpoint.

198.19.128.0/17 is the network segment allocated to the VPCEP service. To use VPCEP, allow access from this network segment.