Accessing Kafka Using a VPC Endpoint Across VPCs

VPCs are logically isolated from each other. If a Kafka instance and a Kafka client are in different VPCs within a region, they cannot communicate with each other. In this case, you can use one of the following methods to access a Kafka instance across VPCs:

Establish a VPC peering connection to allow two VPCs to communicate with each other. For details, see VPC Peering Connection.
Use VPC Endpoint (VPCEP) to establish a cross-VPC connection.

The following describes how to use VPCEP to implement cross-VPC access.

VPCEP provides two types of resources: VPC endpoint services and VPC endpoints.

A VPC endpoint service can be a Kafka instance which is accessed using VPC endpoints.
A VPC endpoint is a secure and private channel for connecting a VPC to a VPC endpoint service.

Figure 1 Working principle of accessing a Kafka instance across VPCs

Creating a VPC Endpoint Service

Log in to the console.
Click in the upper left corner to select a region.

Select the region where your Kafka instance is located.
Click in the upper left corner and choose Application > Distributed Message Service (for Kafka)Application > to open the console of DMS for Kafka.
Click the desired Kafka instance to view its details.
In the Advanced Settings section on the Basic Information tab page, obtain the listeners IP addresses and port IDs of the instance for Cross-VPC Access.

Figure 2 Cross-VPC access–related listeners IP addresses and corresponding port IDs of the Kafka instance
In the Network section on the Basic Information tab page, view the VPC to which the Kafka instance belongs.

Figure 3 Viewing the VPC to which the Kafka instance belongs
Click the VPC to obtain the VPC ID on the VPC console.

Figure 4 Obtaining the VPC ID
Call the VPC Endpoint API to create a VPC endpoint service. For details, see Creating a VPC Endpoint Service.
```
POST https://{endpoint}/v1/{project_id}/vpc-endpoint-services
```
Set the following request parameter to the specified values, and other parameters as required.
- port_id: one of the port IDs obtained in 5.
- vpc_id: VPC ID obtained in 7.
- server_type: VM
- client_port: 9011
- server_port: 9011
- protocol: TCP
- approval_enabled: false
- service_type: interface
- endpoint: VPCEP endpoint obtained from Regions and Endpoints. The region must be the same as that of the Kafka instance.
- project_id: project ID obtained from Obtaining a Project ID. The region must be the same as that of the Kafka instance.
Record the value of service_name in the response. This parameter indicates the name of the VPC endpoint service.
Repeat 8 to create VPC endpoint services for other port IDs obtained in 5 and record the VPC endpoint service names.

(Optional) Adding a Whitelist

The VPC endpoint service can be used across accounts through a whitelist.

If the Kafka client and Kafka instance belong to different accounts, add the ID of the account to which the Kafka client belongs to the whitelist of the endpoint service. For details, see Add a Whitelist Record.

Buying a VPC Endpoint

Click in the upper left corner of the management console. Then choose Network > VPC Endpoint.
Click Buy VPC Endpoint.
Set the following parameters:
- Region: Select the region that the Kafka instance is in.
- Service Category: Select Find a service by name.
- VPC Endpoint Service Name: Enter the VPC endpoint service name recorded in 8 and click Verify. If Service name found is displayed, proceed with subsequent operations.
- VPC: Select the VPC that the Kafka client is in.
- Subnet: Select the subnet that the Kafka client is in.
- Private IP Address: Select Automatic.
Retain the default values for other parameters. For details, see Buying a VPC Endpoint.

Figure 5 VPC endpoint parameters
Click Next.
Confirm the configurations and submit the request.
Go back to the VPC endpoint list and check whether the status of the created VPC endpoint has changed to Accepted. The Accepted state means that the VPC endpoint has been connected to the VPC endpoint service.

Figure 6 Checking the VPC endpoint status
Click the VPC endpoint ID. On the Summary tab page, obtain the private IP address.

You can use the private IP address to access the VPC endpoint service.

Figure 7 Viewing the private IP address
Repeat 1 to 7 to buy a VPC endpoint for each VPC endpoint service created in 9, and view and record the private IP addresses of the VPC endpoint services.

Modifying Parameter advertised.listeners IP

Click in the upper left corner and choose Application > Distributed Message Service (for Kafka) to open the console of DMS for Kafka.
Click the desired Kafka instance to view its details.
On the Advanced Settings section of the Basic Information tab page, click Modify for Cross-VPC Access to change the value of advertised.listeners IP address to the private IP addresses recorded in 7 and 8. Click Save.

Each IP address must match the corresponding port ID. Otherwise, the network will be disconnected.

Figure 8 Changing the advertised.listeners IP addresses

Verifying Connectivity

Check whether messages can be created and retrieved by referring to Connecting to Kafka Using the Client (SASL Disabled) or Connecting to Kafka Using the Client (SASL Enabled).

Notes:

The address for connecting to a Kafka instance is in the format of "advertised.listeners IP:9011". For example, the addresses for connecting to the Kafka instance shown in Figure 8 are 192.168.0.71:9011,192.168.0.11:9011,192.168.0.21:9011.
Configure inbound rules for the security group of the Kafka instance to allow access from 198.19.128.0/17 over port 9011.
If a network access control list (ACL) has been configured for the subnet of this instance, configure inbound rules for the network ACL to allow access from 198.19.128.0/17 and from the subnet used by the VPC endpoint.