Compute
Elastic Cloud Server
Huawei Cloud Flexus
Bare Metal Server
Auto Scaling
Image Management Service
Dedicated Host
FunctionGraph
Cloud Phone Host
Huawei Cloud EulerOS
Networking
Virtual Private Cloud
Elastic IP
Elastic Load Balance
NAT Gateway
Direct Connect
Virtual Private Network
VPC Endpoint
Cloud Connect
Enterprise Router
Enterprise Switch
Global Accelerator
Management & Governance
Cloud Eye
Identity and Access Management
Cloud Trace Service
Resource Formation Service
Tag Management Service
Log Tank Service
Config
OneAccess
Resource Access Manager
Simple Message Notification
Application Performance Management
Application Operations Management
Organizations
Optimization Advisor
IAM Identity Center
Cloud Operations Center
Resource Governance Center
Migration
Server Migration Service
Object Storage Migration Service
Cloud Data Migration
Migration Center
Cloud Ecosystem
KooGallery
Partner Center
User Support
My Account
Billing Center
Cost Center
Resource Center
Enterprise Management
Service Tickets
HUAWEI CLOUD (International) FAQs
ICP Filing
Support Plans
My Credentials
Customer Operation Capabilities
Partner Support Plans
Professional Services
Analytics
MapReduce Service
Data Lake Insight
CloudTable Service
Cloud Search Service
Data Lake Visualization
Data Ingestion Service
GaussDB(DWS)
DataArts Studio
Data Lake Factory
DataArts Lake Formation
IoT
IoT Device Access
Others
Product Pricing Details
System Permissions
Console Quick Start
Common FAQs
Instructions for Associating with a HUAWEI CLOUD Partner
Message Center
Security & Compliance
Security Technologies and Applications
Web Application Firewall
Host Security Service
Cloud Firewall
SecMaster
Anti-DDoS Service
Data Encryption Workshop
Database Security Service
Cloud Bastion Host
Data Security Center
Cloud Certificate Manager
Edge Security
Managed Threat Detection
Blockchain
Blockchain Service
Web3 Node Engine Service
Media Services
Media Processing Center
Video On Demand
Live
SparkRTC
MetaStudio
Storage
Object Storage Service
Elastic Volume Service
Cloud Backup and Recovery
Storage Disaster Recovery Service
Scalable File Service Turbo
Scalable File Service
Volume Backup Service
Cloud Server Backup Service
Data Express Service
Dedicated Distributed Storage Service
Containers
Cloud Container Engine
SoftWare Repository for Container
Application Service Mesh
Ubiquitous Cloud Native Service
Cloud Container Instance
Databases
Relational Database Service
Document Database Service
Data Admin Service
Data Replication Service
GeminiDB
GaussDB
Distributed Database Middleware
Database and Application Migration UGO
TaurusDB
Middleware
Distributed Cache Service
API Gateway
Distributed Message Service for Kafka
Distributed Message Service for RabbitMQ
Distributed Message Service for RocketMQ
Cloud Service Engine
Multi-Site High Availability Service
EventGrid
Dedicated Cloud
Dedicated Computing Cluster
Business Applications
Workspace
ROMA Connect
Message & SMS
Domain Name Service
Edge Data Center Management
Meeting
AI
Face Recognition Service
Graph Engine Service
Content Moderation
Image Recognition
Optical Character Recognition
ModelArts
ImageSearch
Conversational Bot Service
Speech Interaction Service
Huawei HiLens
Video Intelligent Analysis Service
Developer Tools
SDK Developer Guide
API Request Signing Guide
Terraform
Koo Command Line Interface
Content Delivery & Edge Computing
Content Delivery Network
Intelligent EdgeFabric
CloudPond
Intelligent EdgeCloud
Solutions
SAP Cloud
High Performance Computing
Developer Services
ServiceStage
CodeArts
CodeArts PerfTest
CodeArts Req
CodeArts Pipeline
CodeArts Build
CodeArts Deploy
CodeArts Artifact
CodeArts TestPlan
CodeArts Check
CodeArts Repo
Cloud Application Engine
MacroVerse aPaaS
KooMessage
KooPhone
KooDrive

Cross-VPC Access to a Kafka Instance

Updated on 2024-04-07 GMT+08:00

Context

VPCs are logically isolated from each other. If a Kafka instance and a Kafka client are in different VPCs within a region, they cannot communicate with each other. In this case, you can use one of the following methods to access a Kafka instance across VPCs:

  • Establish a VPC peering connection to allow two VPCs to communicate with each other. For details, see section "VPC Peering Connection" in Virtual Private Cloud User Guide.
  • Use VPC Endpoint (VPCEP) to establish a cross-VPC connection.

Scenario

The following describes how to use VPCEP to implement cross-VPC access.

VPCEP provides two types of resources: VPC endpoint services and VPC endpoints.

  • A VPC endpoint service can be a Kafka instance which is accessed using VPC endpoints.
  • A VPC endpoint is a secure and private channel for connecting a VPC to a VPC endpoint service.
Figure 1 Working principle of accessing a Kafka instance across VPCs

Procedure

Figure 2 Process for accessing a Kafka instance across VPCs

Creating a VPC Endpoint Service

  1. Log in to the management console.
  2. Click in the upper left corner to select a region.

    NOTE:

    Select the region where your Kafka instance is located.

  3. Click in the upper left corner and choose Application > Distributed Message Service for Kafka to open the console of DMS for Kafka.
  4. Click the desired Kafka instance to view the instance details.
  5. In the Advanced Settings section on the Basic Information tab page, obtain the listeners IP addresses and port IDs of the instance for Cross-VPC Access.

    Figure 3 Cross-VPC access–related listeners IP addresses and corresponding port IDs of the Kafka instance

  6. In the Network section on the Basic Information tab page, view the VPC to which the Kafka instance belongs.

    Figure 4 Viewing the VPC to which the Kafka instance belongs

  7. Click the VPC to obtain the VPC ID on the VPC console.

    Figure 5 Obtaining the VPC ID

  8. Call the VPC Endpoint API to create a VPC endpoint service. For details, see "Creating a VPC Endpoint Service" in VPC Endpoint API Reference.

    curl -i -k -H 'Accept:application/json' -H 'Content-Type:application/json;charset=utf8' -X POST -H "X-Auth-Token:$token" -d '{"port_id":"38axxxeac","vpc_id":"706xxx888","ports":[{"protocol":"TCP","client_port":9011,"server_port":9011 }],"approval_enabled":false,"service_type":"interface","server_type":"VM"}' https://{endpoint}/v1/{project_id}/vpc-endpoint-services

    Parameter description:

    • token: an access credential issued to an IAM user to bear its identity and permissions. For details on how to obtain a token, see Obtaining a User Token.
    • port_id: one of the port IDs obtained in 5.
    • vpc_id: VPC ID obtained in 7.
    • endpoint: VPCEP endpoint obtained from Regions and Endpoints. The region must be the same as that of the Kafka instance.
    • project_id: project ID obtained from Obtaining a Project ID. The region must be the same as that of the Kafka instance.

    Record the value of service_name in the response. This parameter indicates the name of the VPC endpoint service.

  9. Repeat 8 to create VPC endpoint services for other port IDs obtained in 5 and record the VPC endpoint service names.

(Optional) Adding a Whitelist Record

If the Kafka client and Kafka instance belong to different accounts, add the ID of the account to which the Kafka client belongs to the whitelist of the endpoint service. For details, see Add a Whitelist Record.

Creating a VPC Endpoint

  1. Click in the upper left corner of the management console. Then choose Network > VPC Endpoint.
  2. Click Create VPC Endpoint.
  3. Set the following parameters:

    • Region: Select the region that the Kafka instance is in.
    • Service Category: Select Find a service by name.
    • VPC Endpoint Service Name: Enter the VPC endpoint service name recorded in 8 and click Verify. If Service name found is displayed, proceed with subsequent operations.
    • VPC: Select the VPC that the Kafka client is in.
    • Subnet: Select the subnet that the Kafka client is in.
    • Private IP Address: Select Automatic.

    Retain the default values for other parameters. For details, see Creating a VPC Endpoint.

    Figure 6 VPC endpoint parameters

  4. Click Create Now.
  5. Confirm the configurations and submit the request.
  6. Go back to the VPC endpoint list and check whether the status of the created VPC endpoint has changed to Accepted. The Accepted state means that the VPC endpoint has been connected to the VPC endpoint service.

    Figure 7 Checking the VPC endpoint status

  7. Click the VPC endpoint ID. On the Summary tab page, obtain the private IP address.

    You can use the private IP address to access the VPC endpoint service.

    Figure 8 Viewing the private IP address

  8. Repeat 1 to 7 to create a VPC endpoint for each VPC endpoint service created in 9, and view and record the private IP addresses of the VPC endpoint services.

Changing the advertised.listeners IP Address

  1. Click in the upper left corner and choose Application > Distributed Message Service for Kafka to open the console of DMS for Kafka.
  2. Click the desired Kafka instance to view the instance details.
  3. On the Advanced Settings section of the Basic Information tab page, click Modify for Cross-VPC Access to change the value of advertised.listeners IP address to the private IP addresses recorded in 7 and 8. Click Save.

    NOTICE:

    Each IP address must match the corresponding port ID. Otherwise, the network will be disconnected.

    Figure 9 Changing the advertised.listeners IP addresses

Verifying Connectivity

Check whether messages can be created and retrieved by referring to Accessing a Kafka Instance Without SASL or Accessing a Kafka Instance with SASL.

Notes:

  • The address for connecting to a Kafka instance is in the format of "advertised.listeners IP:9011". For example, the addresses for connecting to the Kafka instance shown in Figure 9 are 192.168.0.92:9011,192.168.0.93:9011,192.168.0.94:9011.
  • Configure inbound rules for the security group of the Kafka instance to allow access from 198.19.128.0/17 over port 9011.
  • If a network access control list (ACL) has been configured for the subnet of this instance, configure inbound rules for the network ACL to allow access from 198.19.128.0/17 and from the subnet used by the VPC endpoint.
NOTE:

198.19.128.0/17 is the network segment allocated to the VPCEP service. To use VPCEP, allow access from this network segment.

We use cookies to improve our site and your experience. By continuing to browse our site you accept our cookie policy. Find out more

Feedback

Feedback

Feedback

0/500

Selected Content

Submit selected content with the feedback