Configuring Kafka Log Push

Constraints

• An API can be bound to only one plug-in policy of the same type in the same environment. A plug-in policy that has been bound to an API cannot be deleted.
• A maximum of five Kafka log push policies can be created on a ROMA Connect instance.
• In the log data to be pushed, the response data does not support Transfer Encoding response header parameters.
• By default, a maximum of 4 KB of logs can be pushed. Excess logs will be truncated.
• The request body and response body in pushed logs are calculated in UTF-8 bytes.

Creating a Kafka Log Push Policy

1. Log in to the ROMA Connect console. On the Instances page, click View Console of an instance.
2. In the navigation pane on the left, choose API Connect > API Policies. On the Policies tab, click Create Policy.
3. On the Select Policy Type page, select Kafka Log Push in the Plug-ins area.
4. On the page displayed, configure plug-in policy information.

   Table 1 Policy configuration

   Parameter	Description
   Name	Enter a policy name. Using naming rules facilitates future search.
   Type	Fixed as Kafka Log Push.
   Scope	Specify the scope to view the policy. Integration application: Each policy belongs to an integration application. Only users who have the permission on the integration application can view and use the policy. All: All users in the current instance can view and use the policy.
   Integration Application	Mandatory for Scope set to Integration application. Select an integration application for the policy. If none is available, click Create Integration Application on the right to create one.
   Description	Describe the policy.
   Policy Content: Configure the policy in a form or script. For details about how to configure a script, see Script Configuration Example.
   Policy Information
   Broker Addresses	Connection addresses of target Kafka brokers to which logs are to be pushed. Use commas (,) to separate multiple addresses.
   Topic	Topic of the target Kafka to report logs to.
   Key	Enter the key value of a message so that the message will be stored in a specified partition of Kafka. It can be used as an ordered message queue. If this parameter is left empty, messages are stored in different message partitions in a distributed manner.
   Retry	Configuration for retrying when logs fail to be pushed to Kafka. Retry Times: the number of retry attempts in case of a failure Range: 0 to 5 Retry Interval: the interval of retry attempts in case of a failure, in seconds Range: 1 to 10
   SASL Configuration
   Security Protocol	Protocol used for connecting to the target Kafka. PLAINTEXT: user authentication protocol of the default access point. SASL_PLAINTEXT: SASL user authentication protocol. SASL_SSL: SSL user authentication protocol.
   Message Tx/Rx Mechanism	Message transmission and receiving mechanism of the target Kafka. The default value is PLAIN.
   SASL Username	Mandatory for Security Protocol set to SASL_PLAINTEXT or SASL_SSL. Username used for SASL or SSL authentication.
   SASL Password	Mandatory for Security Protocol set to SASL_PLAINTEXT or SASL_SSL. Password used for SASL or SSL authentication.
   Confirm SASL Password	Mandatory for Security Protocol set to SASL_PLAINTEXT or SASL_SSL. Enter the SASL or SSL authentication password again.
   Certificate Content	Available for Security Protocol set to SASL_SSL. CA certificate used for SSL authentication.
   Metadata Configuration
   System Metadata	System fields that need to be included in pushed logs. The following fields are selected by default: start_time, request_id, client_ip, request_time, http_status, scheme, request_method, upstream_addr, upstream_status, upstream_response_time, http_x_forwarded_for, http_user_agent, and error_type. You can also specify other system fields.
   Request Data	API request information that needs to be included in pushed logs. The log contains the request header: Specify a header that needs to be included. Use commas (,) to separate multiple fields. The asterisk (*) can be used as a wildcard. The log contains the request QueryString: Specify a query string that needs to be included. Use commas (,) to separate multiple fields. The asterisk (*) can be used as a wildcard. The log contains the request body: If this option is selected, logs will contain the body of API requests.
   Response Data	API response information that needs to be included in pushed logs. The log contains the response header: Specify a header that needs to be included. Use commas (,) to separate multiple fields. The asterisk (*) can be used as a wildcard. The log contains the response body: If this option is selected, logs will contain the body of API request responses.
   Custom Authentication	Custom authentication information that needs to be included in pushed logs. Frontend: Enter a response field of frontend authentication that needs to be included. Separate multiple fields with commas (,). Backend: Enter a response field of backend authentication that needs to be included. Separate multiple fields with commas (,).

5. Click OK.

   After a plug-in policy is created, perform Binding a Plug-in Policy to an API for the policy to take effect for the API.

Binding a Plug-in Policy to an API

1. On the Policies tab, filter policies by Kafka Log Push.
2. Click the name of a policy to go to the details page.
3. On the APIs tab, select the environment of the APIs you want to bind the policy to and click Bind to APIs.
4. On the page displayed, select the APIs to bind the policy to.

   APIs can be filtered by API group and API name.

5. Click OK.

Script Configuration Example

{
  "broker_list": [
    "10.10.10.10:81",
    "10.10.10.11:82",
    "10.10.10.12:83"
  ],
  "topic": "Topic-test",
  "key": "aaa",
  "max_retry_count": 0,
  "retry_backoff": 1,
  "sasl_config": {
    "security_protocol": "PLAINTEXT",
    "sasl_mechanisms": "PLAIN",
    "sasl_username": "",
    "sasl_password": "",
    "ssl_ca_content": ""
  },
  "meta_config": {
    "system": {
      "start_time": true,
      "request_id": true,
      "client_ip": true,
      "api_id": false,
      "user_name": false,
      "app_id": false,
      "request_time": true,
      "http_status": true,
      "server_protocol": false,
      "scheme": true,
      "request_method": true,
      "host": false,
      "api_uri_mode": false,
      "uri": false,
      "request_size": false,
      "response_size": false,
      "upstream_uri": false,
      "upstream_addr": true,
      "upstream_status": true,
      "upstream_connect_time": false,
      "upstream_header_time": false,
      "upstream_response_time": true,
      "http_x_forwarded_for": true,
      "http_user_agent": true,
      "error_type": true,
      "access_model1": false,
      "access_model2": false,
      "inner_time": false,
      "proxy_protocol_vni": false,
      "proxy_protocol_vpce_id": false,
      "proxy_protocol_addr": false,
      "body_bytes_sent": false,
      "api_name": false,
      "app_name": false,
      "provider_app_id": false,
      "all_upstream_response_time": false,
      "region_id": false,
      "auth_type": false,
      "response_source": false,
      "provider_app_name": false,
      "custom_data_log01": false,
      "custom_data_log02": false,
      "custom_data_log03": false,
      "custom_data_log04": false,
      "custom_data_log05": false,
      "custom_data_log06": false,
      "custom_data_log07": false,
      "custom_data_log08": false,
      "custom_data_log09": false,
      "custom_data_log10": false
    },
    "call_data": {
      "log_request_header": true,
      "request_header_filter": "aaa",
      "log_request_query_string": true,
      "request_query_string_filter": "bbb",
      "log_request_body": true,
      "log_response_header": true,
      "response_header_filter": "ccc",
      "log_response_body": true,
      "custom_authorizer": {
        "frontend": [],
        "backend": []
      }
    }
  }
}