Creating a Logstash Configuration File

Constraints

• The size of each configuration file cannot exceed 100 KB.
• A Logstash cluster can have a maximum of 50 configuration files. There can be a maximum of five configuration files that are in the Verifying state at the same time.

Prerequisites

You have obtained information about servers or clusters at the data source and destination, including IP addresses, usernames, and passwords.

Creating a Configuration File

1. Log in to the CSS management console.
2. In the navigation pane on the left, choose Clusters > Logstash.
3. In the displayed cluster list, locate the row that contains the target cluster, and click Configuration Center in the Operation column. The Configuration Center page is displayed.

   Alternatively, in the cluster list, click the cluster name to go to the cluster information page. In the navigation pane on the left, choose Configuration Center.

4. Click Create in the upper right corner. On the Create Configuration File page that is displayed, edit the configuration file.

   Table 1 Creating a configuration file

   Parameter	Description
   Name	User-defined configuration file name. It can contain only letters, digits, hyphens (-), and underscores (_), and must start with a letter. The minimum length is 4 characters.
   Configuration File Content	You can configure the configuration file content by modifying a template, and develop it from scratch. To use a configuration file template, expand System Templates or Custom Templates, select a template based on its description, and click Apply in the Operation column to copy the content of the template to the Configuration File Content area, where you can further modify relevant items. System templates are preset configuration file templates of CSS. For details, see Logstash Configuration File Templates. Custom templates are templates added by users. NOTE: The size of each configuration file cannot exceed 100 KB.
   Hidden Content	For items that you enter in this box, the corresponding strings will be replaced with *** in the configurations. Enter sensitive strings that you want to hide, and press Enter. You can enter a maximum of 20 strings, each with a maximum length of 512 bytes.

5. Click Next to configure Logstash pipeline parameters.

   Table 2 Pipeline parameters

   Parameter	Description
   pipeline.workers	Number of worker threads that will execute the Filters and Outputs stages of the pipeline in parallel. Value range: 1–20 Default value: Number of CPU cores
   pipeline.batch.size	Maximum number of events that a worker thread collects from inputs before attempting to execute its filters and outputs. A larger value is more effective but increases memory overhead. Default value: 125
   pipeline.batch.delay	Maximum wait time for each new event before scheduling small batches to the pipeline worker thread and creating a pipeline event batch. Default value: 50 Unit: milliseconds.
   queue.type	An internal queue for event buffering. memory indicates a traditional memory-based queue. persisted indicates a disk-based ACKed persistent queue. Default value: memory
   queue.checkpoint.writes	Maximum number of written events before a forced checkpoint. This parameter needs to be set only when queue.type is set to persisted. Default value: 1024
   queue.max_bytes	Total capacity of each persistent queue in MB. Ensure that the disk capacity is greater than this value. This parameter needs to be set only when queue.type is set to persisted. Default value: 1024 Unit: MB

6. After the configuration is complete, click Create.

   On the Configuration Center page, you can check the created configuration file. If its status changes to Available, it has been successfully created.

Managing Configuration Files

You can edit existing configuration files, set a configuration file as a custom template, and delete configuration files.