Creating an Elasticsearch Cluster

Scenario

Table 1 lists key parameters that differentiate between different types of clusters.

Prerequisites

You have planned the Elasticsearch clusters that need to be created by following the instructions in Elasticsearch Cluster Planning Suggestions.

Creating a Cluster

1. Log in to the CSS management console.
2. On the Dashboard page, click Create Cluster in the upper right corner. The Create Cluster page is displayed.

   Alternatively, choose Clusters > Elasticsearch in the navigation tree on the left. Click Create Cluster in the upper right corner. The Create Cluster page is displayed.

3. On the Basic Configuration page, configure basic information and resources for the Elasticsearch cluster.

   Table 2 Basic configuration of the Elasticsearch cluster

   Parameter	Description
   Billing Mode	Select Yearly/Monthly or Pay-per-use. Yearly/monthly: You pay for the cluster by year or month, in advance. The service duration range is one month to three years. If you plan to use a cluster for more than nine months, you are advised to purchase a yearly package for a better price. Pay-per-use: You are billed by actual duration of use, with a billing cycle of one hour. For example, 58 minutes of usage will be rounded up to an hour and billed.
   Required Duration	The duration for which the purchased EIP will be used. The duration must be specified if the Billing Mode is set to Yearly/Monthly. Configure automatic renewal if necessary.
   Region	Select the region where the cluster is located. ECSs in different regions cannot communicate with each other over an intranet. For lower network latency and quicker resource access, select the nearest region.
   AZ	Select AZs associated with the cluster region. A maximum of three AZs can be configured. For details about the use of multiple AZs, see Planning Cluster AZs.
   Type	Select Elasticsearch.
   Version	Select a cluster version from the drop-down list box.
   Name	Cluster name, which contains 4 to 32 characters. Only letters, numbers, hyphens (-), and underscores (_) are allowed and the value must start with a letter.
   Nodes	Number of nodes in the cluster. Select a number from 1 to 32. You are advised to configure three or more nodes to ensure high availability of the cluster. If Master node and Client node are both unselected, data nodes will be used for all of the following purposes: cluster management, data storage, cluster access, and data analysis. To ensure reliability, a cluster should have a least three nodes. If Master node is selected but Client node is not, data nodes will be used for data storage, cluster access, and data analysis. If Master node is unselected but Client node is selected, data nodes will be used for data storage and cluster management. If Master node and Client node are both selected, data nodes will be used for data storage only. NOTE: If the number of data nodes in a cluster is not an integer multiple of that of AZs, data in the cluster may be unevenly distributed, affecting data query or write performance.
   CPU Architecture	x86 and Kunpeng are supported. The supported types depend on the actual regional environment.
   Node Specifications	Data node flavor. You can select a specified specification based on your needs. Each cluster supports only one specification. For details, see ECS Types.
   Node Storage Type	If you select EVS for node storage, you need to further select the EVS disk type for data nodes of the cluster. Options include Common I/O, High I/O, Ultra-high I/O, and Extreme SSD. NOTE: If the type of storage in use is not supported, the storage type is not displayed.
   Node Storage Capacity	Data node storage capacity. Its value range varies with node specifications. The node storage capacity must be a multiple of 20. The node storage capacity cannot be reduced once the cluster is created. Choose an appropriate capacity based on service needs.
   Master node	The master node is responsible for cluster management, such as metadata management, index creation and deletion, and shard allocation. It plays a critical role in metadata management, node management, stability guarantee, and cluster operation control for large-scale clusters. After enabling the master node, specify Node Specifications, Nodes, and Node Storage Type. The value of Nodes must be an odd number greater than or equal to 3. Up to nine nodes are supported. The value of Node Storage Capacity is fixed. You can select a storage type based on your needs.
   Client node	Client nodes receive and coordinate external requests, such as search and write requests. They play an important role in handling high-load queries, complex aggregations, managing a large number of shards, and improving cluster scalability. After enabling the client node, specify Node Specifications, Nodes and Node Storage Type. The value of Nodes ranges from 1 to 32. The value of Node Storage Capacity is fixed. You can select a storage type based on your needs.
   Cold data node	Cold data nodes are used to store query latency-insensitive data in large quantities. They offer an effective way to manage large datasets and cut storage costs. After enabling cold data node, configure Node Specifications, Nodes, Node Storage Type, and Node Storage Capacity. The value of Nodes ranges from 1 to 32. Select Node Storage Type and Node Storage Capacity as required. When cold data nodes are enabled, users can switch between cold and hot data nodes. For details, see Switching Between Hot and Cold Storage for an Elasticsearch Cluster. NOTE: If the number of cold data nodes in a cluster is not an integer multiple of that of AZs, data in the cluster may be unevenly distributed, affecting data query or write performance.
   Enterprise Project	When creating a CSS cluster, you can bind an enterprise project to the cluster if you have enabled the enterprise project function. Select an enterprise project from the Enterprise Project drop-down list, or click View Enterprise Project to go to the Enterprise Project Management Service page and check existing enterprise projects.

4. Click Next: Network.
5. On the Network page, configure the network settings and security mode for the Elasticsearch cluster.

   Table 3 Network settings for the Elasticsearch cluster

   Parameter	Description
   VPC	Specify a VPC to isolate the cluster's network. Click View VPC to go to the VPC management console and check the created VPCs or VPCs shared with the current account. If no VPC is available, contact the CSS administrator to create a new VPC. For details, see Creating a VPC and Subnet. NOTE: The VPC must contain CIDRs. Otherwise, cluster creation will fail. By default, a created VPC contains CIDRs.
   Subnet	A subnet provides dedicated network resources that are isolated from other networks, improving network security. Select a subnet needed by the cluster in the current VPC. You may select a subnet in a shared VPC. If the selected VPC supports IPv6, select whether to assign an IPv6 address from the drop-down list. Elasticsearch 7.6.2 and 7.10.2 clusters support automatic assignment of IPv6 addresses.
   Security Group	A security group serves as a virtual firewall that provides access control policies for clusters. Select a security group for the cluster. Click View Security Group to go to the security group list, where you can view details about security groups. NOTE: For enable cluster access, ensure that port 9200 is allowed by the security group. If your cluster version is 7.6.2 or later, ensure that all the ports used for communication between nodes in the same security group are allowed. If such settings cannot be configured, ensure at least the access to port 9300 is allowed. After the port 9300 is enabled, if the cluster disk usage is high, delete expired data to release the disk storage space. For details, see How Do I Clear Expired Data to Release Storage Space?
   Security Mode	Whether to enable the security mode for the cluster. The security mode is enabled by default. In security mode, a cluster's communication is encrypted and access to the cluster requires user authentication. This is why the Administrator Username and Administrator Password of the cluster are needed. The default administrator username is admin. Set and confirm the Administrator Password. This password will be required when you access this cluster. If Security Mode is disabled, a cluster in non-security mode will be created. With such a cluster, access to the cluster will not require user authentication, and data will be transmitted in plaintext using HTTP. Make sure the customer is in a secure environment, and do not expose the cluster access interface to the public network.
   HTTPS Access	HTTPS access can be enabled only when security mode is enabled for the cluster. With HTTPS access enabled, communication will be encrypted when you access the cluster. NOTE: A cluster in security mode uses HTTPS for communication, but its read performance will not be as good as a non-security mode cluster that uses HTTP. The performance loss is estimated at around 20% under high concurrency. If you want fast read performance as well as the isolation and permission control (such as indexes, documents, and fields) enabled by the security mode, you can disable HTTPS Access. After HTTPS Access is disabled, HTTP protocol is used for cluster communication. In this case, data security cannot be ensured and public IP address cannot be used.
   Public IP Address	This parameter is available only when Security Mode and HTTPS Access are enabled. When Public IP Address is enabled, a public IP address is automatically assigned, which will enable access to the security cluster from the Internet. For details, see Configuring Public Network Access for an Elasticsearch Cluster.

6. Click Next: Advanced Settings.
7. On the Advanced Settings page, configure a snapshot policy and other advanced settings for the Elasticsearch cluster.
   1. Set a cluster snapshot policy.

      The cluster snapshot function is enabled by default. You can also disable it by toggling off Cluster Snapshot. To store snapshots automatically created in OBS, an agency will need to be created in order to access OBS. Fees will be incurred for using standard OBS storage.

      Table 4 Basic configuration for a cluster snapshot policy

      Parameter	Description
      OBS Bucket	Select an OBS bucket for storing snapshots from the drop-down list box. You can also click Create Bucket on the right to create an OBS bucket. For details, see Creating a Bucket. The created or existing OBS bucket must meet the following requirements: Storage Class is Standard. Region must be the same as that of the created cluster.
      Backup Path	Storage path of the snapshot in the OBS bucket. The backup path cannot: Contain the following characters: \:*?"<>| Start with a slash (/). Start or end with a period (.). Exceed 1023 characters.
      IAM Agency	To store snapshot data to an OBS bucket, you must have the required OBS access permissions. Select an IAM agency to grant the current account the permission to access and use OBS. If you are configuring an agency for the first time, click Automatically Create IAM Agency to create css-obs-agency. If there is an IAM agency automatically created earlier, you can click One-click authorization to delete the OBS Administrator permissions, and add the following custom policies instead to implement more refined permissions control. "obs:bucket:GetBucketLocation", "obs:object:GetObjectVersion", "obs:object:GetObject", "obs:object:DeleteObject", "obs:bucket:HeadBucket", "obs:bucket:GetBucketStoragePolicy", "obs:object:DeleteObjectVersion", "obs:bucket:ListBucketVersions", "obs:bucket:ListBucket", "obs:object:PutObject" To use Automatically Create IAM Agency and One-click authorization, the following minimum permissions are needed: "iam:agencies:listAgencies", "iam:roles:listRoles", "iam:agencies:getAgency", "iam:agencies:createAgency", "iam:permissions:listRolesForAgency", "iam:permissions:grantRoleToAgency", "iam:permissions:listRolesForAgencyOnProject", "iam:permissions:revokeRoleFromAgency", "iam:roles:createRole" To use an IAM agency, the following minimum permissions are needed: "iam:agencies:listAgencies", "iam:agencies:getAgency", "iam:permissions:listRolesForAgencyOnProject", "iam:permissions:listRolesForAgency"

      Table 5 Setting Automatic Snapshot Creation

      Parameter	Description
      Snapshot Name Prefix	The snapshot name prefix contains 1 to 32 characters and must start with a lowercase letter. Only lowercase letters, digits, hyphens (-), and underscores (_) are allowed. A snapshot name consists of a snapshot name prefix and a timestamp, for example, snapshot-1566921603720.
      Time Zone	Time zone for the backup time, which cannot be changed. Specify Backup Started Time based on the time zone.
      Backup Start Time	The time when the backup starts automatically every day. You can specify this parameter only in full hours, for example, 00:00 or 01:00. The value ranges from 00:00 to 23:00. Select a time from the drop-down list.
      Retained Snapshots	Number of automatic snapshots to be retained. The value ranges from 1 to 90. The system automatically deletes excess snapshots every half hour. (The expiration deletion policy applies only to the snapshots that were automatically taken at the same frequency as the current automated snapshot creation policy.) NOTE: If the snapshot creation interval is short or if the data size of indexes is large, the number of automatic snapshots retained may not reach the value set using this parameter.

   2. Configure advanced settings for the cluster. Select Default or Custom.
      • Default: VPC Endpoint Service, Kibana Public Access, and Tags are disabled by default. You can manually enable these settings after the cluster is created.
      • Custom: You can enable VPC Endpoint Service, Kibana Public Access, and Tags as required.

      VPC Endpoint Service

      VPC Endpoint Service enables you to access resources across Virtual Private Clouds (VPCs) using a dedicated gateway, without exposing network information of servers. When VPC Endpoint Service is enabled, a VPC endpoint will be created by default. You can select Private Domain Name Creation if necessary. Users will be able to access this cluster across VPCs through node IP addresses or a private domain name.

      

      • If a shared VPC and a subnet within this shared VPC were selected earlier for the cluster on the Network page, VPC Endpoint Service cannot be enabled for the cluster.
      • After VPC Endpoint Service is enabled for a cluster, you will be billed per use for the service. For more information, see Billing Modes.

      Table 6 Configuring VPC Endpoint Service

      Parameter	Description
      Private Domain Name Creation	If Private Domain Name Creation is selected, the system generates a node IP address and also automatically creates a private domain name, which enables users to access this cluster from within the same VPC. If it is not selected, only a node IP address is generated.
      Create professional endpoints	Choose whether to create professional endpoints. If unselected, a basic endpoint will be created. If selected, a professional endpoint will be created. NOTE: If the region where the cluster is located does not support professional endpoints, this option is grayed out. By default, a basic endpoint is created.
      IPv4/IPv6 dual stack network	Whether to enable IPv4/IPv6 dual-stack networking. This option is available only when IPv6 is enabled for the VPC subnet of the cluster and you have selected Create professional endpoints earlier.
      VPC Endpoint Service Whitelist	In VPC Endpoint Service Whitelist, you can add accounts that are allowed to access the cluster using a node IP address or private domain name. Click Add to add accounts in Authorized Account ID. If the authorized account ID is set to *, all users are allowed to access the cluster. Click Delete in the Operation column to delete accounts. NOTE: To obtain your authorized account ID, point to your username in the upper right corner, and choose My Credentials. Copy the value of Account ID.

      Kibana Public Access

      This parameter is available only when security mode is enabled for the cluster. By enabling this option, you can obtain a public IP address for accessing Kibana.

      Table 7 Configuring public network access for Kibana

      Parameter	Description
      Bandwidth	Bandwidth for accessing Kibana through a public IP address Value range: 1 to 100. Unit: Mbit/s
      Access Control	If you disable this function, all IP addresses can access Kibana through the public IP address. If you enable this function, only IP addresses or IP address ranges in the whitelist can access Kibana through the public IP address.
      Whitelist	IP addresses or IP address ranges allowed to access the cluster. Use commas (,) to separate multiple IP addresses or ranges. This parameter can be configured only when Access Control is enabled. You are advised to enable the whitelist. NOTE: The whitelist that controls Kibana public network access depends on whitelist support by the ELB service. After you update the whitelist, the new settings take effect immediately for new connections. For existing persistent connections using the IP addresses that have been removed from the whitelist, the new settings take effect in approximately 1 minute after these connections are disconnected.

      Tags

      Adding tags to clusters can help you identify and manage your cluster resources. You can customize tags or use tags preset by Tag Management Service (TMS).

      

      If your organization has configured tag policies for CSS, add cluster tags based on these policies. If a tag does not comply with the tag policies, cluster creation may fail. Contact the administrator to learn more about tag policies.

      Table 8 Tag rules

      Parameter	Description
      Tag Key	Must be unique in a cluster. Enter up to 64 characters. It can contain only numbers, letters, Chinese characters, and the following special characters: _.:=+-@ The value cannot start or end with a space. Cannot be blank.
      Tag Value	Enter up to 64 characters. It can contain only numbers, letters, Chinese characters, and the following special characters: _.:=+-@ The value cannot start or end with a space. Cannot be blank.

8. Click Next: Confirm Configuration. Check the configuration and click Next to create a cluster.
9. Click Back to Cluster List to switch to the Clusters page. The cluster you created is listed on the displayed page and its status is Creating. If the cluster is successfully created, its status will change to Available.

   If cluster creation fails, try creating the cluster again by rectifying the errors returned.

Follow-up Operations

After an Elasticsearch cluster is created, you are advised to optimize the query performance of the cluster to improve efficiency by referring to Cluster Performance Tuning.