El contenido no se encuentra disponible en el idioma seleccionado. Estamos trabajando continuamente para agregar más idiomas. Gracias por su apoyo.

Compute
Elastic Cloud Server
Huawei Cloud Flexus
Bare Metal Server
Auto Scaling
Image Management Service
Dedicated Host
FunctionGraph
Cloud Phone Host
Huawei Cloud EulerOS
Networking
Virtual Private Cloud
Elastic IP
Elastic Load Balance
NAT Gateway
Direct Connect
Virtual Private Network
VPC Endpoint
Cloud Connect
Enterprise Router
Enterprise Switch
Global Accelerator
Management & Governance
Cloud Eye
Identity and Access Management
Cloud Trace Service
Resource Formation Service
Tag Management Service
Log Tank Service
Config
OneAccess
Resource Access Manager
Simple Message Notification
Application Performance Management
Application Operations Management
Organizations
Optimization Advisor
IAM Identity Center
Cloud Operations Center
Resource Governance Center
Migration
Server Migration Service
Object Storage Migration Service
Cloud Data Migration
Migration Center
Cloud Ecosystem
KooGallery
Partner Center
User Support
My Account
Billing Center
Cost Center
Resource Center
Enterprise Management
Service Tickets
HUAWEI CLOUD (International) FAQs
ICP Filing
Support Plans
My Credentials
Customer Operation Capabilities
Partner Support Plans
Professional Services
Analytics
MapReduce Service
Data Lake Insight
CloudTable Service
Cloud Search Service
Data Lake Visualization
Data Ingestion Service
GaussDB(DWS)
DataArts Studio
Data Lake Factory
DataArts Lake Formation
IoT
IoT Device Access
Others
Product Pricing Details
System Permissions
Console Quick Start
Common FAQs
Instructions for Associating with a HUAWEI CLOUD Partner
Message Center
Security & Compliance
Security Technologies and Applications
Web Application Firewall
Host Security Service
Cloud Firewall
SecMaster
Anti-DDoS Service
Data Encryption Workshop
Database Security Service
Cloud Bastion Host
Data Security Center
Cloud Certificate Manager
Edge Security
Managed Threat Detection
Blockchain
Blockchain Service
Web3 Node Engine Service
Media Services
Media Processing Center
Video On Demand
Live
SparkRTC
MetaStudio
Storage
Object Storage Service
Elastic Volume Service
Cloud Backup and Recovery
Storage Disaster Recovery Service
Scalable File Service Turbo
Scalable File Service
Volume Backup Service
Cloud Server Backup Service
Data Express Service
Dedicated Distributed Storage Service
Containers
Cloud Container Engine
SoftWare Repository for Container
Application Service Mesh
Ubiquitous Cloud Native Service
Cloud Container Instance
Databases
Relational Database Service
Document Database Service
Data Admin Service
Data Replication Service
GeminiDB
GaussDB
Distributed Database Middleware
Database and Application Migration UGO
TaurusDB
Middleware
Distributed Cache Service
API Gateway
Distributed Message Service for Kafka
Distributed Message Service for RabbitMQ
Distributed Message Service for RocketMQ
Cloud Service Engine
Multi-Site High Availability Service
EventGrid
Dedicated Cloud
Dedicated Computing Cluster
Business Applications
Workspace
ROMA Connect
Message & SMS
Domain Name Service
Edge Data Center Management
Meeting
AI
Face Recognition Service
Graph Engine Service
Content Moderation
Image Recognition
Optical Character Recognition
ModelArts
ImageSearch
Conversational Bot Service
Speech Interaction Service
Huawei HiLens
Video Intelligent Analysis Service
Developer Tools
SDK Developer Guide
API Request Signing Guide
Terraform
Koo Command Line Interface
Content Delivery & Edge Computing
Content Delivery Network
Intelligent EdgeFabric
CloudPond
Intelligent EdgeCloud
Solutions
SAP Cloud
High Performance Computing
Developer Services
ServiceStage
CodeArts
CodeArts PerfTest
CodeArts Req
CodeArts Pipeline
CodeArts Build
CodeArts Deploy
CodeArts Artifact
CodeArts TestPlan
CodeArts Check
CodeArts Repo
Cloud Application Engine
MacroVerse aPaaS
KooMessage
KooPhone
KooDrive
On this page

Using Cascaded Agents to Collect Static Logs from Local Host to HBase

Updated on 2024-10-09 GMT+08:00

This section describes how to use the Flume client to collect static logs from a local host and save them to the flume_test HBase table. In this scenario, multi-level agents are cascaded.

NOTE:

By default, the cluster network environment is secure and the SSL authentication is not enabled during the data transmission process. For details about how to use the encryption mode, see Configuring an Encrypted Flume Data Collection Task. The configuration applies to scenarios where only the server is configured, for example, Spooldir Source+File Channel+HBase Sink.

Prerequisites

  • The cluster has been installed, including the HBase and Flume services.
  • The Flume client has been installed.
  • The network environment of the cluster is secure.
  • An HBase table has been created by running the create 'flume_test', 'cf' command.
  • The MRS cluster administrator has understood service requirements and prepared HBase administrator flume_hbase.

Procedure

  1. On FusionInsight Manager, choose System > User and choose More > Download Authentication Credential to download the Kerberos certificate file of user flume_hbase and save it to the local host.

    Figure 1 Downloading the authentication credential

  2. Configure the client parameters of the Flume role.

    1. Use the Flume configuration tool on FusionInsight Manager to configure the Flume role client parameters and generate a configuration file.
      1. Log in to FusionInsight Manager and choose Cluster > Services. On the page that is displayed, choose Flume. On the displayed page, click the Configuration Tool tab.
        Figure 2 Choosing Configuration Tool
      2. Set Agent Name to client. Select the source, channel, and sink to be used, drag them to the GUI on the right, and connect them.

        Use SpoolDir Source, File Channel, and Avro Sink.

        Figure 3 Example for the Flume configuration tool
      3. Double-click the source, channel, and sink. Set corresponding configuration parameters by referring to Table 1 based on the actual environment.
        NOTE:
        • If you want to continue using the properties.propretites file by modifying it, log in to FusionInsight Manager, choose Cluster > Name of the desired cluster > Services. On the page that is displayed, choose Flume. On the displayed page, click the Configuration Tool tab, click Import, import the file, and modify the configuration items related to non-encrypted transmission.
        • It is recommended that the numbers of Sources, Channels, and Sinks do not exceed 40 during configuration file import. Otherwise, the response time may be very long.
        Table 1 Parameters to be modified for the Flume role client

        Parameter

        Description

        Example Value

        Name

        The value must be unique and cannot be left blank.

        test

        spoolDir

        Specifies the directory where the file to be collected resides. This parameter cannot be left blank. The directory needs to exist and have the write, read, and execute permissions on the flume running user.

        /srv/BigData/hadoop/data1/zb

        trackerDir

        Specifies the path for storing the metadata of files collected by Flume.

        /srv/BigData/hadoop/data1/tracker

        batchSize

        Specifies the number of events that Flume sends in a batch (number of data pieces). A larger value indicates higher performance and lower timeliness.

        61200

        dataDirs

        Specifies the directory for storing buffer data. The run directory is used by default. Configuring multiple directories on disks can improve transmission efficiency. Use commas (,) to separate multiple directories. If the directory is inside the cluster, the /srv/BigData/hadoop/dataX/flume/data directory can be used. dataX ranges from data1 to dataN. If the directory is outside the cluster, it needs to be independently planned.

        /srv/BigData/hadoop/data1/flume/data

        checkpointDir

        Specifies the directory for storing the checkpoint information, which is under the run directory by default. If the directory is inside the cluster, the /srv/BigData/hadoop/dataX/flume/checkpoint directory can be used. dataX ranges from data1 to dataN. If the directory is outside the cluster, it needs to be independently planned.

        /srv/BigData/hadoop/data1/flume/checkpoint

        transactionCapacity

        Specifies the transaction size, that is, the number of events in a transaction that can be processed by the current Channel. The size cannot be smaller than the batchSize of Source. Setting the same size as batchSize is recommended.

        61200

        hostname

        Specifies the name or IP address of the host whose data is to be sent. This parameter cannot be left blank. Name or IP address must be configured to be the name or IP address that the Avro source associated with it.

        192.168.108.11

        port

        Specifies the port that sends the data. This parameter cannot be left blank. It must be consistent with the port that is monitored by the connected Avro Source.

        21154

        ssl

        Specifies whether to enable the SSL authentication. (You are advised to enable this function to ensure security.)

        Only Sources of the Avro type have this configuration item.

        • true indicates that the function is enabled.
        • false indicates that the client authentication function is not enabled.

        false

      4. Click Export to save the properties.properties configuration file to the local.
    2. Upload the properties.properties file to flume/conf/ under the installation directory of the Flume client.

  3. Configure the server parameters of the Flume role and upload the configuration file to the cluster.

    1. Use the Flume configuration tool on the FusionInsight Manager portal to configure the server parameters and generate the configuration file.
      1. Log in to FusionInsight Manager and choose Cluster > Services. On the page that is displayed, choose Flume. On the displayed page, click the Configuration Tool tab.
        Figure 4 Choosing Configuration Tool
      2. Set Agent Name to server. Select the source, channel, and sink to be used, drag them to the GUI on the right, and connect them.

        For example, use Avro Source, File Channel, and HBase Sink.

        Figure 5 Example for the Flume configuration tool
      3. Double-click the source, channel, and sink. Set corresponding configuration parameters by referring to Table 2 based on the actual environment.
        NOTE:
        • If the server parameters of the Flume role have been configured, you can choose Cluster > Name of the desired cluster > Services > Flume > Instance on FusionInsight Manager. Then select the corresponding Flume role instance and click the Download button behind the flume.config.file parameter on the Instance Configurations page to obtain the existing server parameter configuration file. Choose Cluster > Name of the desired cluster > Services > Flume > Configuration Tool > Import, import the file, and modify the configuration items related to non-encrypted transmission.
        • It is recommended that the numbers of Sources, Channels, and Sinks do not exceed 40 during configuration file import. Otherwise, the response time may be very long.
        • A unique checkpoint directory needs to be configured for each File Channel.
        Table 2 Parameters to be modified for the Flume role server

        Parameter

        Description

        Example Value

        Name

        The value must be unique and cannot be left blank.

        test

        bind

        Specifies the IP address to which Avro Source is bound. This parameter cannot be left blank. It must be configured as the IP address that the server configuration file will upload.

        192.168.108.11

        port

        Specifies the ID of the port that the Avro Source monitors. This parameter cannot be left blank. It must be configured as an unused port.

        21154

        ssl

        Specifies whether to enable the SSL authentication. (You are advised to enable this function to ensure security.)

        Only Sources of the Avro type have this configuration item.

        • true indicates that the function is enabled.
        • false indicates that the client authentication function is not enabled.

        false

        dataDirs

        Specifies the directory for storing buffer data. The run directory is used by default. Configuring multiple directories on disks can improve transmission efficiency. Use commas (,) to separate multiple directories. If the directory is inside the cluster, the /srv/BigData/hadoop/dataX/flume/data directory can be used. dataX ranges from data1 to dataN. If the directory is outside the cluster, it needs to be independently planned.

        /srv/BigData/hadoop/data1/flumeserver/data

        checkpointDir

        Specifies the directory for storing the checkpoint information, which is under the run directory by default. If the directory is inside the cluster, the /srv/BigData/hadoop/dataX/flume/checkpoint directory can be used. dataX ranges from data1 to dataN. If the directory is outside the cluster, it needs to be independently planned.

        /srv/BigData/hadoop/data1/flumeserver/checkpoint

        transactionCapacity

        Specifies the transaction size, that is, the number of events in a transaction that can be processed by the current Channel. The size cannot be smaller than the batchSize of Source. Setting the same size as batchSize is recommended.

        61200

        table

        Specifies the HBase table name. This parameter cannot be left blank.

        flume_test

        columnFamily

        Specifies the HBase column family name. This parameter cannot be left blank.

        cf

        batchSize

        Specifies the maximum number of events written to HBase by Flume in a batch.

        61200

        kerberosPrincipal

        Specifies the Kerberos authentication user, which is mandatory in security versions. This configuration is required only in security clusters.

        flume_hbase

        kerberosKeytab

        Specifies the file path for Kerberos authentication, which is mandatory in security versions. This configuration is required only in security clusters.

        /opt/test/conf/user.keytab

        NOTE:

        Obtain the user.keytab file from the Kerberos certificate file of the user flume_hbase. In addition, ensure that the user who installs and runs the Flume client has the read and write permissions on the user.keytab file.

      4. Click Export to save the properties.properties configuration file to the local.
    2. Log in to FusionInsight Manager and choose Cluster > Name of the desired cluster > Services > Flume. In the Instances tab, click the Flume role whose configuration file is to be uploaded.
      Figure 6 Clicking the Flume role
    3. On the Instance Configurations page, click Upload File next to flume.config.file and select the properties.properties file.
      Figure 7 Uploading a file
      NOTE:
      • An independent server configuration file can be uploaded to each Flume instance.
      • This step is required for updating the configuration file. Modifying the configuration file on the background is an improper operation because the modification will be overwritten after configuration synchronization.
    4. Click Save, and then click OK.
    5. Click Finish.

  1. Verify log transmission.

    1. Go to the directory where the HBase client is installed.

      cd /Client installation directory/ HBase/hbase

      kinit flume_hbase (Enter the password.)

    2. Run the hbase shell command to access the HBase client.
    3. Run the scan 'flume_test' statement. Logs are written in the HBase column family by line.
      hbase(main):001:0> scan 'flume_test'
      ROW                                                          COLUMN+CELL                                                                                                                                                                    
      2017-09-18 16:05:36,394 INFO  [hconnection-0x415a3f6a-shared--pool2-t1] ipc.AbstractRpcClient: RPC Server Kerberos principal name for service=ClientService is hbase/hadoop.<system domain name>@<system domain name>
       default4021ff4a-9339-4151-a4d0-00f20807e76d                 column=cf:pCol, timestamp=1505721909388, value=Welcome to flume                                                                                                                
       incRow                                                      column=cf:iCol, timestamp=1505721909461, value=\x00\x00\x00\x00\x00\x00\x00\x01                                                                                                
      2 row(s) in 0.3660 seconds

Utilizamos cookies para mejorar nuestro sitio y tu experiencia. Al continuar navegando en nuestro sitio, tú aceptas nuestra política de cookies. Descubre más

Feedback

Feedback

Feedback

0/500

Selected Content

Submit selected content with the feedback