All Documentation
MapReduce Service
MapReduce Service
What's New
Function Overview
Service Overview
Infographics
What Is MRS?
Advantages of MRS Compared with Self-Built Hadoop
Application Scenarios
How Do I Select an MRS Version?
Components
List of MRS Component Versions
Alluxio
CarbonData
ClickHouse
Infographics for ClickHouse
ClickHouse
CDL
CDL Basic Principles
Relationship Between CDL and Other Components
DBService
DBService Basic Principles
Relationship Between DBService and Other Components
Flink
Flink Basic Principles
Flink HA Solution
Relationships Between Flink and Other Components
Flink Enhanced Open Source Features
Window
Job Pipeline
Stream SQL Join
Flink CEP in SQL
Flume
Flume Basic Principles
Relationships Between Flume and Other Components
Flume Enhanced Open Source Features
HBase
HBase Basic Principles
HBase HA Solution
Relationship with Other Components
HBase Enhanced Open Source Features
HDFS
HDFS Basic Principles
HDFS HA Solution
Relationship Between HDFS and Other Components
HDFS Enhanced Open Source Features
HetuEngine
HetuEngine Product Overview
Relationships Between HetuEngine and Other Components
Hive
Hive Basic Principles
Hive CBO Principles
Relationships Between Hive and Other Components
Enhanced Open Source Feature
Hudi
Hue
Hue Basic Principles
Relationships Between Hue and Other Components
Hue Enhanced Open Source Features
Impala
IoTDB
IoTDB Basic Principles
Relationships Between IoTDB and Other Components
IoTDB Enhanced Open Source Features
Kafka
Kafka Basic Principles
Relationships Between Kafka and Other Components
Kafka Enhanced Open Source Features
KafkaManager
KrbServer and LdapServer
KrbServer and LdapServer Principles
KrbServer and LdapServer Enhanced Open Source Features
Kudu
Loader
Loader Basic Principles
Relationship Between Loader and Other Components
Loader Enhanced Open Source Features
Manager
Manager Basic Principles
Manager Key Features
MapReduce
MapReduce Basic Principles
Relationship Between MapReduce and Other Components
MapReduce Enhanced Open Source Features
Oozie
Oozie Basic Principles
Oozie Enhanced Open Source Features
OpenTSDB
Presto
Ranger
Ranger Basic Principles
Relationships Between Ranger and Other Components
Spark
Spark Basic Principles
Spark HA Solution
Relationship Among Spark, HDFS, and Yarn
Spark Enhanced Open Source Feature: Optimized SQL Query of Cross-Source Data
Spark2x
Spark2x Basic Principles
Spark2x HA Solution
Spark2x Multi-active Instance
Spark2x Multi-tenant
Relationship Between Spark2x and Other Components
Spark2x Open Source New Features
Spark2x Enhanced Open Source Features
CarbonData Overview
Optimizing SQL Query of Data of Multiple Sources
Storm
Storm Basic Principles
Relationships Between Storm and Other Components
Storm Enhanced Open Source Features
Tez
YARN
YARN Basic Principles
YARN HA Solution
Relationships Between YARN and Other Components
Yarn Enhanced Open Source Features
ZooKeeper
ZooKeeper Basic Principles
Relationships Between ZooKeeper and Other Components
ZooKeeper Enhanced Open Source Features
Functions
Multi-tenant
Security Hardening
Easy Access to Web UIs of Components
Reliability Enhancement
Job Management
Bootstrap Actions
Enterprise Project Management
Metadata
Cluster Management
Cluster Lifecycle Management
Cluster Scaling
Auto Scaling
Task Node Creation
Scaling Up Master Node Specifications
Isolating a Host
Managing Tags
Cluster O&M
Message Notification
Security
Shared Responsibilities
Asset Identification and Management
Identity Authentication and Access Control
Data Protection Technologies
Audit and Logging
Service Resilience
Security Risk Monitoring
Update Management
Security Hardening
Constraints
Technical Support
Billing
Permissions Management
Related Services
Quota Description
Common Concepts
Released Versions
Version Overview
Release Notes
MRS 3.1.2-LTS.3 Version Description
MRS 3.1.5 Version Description
MRS 3.2.0-LTS.1 Version Description
Billing
Overview
Billing Mode
Introduction
Yearly/Monthly
Pay-per-use
Billing Item
Billing Examples
Billing mode change
Introduction
Pay-per-Use to Yearly/Monthly
Renewal
Introduction
Performing a Manual Renewal
Auto Renewal
Bills
Arrears
Stopping Billing
Cost
FAQs
Why Is the Price Not Displayed During MRS Cluster Creation?
How Is the Task Node in an MRS Cluster Billed?
Why Does My Unsubscription from ECS Fail After I Unsubscribe from MRS?
Getting Started
Buying and Using an MRS Cluster
Getting Started with MapReduce Service
Buying a Cluster
Uploading Data
Creating a Job
Terminating a Cluster
Installing and Using the Cluster Client
Using Clusters with Kerberos Authentication Enabled
Using Hadoop from Scratch
Using Kafka from Scratch
Using HBase from Scratch
Modifying MRS Configurations
Configuring Auto Scaling for an MRS Cluster
Configuring Hive with Storage and Compute Decoupled
Submitting Spark Tasks to New Task Nodes
Configuring Thresholds for Alarms
MRS Component Application Development
HBase Application Development
HDFS Application Development
Hive JDBC Application Development
Hive HCatalog Application Development
Kafka Application Development
Flink Application Development
ClickHouse Application Development
Spark Application Development
Practices
User Guide
Preparing a User
Configuring Cloud Service Permissions
Creating an MRS User
Creating a Custom Policy
Synchronizing IAM Users to MRS
Configuring a Cluster
How to Buy an MRS Cluster?
Quickly Buying an MRS Cluster
Quickly Buying a Hadoop Analysis Cluster
Quickly Buying an HBase Query Cluster
Quickly Buying a Kafka Streaming Cluster
Quickly Buying a ClickHouse Cluster
Quickly Buying a Real-time Analysis Cluster
Buying a Custom Cluster
Buying a Custom Topology Cluster
Adding a Tag to a Cluster/Node
Communication Security Authorization
Configuring Auto Scaling Rules
Overview
Configuring Auto Scaling During Cluster Creation
Creating an Auto Scaling Policy for an Existing Cluster
Scenario 1: Using Auto Scaling Rules Alone
Scenario 2: Using Resource Plans Alone
Scenario 3: Using Both Auto Scaling Rules and Resource Plans
Modifying an Auto Scaling Policy
Deleting an Auto Scaling Policy
Enabling or Disabling an Auto Scaling Policy
Viewing an Auto Scaling Policy
Configuring Automation Scripts
Configuring Auto Scaling Metrics
Managing Data Connections
Configuring Data Connections
Configuring an RDS Data Connection
Configuring an RDS Data Connection
Configuring Ranger Data Connections
Configuring a Hive Data Connection
Installing Third-Party Software Using Bootstrap Actions
Viewing Failed MRS Tasks
Viewing Information of a Historical Cluster
Managing Clusters
Logging In to a Cluster
MRS Cluster Node Overview
Logging In to an ECS
Determining Active and Standby Management Nodes
Cluster Overview
Cluster List
Checking the Cluster Status
Viewing Basic Cluster Information
Viewing Cluster Patch Information
Managing Components and Monitoring Hosts
Viewing and Customizing Cluster Monitoring Metrics
Cluster O&M
Importing and Exporting Data
Changing the Subnet of a Cluster
Configuring Message Notification
Checking Health Status
Before You Start
Performing a Health Check
Viewing and Exporting a Health Check Report
Remote O&M
Authorizing O&M
Sharing Logs
Viewing MRS Operation Logs
Changing Billing Mode to Yearly/Monthly
Unsubscribing from a Cluster
Unsubscribing from a Specified Node in a Yearly/Monthly Cluster
Deleting a Cluster
Managing Nodes
Scaling Out a Cluster
Scaling In a Cluster
Removing ClickHouseServer Instance Nodes
Constraints on ClickHouseServer Scale-in
Scaling In ClickHouseServer Nodes
Managing a Host (Node)
Isolating a Host
Canceling Host Isolation
Scaling Up Master Node Specifications
Synchronizing Disk Information
Job Management
Introduction to MRS Jobs
Running a MapReduce Job
Running a SparkSubmit or Spark Job
Running a HiveSQL Job
Running a SparkSql Job
Running a Flink Job
Viewing Job Configuration and Logs
Stopping a Job
Deleting a Job
Using Encrypted OBS Data for Job Running
Configuring Job Notification Rules
Component Management
Object Management
Viewing Configuration
Managing Services
Configuring Service Parameters
Configuring Customized Service Parameters
Synchronizing Service Configuration
Managing Role Instances
Configuring Role Instance Parameters
Synchronizing Role Instance Configuration
Decommissioning and Recommissioning a Role Instance
Starting and Stopping a Cluster
Synchronizing Cluster Configuration
Exporting Cluster Configuration
Performing Rolling Restart
Alarm Management
Viewing the Alarm List
Viewing the Event List
Viewing and Manually Clearing an Alarm
Patch Management
Patch Operation Guide for MRS 3.1.5
Rolling Patches
Restoring Patches for the Isolated Hosts
MRS Patch Description
Fixed the Privilege Escalation Vulnerability of User omm
MRS 2.1.0.11 Patch Description
MRS 3.0.5.1 Patch Description
MRS 2.1.0.10 Patch Description
MRS 2.1.0.9 Patch Description
MRS 2.1.0.8 Patch Description
MRS 2.1.0.7 Patch Description
MRS 2.1.0.6 Patch Description
MRS 2.1.0.3 Patch Description
MRS 2.1.0.2 Patch Description
MRS 2.1.0.1 Patch Description
MRS 2.0.6.1 Patch Description
MRS 2.0.1.3 Patch Description
MRS 2.0.1.2 Patch Description
MRS 2.0.1.1 Patch Description
MRS 1.9.3.3 Patch Description
MRS 1.9.3.1 Patch Description
MRS 1.9.2.2 Patch Description
MRS 1.9.0.8, 1.9.0.9, and 1.9.0.10 Patch Description
MRS 1.9.0.7 Patch Description
MRS 1.9.0.6 Patch Description
MRS 1.9.0.5 Patch Description
MRS 1.8.10.1 Patch Description
Tenant Management
Before You Start
Overview
Creating a Tenant
Creating a Sub-tenant
Deleting a Tenant
Managing a Tenant Directory
Restoring Tenant Data
Creating a Resource Pool
Modifying a Resource Pool
Deleting a Resource Pool
Configuring a Queue
Configuring the Queue Capacity Policy of a Resource Pool
Clearing Configuration of a Queue
Bootstrap Actions
Introduction to Bootstrap Actions
Preparing the Bootstrap Action Script
View Execution Records
Adding a Bootstrap Action
Modifying a Bootstrap Action
Deleting a Bootstrap Action
Sample Scripts
Using an MRS Client
Installing a Client
Installing a Client (MRS 3.x or Later)
Installing a Client (Versions Earlier Than 3.x)
Updating a Client
Updating a Client (Version 3.x or Later)
Updating a Client (Versions Earlier Than 3.x)
Using the Client of Each Component
Using a ClickHouse Client
Using a Flink Client
Using a Flume Client
Using an HBase Client
Using an HDFS Client
Using a Hive Client
Using an Impala Client
Using a Kafka Client
Using a Kudu Client
Using the Oozie Client
Using a Storm Client
Using a Yarn Client
Configuring a Cluster with Decoupled Storage and Compute
MRS Storage-Compute Decoupling
Interconnecting with OBS Using the Cluster Agency Mechanism
Configuring a Storage-Compute Decoupled Cluster (Agency)
Configuring a Storage-Compute Decoupled Cluster (AK/SK)
Configuring the Policy for Clearing Component Data in the Recycle Bin
Interconnecting MRS with OBS Using an Agency
Interconnecting Flink with OBS
Interconnecting Flume with OBS
Interconnecting HDFS with OBS
Interconnecting Hive with OBS
Interconnecting MapReduce with OBS
Interconnecting Spark2x with OBS
Interconnecting Sqoop with External Storage Systems
Interconnecting Hudi with OBS
Configuring Fine-Grained Permissions for MRS Multi-User Access to OBS
Accessing OBS from a Client on a Node Outside the Cluster
FAQ
Example Interconnecting HDFS with OBS
Accessing Web Pages of Open Source Components Managed in MRS Clusters
Web UIs of Open Source Components
Common Ports of Components
Access Through Direct Connect
EIP-based Access
Access Using a Windows ECS
Creating an SSH Channel for Connecting to an MRS Cluster and Configuring the Browser
Accessing Manager
Accessing FusionInsight Manager (MRS 3.x or Later)
Accessing MRS Manager (MRS 2.x or Earlier)
FusionInsight Manager Operation Guide (Applicable to 3.x)
Homepage
Overview
Managing Monitoring Metric Reports
Querying the FusionInsight Manager Version
Cluster
Cluster Management
Overview
Performing a Rolling Restart of a Cluster
Managing Expired Configurations
Downloading the Client
Modifying Cluster Attributes
Managing Cluster Configurations
Managing Static Service Pools
Static Service Resources
Configuring Cluster Static Resources
Viewing Cluster Static Resources
Managing Clients
Managing a Client
Batch Upgrading Clients
Updating the hosts File in Batches
Managing a Service
Overview
Other Service Management Operations
Service Details Page
Performing Active/Standby Switchover of a Role Instance
Resource Monitoring
Collecting Stack Information
Switching Ranger Authentication
Service Configuration
Modifying Service Configuration Parameters
Modifying Custom Configuration Parameters of a Service
Instance Management
Overview
Decommissioning and Recommissioning an Instance
Managing Instance Configurations
Viewing the Instance Configuration File
Instance Group
Managing Instance Groups
Viewing Information About an Instance Group
Configuring Instantiation Group Parameters
Hosts
Host Management Page
Viewing the Host List
Viewing the Host Dashboard
Checking Host Processes and Resources
Host Maintenance Operations
Starting and Stopping All Instances on a Host
Performing a Host Health Check
Configuring Racks for Hosts
Isolating a Host
Exporting Host Information
Resource Overview
Distribution
Trend
Cluster
Host
O&M
Alarms
Overview of Alarms and Events
Configuring the Threshold
Configuring the Alarm Masking Status
Log
Log Online Search
Log Download
Perform a Health Check
Viewing a Health Check Task
Managing Health Check Reports
Modifying Health Check Configuration
Configuring Backup and Backup Restoration
Creating a Backup Task
Creating a Backup Restoration Task
Managing Backup and Backup Restoration Tasks
Audit
Overview
Configuring Audit Log Dumping
Tenant Resources
Multi-Tenancy
Overview
Technical Principles
Multi-Tenant Management
Multi-Tenant Model
Resource Overview
Dynamic Resources
Storage Resources
Multi-Tenancy Usage
Overview
Process Overview
Using the Superior Scheduler
Creating Tenants
Adding a Tenant
Adding a Sub-Tenant
Adding a User and Binding the User to a Tenant Role
Managing Tenants
Managing Tenant Directories
Restoring Tenant Data
Deleting a Tenant
Managing Resources
Adding a Resource Pool
Modifying a Resource Pool
Deleting a Resource Pool
Modifying Queue Resources
Configuring the Queue Capacity Policy of a Resource Pool
Clearing Queue Configurations
Managing Global User Policies
Using the Capacity Scheduler
Creating Tenants
Adding a Tenant
Adding a Sub-Tenant
Adding a User and Binding the User to a Tenant Role
Managing Tenants
Managing Tenant Directories
Restoring Tenant Data
Deleting a Tenant
Clearing Non-associated Queues of a Tenant
Managing Resources
Adding a Resource Pool
Modifying a Resource Pool
Deleting a Resource Pool
Modifying Queue Resources
Configuring the Queue Capacity Policy of a Resource Pool
Clearing Queue Configurations
Switching the Scheduler
System
Configuring Permissions
Managing Users
Creating a User
Modifying User Information
Exporting User Information
Locking a User
Unlocking a User
Deleting a User
Changing a User Password
Initializing a Password
Exporting an Authentication Credential File
Managing User Groups
Managing Roles
Security Policies
Configuring Password Policies
Configuring the Independent Attribute
Configuring Interconnections
Configuring SNMP Northbound Parameters
Configuring Syslog Northbound Parameters
Configuring Monitoring Metric Dumping
Importing a Certificate
OMS Management
Overview of the OMS Page
Modifying OMS Service Configuration Parameters
Viewing Component Packages
Cluster Management
Cluster Mutual Trust Management
Overview of Mutual Trust Between Clusters
Changing Manager's Domain Name
Configuring Cross-Manager Mutual Trust Between Clusters
Assigning User Permissions After Cross-Cluster Mutual Trust Is Configured
Configuring Scheduled Backup of Alarm and Audit Information
Modifying the FusionInsight Manager Routing Table
Switching to the Maintenance Mode
Routine Maintenance
Log Management
About Logs
Manager Log List
Configuring the Log Level and Log File Size
Configuring the Number of Local Audit Log Backups
Viewing Role Instance Logs
Backup and Recovery Management
Introduction
Backing Up Data
Backing Up Manager Data
Backing Up CDL Data
Backing Up ClickHouse Metadata
Backing Up ClickHouse Service Data
Backing Up DBService Data
Backing Up Flink Metadata
Backing Up HBase Metadata
Backing Up HBase Service Data
Backing Up NameNode Data
Backing Up HDFS Service Data
Backing Up Hive Service Data
Backing Up IoTDB Metadata
Backing Up IoTDB Service Data
Backing Up Kafka Metadata
Recovering Data
Restoring Manager Data
Restoring CDL Data
Restoring ClickHouse Metadata
Restoring ClickHouse Service Data
Restoring DBService data
Restoring Flink Metadata
Restoring HBase Metadata
Restoring HBase Service Data
Restoring NameNode Data
Restoring HDFS Service Data
Restoring Hive Service Data
Restoring IoTDB Metadata
Restoring IoTDB Service Data
Restoring Kafka Metadata
Enabling Cross-Cluster Replication
Managing Local Quick Restoration Tasks
Modifying a Backup Task
Viewing Backup and Restoration Tasks
How Do I Configure the Environment When I Create a ClickHouse Backup Task on FusionInsight Manager and Set the Path Type to RemoteHDFS?
Security Management
Security Overview
Right Model
Right Mechanism
Authentication Policies
Permission Verification Policies
User Account List
Default Permission Information
FusionInsight Manager Security Functions
Account Management
Account Security Settings
Unlocking LDAP Users and Management Accounts
Internal an Internal System User
Enabling and Disabling Permission Verification on Cluster Components
Logging In to a Non-Cluster Node Using a Cluster User in Normal Mode
Changing the Password for a System User
Changing the Password for User admin
Changing the Password for an OS User
Changing the Password for a System Internal User
Changing the Password for the Kerberos Administrator
Changing the Password for the OMS Kerberos Administrator
Changing the Passwords of the LDAP Administrator and the LDAP User (Including OMS LDAP)
Changing the Password for the LDAP Administrator
Changing the Password for a Component Running User
Changing the Password for a Database User
Changing the Password of the OMS Database Administrator
Changing the Password for the Data Access User of the OMS Database
Changing the Password for a Component Database User
Resetting the Component Database User Password
Changing the Password for User compdbuser of the DBService Database
Changing or Resetting the Password for User admin of Manager
Certificate Management
Replacing the CA Certificate
Replacing HA Certificates
Security Hardening
Hardening Policies
Configuring a Trusted IP Address to Access LDAP
HFile and WAL Encryption
Configuring Hadoop Security Parameters
Configuring an IP Address Whitelist for Modification Allowed by HBase
Updating a Key for a Cluster
Hardening the LDAP
Configuring Kafka Data Encryption During Transmission
Configuring HDFS Data Encryption During Transmission
Configuring Spark2x Data Encryption During Transmission
Configuring ZooKeeper SSL
Encrypting the Communication Between the Controller and the Agent
Updating SSH Keys for User omm
Security Maintenance
Account Maintenance Suggestions
Password Maintenance Suggestions
Log Maintenance Suggestions
Security Statement
MRS Manager Operation Guide (Applicable to 2.x and Earlier Versions)
Introduction to MRS Manager
Checking Running Tasks
Monitoring Management
Dashboard
Managing Services and Monitoring Hosts
Managing Resource Distribution
Configuring Monitoring Metric Dumping
Alarm Management
Viewing and Manually Clearing an Alarm
Configuring an Alarm Threshold
Configuring Syslog Northbound Interface Parameters
Configuring SNMP Northbound Interface Parameters
Alarm Reference (Applicable to Versions Earlier Than MRS 3.x)
ALM-12001 Audit Log Dump Failure (For MRS 2.x or Earlier)
ALM-12002 HA Resource Abnormal (For MRS 2.x or Earlier)
ALM-12004 OLdap Resource Abnormal (For MRS 2.x or Earlier)
ALM-12005 OKerberos Resource Abnormal (For MRS 2.x or Earlier)
ALM-12006 Node Fault (For MRS 2.x or Earlier)
ALM-12007 Process Fault (For MRS 2.x or Earlier)
ALM-12010 Manager Heartbeat Interruption Between the Active and Standby Nodes (For MRS 2.x or Earlier)
ALM-12011 Data Synchronization Exception Between the Active and Standby Manager Nodes (For MRS 2.x or Earlier)
ALM-12012 NTP Service Abnormal (For MRS 2.x or Earlier)
ALM-12014 Device Partition Lost (For MRS 2.x or Earlier)
ALM-12015 Device Partition File System Read-Only (For MRS 2.x or Earlier)
ALM-12016 CPU Usage Exceeds the Threshold (For MRS 2.x or Earlier)
ALM-12017 Insufficient Disk Capacity (For MRS 2.x or Earlier)
ALM-12018 Memory Usage Exceeds the Threshold (For MRS 2.x or Earlier)
ALM-12027 Host PID Usage Exceeds the Threshold (For MRS 2.x or Earlier)
ALM-12028 Number of Processes in the D State on the Host Exceeds the Threshold (For MRS 2.x or Earlier)
ALM-12031 User omm or Password Is About to Expire (For MRS 2.x or Earlier)
ALM-12032 User ommdba or Password Is About to Expire (For MRS 2.x or Earlier)
ALM-12033 Slow Disk Fault (For MRS 2.x or Earlier)
ALM-12034 Periodic Backup Failure (For MRS 2.x or Earlier)
ALM-12035 Unknown Data Status After Recovery Task Failure (For MRS 2.x or Earlier)
ALM-12037 NTP Server Abnormal (For MRS 2.x or Earlier)
ALM-12038 Monitoring Indicator Dump Failure (For MRS 2.x or Earlier)
ALM-12039 GaussDB Data Is Not Synchronized (For MRS 2.x or Earlier)
ALM-12040 Insufficient System Entropy (For MRS 2.x or Earlier)
ALM-12041 Permission of Key Files Is Abnormal (For MRS 2.x or Earlier)
ALM-12042 Key File Configurations Are Abnormal (For MRS 2.x or Earlier)
ALM-12043 DNS Parsing Duration Exceeds the Threshold (For MRS 2.x or Earlier)
ALM-12045 Read Packet Dropped Rate Exceeds the Threshold (For MRS 2.x or Earlier)
ALM-12046 Write Packet Dropped Rate Exceeds the Threshold (For MRS 2.x or Earlier)
ALM-12047 Read Packet Error Rate Exceeds the Threshold (For MRS 2.x or Earlier)
ALM-12048 Write Packet Error Rate Exceeds the Threshold (For MRS 2.x or Earlier)
ALM-12049 Read Throughput Rate Exceeds the Threshold (For MRS 2.x or Earlier)
ALM-12050 Write Throughput Rate Exceeds the Threshold (For MRS 2.x or Earlier)
ALM-12051 Disk Inode Usage Exceeds the Threshold (For MRS 2.x or Earlier)
ALM-12052 Usage of Temporary TCP Ports Exceeds the Threshold (For MRS 2.x or Earlier)
ALM-12053 File Handle Usage Exceeds the Threshold (For MRS 2.x or Earlier)
ALM-12054 Invalid Certificate File (For MRS 2.x or Earlier)
ALM-12055 Certificate File Is About to Expire (For MRS 2.x or Earlier)
ALM-12180 Disk Card I/O (For MRS 2.x or Earlier)
ALM-12357 Failed to Export Audit Logs to OBS (For MRS 2.x or Earlier)
ALM-13000 ZooKeeper Service Unavailable (For MRS 2.x or Earlier)
ALM-13001 Available ZooKeeper Connections Are Insufficient (For MRS 2.x or Earlier)
ALM-13002 ZooKeeper Memory Usage Exceeds the Threshold (For MRS 2.x or Earlier)
ALM-14000 HDFS Service Unavailable (For MRS 2.x or Earlier)
ALM-14001 HDFS Disk Usage Exceeds the Threshold (For MRS 2.x or Earlier)
ALM-14002 DataNode Disk Usage Exceeds the Threshold (For MRS 2.x or Earlier)
ALM-14003 Number of Lost HDFS Blocks Exceeds the Threshold (For MRS 2.x or Earlier)
ALM-14004 Number of Damaged HDFS Blocks Exceeds the Threshold (For MRS 2.x or Earlier)
ALM-14006 Number of HDFS Files Exceeds the Threshold (For MRS 2.x or Earlier)
ALM-14007 HDFS NameNode Memory Usage Exceeds the Threshold (For MRS 2.x or Earlier)
ALM-14008 HDFS DataNode Memory Usage Exceeds the Threshold (For MRS 2.x or Earlier)
ALM-14009 Number of Faulty DataNodes Exceeds the Threshold (For MRS 2.x or Earlier)
ALM-14010 NameService Is Abnormal (For MRS 2.x or Earlier)
ALM-14011 HDFS DataNode Data Directory Is Not Configured Properly (For MRS 2.x or Earlier)
ALM-14012 HDFS Journalnode Data Is Not Synchronized (For MRS 2.x or Earlier)
ALM-16000 Percentage of Sessions Connected to the HiveServer to the Maximum Number Allowed Exceeds the Threshold (For MRS 2.x or Earlier)
ALM-16001 Hive Warehouse Space Usage Exceeds the Threshold (For MRS 2.x or Earlier)
ALM-16002 Hive SQL Execution Success Rate Is Lower Than the Threshold (For MRS 2.x or Earlier)
ALM-16004 Hive Service Unavailable (For MRS 2.x or Earlier)
ALM-16005 Number of Failed Hive SQL Executions in the Last Period Exceeds the Threshold (For MRS 2.x or Earlier)
ALM-18000 Yarn Service Unavailable (For MRS 2.x or Earlier)
ALM-18002 NodeManager Heartbeat Lost (For MRS 2.x or Earlier)
ALM-18003 NodeManager Unhealthy (For MRS 2.x or Earlier)
ALM-18004 NodeManager Disk Usability Ratio Is Lower Than the Threshold (For MRS 2.x or Earlier)
ALM-18006 MapReduce Job Execution Timeout (For MRS 2.x or Earlier)
ALM-18008 Heap Memory Usage of Yarn ResourceManager Exceeds the Threshold (For MRS 2.x or Earlier)
ALM-18009 Heap Memory Usage of MapReduce JobHistoryServer Exceeds the Threshold (For MRS 2.x or Earlier)
ALM-18010 Number of Pending Yarn Tasks Exceeds the Threshold (For MRS 2.x or Earlier)
ALM-18011 Memory of Pending Yarn Tasks Exceeds the Threshold (For MRS 2.x or Earlier)
ALM-18012 Number of Terminated Yarn Tasks in the Last Period Exceeds the Threshold (For MRS 2.x or Earlier)
ALM-18013 Number of Failed Yarn Tasks in the Last Period Exceeds the Threshold (For MRS 2.x or Earlier)
ALM-19000 HBase Service Unavailable (For MRS 2.x or Earlier)
ALM-19006 HBase Replication Sync Failed (For MRS 2.x or Earlier)
ALM-19007 HBase Merge Queue Exceeds the Threshold (for 2.x and Earlier Versions)
ALM-20002 Hue Service Unavailable (For MRS 2.x or Earlier)
ALM-23001 Loader Service Unavailable (For MRS 2.x or Earlier)
ALM-24000 Flume Service Unavailable (For MRS 2.x or Earlier)
ALM-24001 Flume Agent Is Abnormal (For MRS 2.x or Earlier)
ALM-24003 Flume Client Connection Interrupted (For MRS 2.x or Earlier)
ALM-24004 Flume Fails to Read Data (For MRS 2.x or Earlier)
ALM-24005 Data Transmission by Flume Is Abnormal (For MRS 2.x or Earlier)
ALM-25000 LdapServer Service Unavailable (For MRS 2.x or Earlier)
ALM-25004 Abnormal LdapServer Data Synchronization (For MRS 2.x or Earlier)
ALM-25500 KrbServer Service Unavailable (For MRS 2.x or Earlier)
ALM-26051 Storm Service Unavailable (For MRS 2.x or Earlier)
ALM-26052 Number of Available Supervisors in Storm Is Lower Than the Threshold (For MRS 2.x or Earlier)
ALM-26053 Slot Usage of Storm Exceeds the Threshold (For MRS 2.x or Earlier)
ALM-26054 Heap Memory Usage of Storm Nimbus Exceeds the Threshold (For MRS 2.x or Earlier)
ALM-27001 DBService Unavailable (For MRS 2.x or Earlier)
ALM-27003 DBService Heartbeat Interruption Between the Active and Standby Nodes (For MRS 2.x or Earlier)
ALM-27004 Data Inconsistency Between Active and Standby DBServices (For MRS 2.x or Earlier)
ALM-28001 Spark Service Unavailable (For MRS 2.x or Earlier)
ALM-38000 Kafka Service Unavailable (For MRS 2.x or Earlier)
ALM-38001 Insufficient Kafka Disk Capacity (For MRS 2.x or Earlier)
ALM-38002 Heap Memory Usage of Kafka Exceeds the Threshold (For MRS 2.x or Earlier)
ALM-43001 Spark Service Unavailable (For MRS 2.x or Earlier)
ALM-43006 Heap Memory Usage of the JobHistory Process Exceeds the Threshold (For MRS 2.x or Earlier)
ALM-43007 Non-Heap Memory Usage of the JobHistory Process Exceeds the Threshold (For MRS 2.x or Earlier)
ALM-43008 Direct Memory Usage of the JobHistory Process Exceeds the Threshold (For MRS 2.x or Earlier)
ALM-43009 JobHistory GC Time Exceeds the Threshold (For MRS 2.x or Earlier)
ALM-43010 Heap Memory Usage of the JDBCServer Process Exceeds the Threshold (For MRS 2.x or Earlier)
ALM-43011 Non-Heap Memory Usage of the JDBCServer Process Exceeds the Threshold (For MRS 2.x or Earlier)
ALM-43012 Direct Memory Usage of the JDBCServer Process Exceeds the Threshold (For MRS 2.x or Earlier)
ALM-43013 JDBCServer GC Time Exceeds the Threshold (For MRS 2.x or Earlier)
ALM-44004 Presto Coordinator Resource Group Queuing Tasks Exceed the Threshold (For MRS 2.x or Earlier)
ALM-44005 Presto Coordinator Process GC Time Exceeds the Threshold (For MRS 2.x or Earlier)
ALM-44006 Presto Worker Process GC Time Exceeds the Threshold (For MRS 2.x or Earlier)
ALM-45325 Presto Service Unavailable (For MRS 2.x or Earlier)
Object Management
Managing Objects
Viewing Configurations
Managing Services
Configuring Service Parameters
Configuring Customized Service Parameters
Synchronizing Service Configurations
Managing Role Instances
Configuring Role Instance Parameters
Synchronizing Role Instance Configuration
Decommissioning and Recommissioning a Role Instance
Managing a Host
Isolating a Host
Canceling Host Isolation
Starting or Stopping a Cluster
Synchronizing Cluster Configurations
Exporting Configuration Data of a Cluster
Log Management
About Logs
Manager Log List
Viewing and Exporting Audit Logs
Exporting Service Logs
Configuring Audit Log Exporting Parameters
Health Check Management
Performing a Health Check
Viewing and Exporting a Health Check Report
Configuring the Number of Health Check Reports to Be Reserved
Managing Health Check Reports
DBService Health Check Indicators
Flume Health Check Indicators
HBase Health Check Indicators
Host Health Check Indicators
HDFS Health Check Indicators
Hive Health Check Indicators
Kafka Health Check Indicators
KrbServer Health Check Indicators
LdapServer Health Check Indicators
Loader Health Check Indicators
MapReduce Health Check Indicators
OMS Health Check Indicators
Spark Health Check Indicators
Storm Health Check Indicators
Yarn Health Check Indicators
ZooKeeper Health Check Indicators
Static Service Pool Management
Viewing the Status of a Static Service Pool
Configuring a Static Service Pool
Tenant Management
Overview
Creating a Tenant
Creating a Sub-tenant
Deleting a tenant
Managing a Tenant Directory
Restoring Tenant Data
Creating a Resource Pool
Modifying a Resource Pool
Deleting a Resource Pool
Configuring a Queue
Configuring the Queue Capacity Policy of a Resource Pool
Clearing Configuration of a Queue
Backup and Restoration
Introduction
Backing Up Metadata
Restoring Metadata
Modifying a Backup Task
Viewing Backup and Restoration Tasks
Security Management
Default Users of Clusters with Kerberos Authentication Disabled
Default Users of Clusters with Kerberos Authentication Enabled
Changing the Password of an OS User
Changing the password of user admin
Changing the Password of the Kerberos Administrator
Changing the Passwords of the LDAP Administrator and the LDAP User
Changing the Password of a Component Running User
Changing the Password of the OMS Database Administrator
Changing the Password of the Data Access User of the OMS Database
Changing the Password of a Component Database User
Replacing the HA Certificate
Updating Cluster Keys
Permissions Management
Creating a Role
Creating a User Group
Creating a User
Modifying User Information
Locking a User
Unlocking a User
Deleting a User
Changing the Password of an Operation User
Initializing the Password of a System User
Downloading a User Authentication File
Modifying a Password Policy
MRS Multi-User Permission Management
Users and Permissions of MRS Clusters
Default Users of Clusters with Kerberos Authentication Enabled
Creating a Role
Creating a User Group
Creating a User
Modifying User Information
Locking a User
Unlocking a User
Deleting a User
Changing the Password of an Operation User
Initializing the Password of a System User
Downloading a User Authentication File
Modifying a Password Policy
Configuring Cross-Cluster Mutual Trust Relationships
Configuring Users to Access Resources of a Trusted Cluster
Patch Operation Guide
Patch Operation Guide for Versions
Supporting Rolling Patches
Restoring Patches for the Isolated Hosts
Rolling Restart
Alarm Reference (Applicable to MRS 3.x)
ALM-12001 Audit Log Dumping Failure
ALM-12004 OLdap Resource Abnormal
ALM-12005 OKerberos Resource Abnormal
ALM-12006 Node Fault
ALM-12007 Process Fault
ALM-12010 Manager Heartbeat Interruption Between the Active and Standby Nodes
ALM-12011 Manager Data Synchronization Exception Between the Active and Standby Nodes
ALM-12012 NTP Service Is Abnormal
ALM-12014 Partition Lost
ALM-12015 Partition Filesystem Readonly
ALM-12016 CPU Usage Exceeds the Threshold
ALM-12017 Insufficient Disk Capacity
ALM-12018 Memory Usage Exceeds the Threshold
ALM-12027 Host PID Usage Exceeds the Threshold
ALM-12028 Number of Processes in the D State and Z State on a Host Exceeds the Threshold
ALM-12033 Slow Disk Fault
ALM-12034 Periodical Backup Failure
ALM-12035 Unknown Data Status After Recovery Task Failure
ALM-12037 NTP Server Abnormal
ALM-12038 Monitoring Indicator Dumping Failure
ALM-12039 Active/Standby OMS Databases Not Synchronized
ALM-12040 Insufficient System Entropy
ALM-12041 Incorrect Permission on Key Files
ALM-12042 Incorrect Configuration of Key Files
ALM-12045 Read Packet Dropped Rate Exceeds the Threshold
ALM-12046 Write Packet Dropped Rate Exceeds the Threshold
ALM-12047 Read Packet Error Rate Exceeds the Threshold
ALM-12048 Write Packet Error Rate Exceeds the Threshold
ALM-12049 Network Read Throughput Rate Exceeds the Threshold
ALM-12050 Network Write Throughput Rate Exceeds the Threshold
ALM-12051 Disk Inode Usage Exceeds the Threshold
ALM-12052 TCP Temporary Port Usage Exceeds the Threshold
ALM-12053 Host File Handle Usage Exceeds the Threshold
ALM-12054 Invalid Certificate File
ALM-12055 Certificate File Is About to Expire
ALM-12057 Metadata Not Configured with the Task to Periodically Back Up Data to a Third-Party Server
ALM-12061 Process Usage Exceeds the Threshold
ALM-12062 OMS Parameter Configurations Mismatch with the Cluster Scale
ALM-12063 Unavailable Disk
ALM-12064 Host Random Port Range Conflicts with Cluster Used Port
ALM-12066 Trust Relationships Between Nodes Become Invalid
ALM-12067 Tomcat Resource Is Abnormal
ALM-12068 ACS Resource Exception
ALM-12069 AOS Resource Exception
ALM-12070 Controller Resource Is Abnormal
ALM-12071 Httpd Resource Is Abnormal
ALM-12072 FloatIP Resource Is Abnormal
ALM-12073 CEP Resource Is Abnormal
ALM-12074 FMS Resource Is Abnormal
ALM-12075 PMS Resource Is Abnormal
ALM-12076 GaussDB Resource Is Abnormal
ALM-12077 User omm Expired
ALM-12078 Password of User omm Expired
ALM-12079 User omm Is About to Expire
ALM-12080 Password of User omm Is About to Expire
ALM-12081User ommdba Expired
ALM-12082 User ommdba Is About to Expire
ALM-12083 Password of User ommdba Is About to Expire
ALM-12084 Password of User ommdba Expired
ALM-12085 Service Audit Log Dump Failure
ALM-12087 System Is in the Upgrade Observation Period
ALM-12089 Inter-Node Network Is Abnormal
ALM-12091 Abnormal disaster Resources
ALM-12099 core dump Occurred
ALM-12100 AD Service Connection Failed
ALM-12101 AZ Unhealthy
ALM-12102 AZ HA Component Is Not Deployed Based on DR Requirements
ALM-12103 Executor Resource Exception
ALM-12104 Abnormal Knox Resources
ALM-12110 Failed to get ECS temporary AK/SK
ALM-12172 Failed to Report Metrics to Cloud Eye
ALM-12180 Suspended Disk I/O
ALM-12186 CGroup Task Usage Exceeds the Threshold
ALM-12187 Failed to Expand Disk Partition Capacity
ALM-12188 diskmgt Disk Monitoring Unavailable
ALM-12190 Number of Knox Connections Exceeds the Threshold
ALM-13000 ZooKeeper Service Unavailable
ALM-13001 Available ZooKeeper Connections Are Insufficient
ALM-13002 ZooKeeper Direct Memory Usage Exceeds the Threshold
ALM-13003 GC Duration of the ZooKeeper Process Exceeds the Threshold
ALM-13004 ZooKeeper Heap Memory Usage Exceeds the Threshold
ALM-13005 Failed to Set the Quota of Top Directories of ZooKeeper Components
ALM-13006 Znode Number or Capacity Exceeds the Threshold
ALM-13007 Available ZooKeeper Client Connections Are Insufficient
ALM-13008 ZooKeeper Znode Usage Exceeds the Threshold
ALM-13009 ZooKeeper Znode Capacity Usage Exceeds the Threshold
ALM-13010 Znode Usage of a Directory with Quota Configured Exceeds the Threshold
ALM-14000 HDFS Service Unavailable
ALM-14001 HDFS Disk Usage Exceeds the Threshold
ALM-14002 DataNode Disk Usage Exceeds the Threshold
ALM-14003 Number of Lost HDFS Blocks Exceeds the Threshold
ALM-14006 Number of HDFS Files Exceeds the Threshold
ALM-14007 NameNode Heap Memory Usage Exceeds the Threshold
ALM-14008 DataNode Heap Memory Usage Exceeds the Threshold
ALM-14009 Number of Dead DataNodes Exceeds the Threshold
ALM-14010 NameService Service Is Abnormal
ALM-14011 DataNode Data Directory Is Not Configured Properly
ALM-14012 JournalNode Is Out of Synchronization
ALM-14013 Failed to Update the NameNode FsImage File
ALM-14014 NameNode GC Time Exceeds the Threshold
ALM-14015 DataNode GC Time Exceeds the Threshold
ALM-14016 DataNode Direct Memory Usage Exceeds the Threshold
ALM-14017 NameNode Direct Memory Usage Exceeds the Threshold
ALM-14018 NameNode Non-heap Memory Usage Exceeds the Threshold
ALM-14019 DataNode Non-heap Memory Usage Exceeds the Threshold
ALM-14020 Number of Entries in the HDFS Directory Exceeds the Threshold
ALM-14021 NameNode Average RPC Processing Time Exceeds the Threshold
ALM-14022 NameNode Average RPC Queuing Time Exceeds the Threshold
ALM-14023 Percentage of Total Reserved Disk Space for Replicas Exceeds the Threshold
ALM-14024 Tenant Space Usage Exceeds the Threshold
ALM-14025 Tenant File Object Usage Exceeds the Threshold
ALM-14026 Blocks on DataNode Exceed the Threshold
ALM-14027 DataNode Disk Fault
ALM-14028 Number of Blocks to Be Supplemented Exceeds the Threshold
ALM-14029 Number of Blocks in a Replica Exceeds the Threshold
ALM-14030 HDFS Allows Write of Single-Replica Data
ALM-14031 DataNode Process Is Abnormal
ALM-14032 JournalNode Process Is Abnormal
ALM-14033 ZKFC Process Is Abnormal
ALM-14034 Router Process Is Abnormal
ALM-14035 HttpFS Process Is Abnormal
ALM-16000 Percentage of Sessions Connected to the HiveServer to Maximum Number Allowed Exceeds the Threshold
ALM-16001 Hive Warehouse Space Usage Exceeds the Threshold
ALM-16002 Hive SQL Execution Success Rate Is Lower Than the Threshold
ALM-16003 Background Thread Usage Exceeds the Threshold
ALM-16004 Hive Service Unavailable
ALM-16005 The Heap Memory Usage of the Hive Process Exceeds the Threshold
ALM-16006 The Direct Memory Usage of the Hive Process Exceeds the Threshold
ALM-16007 Hive GC Time Exceeds the Threshold
ALM-16008 Non-Heap Memory Usage of the Hive Process Exceeds the Threshold
ALM-16009 Map Number Exceeds the Threshold
ALM-16045 Hive Data Warehouse Is Deleted
ALM-16046 Hive Data Warehouse Permission Is Modified
ALM-16047 HiveServer Has Been Deregistered from ZooKeeper
ALM-16048 Tez or Spark Library Path Does Not Exist
ALM-17003 Oozie Service Unavailable
ALM-17004 Oozie Heap Memory Usage Exceeds the Threshold
ALM-17005 Oozie Non Heap Memory Usage Exceeds the Threshold
ALM-17006 Oozie Direct Memory Usage Exceeds the Threshold
ALM-17007 Garbage Collection (GC) Time of the Oozie Process Exceeds the Threshold
ALM-17008 Abnormal Connection Between Oozie and ZooKeeper
ALM-17009 Abnormal Connection Between Oozie and DBService
ALM-17010 Abnormal Connection Between Oozie and HDFS
ALM-17011 Abnormal Connection Between Oozie and Yarn
ALM-18000 Yarn Service Unavailable
ALM-18002 NodeManager Heartbeat Lost
ALM-18003 NodeManager Unhealthy
ALM-18008 Heap Memory Usage of ResourceManager Exceeds the Threshold
ALM-18009 Heap Memory Usage of JobHistoryServer Exceeds the Threshold
ALM-18010 ResourceManager GC Time Exceeds the Threshold
ALM-18011 NodeManager GC Time Exceeds the Threshold
ALM-18012 JobHistoryServer GC Time Exceeds the Threshold
ALM-18013 ResourceManager Direct Memory Usage Exceeds the Threshold
ALM-18014 NodeManager Direct Memory Usage Exceeds the Threshold
ALM-18015 JobHistoryServer Direct Memory Usage Exceeds the Threshold
ALM-18016 Non Heap Memory Usage of ResourceManager Exceeds the Threshold
ALM-18017 Non Heap Memory Usage of NodeManager Exceeds the Threshold
ALM-18018 NodeManager Heap Memory Usage Exceeds the Threshold
ALM-18019 Non Heap Memory Usage of JobHistoryServer Exceeds the Threshold
ALM-18020 Yarn Task Execution Timeout
ALM-18021 Mapreduce Service Unavailable
ALM-18022 Insufficient Yarn Queue Resources
ALM-18023 Number of Pending Yarn Tasks Exceeds the Threshold
ALM-18024 Pending Yarn Memory Usage Exceeds the Threshold
ALM-18025 Number of Terminated Yarn Tasks Exceeds the Threshold
ALM-18026 Number of Failed Yarn Tasks Exceeds the Threshold
ALM-19000 HBase Service Unavailable
ALM-19006 HBase Replication Sync Failed
ALM-19007 HBase GC Time Exceeds the Threshold
ALM-19008 Heap Memory Usage of the HBase Process Exceeds the Threshold
ALM-19009 Direct Memory Usage of the HBase Process Exceeds the Threshold
ALM-19011 RegionServer Region Number Exceeds the Threshold
ALM-19012 HBase System Table Directory or File Lost
ALM-19013 Duration of Regions in transaction State Exceeds the Threshold
ALM-19014 Capacity Quota Usage on ZooKeeper Exceeds the Threshold Severely
ALM-19015 Quantity Quota Usage on ZooKeeper Exceeds the Threshold
ALM-19016 Quantity Quota Usage on ZooKeeper Exceeds the Threshold Severely
ALM-19017 Capacity Quota Usage on ZooKeeper Exceeds the Threshold
ALM-19018 HBase Compaction Queue Size Exceeds the Threshold
ALM-19019 Number of HBase HFiles to Be Synchronized Exceeds the Threshold
ALM-19020 Number of HBase WAL Files to Be Synchronized Exceeds the Threshold
ALM-19021 Handler Usage of RegionServer Exceeds the Threshold
ALM-19022 HBase Hotspot Detection Is Unavailable
ALM-19023 Region Traffic Restriction for HBase
ALM-19024 RPC Requests P99 Latency on RegionServer Exceeds the Threshold
ALM-19025 Damaged StoreFile in HBase
ALM-19026 Damaged WAL Files in HBase
ALM-20002 Hue Service Unavailable
ALM-23001 Loader Service Unavailable
ALM-23003 Loader Task Execution Failure
ALM-23004 Loader Heap Memory Usage Exceeds the Threshold
ALM-23005 Loader Non-Heap Memory Usage Exceeds the Threshold
ALM-23006 Loader Direct Memory Usage Exceeds the Threshold
ALM-23007 Garbage Collection (GC) Time of the Loader Process Exceeds the Threshold
ALM-24000 Flume Service Unavailable
ALM-24001 Flume Agent Exception
ALM-24003 Flume Client Connection Interrupted
ALM-24004 Exception Occurs When Flume Reads Data
ALM-24005 Exception Occurs When Flume Transmits Data
ALM-24006 Heap Memory Usage of Flume Server Exceeds the Threshold
ALM-24007 Flume Server Direct Memory Usage Exceeds the Threshold
ALM-24008 Flume Server Non Heap Memory Usage Exceeds the Threshold
ALM-24009 Flume Server Garbage Collection (GC) Time Exceeds the Threshold
ALM-24010 Flume Certificate File Is Invalid or Damaged
ALM-24011 Flume Certificate File Is About to Expire
ALM-24012 Flume Certificate File Has Expired
ALM-24013 Flume MonitorServer Certificate File Is Invalid or Damaged
ALM-24014 Flume MonitorServer Certificate Is About to Expire
ALM-24015 Flume MonitorServer Certificate File Has Expired
ALM-25000 LdapServer Service Unavailable
ALM-25004 Abnormal LdapServer Data Synchronization
ALM-25005 nscd Service Exception
ALM-25006 Sssd Service Exception
ALM-25007 Number of SlapdServer Connections Exceeds the Threshold
ALM-25008 SlapdServer CPU Usage Exceeds the Threshold
ALM-25500 KrbServer Service Unavailable
ALM-26051 Storm Service Unavailable
ALM-26052 Number of Available Supervisors of the Storm Service Is Less Than the Threshold
ALM-26053 Storm Slot Usage Exceeds the Threshold
ALM-26054 Nimbus Heap Memory Usage Exceeds the Threshold
ALM-27001 DBService Service Unavailable
ALM-27003 DBService Heartbeat Interruption Between the Active and Standby Nodes
ALM-27004 Data Inconsistency Between Active and Standby DBServices
ALM-27005 Database Connections Usage Exceeds the Threshold
ALM-27006 Disk Space Usage of the Data Directory Exceeds the Threshold
ALM-27007 Database Enters the Read-Only Mode
ALM-29000 Impala Service Unavailable
ALM-29004 Impalad Process Memory Usage Exceeds the Threshold
ALM-29005 Number of JDBC Connections to Impalad Exceeds the Threshold
ALM-29006 Number of ODBC Connections to Impalad Exceeds the Threshold
ALM-29010 Number of Queries Being Submitted by Impalad Exceeds the Threshold
ALM-29011 Number of Queries Being Executed by Impalad Exceeds the Threshold
ALM-29012 Number of Queries Being Waited by Impalad Exceeds the Threshold
ALM-29100 Kudu Service Unavailable
ALM-29104 Tserver Process Memory Usage Exceeds the Threshold
ALM-29106 Tserver Process CPU Usage Exceeds the Threshold
ALM-29107 Tserver Process Memory Usage Exceeds the Threshold
ALM-38000 Kafka Service Unavailable
ALM-38001 Insufficient Kafka Disk Capacity
ALM-38002 Kafka Heap Memory Usage Exceeds the Threshold
ALM-38004 Kafka Direct Memory Usage Exceeds the Threshold
ALM-38005 GC Duration of the Broker Process Exceeds the Threshold
ALM-38006 Percentage of Kafka Partitions That Are Not Completely Synchronized Exceeds the Threshold
ALM-38007 Status of Kafka Default User Is Abnormal
ALM-38008 Abnormal Kafka Data Directory Status
ALM-38009 Busy Broker Disk I/Os (Applicable to Versions Later Than MRS 3.1.0)
ALM-38009 Kafka Topic Overload (Applicable to MRS 3.1.0 and Earlier Versions)
ALM-38010 Topics with Single Replica
ALM-38011 User Connection Usage on Broker Exceeds the Threshold
ALM-43001 Spark2x Service Unavailable
ALM-43006 Heap Memory Usage of the JobHistory2x Process Exceeds the Threshold
ALM-43007 Non-Heap Memory Usage of the JobHistory2x Process Exceeds the Threshold
ALM-43008 The Direct Memory Usage of the JobHistory2x Process Exceeds the Threshold
ALM-43009 JobHistory2x Process GC Time Exceeds the Threshold
ALM-43010 Heap Memory Usage of the JDBCServer2x Process Exceeds the Threshold
ALM-43011 Non-Heap Memory Usage of the JDBCServer2x Process Exceeds the Threshold
ALM-43012 Direct Heap Memory Usage of the JDBCServer2x Process Exceeds the Threshold
ALM-43013 JDBCServer2x Process GC Time Exceeds the Threshold
ALM-43017 JDBCServer2x Process Full GC Number Exceeds the Threshold
ALM-43018 JobHistory2x Process Full GC Number Exceeds the Threshold
ALM-43019 Heap Memory Usage of the IndexServer2x Process Exceeds the Threshold
ALM-43020 Non-Heap Memory Usage of the IndexServer2x Process Exceeds the Threshold
ALM-43021 Direct Memory Usage of the IndexServer2x Process Exceeds the Threshold
ALM-43022 IndexServer2x Process GC Time Exceeds the Threshold
ALM-43023 IndexServer2x Process Full GC Number Exceeds the Threshold
ALM-44000 Presto Service Unavailable
ALM-44004 Presto Coordinator Resource Group Queuing Tasks Exceed the Threshold
ALM-44005 Presto Coordinator Process GC Time Exceeds the Threshold
ALM-44006 Presto Worker Process GC Time Exceeds the Threshold
ALM-45000 HetuEngine Service Unavailable
ALM-45001 Faulty HetuEngine Compute Instances
ALM-45003 HetuEngine QAS Disk Capacity Is Insufficient
ALM-45175 Average Time for Calling OBS Metadata APIs Is Greater than the Threshold
ALM-45176 Success Rate of Calling OBS Metadata APIs Is Lower than the Threshold
ALM-45177 Success Rate of Calling OBS Data Read APIs Is Lower than the Threshold
ALM-45178 Success Rate of Calling OBS Data Write APIs Is Lower Than the Threshold
ALM-45179 Number of Failed OBS readFully API Calls Exceeds the Threshold
ALM-45180 Number of Failed OBS read API Calls Exceeds the Threshold
ALM-45181 Number of Failed OBS write API Calls Exceeds the Threshold
ALM-45182 Number of Throttled OBS Operations Exceeds the Threshold
ALM-45275 Ranger Service Unavailable
ALM-45276 Abnormal RangerAdmin Status
ALM-45277 RangerAdmin Heap Memory Usage Exceeds the Threshold
ALM-45278 RangerAdmin Direct Memory Usage Exceeds the Threshold
ALM-45279 RangerAdmin Non Heap Memory Usage Exceeds the Threshold
ALM-45280 RangerAdmin GC Duration Exceeds the Threshold
ALM-45281 UserSync Heap Memory Usage Exceeds the Threshold
ALM-45282 UserSync Direct Memory Usage Exceeds the Threshold
ALM-45283 UserSync Non Heap Memory Usage Exceeds the Threshold
ALM-45284 UserSync Garbage Collection (GC) Time Exceeds the Threshold
ALM-45285 TagSync Heap Memory Usage Exceeds the Threshold
ALM-45286 TagSync Direct Memory Usage Exceeds the Threshold
ALM-45287 TagSync Non Heap Memory Usage Exceeds the Threshold
ALM-45288 TagSync Garbage Collection (GC) Time Exceeds the Threshold
ALM-45289 PolicySync Heap Memory Usage Exceeds the Threshold
ALM-45290 PolicySync Direct Memory Usage Exceeds the Threshold
ALM-45291 PolicySync Non-Heap Memory Usage Exceeds the Threshold
ALM-45292 PolicySync GC Duration Exceeds the Threshold
ALM-45325 Presto Service Unavailable
ALM-45326 Number of Presto Coordinator Threads Exceeds the Threshold
ALM-45327 Presto Coordinator Process GC Time Exceeds the Threshold
ALM-45328 Presto Worker Process GC Time Exceeds the Threshold
ALM-45329 Presto Coordinator Resource Group Queuing Tasks Exceed the Threshold
ALM-45330 Number of Presto Worker Threads Exceeds the Threshold
ALM-45425 ClickHouse Service Unavailable
ALM-45426 ClickHouse Service Quantity Quota Usage in ZooKeeper Exceeds the Threshold
ALM-45427 ClickHouse Service Capacity Quota Usage in ZooKeeper Exceeds the Threshold
ALM-45428 ClickHouse Disk I/O Exception
ALM-45429 Table Metadata Synchronization Failed on the Added ClickHouse Node
ALM-45430 Permission Metadata Synchronization Failed on the Added ClickHouse Node
ALM-45431 Improper ClickHouse Instance Distribution for Topology Allocation
ALM-45432 ClickHouse User Synchronization Process Fails
ALM-45433 ClickHouse AZ Topology Exception
ALM-45434 A Single Replica Exists in the ClickHouse Data Table
ALM-45435 Inconsistent Metadata of ClickHouse Tables
ALM-45436 Skew ClickHouse Table Data
ALM-45437 Excessive Parts in the ClickHouse Table
ALM-45438 ClickHouse Disk Usage Exceeds 80%
ALM-45439 ClickHouse Node Enters the Read-Only Mode
ALM-45440 Inconsistency Between ClickHouse Replicas
ALM-45441 Zookeeper Disconnected
ALM-45442 Too Many Concurrent SQL Statements
ALM-45443 Slow SQL Queries in the Cluster
ALM-45444 Abnormal ClickHouse Process
ALM-45475 A Single Replica Exists in the Kudu Data Table
ALM-45476 Kudu Failed to Enter the Maintenance Mode
ALM-45477 Failed to Restore Data After a Disk of Kudu Is Replaced
ALM-45478 Kudu Failed Data Balancing
ALM-45479 Number of Tablets of the Tserver Process Exceeds the Threshold
ALM-45480 Tablet Leaders of a Tserver Process Are Unevenly Distributed
ALM-45585 IoTDB Service Unavailable
ALM-45586 IoTDBServer Heap Memory Usage Exceeds the Threshold
ALM-45587 IoTDBServer GC Duration Exceeds the Threshold
ALM-45588 IoTDBServer Direct Memory Usage Exceeds the Threshold
ALM-45589 ConfigNode Heap Memory Usage Exceeds the Threshold
ALM-45590 ConfigNode GC Duration Exceeds the Threshold
ALM-45591 ConfigNode Direct Memory Usage Exceeds the Threshold
ALM-45592 IoTDBServer RPC Execution Duration Exceeds the Threshold
ALM-45593 IoTDBServer Flush Execution Duration Exceeds the Threshold
ALM-45594 IoTDBServer Intra-Space Merge Duration Exceeds the Threshold
ALM-45595 IoTDBServer Cross-Space Merge Duration Exceeds the Threshold
ALM-45596 Procedure Execution Failed
ALM-45615 CDL Service Unavailable
ALM-45616 CDL Job Execution Exception
ALM-45617 Data Queued in the CDL Replication Slot Exceeds the Threshold
ALM-45635 FlinkServer Job Execution Failure
ALM-45636 FlinkServer Job Checkpoints Keep Failing
ALM-45636 Flink Job Checkpoints Keep Failing
ALM-45637 FlinkServer Task Is Continuously Under Back Pressure
ALM-45638 Number of Restarts After FlinkServer Job Failures Exceeds the Threshold
ALM-45638 Number of Restarts After Flink Job Failures Exceeds the Threshold
ALM-45639 Checkpointing of a Flink Job Times Out
ALM-45640 FlinkServer Heartbeat Interruption Between the Active and Standby Nodes
ALM-45641 Data Synchronization Exception Between the Active and Standby FlinkServer Nodes
ALM-45642 RocksDB Continuously Triggers Write Traffic Limiting
ALM-45643 MemTable Size of RocksDB Continuously Exceeds the Threshold
ALM-45644 Number of SST Files at Level 0 of RocksDB Continuously Exceeds the Threshold
ALM-45645 Pending Flush Size of RocksDB Continuously Exceeds the Threshold
ALM-45646 Pending Compaction Size of RocksDB Continuously Exceeds the Threshold
ALM-45647 Estimated Pending Compaction Size of RocksDB Continuously Exceeds the Threshold
ALM-45648 RocksDB Frequently Encounters Write-Stopped
ALM-45649 P95 Latency of RocksDB Get Requests Continuously Exceeds the Threshold
ALM-45650 P95 Latency of RocksDB Write Requests Continuously Exceeds the Threshold
ALM-45652 Flink Service Unavailable
ALM-45653 Invalid Flink HA Certificate File
ALM-45654 Flink HA Certificate Is About to Expire
ALM-45655 Flink HA Certificate File Has Expired
ALM-45736 Guardian Service Unavailable
ALM-45737 TokenServer Heap Memory Usage Exceeds the Threshold
ALM-45738 TokenServer Direct Memory Usage Exceeds the Threshold
ALM-45739 TokenServer Non-Heap Memory Usage Exceeds the Threshold
ALM-45740 TokenServer GC Duration Exceeds the Threshold
ALM-50201 Doris Service Unavailable
ALM-50202 FE CPU Usage Exceeds the Threshold
ALM-50203 FE Memory Usage Exceeds the Threshold
ALM-50205 BE CPU Usage Exceeds the Threshold
ALM-50206 BE Memory Usage Exceeds the Threshold
ALM-50207 Ratio of Connections to the FE MySQL Port to the Maximum Connections Allowed Exceeds the Threshold
ALM-50208 Failures to Clear Historical Metadata Image Files Exceed the Threshold
ALM-50209 Failures to Generate Metadata Image Files Exceed the Threshold
ALM-50210 Maximum Compaction Score of All BE Nodes Exceeds the Threshold
ALM-50211 FE Queue Length of BE Periodic Report Tasks Exceeds the Threshold
ALM-50212 Accumulated Old-Generation GC Duration of the FE Process Exceeds the Threshold
ALM-50213 Number of Tasks Queuing in the FE Thread Pool for Interacting with BE Exceeds the Threshold
ALM-50214 Number of Tasks Queuing in the FE Thread Pool for Task Processing Exceeds the Threshold
ALM-50215 Longest Duration of RPC Requests Received by Each FE Thrift Method Exceeds the Threshold
ALM-50216 Memory Usage of the FE Node Exceeds the Threshold
ALM-50217 Heap Memory Usage of the FE Node Exceeds the Threshold
ALM-50219 Length of the Queue in the Thread Pool for Query Execution Exceeds the Threshold
ALM-50220 Error Rate of TCP Packet Receiving Exceeds the Threshold
ALM-50221 BE Data Disk Usage Exceeds the Threshold
ALM-50222 Disk Status of a Specified Data Directory on BE Is Abnormal
ALM-50223 Maximum Memory Required by BE Is Greater Than the Remaining Memory of the Machine
ALM-50224 Failures a Certain Task Type on BE Are Increasing
ALM-50225 FE Instance Fault
ALM-50226 BE Instance Fault
ALM-50401 Number of JobServer Jobs Waiting to Be Executed Exceeds the Threshold
ALM-50402 JobGateway Service Unavailable
RISK_ALARM_001 Cluster Network Risks
Security Description
Security Configuration Suggestions for Clusters with Kerberos Authentication Disabled
Security Authentication Principles and Mechanisms
Security Notice
Guide for Fixing the Apache Log4j2 Remote Code Execution Vulnerability (CVE-2021-44228)
MRS Fastjson Vulnerability Remediation Guide
Overview
Impact
Remediating Manager Web
Remediating Manager Controller
Remediating Manager NodeAgent
Remediating Kafka
Remediating Flink
High-Risk Operations
Interconnecting Jupyter Notebook with MRS Using Custom Python
Overview
Installing a Client on a Node Outside the Cluster
Installing Python 3
Configuring the MRS Client
Installing Jupyter Notebook
Verifying that Jupyter Notebook Can Access MRS
FAQs
Appendix
ECS Specifications Used by MRS
BMS Specifications Used by MRS
Data Migration Solution
Making Preparations
Exporting Metadata
Copying Data
Restoring Data
Precautions for MRS 3.x
Installing the Flume Client
Installing the Flume Client on Clusters of Versions Earlier Than MRS 3.x
Installing the Flume Client on MRS 3.x or Later Clusters
Component Operation Guide (Normal)
Using Alluxio
Configuring an Underlying Storage System
Accessing Alluxio Using a Data Application
Common Operations of Alluxio
Using CarbonData (for Versions Earlier Than MRS 3.x)
Using CarbonData from Scratch
About CarbonData Table
Creating a CarbonData Table
Deleting a CarbonData Table
Using CarbonData (for MRS 3.x or Later)
Overview
CarbonData Overview
Main Specifications of CarbonData
Common CarbonData Parameters
CarbonData Operation Guide
CarbonData Quick Start
CarbonData Table Management
About CarbonData Table
Creating a CarbonData Table
Deleting a CarbonData Table
Modify the CarbonData Table
CarbonData Table Data Management
Loading Data
Deleting Segments
Combining Segments
CarbonData Data Migration
Migrating Data on CarbonData from Spark 1.5 to Spark2x
CarbonData Performance Tuning
Tuning Guidelines
Suggestions for Creating CarbonData Tables
Configurations for Performance Tuning
CarbonData Access Control
CarbonData Syntax Reference
DDL
CREATE TABLE
CREATE TABLE As SELECT
DROP TABLE
SHOW TABLES
ALTER TABLE COMPACTION
TABLE RENAME
ADD COLUMNS
DROP COLUMNS
CHANGE DATA TYPE
REFRESH TABLE
REGISTER INDEX TABLE
DML
LOAD DATA
UPDATE CARBON TABLE
DELETE RECORDS from CARBON TABLE
INSERT INTO CARBON TABLE
DELETE SEGMENT by ID
DELETE SEGMENT by DATE
SHOW SEGMENTS
CREATE SECONDARY INDEX
SHOW SECONDARY INDEXES
DROP SECONDARY INDEX
CLEAN FILES
SET/RESET
Operation Concurrent Execution
API
Spatial Indexes
CarbonData Troubleshooting
Filter Result Is not Consistent with Hive when a Big Double Type Value Is Used in Filter
Query Performance Deterioration
CarbonData FAQ
Why Is Incorrect Output Displayed When I Perform Query with Filter on Decimal Data Type Values?
How to Avoid Minor Compaction for Historical Data?
How to Change the Default Group Name for CarbonData Data Loading?
Why Does INSERT INTO CARBON TABLE Command Fail?
Why Is the Data Logged in Bad Records Different from the Original Input Data with Escape Characters?
Why Data Load Performance Decreases due to Bad Records?
Why INSERT INTO/LOAD DATA Task Distribution Is Incorrect and the Opened Tasks Are Less Than the Available Executors when the Number of Initial ExecutorsIs Zero?
Why Does CarbonData Require Additional Executors Even Though the Parallelism Is Greater Than the Number of Blocks to Be Processed?
Why Data loading Fails During off heap?
Why Do I Fail to Create a Hive Table?
How Do I Logically Split Data Across Different Namespaces?
Why Missing Privileges Exception is Reported When I Perform Drop Operation on Databases?
Why the UPDATE Command Cannot Be Executed in Spark Shell?
How Do I Configure Unsafe Memory in CarbonData?
Why Exception Occurs in CarbonData When Disk Space Quota is Set for Storage Directory in HDFS?
Why Does Data Query or Loading Fail and "org.apache.carbondata.core.memory.MemoryException: Not enough memory" Is Displayed?
Why Do Files of a Carbon Table Exist in the Recycle Bin Even If the drop table Command Is Not Executed When Mis-deletion Prevention Is Enabled?
Using ClickHouse
Using ClickHouse from Scratch
ClickHouse Table Engine Overview
Creating a ClickHouse Table
ClickHouse Data Type
Configuring Interconnection Between ClickHouse and OBS
Enabling the mysql_port Configuration for ClickHouse
Common ClickHouse SQL Syntax
CREATE DATABASE: Creating a Database
CREATE TABLE: Creating a Table
INSERT INTO: Inserting Data into a Table
SELECT: Querying Table Data
ALTER TABLE: Modifying a Table Structure
ALTER TABLE: Modifying Table Data
DESC: Querying a Table Structure
DROP: Deleting a Table
SHOW: Displaying Information About Databases and Tables
Migrating ClickHouse Data
Accessing RDS MySQL Using ClickHouse
Importing GaussDB(DWS) Data to a ClickHouse Table
Using ClickHouse to Import and Export Data
Synchronizing Kafka Data to ClickHouse
Using the ClickHouse Data Migration Tool
User Management and Authentication
ClickHouse User and Permission Management
Interconnecting ClickHouse With OpenLDAP for Authentication
ClickHouse Cluster Management
ClickHouse Cluster Configuration
Expanding the Data Disk Capacity of a ClickHouse Node
Accessing ClickHouse Through ELB
Backing Up and Restoring ClickHouse Data Using a Data File
ClickHouse Log Overview
ClickHouse Performance Tuning
Solution to the "Too many parts" Error in Data Tables
Accelerating Merge Operations
Accelerating TTL Operations
ClickHouse FAQ
How Do I Do If the Disk Status Displayed in the System.disks Table Is fault or abnormal?
How Do I Migrate Data from Hive/HDFS to ClickHouse?
How Do I Migrate Data from OBS/S3 to ClickHouse?
An Error Is Reported in Logs When the Auxiliary ZooKeeper or Replica Data Is Used to Synchronize Table Data
How Do I Grant the Select Permission at the Database Level to ClickHouse Users?
Using DBService
DBService Log Overview
Using Flink
Using Flink from Scratch
Viewing Flink Job Information
Configuring Flink Service Parameters
Configuring Flink Security Features
Security Features
Authentication and Encryption
Configuring Kafka
Configuring Pipeline
Configuring and Developing a Flink Visualization Job
Introduction to Flink Web UI
Flink Web UI Permission Management
Creating a FlinkServer Role
Accessing the Flink Web UI
Creating an Application
Creating a Cluster Connection
Creating a Data Connection
Creating a Stream Table
Creating a Job
Configuring and Managing UDFs
Flink Log Overview
Flink Performance Tuning
Memory Configuration Optimization
Configuring DOP
Configuring Process Parameters
Optimizing the Design of Partitioning Method
Configuring the Netty Network Communication
Experience Summary
Common Flink Shell Commands
Reference
Example of Issuing a Certificate
Flink Restart Policy
Using Flume
Using Flume from Scratch
Overview
Installing the Flume Client
Installing the Flume Client on Clusters of Versions Earlier Than MRS 3.x
Installing the Flume Client on MRS 3.x or Later Clusters
Viewing Flume Client Logs
Stopping or Uninstalling the Flume Client
Using the Encryption Tool of the Flume Client
Flume Service Configuration Guide
Flume Configuration Parameter Description
Using Environment Variables in the properties.properties File
Non-Encrypted Transmission
Configuring Non-encrypted Transmission
Typical Scenario: Collecting Local Static Logs and Uploading Them to Kafka
Typical Scenario: Collecting Local Static Logs and Uploading Them to HDFS
Typical Scenario: Collecting Local Dynamic Logs and Uploading Them to HDFS
Typical Scenario: Collecting Logs from Kafka and Uploading Them to HDFS
Typical Scenario: Collecting Logs from Kafka and Uploading Them to HDFS Through the Flume Client
Typical Scenario: Collecting Local Static Logs and Uploading Them to HBase
Encrypted Transmission
Configuring the Encrypted Transmission
Typical Scenario: Collecting Local Static Logs and Uploading Them to HDFS
Viewing Flume Client Monitoring Information
Connecting Flume to Kafka in Security Mode
Connecting Flume with Hive in Security Mode
Configuring the Flume Service Model
Overview
Service Model Configuration Guide
Introduction to Flume Logs
Flume Client Cgroup Usage Guide
Secondary Development Guide for Flume Third-Party Plug-ins
Configuring the Flume Customized Script
Common Issues About Flume
Using HBase
Using HBase from Scratch
Using an HBase Client
Creating HBase Roles
Configuring HBase Replication
Configuring HBase Parameters
Enabling Cross-Cluster Copy
Using the ReplicationSyncUp Tool
Using HIndex
Introduction to HIndex
Loading Index Data in Batches
Using an Index Generation Tool
Migrating Index Data
Configuring an RSGroup
Configuring HBase DR
Configuring HBase Data Compression and Encoding
Performing an HBase DR Service Switchover
Performing an HBase DR Active/Standby Cluster Switchover
Community BulkLoad Tool
In-House Enhanced BulkLoad Tool
Importing Data in a Customized Manner
Importing Data in Batches
Combining Rowkeys
Implementing Custom RowKeys
Combining Fields
Specifying Field Data Types
Defining Inapplicable Data Rows
Importing Data with Indexes in a Customized Manner
Creating a Secondary Index When Importing Data In Batches
Combining Rowkeys
Implementing Custom RowKeys
Combining Fields
Specifying Field Data Type
Defining Inapplicable Data Row
Updating Rows in Batches
Deleting Rows in Batches
Collecting Statistics on Rows
Configuring the MOB
Configuring Secure HBase Replication
Configuring Region In Transition Recovery Chore Service
Using a Secondary Index
HBase Log Overview
HBase Performance Tuning
Improving the BulkLoad Efficiency
Improving Put Performance
Optimizing Put and Scan Performance
Improving Real-time Data Write Performance
Improving Real-time Data Read Performance
Optimizing JVM Parameters
Common Issues About HBase
Why Does a Client Keep Failing to Connect to a Server for a Long Time?
Operation Failures Occur in Stopping BulkLoad On the Client
Why May a Table Creation Exception Occur When HBase Deletes or Creates the Same Table Consecutively?
Why Other Services Become Unstable If HBase Sets up A Large Number of Connections over the Network Port?
Why Does the HBase BulkLoad Task (One Table Has 26 TB Data) Consisting of 210,000 Map Tasks and 10,000 Reduce Tasks Fail?
How Do I Restore a Region in the RIT State for a Long Time?
Why Does HMaster Exits Due to Timeout When Waiting for the Namespace Table to Go Online?
Why Does SocketTimeoutException Occur When a Client Queries HBase?
Why Modified and Deleted Data Can Still Be Queried by Using the Scan Command?
Why "java.lang.UnsatisfiedLinkError: Permission denied" exception thrown while starting HBase shell?
When does the RegionServers listed under "Dead Region Servers" on HMaster WebUI gets cleared?
Why Are Different Query Results Returned After I Use Same Query Criteria to Query Data Successfully Imported by HBase bulkload?
What Should I Do If I Fail to Create Tables Due to the FAILED_OPEN State of Regions?
How Do I Delete Residual Table Names in the /hbase/table-lock Directory of ZooKeeper?
Why Does HBase Become Faulty When I Set a Quota for the Directory Used by HBase in HDFS?
Why HMaster Times Out While Waiting for Namespace Table to be Assigned After Rebuilding Meta Using OfflineMetaRepair Tool and Startups Failed
Why Messages Containing FileNotFoundException and no lease Are Frequently Displayed in the HMaster Logs During the WAL Splitting Process?
Why Does the ImportTsv Tool Display "Permission denied" When the Same Linux User as and a Different Kerberos User from the Region Server Are Used?
Insufficient Rights When a Tenant Accesses Phoenix
Insufficient Rights When a Tenant Uses the HBase Bulkload Function
What Can I Do When HBase Fails to Recover a Task and a Message Is Displayed Stating "Rollback recovery failed"?
How Do I Fix Region Overlapping?
Why Does RegionServer Fail to Be Started When GC Parameters Xms and Xmx of HBase RegionServer Are Set to 31 GB?
Why Does the LoadIncrementalHFiles Tool Fail to Be Executed and "Permission denied" Is Displayed When Nodes in a Cluster Are Used to Import Data in Batches?
Why Is the Error Message "import argparse" Displayed When the Phoenix sqlline Script Is Used?
How Do I Deal with the Restrictions of the Phoenix BulkLoad Tool?
Why a Message Is Displayed Indicating that the Permission is Insufficient When CTBase Connects to the Ranger Plug-ins?
Using HDFS
Using Hadoop from Scratch
Configuring HDFS Parameters
Configuring Memory Management
Creating an HDFS Role
Using the HDFS Client
Running the DistCp Command
Overview of HDFS File System Directories
Changing the DataNode Storage Directory
Configuring HDFS Directory Permission
Configuring NFS
Planning HDFS Capacity
Configuring ulimit for HBase and HDFS
Configuring HDFS DataNode Data Balancing
Configuring Replica Replacement Policy for Heterogeneous Capacity Among DataNodes
Configuring the Number of Files in a Single HDFS Directory
Configuring the Recycle Bin Mechanism
Setting Permissions on Files and Directories
Setting the Maximum Lifetime and Renewal Interval of a Token
Configuring the Damaged Disk Volume
Configuring Encrypted Channels
Reducing the Probability of Abnormal Client Application Operation When the Network Is Not Stable
Configuring the NameNode Blacklist
Optimizing HDFS NameNode RPC QoS
Optimizing HDFS DataNode RPC QoS
Configuring LZC Compression
Configuring Reserved Percentage of Disk Usage on DataNodes
Configuring HDFS NodeLabel
Configuring HDFS Mover
Using HDFS AZ Mover
Configuring HDFS DiskBalancer
Configuring the Observer NameNode to Process Read Requests
Performing Concurrent Operations on HDFS Files
Introduction to HDFS Logs
HDFS Performance Tuning
Improving Write Performance
Improving Read Performance Using Client Metadata Cache
Improving the Connection Between the Client and NameNode Using Current Active Cache
FAQ
NameNode Startup Is Slow
DataNode Is Normal but Cannot Report Data Blocks
HDFS WebUI Cannot Properly Update Information About Damaged Data
Why Do DistCp Commands Fail to Run in a Security Cluster and What Should I Do When Exceptions Are Thrown?
What Should I Do If DataNode Fails to Be Started When the Number of Disks Defined in dfs.datanode.data.dir Equals the Value of dfs.datanode.failed.volumes.tolerated?
Failed to Calculate the Capacity of a DataNode when Multiple data.dir Directories Are Configured in a Disk Partition
Standby NameNode Fails to Be Restarted When the System Is Powered off During Metadata (Namespace) Storage
What Should I Do If Data in the Cache Is Lost When the System Is Powered Off During Small File Storage?
Why Does Array Border-crossing Occur During FileInputFormat Split?
Why Is the Storage Type of File Copies DISK When the Tiered Storage Policy Is LAZY_PERSIST?
How Do I Handle the Problem that HDFS Client Is Irresponsive When the NameNode Is Overloaded for a Long Time?
Can I Delete or Modify the Data Storage Directory in DataNode?
Blocks Miss on the NameNode UI After the Successful Rollback
Why Is "java.net.SocketException: No buffer space available" Reported When Data Is Written to HDFS
Why are There Two Standby NameNodes After the active NameNode Is Restarted?
When Does a Balance Process in HDFS, Shut Down and Fail to be Executed Again?
"This page can't be displayed" Is Displayed When Internet Explorer Fails to Access the Native HDFS UI
NameNode Fails to Be Restarted Due to EditLog Discontinuity
Using Hive
Using Hive from Scratch
Configuring Hive Parameters
Hive SQL
Permission Management
Hive Permission
Creating a Hive Role
Configuring Permissions for Hive Tables, Columns, or Databases
Configuring Permissions to Use Other Components for Hive
Using a Hive Client
Using HDFS Colocation to Store Hive Tables
Using the Hive Column Encryption Function
Customizing Row Separators
Configuring Hive on HBase in Across Clusters with Mutual Trust Enabled
Deleting Single-Row Records from Hive on HBase
Configuring HTTPS/HTTP-based REST APIs
Enabling or Disabling the Transform Function
Access Control of a Dynamic Table View on Hive
Specifying Whether the ADMIN Permissions Is Required for Creating Temporary Functions
Using Hive to Read Data in a Relational Database
Supporting Traditional Relational Database Syntax in Hive
Creating User-Defined Hive Functions
Enhancing beeline Reliability
Viewing Table Structures Using the show create Statement as Users with the select Permission
Writing a Directory into Hive with the Old Data Removed to the Recycle Bin
Inserting Data to a Directory That Does Not Exist
Creating Databases and Tables in the Default Database Only as the Hive Administrator
Disabling of Specifying the location Keyword When Creating an Internal Hive Table
Enabling the Function of Creating a Foreign Table in a Directory That Can Only Be Read
Authorizing Over 32 Roles in Hive
Restricting the Maximum Number of Maps for Hive Tasks
HiveServer Lease Isolation
Switching the Hive Execution Engine to Tez
Hive Supporting Cold and Hot Storage of Partitioned Metadata
Hive Supporting ZSTD Compression Formats
Hive Log Overview
Hive Performance Tuning
Creating Table Partitions
Optimizing Join
Optimizing Group By
Optimizing Data Storage
Optimizing SQL Statements
Optimizing the Query Function Using Hive CBO
Common Issues About Hive
How Do I Delete UDFs on Multiple HiveServers at the Same Time?
Why Cannot the DROP operation Be Performed on a Backed-up Hive Table?
How to Perform Operations on Local Files with Hive User-Defined Functions
How Do I Forcibly Stop MapReduce Jobs Executed by Hive?
Table Creation Fails Because Hive Complex Fields' Names Contain Special Characters
How Do I Monitor the Hive Table Size?
How Do I Prevent Key Directories from Data Loss Caused by Misoperations of the insert overwrite Statement?
Why Is Hive on Spark Task Freezing When HBase Is Not Installed?
Error Reported When the WHERE Condition Is Used to Query Tables with Excessive Partitions in FusionInsight Hive
Why Cannot I Connect to HiveServer When I Use IBM JDK to Access the Beeline Client?
Description of Hive Table Location (Either Be an OBS or HDFS Path)
Why Cannot Data Be Queried After the MapReduce Engine Is Switched After the Tez Engine Is Used to Execute Union-related Statements?
Why Does Hive Not Support Concurrent Data Writing to the Same Table or Partition?
Why Does Hive Not Support Vectorized Query?
Why Does Metadata Still Exist When the HDFS Data Directory of the Hive Table Is Deleted by Mistake?
How Do I Disable the Logging Function of Hive?
Why Hive Tables in the OBS Directory Fail to Be Deleted?
Hive Configuration Problems
Using Hudi
Getting Started
Basic Operations
Hudi Table Schema
Write
Batch Write
Stream Write
Synchronizing Hudi Table Data to Hive
Read
Overview
Reading COW Table Views
Reading MOR Table Views
Data Management and Maintenance
Clustering
Cleaning
Compaction
Savepoint
Single-Table Concurrency Control
Using the Hudi Client
Operating a Hudi Table Using hudi-cli.sh
Configuration Reference
Overview
Write Configuration
Configuration of Hive Table Synchronization
Index Configuration
Storage Configuration
Compaction and Cleaning Configurations
Single-Table Concurrency Control Configuration
Hudi Performance Tuning
Common Issues About Hudi
Data Write
Parquet/Avro schema Is Reported When Updated Data Is Written
UnsupportedOperationException Is Reported When Updated Data Is Written
SchemaCompatabilityException Is Reported When Updated Data Is Written
What Should I Do If Hudi Consumes Much Space in a Temporary Folder During Upsert?
Hudi Fails to Write Decimal Data with Lower Precision
Data Collection
IllegalArgumentException Is Reported When Kafka Is Used to Collect Data
HoodieException Is Reported When Data Is Collected
HoodieKeyException Is Reported When Data Is Collected
Hive Synchronization
SQLException Is Reported During Hive Data Synchronization
HoodieHiveSyncException Is Reported During Hive Data Synchronization
SemanticException Is Reported During Hive Data Synchronization
Using Hue (Versions Earlier Than MRS 3.x)
Using Hue from Scratch
Accessing the Hue Web UI
Hue Common Parameters
Using HiveQL Editor on the Hue Web UI
Using the Metadata Browser on the Hue Web UI
Using File Browser on the Hue Web UI
Using Job Browser on the Hue Web UI
Using Hue (MRS 3.x or Later)
Using Hue from Scratch
Accessing the Hue Web UI
Hue Common Parameters
Using HiveQL Editor on the Hue Web UI
Using the SparkSql Editor on the Hue Web UI
Using the Metadata Browser on the Hue Web UI
Using File Browser on the Hue Web UI
Using Job Browser on the Hue Web UI
Using HBase on the Hue Web UI
Typical Scenarios
HDFS on Hue
Configuring HDFS Cold and Hot Data Migration
Hive on Hue
Oozie on Hue
Hue Log Overview
Common Issues About Hue
Why Do HQL Statements Fail to Execute in Hue Using Internet Explorer?
Why Does the use database Statement Become Invalid in Hive?
Why Do HDFS Files Fail to Access Through the Hue Web UI?
Why Do Large Files Fail to Upload on the Hue Page
Why Is the Hue Native Page Cannot Be Properly Displayed If the Hive Service Is Not Installed in a Cluster?
How Do I Solve the Problem of Setting the Time Zone of the Oozie Editor on the Hue Web UI?
Using Impala
Using Impala from Scratch
Common Impala Parameters
Accessing the Impala Web UI
Using Impala to Operate Kudu
Interconnecting Impala with External LDAP
Enabling and Configuring a Dynamic Resource Pool for Impala
Using the Impala Query Management Page
Using Kafka
Using Kafka from Scratch
Managing Kafka Topics
Querying Kafka Topics
Managing Kafka User Permissions
Managing Messages in Kafka Topics
Synchronizing Binlog-based MySQL Data to the MRS Cluster
Creating a Kafka Role
Kafka Common Parameters
Safety Instructions on Using Kafka
Kafka Specifications
Using the Kafka Client
Configuring Kafka HA and High Reliability Parameters
Changing the Broker Storage Directory
Checking the Consumption Status of Consumer Group
Kafka Balancing Tool Instructions
Balancing Data After Kafka Node Scale-Out
Kafka Token Authentication Mechanism Tool Usage
Using Kafka UI
Accessing Kafka UI
Kafka UI Overview
Creating a Topic on Kafka UI
Migrating a Partition on Kafka UI
Managing Topics on Kafka UI
Viewing Brokers on Kafka UI
Viewing a Consumer Group on Kafka UI
Introduction to Kafka Logs
Performance Tuning
Kafka Performance Tuning
Kafka Feature Description
Migrating Data Between Kafka Nodes
Common Issues About Kafka
How Do I Solve the Problem that Kafka Topics Cannot Be Deleted?
Using KafkaManager
Introduction to KafkaManager
Accessing the KafkaManager Web UI
Managing Kafka Clusters
Kafka Cluster Monitoring Management
Using Loader
Using Loader from Scratch
How to Use Loader
Common Loader Parameters
Creating a Loader Role
Loader Link Configuration
Managing Loader Links (Versions Earlier Than MRS 3.x)
Managing Loader Links (MRS 3.x and Later Versions)
Source Link Configurations of Loader Jobs
Destination Link Configurations of Loader Jobs
Managing Loader Jobs
Preparing a Driver for MySQL Database Link
Importing Data
Overview
Importing Data Using Loader
Typical Scenario: Importing Data from an SFTP Server to HDFS or OBS
Typical Scenario: Importing Data from an SFTP Server to HBase
Typical Scenario: Importing Data from an SFTP Server to Hive
Typical Scenario: Importing Data from an FTP Server to HBase
Typical Scenario: Importing Data from a Relational Database to HDFS or OBS
Typical Scenario: Importing Data from a Relational Database to HBase
Typical Scenario: Importing Data from a Relational Database to Hive
Typical Scenario: Importing Data from HDFS or OBS to HBase
Typical Scenario: Importing Data from a Relational Database to ClickHouse
Typical Scenario: Importing Data from HDFS to ClickHouse
Exporting Data
Overview
Using Loader to Export Data
Typical Scenario: Exporting Data from HDFS or OBS to an SFTP Server
Typical Scenario: Exporting Data from HBase to an SFTP Server
Typical Scenario: Exporting Data from Hive to an SFTP Server
Typical Scenario: Exporting Data from HDFS or OBS to a Relational Database
Typical Scenario: Exporting Data from HBase to a Relational Database
Typical Scenario: Exporting Data from Hive to a Relational Database
Typical Scenario: Importing Data from HBase to HDFS or OBS
Managing Jobs
Migrating Loader Jobs in Batches
Deleting Loader Jobs in Batches
Importing Loader Jobs in Batches
Exporting Loader Jobs in Batches
Viewing Historical Job Information
Operator Help
Overview
Input Operators
CSV File Input
Fixed File Input
Table Input
HBase Input
HTML Input
Hive input
Spark Input
Conversion Operators
Long Date Conversion
Null Value Conversion
Constant Field Addition
Random Value Conversion
Concat Fields
Extract Fields
Modulo Integer
String Cut
EL Operation
String Operations
String Reverse
String Trim
Filter Rows
Update Fields Operator
Output Operators
Hive output
Spark Output
Table Output
File Output
HBase Output
ClickHouse Output
Associating, Editing, Importing, or Exporting the Field Configuration of an Operator
Using Macro Definitions in Configuration Items
Operator Data Processing Rules
Client Tools
Running a Loader Job by Using Commands
loader-tool Usage Guide
loader-tool Usage Example
schedule-tool Usage Guide
schedule-tool Usage Example
Using loader-backup to Back Up Job Data
Open Source sqoop-shell Tool Usage Guide
Example for Using the Open-Source sqoop-shell Tool (SFTP-HDFS)
Example for Using the Open-Source sqoop-shell Tool (Oracle-HBase)
Loader Log Overview
Example: Using Loader to Import Data from OBS to HDFS
Common Issues About Loader
Data Cannot Be Saved in Internet Explorer 10 or 11
Differences Among Connectors Used During the Process of Importing Data from the Oracle Database to HDFS
Using Kudu
Using Kudu from Scratch
Accessing the Kudu Web UI
Using MapReduce
Configuring the Log Archiving and Clearing Mechanism
Reducing Client Application Failure Rate
Transmitting MapReduce Tasks from Windows to Linux
Configuring the Distributed Cache
Configuring the MapReduce Shuffle Address
Configuring the Cluster Administrator List
Introduction to MapReduce Logs
MapReduce Performance Tuning
Optimization Configuration for Multiple CPU Cores
Determining the Job Baseline
Streamlining Shuffle
AM Optimization for Big Tasks
Speculative Execution
Using Slow Start
Optimizing Performance for Committing MR Jobs
Common Issues About MapReduce
After an Active/Standby Switchover of ResourceManager Occurs, a Task Is Interrupted and Runs for a Long Time
Why Does a MapReduce Task Stay Unchanged for a Long Time?
Why the Client Hangs During Job Running?
Why Cannot HDFS_DELEGATION_TOKEN Be Found in the Cache?
How Do I Set the Task Priority When Submitting a MapReduce Task?
Why Physical Memory Overflow Occurs If a MapReduce Task Fails?
After the Address of MapReduce JobHistoryServer Is Changed, Why the Wrong Page is Displayed When I Click the Tracking URL on the ResourceManager WebUI?
MapReduce Job Failed in Multiple NameService Environment
Why a Fault MapReduce Node Is Not Blacklisted?
Using OpenTSDB
Using an MRS Client to Operate OpenTSDB Metric Data
Running the curl Command to Operate OpenTSDB
Using Oozie
Using Oozie from Scratch
Using the Oozie Client
Enabling Oozie High Availability (HA)
Using Oozie Client to Submit an Oozie Job
Submitting a Hive Job
Submitting a Spark2x Job
Submitting a Loader Job
Submitting a DistCp Job
Submitting Other Jobs
Using Hue to Submit an Oozie Job
Creating a Workflow
Submitting a Workflow Job
Submitting a Hive2 Job
Submitting a Spark2x Job
Submitting a Java Job
Submitting a Loader Job
Submitting a MapReduce Job
Submitting a Sub-workflow Job
Submitting a Shell Job
Submitting an HDFS Job
Submitting a Streaming Job
Submitting a DistCp Job
Example of Mutual Trust Operations
Submitting an SSH Job
Submitting a Hive Script
Submitting a Coordinator Periodic Scheduling Job
Submitting a Bundle Batch Processing Job
Querying Job Execution Results
Oozie Log Overview
Common Issues About Oozie
Oozie Scheduled Tasks Are Not Executed on Time
Why Update of the share lib Directory of Oozie on HDFS Does Not Take Effect?
Common Oozie Troubleshooting Methods
Using Presto
Accessing the Presto Web UI
Using a Client to Execute Query Statements
Presto FAQ
How Do I Configure Multiple Hive Connections for Presto?
Using Ranger (MRS 1.9.2)
Creating a Ranger Cluster
Accessing the Ranger Web UI and Synchronizing Unix Users to the Ranger Web UI
Configuring Hive/Impala Access Permissions in Ranger
Configuring HBase Access Permissions in Ranger
Using Ranger (MRS 3.x)
Logging In to the Ranger Web UI
Enabling Ranger Authentication
Configuring Component Permission Policies
Viewing Ranger Audit Information
Configuring a Security Zone
Viewing Ranger Permission Information
Adding a Ranger Access Permission Policy for HDFS
Adding a Ranger Access Permission Policy for HBase
Adding a Ranger Access Permission Policy for Hive
Adding a Ranger Access Permission Policy for Yarn
Adding a Ranger Access Permission Policy for Spark2x
Adding a Ranger Access Permission Policy for Kafka
Adding a Ranger Access Permission Policy for Storm
Ranger Log Overview
Common Issues About Ranger
Why Ranger Startup Fails During the Cluster Installation?
How Do I Determine Whether the Ranger Authentication Is Used for a Service?
Why Cannot a New User Log In to Ranger After Changing the Password?
When an HBase Policy Is Added or Modified on Ranger, Wildcard Characters Cannot Be Used to Search for Existing HBase Tables
Using Spark
Precautions
Getting Started with Spark
Getting Started with Spark SQL
Using the Spark Client
Accessing the Spark Web UI
Interconnecting Spark with OpenTSDB
Creating a Table and Associating It with OpenTSDB
Inserting Data to the OpenTSDB Table
Querying an OpenTSDB Table
Modifying the Default Configuration Data
Using Spark2x
Precautions
Basic Operation
Getting Started
Configuring Parameters Rapidly
Common Parameters
Spark on HBase Overview and Basic Applications
Spark on HBase V2 Overview and Basic Applications
SparkSQL Permission Management(Security Mode)
Spark SQL Permissions
Creating a Spark SQL Role
Configuring Permissions for SparkSQL Tables, Columns, and Databases
Configuring Permissions for SparkSQL to Use Other Components
Configuring the Client and Server
Scenario-Specific Configuration
Configuring Multi-active Instance Mode
Configuring the Multi-tenant Mode
Configuring the Switchover Between the Multi-active Instance Mode and the Multi-tenant Mode
Configuring the Size of the Event Queue
Configuring Executor Off-Heap Memory
Enhancing Stability in a Limited Memory Condition
Viewing Aggregated Container Logs on the Web UI
Configuring Environment Variables in Yarn-Client and Yarn-Cluster Modes
Configuring the Default Number of Data Blocks Divided by SparkSQL
Configuring the Compression Format of a Parquet Table
Configuring the Number of Lost Executors Displayed in WebUI
Setting the Log Level Dynamically
Configuring Whether Spark Obtains HBase Tokens
Configuring LIFO for Kafka
Configuring Reliability for Connected Kafka
Configuring Streaming Reading of Driver Execution Results
Filtering Partitions without Paths in Partitioned Tables
Configuring Spark2x Web UI ACLs
Configuring Vector-based ORC Data Reading
Broaden Support for Hive Partition Pruning Predicate Pushdown
Hive Dynamic Partition Overwriting Syntax
Configuring the Column Statistics Histogram to Enhance the CBO Accuracy
Configuring Local Disk Cache for JobHistory
Configuring Spark SQL to Enable the Adaptive Execution Feature
Configuring Event Log Rollover
Adapting to the Third-party JDK When Ranger Is Used
Spark2x Logs
Obtaining Container Logs of a Running Spark Application
Small File Combination Tools
Using CarbonData for First Query
Spark2x Performance Tuning
Spark Core Tuning
Data Serialization
Optimizing Memory Configuration
Setting the DOP
Using Broadcast Variables
Using the external shuffle service to improve performance
Configuring Dynamic Resource Scheduling in Yarn Mode
Configuring Process Parameters
Designing the Direction Acyclic Graph (DAG)
Experience
Spark SQL and DataFrame Tuning
Optimizing the Spark SQL Join Operation
Improving Spark SQL Calculation Performance Under Data Skew
Optimizing Spark SQL Performance in the Small File Scenario
Optimizing the INSERT...SELECT Operation
Multiple JDBC Clients Concurrently Connecting to JDBCServer
Optimizing Memory when Data Is Inserted into Dynamic Partitioned Tables
Optimizing Small Files
Optimizing the Aggregate Algorithms
Optimizing Datasource Tables
Merging CBO
Optimizing SQL Query of Data of Multiple Sources
SQL Optimization for Multi-level Nesting and Hybrid Join
Spark Streaming Tuning
Common Issues About Spark2x
Spark Core
How Do I View Aggregated Spark Application Logs?
Why Is the Return Code of Driver Inconsistent with Application State Displayed on ResourceManager WebUI?
Why Cannot Exit the Driver Process?
Why Does FetchFailedException Occur When the Network Connection Is Timed out
How to Configure Event Queue Size If Event Queue Overflows?
What Can I Do If the getApplicationReport Exception Is Recorded in Logs During Spark Application Execution and the Application Does Not Exit for a Long Time?
What Can I Do If "Connection to ip:port has been quiet for xxx ms while there are outstanding requests" Is Reported When Spark Executes an Application and the Application Ends?
Why Do Executors Fail to be Removed After the NodeManeger Is Shut Down?
What Can I Do If the Message "Password cannot be null if SASL is enabled" Is Displayed?
What Should I Do If the Message "Failed to CREATE_FILE" Is Displayed in the Restarted Tasks When Data Is Inserted Into the Dynamic Partition Table?
Why Tasks Fail When Hash Shuffle Is Used?
What Can I Do If the Error Message "DNS query failed" Is Displayed When I Access the Aggregated Logs Page of Spark Applications?
What Can I Do If Shuffle Fetch Fails Due to the "Timeout Waiting for Task" Exception?
Why Does the Stage Retry due to the Crash of the Executor?
Why Do the Executors Fail to Register Shuffle Services During the Shuffle of a Large Amount of Data?
Why Does the Out of Memory Error Occur in NodeManager During the Execution of Spark Applications
Why Does the Realm Information Fail to Be Obtained When SparkBench is Run on HiBench for the Cluster in Security Mode?
Spark SQL and DataFrame
What Do I have to Note When Using Spark SQL ROLLUP and CUBE?
Why Spark SQL Is Displayed as a Temporary Table in Different Databases?
How to Assign a Parameter Value in a Spark Command?
What Directory Permissions Do I Need to Create a Table Using SparkSQL?
Why Do I Fail to Delete the UDF Using Another Service?
Why Cannot I Query Newly Inserted Data in a Parquet Hive Table Using SparkSQL?
How to Use Cache Table?
Why Are Some Partitions Empty During Repartition?
Why Does 16 Terabytes of Text Data Fails to Be Converted into 4 Terabytes of Parquet Data?
Why the Operation Fails When the Table Name Is TABLE?
Why Is a Task Suspended When the ANALYZE TABLE Statement Is Executed and Resources Are Insufficient?
If I Access a parquet Table on Which I Do not Have Permission, Why a Job Is Run Before "Missing Privileges" Is Displayed?
Why Do I Fail to Modify MetaData by Running the Hive Command?
Why Is "RejectedExecutionException" Displayed When I Exit Spark SQL?
How Do I Do If I Incidentally Kill the JDBCServer Process During Health Check?
Why No Result Is found When 2016-6-30 Is Set in the Date Field as the Filter Condition?
Why Does the --hivevar Option I Specified in the Command for Starting spark-beeline Fail to Take Effect?
Why Is the "Code of method ... grows beyond 64 KB" Error Message Displayed When I Run Complex SQL Statements?
Why Is Memory Insufficient if 10 Terabytes of TPCDS Test Suites Are Consecutively Run in Beeline/JDBCServer Mode?
Why Functions Cannot Be Used When Different JDBCServers Are Connected?
Why Does an Exception Occur When I Drop Functions Created Using the Add Jar Statement?
Why Does Spark2x Have No Access to DataSource Tables Created by Spark1.5?
Why Does Spark-beeline Fail to Run and Error Message "Failed to create ThriftService instance" Is Displayed?
Why Cannot I Query Newly Inserted Data in an ORC Hive Table Using Spark SQL?
Spark Streaming
Same DAG Log Is Recorded Twice for a Streaming Task
What Can I Do If Spark Streaming Tasks Are Blocked?
What Should I Pay Attention to When Optimizing Spark Streaming Task Parameters?
Why Does the Spark Streaming Application Fail to Be Submitted After the Token Validity Period Expires?
Why does Spark Streaming Application Fail to Restart from Checkpoint When It Creates an Input Stream Without Output Logic?
Why Is the Input Size Corresponding to Batch Time on the Web UI Set to 0 Records When Kafka Is Restarted During Spark Streaming Running?
Why the Job Information Obtained from the restful Interface of an Ended Spark Application Is Incorrect?
Why Cannot I Switch from the Yarn Web UI to the Spark Web UI?
What Can I Do If an Error Occurs when I Access the Application Page Because the Application Cached by HistoryServer Is Recycled?
Why Is not an Application Displayed When I Run the Application with the Empty Part File?
Why Does Spark2x Fail to Export a Table with the Same Field Name?
Why JRE fatal error after running Spark application multiple times?
Native Spark2x UI Fails to Be Accessed or Is Incorrectly Displayed when Internet Explorer Is Used for Access
How Does Spark2x Access External Cluster Components?
Why Does the Foreign Table Query Fail When Multiple Foreign Tables Are Created in the Same Directory?
Why Is the Native Page of an Application in Spark2x JobHistory Displayed Incorrectly?
Why Do I Fail to Create a Table in the Specified Location on OBS After Logging to spark-beeline?
Spark Shuffle Exception Handling
Using Sqoop
Using Sqoop from Scratch
Adapting Sqoop 1.4.7 to MRS 3.x Clusters
Common Sqoop Commands and Parameters
Common Issues About Sqoop
What Should I Do If Class QueryProvider Is Unavailable?
What Should I Do If Method getHiveClient Does Not Exist?
How Do I Do If PostgreSQL or GaussDB Fails to Connect?
What Should I Do If Data Failed to Be Synchronized to a Hive Table on the OBS Using hive-table?
What Should I Do If Data Failed to Be Synchronized to an ORC or Parquet Table Using hive-table?
What Should I Do If Data Failed to Be Synchronized Using hive-table?
What Should I Do If Data Failed to Be Synchronized to a Hive Parquet Table Using HCatalog?
What Should I Do If the Data Type of Fields timestamp and data Is Incorrect During Data Synchronization Between Hive and MySQL?
Using Storm
Using Storm from Scratch
Using the Storm Client
Submitting Storm Topologies on the Client
Accessing the Storm Web UI
Managing Storm Topologies
Querying Storm Topology Logs
Storm Common Parameters
Configuring a Storm Service User Password Policy
Migrating Storm Services to Flink
Overview
Completely Migrating Storm Services
Performing Embedded Service Migration
Migrating Services of External Security Components Interconnected with Storm
Storm Log Introduction
Performance Tuning
Storm Performance Tuning
Using Tez
Precautions
Common Tez Parameters
Accessing TezUI
Log Overview
Common Issues
TezUI Cannot Display Tez Task Execution Details
Error Occurs When a User Switches to the Tez Web UI
Yarn Logs Cannot Be Viewed on the TezUI Page
Table Data Is Empty on the TezUI HiveQueries Page
Using YARN
Common YARN Parameters
Creating Yarn Roles
Using the YARN Client
Configuring Resources for a NodeManager Role Instance
Changing NodeManager Storage Directories
Configuring Strict Permission Control for Yarn
Configuring Container Log Aggregation
Using CGroups with YARN
Configuring the Number of ApplicationMaster Retries
Configure the ApplicationMaster to Automatically Adjust the Allocated Memory
Configuring the Access Channel Protocol
Configuring Memory Usage Detection
Configuring the Additional Scheduler WebUI
Configuring Yarn Restart
Configuring ApplicationMaster Work Preserving
Configuring the Localized Log Levels
Configuring Users That Run Tasks
Yarn Log Overview
Yarn Performance Tuning
Preempting a Task
Setting the Task Priority
Optimizing Node Configuration
Common Issues About Yarn
Why Mounted Directory for Container is Not Cleared After the Completion of the Job While Using CGroups?
Why the Job Fails with HDFS_DELEGATION_TOKEN Expired Exception?
Why Are Local Logs Not Deleted After YARN Is Restarted?
Why the Task Does Not Fail Even Though AppAttempts Restarts for More Than Two Times?
Why Is an Application Moved Back to the Original Queue After ResourceManager Restarts?
Why Does Yarn Not Release the Blacklist Even All Nodes Are Added to the Blacklist?
Why Does the Switchover of ResourceManager Occur Continuously?
Why Does a New Application Fail If a NodeManager Has Been in Unhealthy Status for 10 Minutes?
Why Does an Error Occur When I Query the ApplicationID of a Completed or Non-existing Application Using the RESTful APIs?
Why May A Single NodeManager Fault Cause MapReduce Task Failures in the Superior Scheduling Mode?
Why Are Applications Suspended After They Are Moved From Lost_and_Found Queue to Another Queue?
How Do I Limit the Size of Application Diagnostic Messages Stored in the ZKstore?
Why Does a MapReduce Job Fail to Run When a Non-ViewFS File System Is Configured as ViewFS?
Why Do Reduce Tasks Fail to Run in Some OSs After the Native Task Feature is Enabled?
Using ZooKeeper
Using ZooKeeper from Scratch
Common ZooKeeper Parameters
Using a ZooKeeper Client
Configuring the ZooKeeper Permissions
ZooKeeper Log Overview
Common Issues About ZooKeeper
Why Do ZooKeeper Servers Fail to Start After Many znodes Are Created?
Why Does the ZooKeeper Server Display the java.io.IOException: Len Error Log?
Why Four Letter Commands Don't Work With Linux netcat Command When Secure Netty Configurations Are Enabled at Zookeeper Server?
How Do I Check Which ZooKeeper Instance Is a Leader?
Why Cannot the Client Connect to ZooKeeper using the IBM JDK?
What Should I Do When the ZooKeeper Client Fails to Refresh a TGT?
Why Is Message "Node does not exist" Displayed when A Large Number of Znodes Are Deleted Using the deleteallCommand
Appendix
Modifying Cluster Service Configuration Parameters
Accessing Manager
Accessing MRS Manager (Versions Earlier Than MRS 3.x)
Accessing FusionInsight Manager (MRS 3.x or Later)
Using an MRS Client
Installing a Client (MRS 3.x or Later)
Installing a Client (Versions Earlier Than 3.x)
Updating a Client (Version 3.x or Later)
Updating a Client (Versions Earlier Than 3.x)
Component Operation Guide (LTS)
Using CarbonData
Overview
CarbonData Overview
Main Specifications of CarbonData
Common CarbonData Parameters
CarbonData Operation Guide
CarbonData Quick Start
CarbonData Table Management
About CarbonData Table
Creating a CarbonData Table
Deleting a CarbonData Table
Modify the CarbonData Table
CarbonData Table Data Management
Loading Data
Deleting Segments
Combining Segments
CarbonData Data Migration
Migrating Data on CarbonData from Spark 1.5 to Spark2x
CarbonData Performance Tuning
Tuning Guidelines
Suggestions for Creating CarbonData Tables
Configurations for Performance Tuning
CarbonData Access Control
CarbonData Syntax Reference
DDL
CREATE TABLE
CREATE TABLE As SELECT
DROP TABLE
SHOW TABLES
ALTER TABLE COMPACTION
TABLE RENAME
ADD COLUMNS
DROP COLUMNS
CHANGE DATA TYPE
REFRESH TABLE
REGISTER INDEX TABLE
DML
LOAD DATA
UPDATE CARBON TABLE
DELETE RECORDS from CARBON TABLE
INSERT INTO CARBON TABLE
DELETE SEGMENT by ID
DELETE SEGMENT by DATE
SHOW SEGMENTS
CREATE SECONDARY INDEX
SHOW SECONDARY INDEXES
DROP SECONDARY INDEX
CLEAN FILES
SET/RESET
Operation Concurrent Execution
API
Spatial Indexes
CarbonData Troubleshooting
Filter Result Is not Consistent with Hive when a Big Double Type Value Is Used in Filter
Query Performance Deterioration
CarbonData FAQ
Why Is Incorrect Output Displayed When I Perform Query with Filter on Decimal Data Type Values?
How to Avoid Minor Compaction for Historical Data?
How to Change the Default Group Name for CarbonData Data Loading?
Why Does INSERT INTO CARBON TABLE Command Fail?
Why Is the Data Logged in Bad Records Different from the Original Input Data with Escape Characters?
Why Data Load Performance Decreases due to Bad Records?
Why INSERT INTO/LOAD DATA Task Distribution Is Incorrect and the Opened Tasks Are Less Than the Available Executors when the Number of Initial ExecutorsIs Zero?
Why Does CarbonData Require Additional Executors Even Though the Parallelism Is Greater Than the Number of Blocks to Be Processed?
Why Data loading Fails During off heap?
Why Do I Fail to Create a Hive Table?
How Do I Logically Split Data Across Different Namespaces?
Why Missing Privileges Exception is Reported When I Perform Drop Operation on Databases?
Why the UPDATE Command Cannot Be Executed in Spark Shell?
How Do I Configure Unsafe Memory in CarbonData?
Why Exception Occurs in CarbonData When Disk Space Quota is Set for Storage Directory in HDFS?
Why Does Data Query or Loading Fail and "org.apache.carbondata.core.memory.MemoryException: Not enough memory" Is Displayed?
Why Do Files of a Carbon Table Exist in the Recycle Bin Even If the drop table Command Is Not Executed When Mis-deletion Prevention Is Enabled?
Using CDL
CDL Usage Instructions
Using CDL from Scratch
Creating a CDL User
Preparing for Creating a CDL Job
Logging In to the CDLService WebUI
Uploading a Driver File
Creating a Database Link
Managing ENV
Configuring Heartbeat and Data Consistency Check for a Synchronization Task
Creating a CDL Job
Creating a CDL Data Synchronization Job
Creating a CDL Data Comparison Job
Common CDL Jobs
Synchronizing Data from PgSQL to Kafka
Synchronizing Data from MySQL to Hudi
Synchronizing Data from PgSQL to Hudi
Synchronizing Data from ThirdKafka to Hudi
Synchronizing Data from Hudi to DWS
Synchronizing Data from Hudi to ClickHouse
CDL Log Overview
CDL FAQs
Hudi Does Not Receive Data After a CDL Job Is Executed
Error 104 or 143 Is Reported After a CDL Job Runs for a Period of Time
Error Is Reported When the Job of Capturing Data From PgSQL to Hudi Is Started
Error 403 Is Reported When a CDL Job Is Stopped
Using ClickHouse
Using ClickHouse from Scratch
ClickHouse Table Engine Overview
Creating a ClickHouse Table
ClickHouse Data Type
Enabling the Read-Only Mode of the ClickHouse Table
Configuring Interconnection Between ClickHouse and OBS
Enabling the mysql_port Configuration for ClickHouse
Common ClickHouse SQL Syntax
CREATE DATABASE: Creating a Database
CREATE TABLE: Creating a Table
INSERT INTO: Inserting Data into a Table
SELECT: Querying Table Data
ALTER TABLE: Modifying a Table Structure
DESC: Querying a Table Structure
DROP: Deleting a Table
SHOW: Displaying Information About Databases and Tables
Migrating ClickHouse Data
Accessing RDS MySQL Using ClickHouse
Importing DWS Data to a ClickHouse Table
Using ClickHouse to Import and Export Data
Synchronizing Kafka Data to ClickHouse
Using the ClickHouse Data Migration Tool
Using the Migration Tool to Quickly Migrate ClickHouse Cluster Data
User Management and Authentication
ClickHouse User and Permission Management
Configuring the Password of the Default Account of a ClickHouse Cluster (MRS 3.1.2)
Interconnecting ClickHouse With OpenLDAP for Authentication
ClickHouse Multi-Tenancy
Overview
Enabling the CPU Priority Feature
Managing ClickHouse Tenants
Modifying the Memory Limit of ClickHouse on a ClickHouseServer Node
ClickHouse Cluster Management
Expanding the Data Disk Capacity of a ClickHouse Node
Accessing ClickHouse Through ELB
Monitoring of Slow ClickHouse Query Statements and Replication Table Data Synchronization
Slow Query Statement Monitoring
Replication Table Data Synchronization Monitoring
Interconnecting ClickHouse with HDFS
Backing Up and Restoring ClickHouse Data Using a Data File
ClickHouse Log Overview
ClickHouse Performance Tuning
Solution to the "Too many parts" Error in Data Tables
Accelerating Merge Operations
Accelerating TTL Operations
ClickHouse FAQ
How Do I Do If the Disk Status Displayed in the System.disks Table Is fault or abnormal?
How Do I Migrate Data from Hive/HDFS to ClickHouse?
How Do I Migrate Data from OBS/S3 to ClickHouse?
An Error Is Reported in Logs When the Auxiliary ZooKeeper or Replica Data Is Used to Synchronize Table Data
How Do I Grant the Select Permission at the Database Level to ClickHouse Users?
Using DBService
Configuring SSL for the HA Module
Restoring SSL for the HA Module
Configuring the Timeout Interval of DBService Backup Tasks
DBService Log Overview
Using Flink
Using Flink from Scratch
Viewing Flink Job Information
Configuring Flink Service Parameters
Configuring Flink Security Features
Security Features
Authentication and Encryption
Configuring Kafka
Configuring Pipeline
Configuring and Developing a Flink Visualization Job
Introduction to Flink Web UI
Flink Web UI Permission Management
Creating a FlinkServer Role
Accessing the Flink Web UI
Creating an Application
Creating a Cluster Connection
Creating a Data Connection
Creating a Stream Table
Creating a Job
Configuring and Managing UDFs
Importing and Exporting Jobs
Configuring Interconnection Between FlinkServer and Other Components
Interconnecting FlinkServer with ClickHouse
Interconnecting FlinkServer with GaussDB(DWS)
Interconnecting FlinkServer with HBase
Interconnecting FlinkServer with HDFS
Interconnecting FlinkServer with Hive
Interconnecting FlinkServer with Hudi
Interconnecting FlinkServer with Kafka
Configuring the Function of Deleting Residual Task Running Information
Flink Log Overview
Flink Performance Tuning
Memory Configuration Optimization
Configuring DOP
Configuring Process Parameters
Optimizing the Design of Partitioning Method
Configuring the Netty Network Communication
Experience Summary
Common Flink Shell Commands
Flink Restart Policy
Using Flume
Using Flume from Scratch
Overview
Installing the Flume Client
Installing the Flume Client
Viewing Flume Client Logs
Stopping or Uninstalling the Flume Client
Using the Encryption Tool of the Flume Client
Flume Service Configuration Guide
Flume Configuration Parameter Description
Using Environment Variables in the properties.properties File
Non-Encrypted Transmission
Configuring Non-encrypted Transmission
Typical Scenario: Collecting Local Static Logs and Uploading Them to Kafka
Typical Scenario: Collecting Local Static Logs and Uploading Them to HDFS
Typical Scenario: Collecting Local Dynamic Logs and Uploading Them to HDFS
Typical Scenario: Collecting Logs from Kafka and Uploading Them to HDFS
Typical Scenario: Collecting Logs from Kafka and Uploading Them to HDFS Through the Flume Client
Typical Scenario: Collecting Local Static Logs and Uploading Them to HBase
Encrypted Transmission
Configuring the Encrypted Transmission
Typical Scenario: Collecting Local Static Logs and Uploading Them to HDFS
Viewing Flume Client Monitoring Information
Connecting Flume to Kafka in Security Mode
Connecting Flume with Hive in Security Mode
Configuring the Flume Service Model
Overview
Service Model Configuration Guide
Introduction to Flume Logs
Flume Client Cgroup Usage Guide
Secondary Development Guide for Flume Third-Party Plug-ins
Configuring the Flume Customized Script
Common Issues About Flume
Using HBase
Using HBase from Scratch
Using an HBase Client
Creating HBase Roles
Configuring HBase Replication
Enabling Cross-Cluster Copy
Using the ReplicationSyncUp Tool
In-House Enhanced Phoenix
CsvBulkloadTool Supports Parsing User-Defined Delimiters in Data Files
Using HIndex
Introduction to HIndex
Loading Index Data in Batches
Using an Index Generation Tool
Configuring an RSGroup
Configuring HBase DR
Configuring HBase Data Compression and Encoding
Performing an HBase DR Service Switchover
Performing an HBase DR Active/Standby Cluster Switchover
Community BulkLoad Tool
In-House Enhanced BulkLoad Tool
Importing Data in a Customized Manner
Importing Data in Batches
Combining Rowkeys
Implementing Custom RowKeys
Combining Fields
Specifying Field Data Types
Defining Inapplicable Data Rows
Importing Data with Indexes in a Customized Manner
Creating a Secondary Index When Importing Data In Batches
Combining Rowkeys
Implementing Custom RowKeys
Combining Fields
Specifying Field Data Type
Defining Inapplicable Data Row
Updating Rows in Batches
Deleting Rows in Batches
Collecting Statistics on Rows
Configuring the MOB
Configuring Secure HBase Replication
Configuring Region In Transition Recovery Chore Service
Using a Secondary Index
HBase Log Overview
HBase Performance Tuning
Improving the BulkLoad Efficiency
Improving Put Performance
Optimizing Put and Scan Performance