All Documents
MapReduce ServiceMapReduce Service
- What's New
- Function Overview
-
Service Overview
- Infographics
- What Is MRS?
- Advantages of MRS Compared with Self-Built Hadoop
- Application Scenarios
- Components
- Functions
- Constraints
- Billing
- Permissions Management
- Related Services
- Quota Description
- Common Concepts
-
Getting Started
- Buying and Using an MRS Cluster
- Installing and Using the Cluster Client
- Using Clusters with Kerberos Authentication Enabled
- Using Hadoop from Scratch
- Using Kafka from Scratch
- Using HBase from Scratch
- Modifying MRS Configurations
- Configuring Auto Scaling for an MRS Cluster
- Configuring Hive with Storage and Compute Decoupled
- Submitting Spark Tasks to New Task Nodes
-
User Guide
- Preparing a User
-
Configuring a Cluster
- Methods of Purchasing MRS Clusters
- Quick Purchase of a Cluster
- Purchasing a Custom Cluster
- Purchasing a Custom Topology Cluster
- Adding a Tag to a Cluster
- Communication Security Authorization
- Configuring an Auto Scaling Rule
- Managing Data Connections
- Installing the Third-Party Software Using Bootstrap Actions
- Viewing Failed MRS Tasks
- Viewing Information of a Historical Cluster
-
Managing Clusters
- Logging In to a Cluster
- Cluster Overview
- Cluster O&M
- Managing Nodes
-
Job Management
- Introduction to MRS Jobs
- Running a MapReduce Job
- Running a SparkSubmit or Spark Job
- Running a HiveSQL Job
- Running a SparkSql Job
- Running a Flink Job
- Running a Kafka Job
- Viewing Job Configuration and Logs
- Stopping a Job
- Deleting a Job
- Using Encrypted OBS Data for Job Running
- Configuring Job Notification Rules
-
Component Management
- Object Management
- Viewing Configuration
- Managing Services
- Configuring Service Parameters
- Configuring Customized Service Parameters
- Synchronizing Service Configuration
- Managing Role Instances
- Configuring Role Instance Parameters
- Synchronizing Role Instance Configuration
- Decommissioning and Recommissioning a Role Instance
- Starting and Stopping a Cluster
- Synchronizing Cluster Configuration
- Exporting Cluster Configuration
- Performing Rolling Restart
- Alarm Management
-
Patch Management
- Patch Operation Guide for MRS 1.9.2
- Rolling Patches
- Restoring Patches for the Isolated Hosts
-
MRS Patch Description
- Fixed the Privilege Escalation Vulnerability of User omm
- MRS 1.8.10.1 Patch Description
- MRS 1.9.0.5 Patch Description
- MRS 1.9.0.6 Patch Description
- MRS 1.9.0.7 Patch Description
- MRS 1.9.0.8 Patch Description
- MRS 1.9.2.2 Patch Description
- MRS 1.9.3.1 Patch Description
- MRS 1.9.3.3 Patch Description
- MRS 2.0.1.1 Patch Description
- MRS 2.0.1.2 Patch Description
- MRS 2.0.1.3 Patch Description
- MRS 2.0.6.1 Patch Description
- MRS 2.1.0.1 Patch Description
- MRS 2.1.0.2 Patch Description
- MRS 2.1.0.3 Patch Description
- MRS 2.1.0.6 Patch Description
- MRS 2.1.0.7 Patch Description
- MRS 2.1.0.8 Patch Description
- MRS 2.1.0.9 Patch Description
- MRS 2.1.0.10 Patch Description
- MRS 2.1.0.11 Patch Description
- MRS 3.0.5.1 Patch Description
-
Tenant Management
- Before You Start
- Overview
- Creating a Tenant
- Creating a Sub-tenant
- Deleting a Tenant
- Managing a Tenant Directory
- Restoring Tenant Data
- Creating a Resource Pool
- Modifying a Resource Pool
- Deleting a Resource Pool
- Configuring a Queue
- Configuring the Queue Capacity Policy of a Resource Pool
- Clearing Configuration of a Queue
- Using an MRS Client
- Configuring a Cluster with Storage and Compute Decoupled
- Accessing Web Pages of Open Source Components Managed in MRS Clusters
- Interconnecting Jupyter Notebook with MRS Using Custom Python
- Accessing Manager
-
FusionInsight Manager Operation Guide (Applicable to 3.x)
- Getting Started
- Homepage
-
Cluster
- Cluster Management
- Managing a Service
- Instance Management
- Hosts
- O&M
- Audit
- Tenant Resources
- System Configuration
- Cluster Management
- Log Management
- Backup and Recovery Management
-
Security Management
- Security Overview
- Account Management
- Certificate Management
-
Security Hardening
- Hardening Policy
- Configuring a Trusted IP Address to Access LDAP
- HFile and WAL Encryption
- Security Configuration
- Configuring an IP Address Whitelist for Modifications Allowed by HBase
- Updating a Key for a Cluster
- Hardening the LDAP
- Configuring Kafka Data Encryption During Transmission
- Configuring HDFS Data Encryption During Transmission
- Encrypting the Communication Between Controller and Agent
- Updating SSH Keys for User omm
- Security Maintenance
- Security Statement
-
Alarm Reference (Applicable to MRS 3.x)
- ALM-12001 Audit Log Dumping Failure
- ALM-12004 OLdap Resource Abnormal
- ALM-12005 OKerberos Resource Abnormal
- ALM-12006 Node Fault
- ALM-12007 Process Fault
- ALM-12010 Manager Heartbeat Interruption Between the Active and Standby Nodes
- ALM-12011 Manager Data Synchronization Exception Between the Active and Standby Nodes
- ALM-12014 Partition Lost
- ALM-12015 Partition Filesystem Readonly
- ALM-12016 CPU Usage Exceeds the Threshold
- ALM-12017 Insufficient Disk Capacity
- ALM-12018 Memory Usage Exceeds the Threshold
- ALM-12027 Host PID Usage Exceeds the Threshold
- ALM-12028 The number of processes that are in the D state on the host exceeds the threshold
- ALM-12033 Slow Disk Fault
- ALM-12034 Periodical Backup Failure
- ALM-12035 Unknown Data Status After Recovery Task Failure
- ALM-12038 Monitoring Indicator Dumping Failure
- ALM-12039 Active/Standby OMS Databases Not Synchronized
- ALM-12040 Insufficient System Entropy
- ALM-12041 Incorrect Permission on Key Files
- ALM-12042 Incorrect Configuration of Key Files
- ALM-12045 Network Read Packet Dropped Rate Exceeds the Threshold
- ALM-12046 Network Write Packet Dropped Rate Exceeds the Threshold
- ALM-12047 Network Read Packet Error Rate Exceeds the Threshold
- ALM-12048 Network Write Packet Error Rate Exceeds the Threshold
- ALM-12049 Network Read Throughput Rate Exceeds the Threshold
- ALM-12050 Network Write Throughput Rate Exceeds the Threshold
- ALM-12051 Disk Inode Usage Exceeds the Threshold
- ALM-12052 TCP Temporary Port Usage Exceeds the Threshold
- ALM-12053 Host File Handle Usage Exceeds the Threshold
- ALM-12054 Invalid Certificate File
- ALM-12055 The Certificate File Is About to Expire
- ALM-12057 Metadata Not Configured with the Task to Periodically Back Up Data to a Third-Party Server
- ALM-12061 Process Usage Exceeds the Threshold
- ALM-12062 OMS Parameter Configurations Mismatch with the Cluster Scale
- ALM-12063 Unavailable Disk
- ALM-12064 Host Random Port Range Conflicts with Cluster Used Port
- ALM-12066 Trust Relationships Between Nodes Become Invalid
- ALM-12067 Tomcat Resource Is Abnormal
- ALM-12068 ACS Resource Is Abnormal
- ALM-12069 AOS Resource Is Abnormal
- ALM-12070 Controller Resource Is Abnormal
- ALM-12071 Httpd Resource Is Abnormal
- ALM-12072 FloatIP Resource Is Abnormal
- ALM-12073 CEP Resource Is Abnormal
- ALM-12074 FMS Resource Is Abnormal
- ALM-12075 PMS Resource Is Abnormal
- ALM-12076 GaussDB Resource Is Abnormal
- ALM-12077 User omm Expired
- ALM-12078 Password of User omm Expired
- ALM-12079 User omm Is About to Expire
- ALM-12080 Password of User omm Is About to Expire
- ALM-12081User ommdba Expired
- ALM-12082 User ommdba Is About to Expire
- ALM-12083 Password of User ommdba Is About to Expire
- ALM-12084 Password of User ommdba Expired
- ALM-12085 Service Audit Log Dump Failure
- ALM-12087 System Is in the Upgrade Observation Period
- ALM-12089 Inter-Node Network Is Abnormal
- ALM-12101 AZ Unhealthy
- ALM-12102 AZ HA Component Is Not Deployed Based on DR Requirements
- ALM-12110 Failed to get ECS temporary ak/sk
- ALM-13000 ZooKeeper Service Unavailable
- ALM-13001 Available ZooKeeper Connections Are Insufficient
- ALM-13002 ZooKeeper Direct Memory Usage Exceeds the Threshold
- ALM-13003 GC Duration of the ZooKeeper Process Exceeds the Threshold
- ALM-13004 ZooKeeper Heap Memory Usage Exceeds the Threshold
- ALM-13005 Failed to Set the Quota of Top Directories of ZooKeeper Components
- ALM-13006 Znode Number or Capacity Exceeds the Threshold
- ALM-13007 Available ZooKeeper Client Connections Are Insufficient
- ALM-13008 ZooKeeper Znode Usage Exceeds the Threshold
- ALM-13009 ZooKeeper Znode Capacity Usage Exceeds the Threshold
- ALM-13010 Znode Usage of a Directory with Quota Configured Exceeds the Threshold
- ALM-14000 HDFS Service Unavailable
- ALM-14001 HDFS Disk Usage Exceeds the Threshold
- ALM-14002 DataNode Disk Usage Exceeds the Threshold
- ALM-14003 Number of Lost HDFS Blocks Exceeds the Threshold
- ALM-14006 Number of HDFS Files Exceeds the Threshold
- ALM-14007 NameNode Heap Memory Usage Exceeds the Threshold
- ALM-14008 DataNode Heap Memory Usage Exceeds the Threshold
- ALM-14009 Number of Dead DataNodes Exceeds the Threshold
- ALM-14010 NameService Service Is Abnormal
- ALM-14011 DataNode Data Directory Is Not Configured Properly
- ALM-14012 JournalNode Is Out of Synchronization
- ALM-14013 Failed to Update the NameNode FsImage File
- ALM-14014 NameNode GC Time Exceeds the Threshold
- ALM-14015 DataNode GC Time Exceeds the Threshold
- ALM-14016 DataNode Direct Memory Usage Exceeds the Threshold
- ALM-14017 NameNode Direct Memory Usage Exceeds the Threshold
- ALM-14018 NameNode Non-heap Memory Usage Exceeds the Threshold
- ALM-14019 DataNode Non-heap Memory Usage Exceeds the Threshold
- ALM-14020 Number of Entries in the HDFS Directory Exceeds the Threshold
- ALM-14021 NameNode Average RPC Processing Time Exceeds the Threshold
- ALM-14022 NameNode Average RPC Queuing Time Exceeds the Threshold
- ALM-14023 Percentage of Total Reserved Disk Space for Replicas Exceeds the Threshold
- ALM-14024 Tenant Space Usage Exceeds the Threshold
- ALM-14025 Tenant File Object Usage Exceeds the Threshold
- ALM-14026 Blocks on DataNode Exceed the Threshold
- ALM-14027 DataNode Disk Fault
- ALM-14028 Number of Blocks to Be Supplemented Exceeds the Threshold
- ALM-14029 Number of Blocks in a Replica Exceeds the Threshold
- ALM-16000 Percentage of Sessions Connected to the HiveServer to Maximum Number Allowed Exceeds the Threshold
- ALM-16001 Hive Warehouse Space Usage Exceeds the Threshold
- ALM-16002 Hive SQL Execution Success Rate Is Lower Than the Threshold
- ALM-16003 Background Thread Usage Exceeds the Threshold
- ALM-16004 Hive Service Unavailable
- ALM-16005 The Heap Memory Usage of the Hive Process Exceeds the Threshold
- ALM-16006 The Direct Memory Usage of the Hive Process Exceeds the Threshold
- ALM-16007 Hive GC Time Exceeds the Threshold
- ALM-16008 Non-Heap Memory Usage of the Hive Process Exceeds the Threshold
- ALM-16009 Map Number Exceeds the Threshold
- ALM-16045 Hive Data Warehouse Is Deleted
- ALM-16046 Hive Data Warehouse Permission Is Modified
- ALM-16047 HiveServer Has Been Deregistered from ZooKeeper
- ALM-16048 Tez or Spark Library Path Does Not Exist
- ALM-17003 Oozie Service Unavailable
- ALM-17004 Oozie Heap Memory Usage Exceeds the Threshold
- ALM-17005 Oozie Non Heap Memory Usage Exceeds the Threshold
- ALM-17006 Oozie Direct Memory Usage Exceeds the Threshold
- ALM-17007 Garbage Collection (GC) Time of the Oozie Process Exceeds the Threshold
- ALM-18000 Yarn Service Unavailable
- ALM-18002 NodeManager Heartbeat Lost
- ALM-18003 NodeManager Unhealthy
- ALM-18008 Heap Memory Usage of ResourceManager Exceeds the Threshold
- ALM-18009 Heap Memory Usage of JobHistoryServer Exceeds the Threshold
- ALM-18010 ResourceManager GC Time Exceeds the Threshold
- ALM-18011 NodeManager GC Time Exceeds the Threshold
- ALM-18012 JobHistoryServer GC Time Exceeds the Threshold
- ALM-18013 ResourceManager Direct Memory Usage Exceeds the Threshold
- ALM-18014 NodeManager Direct Memory Usage Exceeds the Threshold
- ALM-18015 JobHistoryServer Direct Memory Usage Exceeds the Threshold
- ALM-18016 Non Heap Memory Usage of ResourceManager Exceeds the Threshold
- ALM-18017 Non Heap Memory Usage of NodeManager Exceeds the Threshold
- ALM-18018 NodeManager Heap Memory Usage Exceeds the Threshold
- ALM-18019 Non Heap Memory Usage of JobHistoryServer Exceeds the Threshold
- ALM-18020 Yarn Task Execution Timeout
- ALM-18021 Mapreduce Service Unavailable
- ALM-18022 Insufficient Yarn Queue Resources
- ALM-18023 Number of Pending Yarn Tasks Exceeds the Threshold
- ALM-18024 Pending Yarn Memory Usage Exceeds the Threshold
- ALM-18025 Number of Terminated Yarn Tasks Exceeds the Threshold
- ALM-18026 Number of Failed Yarn Tasks Exceeds the Threshold
- ALM-19000 HBase Service Unavailable
- ALM-19006 HBase Replication Sync Failed
- ALM-19007 HBase GC Time Exceeds the Threshold
- ALM-19008 Heap Memory Usage of the HBase Process Exceeds the Threshold
- ALM-19009 Direct Memory Usage of the HBase Process Exceeds the Threshold
- ALM-19011 RegionServer Region Number Exceeds the Threshold
- ALM-19012 HBase System Table Directory or File Lost
- ALM-19013 Duration of Regions in transaction State Exceeds the Threshold
- ALM-19014 Capacity Quota Usage on ZooKeeper Exceeds the Threshold Severely
- ALM-19015 Quantity Quota Usage on ZooKeeper Exceeds the Threshold
- ALM-19016 Quantity Quota Usage on ZooKeeper Exceeds the Threshold Severely
- ALM-19017 Capacity Quota Usage on ZooKeeper Exceeds the Threshold
- ALM-19018 HBase Compaction Queue Exceeds the Threshold
- ALM-19019 Number of HBase HFiles to Be Synchronized Exceeds the Threshold
- ALM-19020 Number of HBase WAL Files to Be Synchronized Exceeds the Threshold
- ALM-20002 Hue Service Unavailable
- ALM-24000 Flume Service Unavailable
- ALM-24001 Flume Agent Exception
- ALM-24003 Flume Client Connection Interrupted
- ALM-24004 Exception Occurs When Flume Reads Data
- ALM-24005 Exception Occurs When Flume Transmits Data
- ALM-24006 Heap Memory Usage of Flume Server Exceeds the Threshold
- ALM-24007 Flume Server Direct Memory Usage Exceeds the Threshold
- ALM-24008 Flume Server Non-Heap Memory Usage Exceeds the Threshold
- ALM-24009 Flume Server Garbage Collection (GC) Time Exceeds the Threshold
- ALM-24010 Flume Certificate File Is Invalid or Damaged
- ALM-24011 Flume Certificate File Is About to Expire
- ALM-24012 Flume Certificate File Has Expired
- ALM-24013 Flume MonitorServer Certificate File Is Invalid or Damaged
- ALM-24014 Flume MonitorServer Certificate Is About to Expire
- ALM-24015 Flume MonitorServer Certificate File Has Expired
- ALM-25000 LdapServer Service Unavailable
- ALM-25004 Abnormal LdapServer Data Synchronization
- ALM-25005 nscd Service Exception
- ALM-25006 Sssd Service Exception
- ALM-25500 KrbServer Service Unavailable
- ALM-26051 Storm Service Unavailable
- ALM-26052 Number of Available Supervisors of the Storm Service Is Less Than the Threshold
- ALM-26053 Storm Slot Usage Exceeds the Threshold
- ALM-26054 Nimbus Heap Memory Usage Exceeds the Threshold
- ALM-27001 DBService Service Unavailable
- ALM-27003 DBService Heartbeat Interruption Between the Active and Standby Nodes
- ALM-27004 Data Inconsistency Between Active and Standby DBServices
- ALM-27005 Database Connections Usage Exceeds the Threshold
- ALM-27006 Disk Space Usage of the Data Directory Exceeds the Threshold
- ALM-27007 Database Enters the Read-Only Mode
- ALM-29000 Impala Service Unavailable
- ALM-29004 Impalad Process Memory Usage Exceeds the Threshold
- ALM-29005 Number of JDBC Connections to Impalad Exceeds the Threshold
- ALM-29006 Number of ODBC Connections to Impalad Exceeds the Threshold
- ALM-29100 Kudu Service Unavailable
- ALM-29104 Tserver Process Memory Usage Exceeds the Threshold
- ALM-29106 Tserver Process CPU Usage Exceeds the Threshold
- ALM-29107 Tserver Process Memory Usage Exceeds the Threshold
- ALM-38000 Kafka Service Unavailable
- ALM-38001 Insufficient Kafka Disk Capacity
- ALM-38002 Kafka Heap Memory Usage Exceeds the Threshold
- ALM-38004 Kafka Direct Memory Usage Exceeds the Threshold
- ALM-38005 GC Duration of the Broker Process Exceeds the Threshold
- ALM-38006 Percentage of Kafka Partitions That Are Not Completely Synchronized Exceeds the Threshold
- ALM-38007 Status of Kafka Default User Is Abnormal
- ALM-38008 Abnormal Kafka Data Directory Status
- ALM-38009 Busy Broker Disk I/Os
- ALM-38010 Topics with Single Replica
- ALM-43001 Spark2x Service Unavailable
- ALM-43006 Heap Memory Usage of the JobHistory2x Process Exceeds the Threshold
- ALM-43007 Non-Heap Memory Usage of the JobHistory2x Process Exceeds the Threshold
- ALM-43008 The Direct Memory Usage of the JobHistory2x Process Exceeds the Threshold
- ALM-43009 JobHistory2x Process GC Time Exceeds the Threshold
- ALM-43010 Heap Memory Usage of the JDBCServer2x Process Exceeds the Threshold
- ALM-43011 Non-Heap Memory Usage of the JDBCServer2x Process Exceeds the Threshold
- ALM-43012 Direct Heap Memory Usage of the JDBCServer2x Process Exceeds the Threshold
- ALM-43013 JDBCServer2x Process GC Time Exceeds the Threshold
- ALM-43017 JDBCServer2x Process Full GC Number Exceeds the Threshold
- ALM-43018 JobHistory2x Process Full GC Number Exceeds the Threshold
- ALM-43019 Heap Memory Usage of the IndexServer2x Process Exceeds the Threshold
- ALM-43020 Non-Heap Memory Usage of the IndexServer2x Process Exceeds the Threshold
- ALM-43021 Direct Memory Usage of the IndexServer2x Process Exceeds the Threshold
- ALM-43022 IndexServer2x Process GC Time Exceeds the Threshold
- ALM-43023 IndexServer2x Process Full GC Number Exceeds the Threshold
- ALM-44004 Presto Coordinator Resource Group Queuing Tasks Exceed the Threshold
- ALM-44005 Presto Coordinator Process GC Time Exceeds the Threshold
- ALM-44006 Presto Worker Process GC Time Exceeds the Threshold
- ALM-45175 Average Time for Calling OBS Metadata APIs Is Greater than the Threshold
- ALM-45176 Success Rate of Calling OBS Metadata APIs Is Lower than the Threshold
- ALM-45177 Success Rate of Calling OBS Data Read APIs Is Lower than the Threshold
- ALM-45178 Success Rate of Calling OBS Data Write APIs Is Lower Than the Threshold
- ALM-45275 Ranger Service Unavailable
- ALM-45276 Abnormal RangerAdmin status
- ALM-45277 RangerAdmin Heap Memory Usage Exceeds the Threshold
- ALM-45278 RangerAdmin Direct Memory Usage Exceeds the Threshold
- ALM-45279 RangerAdmin Non Heap Memory Usage Exceeds the Threshold
- ALM-45280 RangerAdmin GC Duration Exceeds the Threshold
- ALM-45281 UserSync Heap Memory Usage Exceeds the Threshold
- ALM-45282 UserSync Direct Memory Usage Exceeds the Threshold
- ALM-45283 UserSync Non Heap Memory Usage Exceeds the Threshold
- ALM-45284 UserSync Garbage Collection (GC) Time Exceeds the Threshold
- ALM-45285 TagSync Heap Memory Usage Exceeds the Threshold
- ALM-45286 TagSync Direct Memory Usage Exceeds the Threshold
- ALM-45287 TagSync Non Heap Memory Usage Exceeds the Threshold
- ALM-45288 TagSync Garbage Collection (GC) Time Exceeds the Threshold
- ALM-45425 ClickHouse Service Unavailable
- ALM-45426 ClickHouse Service Quantity Quota Usage in ZooKeeper Exceeds the Threshold
- ALM-45427 ClickHouse Service Capacity Quota Usage in ZooKeeper Exceeds the Threshold
- ALM-45736 Guardian Service Unavailable
-
MRS Manager Operation Guide (Applicable to 2.x and Earlier Versions)
- Introduction to MRS Manager
- Checking Running Tasks
- Monitoring Management
- Alarm Management
-
Alarm Reference (Applicable to Versions Earlier Than MRS 3.x)
- ALM-12001 Audit Log Dump Failure
- ALM-12002 HA Resource Is Abnormal
- ALM-12004 OLdap Resource Is Abnormal
- ALM-12005 OKerberos Resource Is Abnormal
- ALM-12006 Node Fault
- ALM-12007 Process Fault
- ALM-12010 Manager Heartbeat Interruption Between the Active and Standby Nodes
- ALM-12011 Data Synchronization Exception Between the Active and Standby Manager Nodes
- ALM-12012 NTP Service Is Abnormal
- ALM-12016 CPU Usage Exceeds the Threshold
- ALM-12017 Insufficient Disk Capacity
- ALM-12018 Memory Usage Exceeds the Threshold
- ALM-12027 Host PID Usage Exceeds the Threshold
- ALM-12028 Number of Processes in the D State on the Host Exceeds the Threshold
- ALM-12031 User omm or Password Is About to Expire
- ALM-12032 User ommdba or Password Is About to Expire
- ALM-12033 Slow Disk Fault
- ALM-12034 Periodic Backup Failure
- ALM-12035 Unknown Data Status After Recovery Task Failure
- ALM-12037 NTP Server Is Abnormal
- ALM-12038 Monitoring Indicator Dump Failure
- ALM-12039 GaussDB Data Is Not Synchronized
- ALM-12040 Insufficient System Entropy
- ALM-13000 ZooKeeper Service Unavailable
- ALM-13001 Available ZooKeeper Connections Are Insufficient
- ALM-13002 ZooKeeper Memory Usage Exceeds the Threshold
- ALM-14000 HDFS Service Unavailable
- ALM-14001 HDFS Disk Usage Exceeds the Threshold
- ALM-14002 DataNode Disk Usage Exceeds the Threshold
- ALM-14003 Number of Lost HDFS Blocks Exceeds the Threshold
- ALM-14004 Number of Damaged HDFS Blocks Exceeds the Threshold
- ALM-14006 Number of HDFS Files Exceeds the Threshold
- ALM-14007 HDFS NameNode Memory Usage Exceeds the Threshold
- ALM-14008 HDFS DataNode Memory Usage Exceeds the Threshold
- ALM-14009 Number of Faulty DataNodes Exceeds the Threshold
- ALM-14010 NameService Service Is Abnormal
- ALM-14011 HDFS DataNode Data Directory Is Not Configured Properly
- ALM-14012 HDFS JournalNode Data Is Not Synchronized
- ALM-16000 Percentage of Sessions Connected to the HiveServer to the Maximum Number Allowed Exceeds the Threshold
- ALM-16001 Hive Warehouse Space Usage Exceeds the Threshold
- ALM-16002 Hive SQL Execution Success Rate Is Lower Than the Threshold
- ALM-16004 Hive Service Unavailable
- ALM-18000 Yarn Service Unavailable
- ALM-18002 NodeManager Heartbeat Lost
- ALM-18003 NodeManager Unhealthy
- ALM-18004 NodeManager Disk Usability Ratio Is Lower Than the Threshold
- ALM-18006 MapReduce Job Execution Timeout
- ALM-19000 HBase Service Unavailable
- ALM-19006 HBase Replication Sync Failed
- ALM-25000 LdapServer Service Unavailable
- ALM-25004 Abnormal LdapServer Data Synchronization
- ALM-25500 KrbServer Service Unavailable
- ALM-27001 DBService Is Unavailable
- ALM-27003 DBService Heartbeat Interruption Between the Active and Standby Nodes
- ALM-27004 Data Inconsistency Between Active and Standby DBServices
- ALM-28001 Spark Service Unavailable
- ALM-26051 Storm Service Unavailable
- ALM-26052 Number of Available Supervisors in Storm Is Lower Than the Threshold
- ALM-26053 Slot Usage of Storm Exceeds the Threshold
- ALM-26054 Heap Memory Usage of Storm Nimbus Exceeds the Threshold
- ALM-38000 Kafka Service Unavailable
- ALM-38001 Insufficient Kafka Disk Space
- ALM-38002 Heap Memory Usage of Kafka Exceeds the Threshold
- ALM-24000 Flume Service Unavailable
- ALM-24001 Flume Agent Is Abnormal
- ALM-24003 Flume Client Connection Failure
- ALM-24004 Flume Fails to Read Data
- ALM-24005 Data Transmission by Flume Is Abnormal
- ALM-12041 Permission of Key Files Is Abnormal
- ALM-12042 Key File Configurations Are Abnormal
- ALM-23001 Loader Service Unavailable
- ALM-12357 Failed to Export Audit Logs to OBS
- ALM-12014 Device Partition Lost
- ALM-12015 Device Partition File System Read-Only
- ALM-12043 DNS Parsing Duration Exceeds the Threshold
- ALM-12045 Read Packet Dropped Rate Exceeds the Threshold
- ALM-12046 Write Packet Dropped Rate Exceeds the Threshold
- ALM-12047 Read Packet Error Rate Exceeds the Threshold
- ALM-12048 Write Packet Error Rate Exceeds the Threshold
- ALM-12049 Read Throughput Rate Exceeds the Threshold
- ALM-12050 Write Throughput Rate Exceeds the Threshold
- ALM-12051 Disk Inode Usage Exceeds the Threshold
- ALM-12052 Usage of Temporary TCP Ports Exceeds the Threshold
- ALM-12053 File Handle Usage Exceeds the Threshold
- ALM-12054 The Certificate File Is Invalid
- ALM-12055 The Certificate File Is About to Expire
- ALM-18008 Heap Memory Usage of Yarn ResourceManager Exceeds the Threshold
- ALM-18009 Heap Memory Usage of MapReduce JobHistoryServer Exceeds the Threshold
- ALM-20002 Hue Service Unavailable
- ALM-43001 Spark Service Unavailable
- ALM-43006 Heap Memory Usage of the JobHistory Process Exceeds the Threshold
- ALM-43007 Non-Heap Memory Usage of the JobHistory Process Exceeds the Threshold
- ALM-43008 Direct Memory Usage of the JobHistory Process Exceeds the Threshold
- ALM-43009 JobHistory GC Time Exceeds the Threshold
- ALM-43010 Heap Memory Usage of the JDBCServer Process Exceeds the Threshold
- ALM-43011 Non-Heap Memory Usage of the JDBCServer Process Exceeds the Threshold
- ALM-43012 Direct Memory Usage of the JDBCServer Process Exceeds the Threshold
- ALM-43013 JDBCServer GC Time Exceeds the Threshold
- ALM-44004 Presto Coordinator Resource Group Queuing Tasks Exceed the Threshold
- ALM-44005 Presto Coordinator Process GC Time Exceeds the Threshold
- ALM-44006 Presto Worker Process GC Time Exceeds the Threshold
- ALM-18010 Number of Pending Yarn Tasks Exceeds the Threshold
- ALM-18011 Memory of Pending Yarn Tasks Exceeds the Threshold
- ALM-18012 Number of Terminated Yarn Tasks in the Last Period Exceeds the Threshold
- ALM-18013 Number of Failed Yarn Tasks in the Last Period Exceeds the Threshold
- ALM-16005 Number of Failed Hive SQL Executions in the Last Period Exceeds the Threshold
-
Object Management
- Managing Objects
- Viewing Configurations
- Managing Services
- Configuring Service Parameters
- Configuring Customized Service Parameters
- Synchronizing Service Configurations
- Managing Role Instances
- Configuring Role Instance Parameters
- Synchronizing Role Instance Configuration
- Decommissioning and Recommissioning a Role Instance
- Managing a Host
- Isolating a Host
- Canceling Host Isolation
- Starting or Stopping a Cluster
- Synchronizing Cluster Configurations
- Exporting Configuration Data of a Cluster
- Log Management
-
Health Check Management
- Performing a Health Check
- Viewing and Exporting a Health Check Report
- Configuring the Number of Health Check Reports to Be Reserved
- Managing Health Check Reports
- DBService Health Check Indicators
- Flume Health Check Indicators
- HBase Health Check Indicators
- Host Health Check Indicators
- HDFS Health Check Indicators
- Hive Health Check Indicators
- Kafka Health Check Indicators
- KrbServer Health Check Indicators
- LdapServer Health Check Indicators
- Loader Health Check Indicators
- MapReduce Health Check Indicators
- OMS Health Check Indicators
- Spark Health Check Indicators
- Storm Health Check Indicators
- Yarn Health Check Indicators
- ZooKeeper Health Check Indicators
- Static Service Pool Management
-
Tenant Management
- Overview
- Creating a Tenant
- Creating a Sub-tenant
- Deleting a tenant
- Managing a Tenant Directory
- Restoring Tenant Data
- Creating a Resource Pool
- Modifying a Resource Pool
- Deleting a Resource Pool
- Configuring a Queue
- Configuring the Queue Capacity Policy of a Resource Pool
- Clearing Configuration of a Queue
- Backup and Restoration
-
Security Management
- Default Users of Clusters with Kerberos Authentication Disabled
- Default Users of Clusters with Kerberos Authentication Enabled
- Changing the Password of an OS User
- Changing the password of user admin
- Changing the Password of the Kerberos Administrator
- Changing the Passwords of the LDAP Administrator and the LDAP User
- Changing the Password of a Component Running User
- Changing the Password of the OMS Database Administrator
- Changing the Password of the Data Access User of the OMS Database
- Changing the Password of a Component Database User
- Replacing the HA Certificate
- Updating Cluster Keys
- Permissions Management
-
MRS Multi-User Permission Management
- Users and Permissions of MRS Clusters
- Default Users of Clusters with Kerberos Authentication Enabled
- Creating a Role
- Creating a User Group
- Creating a User
- Modifying User Information
- Locking a User
- Unlocking a User
- Deleting a User
- Changing the Password of an Operation User
- Initializing the Password of a System User
- Downloading a User Authentication File
- Modifying a Password Policy
- Configuring Cross-Cluster Mutual Trust Relationships
- Configuring Users to Access Resources of a Trusted Cluster
- Configuring Fine-Grained Permissions for MRS Multi-User Access to OBS
- Patch Operation Guide
- Restoring Patches for the Isolated Hosts
- Rolling Restart
- Security Description
- High-Risk Operations
- Appendix
-
Component Operation Guide
- Using Alluxio
- Using CarbonData (for Versions Earlier Than MRS 3.x)
-
Using CarbonData (for MRS 3.x or Later)
- Overview
- Configuration Reference
- CarbonData Operation Guide
- CarbonData Performance Tuning
- CarbonData Access Control
- CarbonData Syntax Reference
- CarbonData Troubleshooting
-
CarbonData FAQ
- Why Is Incorrect Output Displayed When I Perform Query with Filter on Decimal Data Type Values?
- How to Avoid Minor Compaction for Historical Data?
- How to Change the Default Group Name for CarbonData Data Loading?
- Why Does INSERT INTO CARBON TABLE Command Fail?
- Why Is the Data Logged in Bad Records Different from the Original Input Data with Escape Characters?
- Why Data Load Performance Decreases due to Bad Records?
- Why INSERT INTO/LOAD DATA Task Distribution Is Incorrect and the Opened Tasks Are Less Than the Available Executors when the Number of Initial ExecutorsIs Zero?
- Why Does CarbonData Require Additional Executors Even Though the Parallelism Is Greater Than the Number of Blocks to Be Processed?
- Why Data loading Fails During off heap?
- Why Do I Fail to Create a Hive Table?
- Why CarbonData tables created in V100R002C50RC1 not reflecting the privileges provided in Hive Privileges for non-owner?
- How Do I Logically Split Data Across Different Namespaces?
- Why Missing Privileges Exception is Reported When I Perform Drop Operation on Databases?
- Why the UPDATE Command Cannot Be Executed in Spark Shell?
- How Do I Configure Unsafe Memory in CarbonData?
- Why Exception Occurs in CarbonData When Disk Space Quota is Set for Storage Directory in HDFS?
- Why Does Data Query or Loading Fail and "org.apache.carbondata.core.memory.MemoryException: Not enough memory" Is Displayed?
- Why Do Files of a Carbon Table Exist in the Recycle Bin Even If the drop table Command Is Not Executed When Mis-deletion Prevention Is Enabled?
- Using ClickHouse
- Using DBService
-
Using Flink
- Using Flink from Scratch
- Viewing Flink Job Information
- Flink Configuration Management
- Security Configuration
- Security Hardening
- Security Statement
- Using the Flink Web UI
- Flink Log Overview
- Flink Performance Tuning
- Common Flink Shell Commands
- Reference
-
Using Flume
- Using Flume from Scratch
- Overview
- Installing the Flume Client
- Viewing Flume Client Logs
- Stopping or Uninstalling the Flume Client
- Using the Encryption Tool of the Flume Client
- Flume Service Configuration Guide
- Flume Configuration Parameter Description
- Using Environment Variables in the properties.properties File
-
Non-Encrypted Transmission
- Configuring Non-encrypted Transmission
- Typical Scenario: Collecting Local Static Logs and Uploading Them to Kafka
- Typical Scenario: Collecting Local Static Logs and Uploading Them to HDFS
- Typical Scenario: Collecting Local Dynamic Logs and Uploading Them to HDFS
- Typical Scenario: Collecting Logs from Kafka and Uploading Them to HDFS
- Typical Scenario: Collecting Logs from Kafka and Uploading Them to HDFS Through the Flume Client
- Typical Scenario: Collecting Local Static Logs and Uploading Them to HBase
- Encrypted Transmission
- Viewing Flume Client Monitoring Information
- Connecting Flume to Kafka in Security Mode
- Connecting Flume with Hive in Security Mode
- Configuring the Flume Service Model
- Introduction to Flume Logs
- Flume Client Cgroup Usage Guide
- Secondary Development Guide for Flume Third-Party Plug-ins
- Configuring the Flume Customized Script
- Common Issues About Flume
-
Using HBase
- Using HBase from Scratch
- Using an HBase Client
- Creating HBase Roles
- Configuring HBase Replication
- Configuring HBase Parameters
- Enabling Cross-Cluster Copy
- Using the ReplicationSyncUp Tool
- GeoMesa Command Line
- Using HIndex
- Configuring HBase DR
- Configuring HBase Data Compression and Encoding
- Performing an HBase DR Service Switchover
- Performing an HBase DR Active/Standby Cluster Switchover
- Community BulkLoad Tool
- In-House Enhanced BulkLoad Tool
- Configuring the MOB
- Configuring Secure HBase Replication
- Configuring Region In Transition Recovery Chore Service
- Using a Secondary Index
- HBase Log Overview
- HBase Performance Tuning
-
Common Issues About HBase
- Why Does a Client Keep Failing to Connect to a Server for a Long Time?
- Operation Failures Occur in Stopping BulkLoad On the Client
- Why May a Table Creation Exception Occur When HBase Deletes or Creates the Same Table Consecutively?
- Why Other Services Become Unstable If HBase Sets up A Large Number of Connections over the Network Port?
- Why Does the HBase BulkLoad Task (One Table Has 26 TB Data) Consisting of 210,000 Map Tasks and 10,000 Reduce Tasks Fail?
- How Do I Restore a Region in the RIT State for a Long Time?
- Why Does HMaster Exits Due to Timeout When Waiting for the Namespace Table to Go Online?
- Why Does SocketTimeoutException Occur When a Client Queries HBase?
- Why Modified and Deleted Data Can Still Be Queried by Using the Scan Command?
- Why "java.lang.UnsatisfiedLinkError: Permission denied" exception thrown while starting HBase shell?
- When does the RegionServers listed under "Dead Region Servers" on HMaster WebUI gets cleared?
- Why Are Different Query Results Returned After I Use Same Query Criteria to Query Data Successfully Imported by HBase bulkload?
- What Should I Do If I Fail to Create Tables Due to the FAILED_OPEN State of Regions?
- How Do I Delete Residual Table Names in the /hbase/table-lock Directory of ZooKeeper?
- Why Does HBase Become Faulty When I Set a Quota for the Directory Used by HBase in HDFS?
- Why HMaster Times Out While Waiting for Namespace Table to be Assigned After Rebuilding Meta Using OfflineMetaRepair Tool and Startups Failed
- Why Messages Containing FileNotFoundException and no lease Are Frequently Displayed in the HMaster Logs During the WAL Splitting Process?
- Why Does the ImportTsv Tool Display "Permission denied" When the Same Linux User as and a Different Kerberos User from the Region Server Are Used?
- Insufficient Rights When a Tenant Accesses Phoenix
- Insufficient Rights When a Tenant Uses the HBase Bulkload Function
- What Can I Do When HBase Fails to Recover a Task and a Message Is Displayed Stating "Rollback recovery failed"?
- How Do I Fix Region Overlapping?
- Why Does RegionServer Fail to Be Started When GC Parameters Xms and Xmx of HBase RegionServer Are Set to 31 GB?
- Why Does the LoadIncrementalHFiles Tool Fail to Be Executed and "Permission denied" Is Displayed When Nodes in a Cluster Are Used to Import Data in Batches?
- Why Is the Error Message "import argparse" Displayed When the Phoenix sqlline Script Is Used?
- How Do I Deal with the Restrictions of the Phoenix BulkLoad Tool?
- Why a Message Is Displayed Indicating that the Permission is Insufficient When CTBase Connects to the Ranger Plug-ins?
-
Using HDFS
- Using Hadoop from Scratch
- Configuring Memory Management
- Creating an HDFS Role
- Using the HDFS Client
- Running the DistCp Command
- Overview of HDFS File System Directories
- Changing the DataNode Storage Directory
- Configuring HDFS Directory Permission
- Configuring NFS
- Planning HDFS Capacity
- Configuring ulimit for HBase and HDFS
- Balancing DataNode Capacity
- Configuring Replica Replacement Policy for Heterogeneous Capacity Among DataNodes
- Configuring the Number of Files in a Single HDFS Directory
- Configuring the Recycle Bin Mechanism
- Setting Permissions on Files and Directories
- Setting the Maximum Lifetime and Renewal Interval of a Token
- Configuring the Damaged Disk Volume
- Configuring Encrypted Channels
- Reducing the Probability of Abnormal Client Application Operation When the Network Is Not Stable
- Configuring the NameNode Blacklist
- Optimizing HDFS NameNode RPC QoS
- Optimizing HDFS DataNode RPC QoS
- Configuring LZC Compression
- Configuring Reserved Percentage of Disk Usage on DataNodes
- Configuring HDFS NodeLabel
- Configuring HDFS Mover
- Using HDFS AZ Mover
- Configuring HDFS DiskBalancer
- Configuring the Observer NameNode to Process Read Requests
- Performing Concurrent Operations on HDFS Files
- Introduction to HDFS Logs
- HDFS Performance Tuning
-
FAQ
- NameNode Startup Is Slow
- DataNode Is Normal but Cannot Report Data Blocks
- HDFS WebUI Cannot Properly Update Information About Damaged Data
- Why Does the Distcp Command Fail in the Secure Cluster, Causing an Exception?
- Why Does DataNode Fail to Start When the Number of Disks Specified by dfs.datanode.data.dir Equals dfs.datanode.failed.volumes.tolerated?
- Failed to Calculate the Capacity of a DataNode when Multiple data.dir Directories Are Configured in a Disk Partition
- Standby NameNode Fails to Be Restarted When the System Is Powered off During Metadata (Namespace) Storage
- Why Data in the Buffer Is Lost If a Power Outage Occurs During Storage of Small Files
- Why Does Array Border-crossing Occur During FileInputFormat Split?
- Why Is the Storage Type of File Copies DISK When the Tiered Storage Policy Is LAZY_PERSIST?
- The HDFS Client Is Unresponsive When the NameNode Is Overloaded for a Long Time
- Can I Delete or Modify the Data Storage Directory in DataNode?
- Blocks Miss on the NameNode UI After the Successful Rollback
- Why Is "java.net.SocketException: No buffer space available" Reported When Data Is Written to HDFS
- Why are There Two Standby NameNodes After the active NameNode Is Restarted?
- When Does a Balance Process in HDFS, Shut Down and Fail to be Executed Again?
- "This page can't be displayed" Is Displayed When Internet Explorer Fails to Access the Native HDFS UI
- NameNode Fails to Be Restarted Due to EditLog Discontinuity
-
Using Hive
- Using Hive from Scratch
- Configuring Hive Parameters
- Hive SQL
- Permission Management
- Using a Hive Client
- Using HDFS Colocation to Store Hive Tables
- Using the Hive Column Encryption Function
- Customizing Row Separators
- Configuring Hive on HBase in Across Clusters with Mutual Trust Enabled
- Deleting Single-Row Records from Hive on HBase
- Configuring HTTPS/HTTP-based REST APIs
- Enabling or Disabling the Transform Function
- Access Control of a Dynamic Table View on Hive
- Specifying Whether the ADMIN Permissions Is Required for Creating Temporary Functions
- Using Hive to Read Data in a Relational Database
- Supporting Traditional Relational Database Syntax in Hive
- Creating User-Defined Hive Functions
- Enhancing beeline Reliability
- Viewing Table Structures Using the show create Statement as Users with the select Permission
- Writing a Directory into Hive with the Old Data Removed to the Recycle Bin
- Inserting Data to a Directory That Does Not Exist
- Creating Databases and Creating Tables in the Default Database Only as the Hive Administrator
- Disabling of Specifying the location Keyword When Creating an Internal Hive Table
- Enabling the Function of Creating a Foreign Table in a Directory That Can Only Be Read
- Authorizing Over 32 Roles in Hive
- Restricting the Maximum Number of Maps for Hive Tasks
- HiveServer Lease Isolation
- Hive Supporting Transactions
- Switching the Hive Execution Engine to Tez
- Interconnecting Hive with External Self-Built Relational Databases
- Hive Materialized View
- Interconnecting Hive with External LDAP
- Hive Log Overview
- Hive Performance Tuning
-
Common Issues About Hive
- How Do I Delete UDFs on Multiple HiveServers at the Same Time?
- Why Cannot the DROP operation Be Performed on a Backed-up Hive Table?
- How to Perform Operations on Local Files with Hive User-Defined Functions
- How Do I Forcibly Stop MapReduce Jobs Executed by Hive?
- Table Creation Fails Because Hive Complex Fields' Names Contain Special Characters
- How Do I Monitor the Hive Table Size?
- How Do I Prevent Key Directories from Data Loss Caused by Misoperations of the insert overwrite Statement?
- Why Is Hive on Spark Task Freezing When HBase Is Not Installed?
- Error Reported When the WHERE Condition Is Used to Query Tables with Excessive Partitions in FusionInsight Hive
- Why Cannot I Connect to HiveServer When I Use IBM JDK to Access the Beeline Client?
- Description of Hive Table Location (Either Be an OBS or HDFS Path)
- Why Cannot Data Be Queried After the MapReduce Engine Is Switched After the Tez Engine Is Used to Execute Union-related Statements?
- Why Does Hive Not Support Concurrent Data Writing to the Same Table or Partition?
- Why Does Hive Not Support Vectorized Query?
- Why Does Metadata Still Exist When the HDFS Data Directory of the Hive Table Is Deleted by Mistake?
- How Do I Disable the Logging Function of Hive?
- Why Hive Tables in the OBS Directory Fail to Be Deleted?
- Hive Configuration Problems
-
Using Hudi
- Getting Started
- Basic Operations
- Hudi Performance Tuning
-
Common Issues About Hudi
-
Data Write
- Parquet/Avro schema Is Reported When Updated Data Is Written
- UnsupportedOperationException Is Reported When Updated Data Is Written
- SchemaCompatabilityException Is Reported When Updated Data Is Written
- What Should I Do If Hudi Consumes Much Space in a Temporary Folder During Upsert?
- Hudi Fails to Write Decimal Data with Lower Precision
- Data Collection
- Hive Synchronization
-
Data Write
- Using Hue (Versions Earlier Than MRS 3.x)
-
Using Hue (MRS 3.x or Later)
- Using Hue from Scratch
- Accessing the Hue Web UI
- Hue Common Parameters
- Using HiveQL Editor on the Hue Web UI
- Using the SparkSql Editor on the Hue Web UI
- Using the Metadata Browser on the Hue Web UI
- Using File Browser on the Hue Web UI
- Using Job Browser on the Hue Web UI
- Using HBase on the Hue Web UI
- Typical Scenarios
- Hue Log Overview
-
Common Issues About Hue
- How Do I Solve the Problem that HQL Fails to Be Executed in Hue Using Internet Explorer?
- Why Does the use database Statement Become Invalid When Hive Is Used?
- What Can I Do If HDFS Files Fail to Be Accessed Using Hue WebUI?
- What Can I Do If a Large File Fails to Be Uploaded on the Hue Page?
- Why Is the Hue Native Page Cannot Be Properly Displayed If the Hive Service Is Not Installed in a Cluster?
- How Do I Solve the Problem of Setting the Time Zone of the Oozie Editor on the Hue Web UI?
- Using Impala
-
Using Kafka
- Using Kafka from Scratch
- Managing Kafka Topics
- Querying Kafka Topics
- Managing Kafka User Permissions
- Managing Messages in Kafka Topics
- Synchronizing Binlog-based MySQL Data to the MRS Cluster
- Creating a Kafka Role
- Kafka Common Parameters
- Safety Instructions on Using Kafka
- Kafka Specifications
- Using the Kafka Client
- Configuring Kafka HA and High Reliability Parameters
- Changing the Broker Storage Directory
- Checking the Consumption Status of Consumer Group
- Kafka Balancing Tool Instructions
- Balancing Data After Kafka Node Scale-Out
- Kafka Token Authentication Mechanism Tool Usage
- Introduction to Kafka Logs
- Performance Tuning
- Kafka Feature Description
- Migrating Data Between Kafka Nodes
- Common Issues About Kafka
- Using KafkaManager
- Using Kudu
-
Using Loader
- Using Loader from Scratch
- How to Use Loader
- Loader Link Configuration
- Managing Loader Links (Versions Earlier Than MRS 3.x)
- Source Link Configurations of Loader Jobs
- Destination Link Configurations of Loader Jobs
- Managing Loader Jobs
- Preparing a Driver for MySQL Database Link
- Loader Log Overview
- Example: Using Loader to Import Data from OBS to HDFS
- Common Issues About Loader
-
Using MapReduce
- Configuring the Log Archiving and Clearing Mechanism
- Reducing Client Application Failure Rate
- Transmitting MapReduce Tasks from Windows to Linux
- Configuring the Distributed Cache
- Configuring the MapReduce Shuffle Address
- Configuring the Cluster Administrator List
- Introduction to MapReduce Logs
- MapReduce Performance Tuning
-
Common Issues About MapReduce
- Why Does It Take a Long Time to Run a Task Upon ResourceManager Active/Standby Switchover?
- Why Does a MapReduce Task Stay Unchanged for a Long Time?
- Why the Client Hangs During Job Running?
- Why Cannot HDFS_DELEGATION_TOKEN Be Found in the Cache?
- How Do I Set the Task Priority When Submitting a MapReduce Task?
- Why Physical Memory Overflow Occurs If a MapReduce Task Fails?
- After the Address of MapReduce JobHistoryServer Is Changed, Why the Wrong Page is Displayed When I Click the Tracking URL on the ResourceManager WebUI?
- MapReduce Job Failed in Multiple NameService Environment
- Why a Fault MapReduce Node Is Not Blacklisted?
-
Using Oozie
- Using Oozie from Scratch
- Using the Oozie Client
- Using Oozie Client to Submit an Oozie Job
-
Using Hue to Submit an Oozie Job
- Creating a Workflow
-
Submitting a Workflow Job
- Submitting a Hive2 Job
- Submitting a Spark2x Job
- Submitting a Java Job
- Submitting a Loader Job
- Submitting a MapReduce Job
- Submitting a Sub-workflow Job
- Submitting a Shell Job
- Submitting an HDFS Job
- Submitting a Streaming Job
- Submitting a DistCp Job
- Example of Mutual Trust Operations
- Submitting an SSH Job
- Submitting a Hive Script
- Submitting a Coordinator Periodic Scheduling Job
- Submitting a Bundle Batch Processing Job
- Querying the Operation Results
- Oozie Log Overview
- Common Issues About Oozie
- Using OpenTSDB
- Using Presto
- Using Ranger (MRS 1.9.2)
-
Using Ranger (MRS 3.x)
- Logging In to the Ranger Web UI
- Enabling Ranger Authentication
- Configuring Component Permission Policies
- Viewing Ranger Audit Information
- Configuring a Security Zone
- Changing the Ranger Data Source to LDAP for a Normal Cluster
- Viewing Ranger Permission Information
- Adding a Ranger Access Permission Policy for HDFS
- Adding a Ranger Access Permission Policy for HBase
- Adding a Ranger Access Permission Policy for Hive
- Adding a Ranger Access Permission Policy for Yarn
- Adding a Ranger Access Permission Policy for Spark2x
- Adding a Ranger Access Permission Policy for Kafka
- Adding a Ranger Access Permission Policy for Storm
- Ranger Log Overview
-
Common Issues About Ranger
- Why Ranger Startup Fails During the Cluster Installation?
- How Do I Determine Whether the Ranger Authentication Is Used for a Service?
- Why Cannot a New User Log In to Ranger After Changing the Password?
- When an HBase Policy Is Added or Modified on Ranger, Wildcard Characters Cannot Be Used to Search for Existing HBase Tables
- Using Spark
-
Using Spark2x
- Precautions
-
Basic Operation
- Getting Started
- Configuring Parameters Rapidly
- Common Parameters
- Spark on HBase Overview and Basic Applications
- Spark on HBase V2 Overview and Basic Applications
- SparkSQL Permission Management(Security Mode)
-
Scenario-Specific Configuration
- Configuring Multi-active Instance Mode
- Configuring the Multi-tenant Mode
- Configuring the Switchover Between the Multi-active Instance Mode and the Multi-tenant Mode
- Configuring the Size of the Event Queue
- Configuring Executor Off-Heap Memory
- Enhancing Stability in a Limited Memory Condition
- Viewing Aggregated Container Logs on the Web UI
- Configuring Environment Variables in Yarn-Client and Yarn-Cluster Modes
- Configuring the Default Number of Data Blocks Divided by SparkSQL
- Configuring the Compression Format of a Parquet Table
- Configuring the Number of Lost Executors Displayed in WebUI
- Setting the Log Level Dynamically
- Configuring Whether Spark Obtains HBase Tokens
- Configuring LIFO for Kafka
- Configuring Reliability for Connected Kafka
- Configuring Streaming Reading of Driver Execution Results
- Filtering Partitions without Paths in Partitioned Tables
- Configuring Spark2x Web UI ACLs
- Configuring Vector-based ORC Data Reading
- Broaden Support for Hive Partition Pruning Predicate Pushdown
- Hive Dynamic Partition Overwriting Syntax
- Configuring the Column Statistics Histogram to Enhance the CBO Accuracy
- Configuring Local Disk Cache for JobHistory
- Configuring Spark SQL to Enable the Adaptive Execution Feature
- Configuring Event Log Rollover
- Adapting to the Third-party JDK When Ranger Is Used
- Spark2x Logs
- Obtaining Container Logs of a Running Spark Application
- Small File Combination Tools
- Using CarbonData for First Query
-
Spark2x Performance Tuning
- Spark Core Tuning
-
Spark SQL and DataFrame Tuning
- Optimizing the Spark SQL Join Operation
- Improving Spark SQL Calculation Performance Under Data Skew
- Optimizing Spark SQL Performance in the Small File Scenario
- Optimizing the INSERT...SELECT Operation
- Multiple JDBC Clients Concurrently Connecting to JDBCServer
- Optimizing Memory when Data Is Inserted into Dynamic Partitioned Tables
- Optimizing Small Files
- Optimizing the Aggregate Algorithms
- Optimizing Datasource Tables
- Merging CBO
- Optimizing SQL Query of Data of Multiple Sources
- SQL Optimization for Multi-level Nesting and Hybrid Join
- Spark Streaming Tuning
-
Common Issues About Spark2x
-
Spark Core
- How Do I View Aggregated Spark Application Logs?
- Why Is the Return Code of Driver Inconsistent with Application State Displayed on ResourceManager WebUI?
- Why Cannot Exit the Driver Process?
- Why Does FetchFailedException Occur When the Network Connection Is Timed out
- How to Configure Event Queue Size If Event Queue Overflows?
- What Can I Do If the getApplicationReport Exception Is Recorded in Logs During Spark Application Execution and the Application Does Not Exit for a Long Time?
- What Can I Do If "Connection to ip:port has been quiet for xxx ms while there are outstanding requests" Is Reported When Spark Executes an Application and the Application Ends?
- Why Do Executors Fail to be Removed After the NodeManeger Is Shut Down?
- What Can I Do If the Message "Password cannot be null if SASL is enabled" Is Displayed?
- What Should I Do If the Message "Failed to CREATE_FILE" Is Displayed in the Restarted Tasks When Data Is Inserted Into the Dynamic Partition Table?
- Why Tasks Fail When Hash Shuffle Is Used?
- What Can I Do If the Error Message "DNS query failed" Is Displayed When I Access the Aggregated Logs Page of Spark Applications?
- What Can I Do If Shuffle Fetch Fails Due to the "Timeout Waiting for Task" Exception?
- Why Does the Stage Retry due to the Crash of the Executor?
- Why Do the Executors Fail to Register Shuffle Services During the Shuffle of a Large Amount of Data?
- Why Does the Out of Memory Error Occur in NodeManager During the Execution of Spark Applications
- Why Does the Realm Information Fail to Be Obtained When SparkBench is Run on HiBench for the Cluster in Security Mode?
-
Spark SQL and DataFrame
- What Do I have to Note When Using Spark SQL ROLLUP and CUBE?
- Why Spark SQL Is Displayed as a Temporary Table in Different Databases?
- How to Assign a Parameter Value in a Spark Command?
- What Directory Permissions Do I Need to Create a Table Using SparkSQL?
- Why Do I Fail to Delete the UDF Using Another Service?
- Why Cannot I Query Newly Inserted Data in a Parquet Hive Table Using SparkSQL?
- How to Use Cache Table?
- Why Are Some Partitions Empty During Repartition?
- Why Does 16 Terabytes of Text Data Fails to Be Converted into 4 Terabytes of Parquet Data?
- Why the Operation Fails When the Table Name Is TABLE?
- Why Is a Task Suspended When the ANALYZE TABLE Statement Is Executed and Resources Are Insufficient?
- If I Access a parquet Table on Which I Do not Have Permission, Why a Job Is Run Before "Missing Privileges" Is Displayed?
- Why Do I Fail to Modify MetaData by Running the Hive Command?
- Why Is "RejectedExecutionException" Displayed When I Exit Spark SQL?
- What Should I Do If the JDBCServer Process is Mistakenly Killed During a Health Check?
- Why No Result Is found When 2016-6-30 Is Set in the Date Field as the Filter Condition?
- Why Does the "--hivevar" Option I Specified in the Command for Starting spark-beeline Fail to Take Effect?
- Why Does the "Permission denied" Exception Occur When I Create a Temporary Table or View in Spark-beeline?
- Why Is the "Code of method ... grows beyond 64 KB" Error Message Displayed When I Run Complex SQL Statements?
- Why Is Memory Insufficient if 10 Terabytes of TPCDS Test Suites Are Consecutively Run in Beeline/JDBCServer Mode?
- Why Are Some Functions Not Available when Another JDBCServer Is Connected?
- Why Does an Exception Occur When I Drop Functions Created Using the Add Jar Statement?
- Why Does Spark2x Have No Access to DataSource Tables Created by Spark1.5?
- Why Does Spark-beeline Fail to Run and Error Message "Failed to create ThriftService instance" Is Displayed?
- Why Cannot I Query Newly Inserted Data in an ORC Hive Table Using Spark SQL?
-
Spark Streaming
- Streaming Task Prints the Same DAG Log Twice
- What Can I Do If Spark Streaming Tasks Are Blocked?
- What Should I Pay Attention to When Optimizing Spark Streaming Task Parameters?
- Why Does the Spark Streaming Application Fail to Be Submitted After the Token Validity Period Expires?
- Why does Spark Streaming Application Fail to Restart from Checkpoint When It Creates an Input Stream Without Output Logic?
- Why Is the Input Size Corresponding to Batch Time on the Web UI Set to 0 Records When Kafka Is Restarted During Spark Streaming Running?
- Why the Job Information Obtained from the restful Interface of an Ended Spark Application Is Incorrect?
- Why Cannot I Switch from the Yarn Web UI to the Spark Web UI?
- What Can I Do If an Error Occurs when I Access the Application Page Because the Application Cached by HistoryServer Is Recycled?
- Why Is not an Application Displayed When I Run the Application with the Empty Part File?
- Why Does Spark2x Fail to Export a Table with the Same Field Name?
- Why JRE fatal error after running Spark application multiple times?
- "This page can't be displayed" Is Displayed When Internet Explorer Fails to Access the Native Spark2x UI
- How Does Spark2x Access External Cluster Components?
- Why Does the Foreign Table Query Fail When Multiple Foreign Tables Are Created in the Same Directory?
- What Should I Do If the Native Page of an Application of Spark2x JobHistory Fails to Display During Access to the Page
- Why Do I Fail to Create a Table in the Specified Location on OBS After Logging to spark-beeline?
- Spark Shuffle Exception Handling
-
Spark Core
-
Using Sqoop
- Using Sqoop from Scratch
- Adapting Sqoop 1.4.7 to MRS 3.x Clusters
- Common Sqoop Commands and Parameters
-
Common Issues About Sqoop
- What Should I Do If Class QueryProvider Is Unavailable?
- What Should I Do If Method getHiveClient Does Not Exist?
- What Should I Do If PostgreSQL or GaussDB Failed to Be Connected?
- What Should I Do If Data Failed to Be Synchronized to a Hive Table on the OBS Using hive-table?
- What Should I Do If Data Failed to Be Synchronized to an ORC or Parquet Table Using hive-table?
- What Should I Do If Data Failed to Be Synchronized Using hive-table?
- What Should I Do If Data Failed to Be Synchronized to a Hive Parquet Table Using HCatalog?
- What Should I Do If the Data Type of Fields timestamp and data Is Incorrect During Data Synchronization Between Hive and MySQL?
- What Should I do If HBaseAdmin.<init> Cannot Be Found When Data Is Read from the MySQL Database to HBase
-
Using Storm
- Using Storm from Scratch
- Using the Storm Client
- Submitting Storm Topologies on the Client
- Accessing the Storm Web UI
- Managing Storm Topologies
- Querying Storm Topology Logs
- Storm Common Parameters
- Configuring a Storm Service User Password Policy
- Migrating Storm Services to Flink
- Storm Log Introduction
- Performance Tuning
- Using Tez
-
Using Yarn
- Common YARN Parameters
- Creating Yarn Roles
- Using the YARN Client
- Configuring Resources for a NodeManager Role Instance
- Changing NodeManager Storage Directories
- Configuring Strict Permission Control for Yarn
- Configuring Container Log Aggregation
- Using CGroups with YARN
- Configuring the Number of ApplicationMaster Retries
- Configure the ApplicationMaster to Automatically Adjust the Allocated Memory
- Configuring the Access Channel Protocol
- Configuring Memory Usage Detection
- Configuring the Additional Scheduler WebUI
- Configuring Yarn Restart
- Configuring ApplicationMaster Work Preserving
- Configuring the Localized Log Levels
- Configuring Users That Run Tasks
- Yarn Log Overview
- Yarn Performance Tuning
-
Common Issues About Yarn
- Why Mounted Directory for Container is Not Cleared After the Completion of the Job While Using CGroups?
- Why the Job Fails with HDFS_DELEGATION_TOKEN Expired Exception?
- Why Are Local Logs Not Deleted After YARN Is Restarted?
- Why the Task Does Not Fail Even Though AppAttempts Restarts for More Than Two Times?
- Why Is an Application Moved Back to the Original Queue After ResourceManager Restarts?
- Why Does Yarn Not Release the Blacklist Even All Nodes Are Added to the Blacklist?
- Why Does the Switchover of ResourceManager Occur Continuously?
- Why Does a New Application Fail If a NodeManager Has Been in Unhealthy Status for 10 Minutes?
- Why Does an Error Occur When I Query the ApplicationID of a Completed or Non-existing Application Using the RESTful APIs?
- Why May A Single NodeManager Fault Cause MapReduce Task Failures in the Superior Scheduling Mode?
- Why Are Applications Suspended After They Are Moved From Lost_and_Found Queue to Another Queue?
- How Do I Limit the Size of Application Diagnostic Messages Stored in the ZKstore?
- Why Does a MapReduce Job Fail to Run When a Non-ViewFS File System Is Configured as ViewFS?
- Why Do Reduce Tasks Fail to Run in Some OSs After the Native Task Feature is Enabled?
-
Using ZooKeeper
- Using ZooKeeper from Scratch
- Common ZooKeeper Parameters
- Using a ZooKeeper Client
- Configuring the ZooKeeper Permissions
- ZooKeeper Log Overview
-
Common Issues About ZooKeeper
- Why Do ZooKeeper Servers Fail to Start After Many znodes Are Created?
- Why Does the ZooKeeper Server Display the java.io.IOException: Len Error Log?
- Why Four Letter Commands Don't Work With Linux netcat Command When Secure Netty Configurations Are Enabled at Zookeeper Server?
- How Do I Check Which ZooKeeper Instance Is a Leader?
- Why Cannot the Client Connect to ZooKeeper using the IBM JDK?
- What Should I Do When the ZooKeeper Client Fails to Refresh a TGT?
- Why Is Message "Node does not exist" Displayed when A Large Number of Znodes Are Deleted Using the deleteallCommand
- Appendix
-
Best Practices
- Data Analytics
-
Data Migration
- Data Migration Solution
- Information Collection Before Data Migrated to MRS
- Preparing the Network Before Data Migration to MRS
- Migrating Data from Hadoop to MRS
- Migrating Data from HBase to MRS
- Migrating Data from Hive to MRS
- Network Port Requirements for Data Migration
- Offline Data Migration - Using BulkLoad to Batch Import Data to HBase
- Migrating Data from MySQL to an MRS Hive Partition Table
- Migrating Data from MRS HDFS to OBS
- Data Backup and Restoration
-
Developer Guide (Earlier than 3.x)
- Before You Start
- Method of Building an MRS Sample Project
-
HBase Application Development
- Overview
- Environment Preparation
-
Application Development
- Development Guidelines in Typical Scenarios
- Creating the Configuration Object
- Creating the Connection Object
- Creating a Table
- Deleting a Table
- Modifying a Table
- Inserting Data
- Deleting Data
- Reading Data Using Get
- Reading Data Using Scan
- Using a Filter
- Adding a Secondary Index
- Enabling/Disabling a Secondary Index
- Querying a List of Secondary Indexes
- Using a Secondary Index to Read Data
- Deleting a Secondary Index
- Writing Data into a MOB Table
- Reading MOB Data
- Multi-Point Region Splitting
- ACL Security Configuration
- Application Commissioning
- More Information
- HBase APIs
- FAQs
- Development Specifications
- Hive Application Development
- MapReduce Application Development
- HDFS Application Development
-
Spark Application Development
- Overview
-
Environment Preparation
- Environment Overview
- Preparing a Development User
- Preparing a Java Development Environment
- Preparing a Scala Development Environment
- Preparing a Python Development Environment
- Preparing an Operating Environment
- Downloading and Importing a Sample Project
- (Optional) Creating a Project
- Preparing the Authentication Mechanism Code
-
Application Development
- Spark Core Application
- Spark SQL Application
- Spark Streaming Application
- Application for Accessing Spark SQL Through JDBC
- Spark on HBase Application
- Reading Data from HBase and Writing Data Back to HBase
- Reading Data from Hive and Write Data to HBase
- Using Streaming to Read Data from Kafka and Write Data to HBase
- Application for Connecting Spark Streaming to Kafka0-10
- Structured Streaming Application
- Application Commissioning
-
Application Tuning
-
Spark Core Tuning
- Data Serialization
- Memory Configuration Optimization
- Setting a Degree of Parallelism
- Using Broadcast Variables
- Using the External Shuffle Service to Improve Performance
- Configuring Dynamic Resource Scheduling in Yarn Mode
- Configuring Process Parameters
- Designing a Direction Acyclic Graph (DAG)
- Experience Summary
- SQL and DataFrame Tuning
- Spark Streaming Tuning
- Spark CBO Tuning
-
Spark Core Tuning
- Spark APIs
-
FAQs
- How Do I Add a Dependency Package with Customized Codes?
- How Do I Handle the Dependency Package That Is Automatically Loaded?
- Why the "Class Does not Exist" Error Is Reported While the SparkStreamingKafka Project Is Running?
- Why a Spark Core Application Is Suspended Instead of Being Exited When Driver Memory Is Insufficient to Store Collected Intensive Data?
- Why the Name of the Spark Application Submitted in Yarn-Cluster Mode Does not Take Effect?
- How Do I Submit the Spark Application Using Java Commands?
- How Does the Permission Control Mechanism Work for the UDF Function in SparkSQL?
- Why Does Kafka Fail to Receive the Data Written Back by Spark Streaming?
- How Do I Perform Remote Debugging Using IDEA?
- A Message Stating "Problem performing GSS wrap" Is Displayed When IBM JDK Is Used
- Why Does the ApplicationManager Fail to Be Terminated When Data Is Being Processed in the Structured Streaming Cluster Mode?
- What Should I Do If FileNotFoundException Occurs When spark-submit Is Used to Submit a Job in Spark on Yarn Client Mode?
- What Should I Do If the "had a not serializable result" Error Is Reported When a Spark Task Reads HBase Data?
- Development Specifications
- Storm Application Development
-
Kafka Application Development
- Overview
- Environment Preparation
-
Application Development
- Typical Application Scenario
- Old Producer API Usage Sample
- Old Consumer API Usage Sample
- Producer API Usage Sample
- Consumer API Usage Sample
- Multi-Thread Producer API Usage Sample
- Multi-Thread Consumer API Usage Sample
- SimpleConsumer API Usage Sample
- Description of the Sample Project Configuration File
- Application Commissioning
- Kafka APIs
- FAQs
- Development Specifications
- Presto Application Development