Hive

创建hive catalog

通过连接Hive Metastore，或者兼容Hive Metastore的元数据服务，Doris可以自动获取Hive的库表信息，并进行数据查询。

除了Hive外，很多其他系统也会使用Hive Metastore存储元数据。所以通过Hive Catalog，我们不仅能访问Hive，也能访问使用Hive Metastore作为元数据存储的系统。

• 创建。
  • Hive On OBS

    CREATE CATALOG hive_catalog PROPERTIES (
        'type'='hms',
        'hive.metastore.uris' = 'thrift://127.x.x.x:port',
        'AWS_ACCESS_KEY' = 'ak',
        'AWS_SECRET_KEY' = 'sk',
        'AWS_ENDPOINT' = 'obs.cn-north-4.myhuaweicloud.com',
        'AWS_REGION' = 'cn-north-4',
        'yarn.resourcemanager.address' = '192.X.X.X:port',   
        'yarn.resourcemanager.principal' = 'mapred/hadoop.hadoop.com@HADOOP.COM'
    );

  • Hive On HDFS

    CREATE CATALOG hive_catalog PROPERTIES (
        'type'='hms',
        'hive.metastore.uris' = 'thrift://127.x.x.x:port',
        'dfs.nameservices'='hacluster',
        'dfs.ha.namenodes.hacluster'='3,4',
        'dfs.namenode.rpc-address.hacluster.3'='192.x.x.x:port',
        'dfs.namenode.rpc-address.hacluster.4'='192.x.x.x:port',
        'dfs.client.failover.proxy.provider.hacluster'='org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider',
        'yarn.resourcemanager.address' = '192.X.X.X:port',   
        'yarn.resourcemanager.principal' = 'mapred/hadoop.hadoop.com@HADOOP.COM'
    );

• 创建后查询：

  select * from hive_catalog.DB.test_table;

元数据缓存设置

创建Catalog时可以采用参数file.meta.cache.ttl-second来设置Hive分区文件缓存自动失效时间，也可以将该值设置为0来禁用分区文件缓存，时间单位为：秒。示例如下：

CREATE CATALOG hive_catalog PROPERTIES (
    'type'='hms',
    'hive.metastore.uris' = 'thrift://127.x.x.x:port',
    'AWS_ACCESS_KEY' = 'ak',
    'AWS_SECRET_KEY' = 'sk',
    'AWS_ENDPOINT' = 'obs.cn-north-4.myhuaweicloud.com',
    'AWS_REGION' = 'cn-north-4',
    'file.meta.cache.ttl-second' = '60',
    'yarn.resourcemanager.address' = '192.X.X.X:port',   
    'yarn.resourcemanager.principal' = 'mapred/hadoop.hadoop.com@HADOOP.COM'
);

Hive版本

Doris可以正确访问不同Hive版本中的Hive Metastore。在默认情况下，Doris会以Hive2.3版本的兼容接口访问Hive Metastore。你也可以在创建Catalog时指定hive的版本。如访问Hive1.1.0版本：

CREATE CATALOG hive_catalog PROPERTIES (
    'type'='hms',
    'hive.metastore.uris' = 'thrift://127.x.x.x:port',
    'AWS_ACCESS_KEY' = 'ak',
    'AWS_SECRET_KEY' = 'sk',
    'AWS_ENDPOINT' = 'obs.cn-north-4.myhuaweicloud.com',
    'AWS_REGION' = 'cn-north-4',
    'hive.version' = '1.1.0',
    'yarn.resourcemanager.address' = '192.X.X.X:port',   
    'yarn.resourcemanager.principal' = 'mapred/hadoop.hadoop.com@HADOOP.COM'
);

参数说明

列类型映射