Livy部署

前置条件说明

1. JDK先配置好(1.8)
2. 安装大数据组件客户端(HDFS\YARN\HIVE\SPARK)

安装机器

前提

1. CDH中需要有两个Hadoop集群，每个集群下部署一个livy (CDH机器允许部署的情况)
2. CDH机器不允许部署的情况，参考https://deepexi.yuque.com/docs/share/cbb7e60f-7188-41ef-8a72-219f269c07aa?#，将部署机器加入CDH集群管理即可

安装包解压

将安装包解压到opt目录下

[root@tools]# unzip apache-livy-0.7.0-incubating-bin.zip -d /opt

修改livy配置文件

[root@tools]# cd /opt/apache-livy-0.7.0-incubating-bin/conf/
[root@tools]# cp livy.conf.template livy.conf
[root@tools]# vim livy.conf

修改配置项（非kerberos直接复制以下配置粘贴保存）：

livy.spark.master = yarn

livy.spark.deployMode = cluster

livy.environment = production

livy.impersonation.enabled = true

livy.server.csrf_protection.enabled true

livy.server.port = 8998

livy.server.session.timeout = 3600000

livy.server.recovery.mode = recovery

livy.server.recovery.state-store=filesystem

livy.server.recovery.state-store.url=/tmp/livy

kerberos集群需增加的配置（kerberos用户以及认证文件找相关运维人员要）：

livy.server.launch.kerberos.keytab = /etc/daas/kerberos/prd/kafka.keytab

livy.server.launch.kerberos.principal = kafka/hadoop03.daas.com@DAAS.COM

livy.server.access-control.enabled = false

livy.server.auth.type = kerberos

livy.server.auth.kerberos.principal = HTTP/hadoop1.daas.com@DAAS.COM

livy.server.auth.kerberos.keytab =/etc/daas/kerberos/dev/http.keytab

livy.server.auth.kerberos.name-rules = DEFAULT

修改livy环境变量配置文件

[root@tools]# cp livy-env.sh.template livy-env.sh
[root@tools]# vim livy-env.sh

修改配置项（以下配置需根据具体环境改动）：

export JAVA_HOME=/usr/java/default

export HADOOP_HOME=/opt/cloudera/parcels/CDH/lib/hadoop

export SPARK_CONF_DIR=/opt/cloudera/parcels/CDH/lib/spark/conf

export SPARK_HOME=/opt/cloudera/parcels/CDH/lib/spark

export HADOOP_CONF_DIR=/opt/cloudera/parcels/CDH/lib/hadoop/etc/hadoop

export LIVY_LOG_DIR=/var/log/livy

export LIVY_PID_DIR=/var/run/livy

export LIVY_SERVER_JAVA_OPTS="-Xmx8g"

修改livy黑名单配置文件、目录权限修改

[root@tools]# cp spark-blacklist.conf.template spark-blacklist.conf
[root@tools]#chmod -R 775 /opt/apache-livy-0.7.0-incubating-bin

配置数据规则引擎运行环境

hadoop fs -mkdir /griffin/
hadoop fs -put -f /opt/griffin-measure.jar /griffin/
hadoop fs -mkdir -p /home/spark_conf/
hadoop fs -put -f /opt/cloudera/parcels/CDH/lib/hive/conf/hive-site.xml /home/spark_conf/

启动livy

[root@tools]# sh  /opt/apache-livy-0.7.0-incubating-bin/bin/livy-server start

livy其它命令相关

停止命令：sh bin/livy-server stop

状态命令：sh bin/livy-server status

日志查看命令：tail -200f /var/log/livy/livy-root-server.out

显示以下界面无报错则正常：

图1 livy其它命令相关