Implementing Bidirectional Data Exchange with HBase (Python)

Function

You can use Spark to call an HBase API to operate HBase table1 and write the data analysis result of table1 to HBase table2.

Sample Code

PySpark does not provide HBase APIs. In this example, Python is used to invoke Java code to implement required operations.

The following code snippets are used as an example. For complete codes, see SparkHbasetoHbasePythonExample.

# -*- coding:utf-8 -*-

from py4j.java_gateway import java_import
from pyspark.sql import SparkSession

# Create the SparkSession and set kryo serialization.
spark = SparkSession\
        .builder\
        .appName("SparkHbasetoHbase") \
        .config("spark.serializer", "org.apache.spark.serializer.KryoSerializer") \
        .config("spark.kryo.registrator", "com.huawei.bigdata.spark.examples.MyRegistrator") \
        .getOrCreate()

# Import the class to be run to sc._jvm
java_import(spark._jvm, 'com.huawei.bigdata.spark.examples.SparkHbasetoHbase')

# Create class instances and invoke the method.
spark._jvm.SparkHbasetoHbase().hbasetohbase(spark._jsc)

# Stop SparkSession.
spark.stop()