Delta Lake#

Delta lake is an open-source project that enables building a Lakehouse Architecture on top of existing storage systems such as S3, ADLS, GCS, and HDFS.

Tip

This article assumes that you have mastered the basic knowledge and operation of Delta Lake. For the knowledge about delta lake not mentioned in this article, you can obtain it from its Official Documentation.

By using kyuubi, we can run SQL queries towards delta lake which is more convenient, easy to understand, and easy to expand than directly using spark to manipulate delta lake.

Delta Lake Integration#

To enable the integration of kyuubi spark sql engine and delta lake through Apache Spark Datasource V2 and Catalog APIs, you need to:

Dependencies#

The classpath of kyuubi spark sql engine with delta lake supported consists of

  1. kyuubi-spark-sql-engine-1.7.0-SNAPSHOT_2.12.jar, the engine jar deployed with kyuubi distributions

  2. a copy of spark distribution

  3. delta-core & delta-storage, which can be found in the Maven Central

In order to make the delta packages visible for the runtime classpath of engines, we can use one of these methods:

  1. Put the delta packages into $SPARK_HOME/jars directly

  2. Set spark.jars=/path/to/delta-core,/path/to/delta-storage

Warning

Please mind the compatibility of different Delta Lake and Spark versions, which can be confirmed on the page of delta release notes.

Configurations#

To activate functionality of delta lake, we can set the following configurations:

spark.sql.extensions=io.delta.sql.DeltaSparkSessionExtension
spark.sql.catalog.spark_catalog=org.apache.spark.sql.delta.catalog.DeltaCatalog

Delta Lake Operations#

As for end-users, who only use a pure SQL interface, there aren’t much differences between using a delta table and a regular hive table. Unless you are going to use some advanced features, but they are still SQL, just more syntax added.

Taking CREATE A TABLE as a example,

CREATE TABLE IF NOT EXISTS kyuubi_delta (
  id INT,
  name STRING,
  org STRING,
  url STRING,
  start TIMESTAMP
) USING DELTA;