Delta lake is an open-source project that enables building a Lakehouse Architecture on top of existing storage systems such as S3, ADLS, GCS, and HDFS.
This article assumes that you have mastered the basic knowledge and operation of Delta Lake. For the knowledge about delta lake not mentioned in this article, you can obtain it from its Official Documentation.
By using kyuubi, we can run SQL queries towards delta lake which is more convenient, easy to understand, and easy to expand than directly using spark to manipulate delta lake.
Delta Lake Integration#
To enable the integration of kyuubi spark sql engine and delta lake through Apache Spark Datasource V2 and Catalog APIs, you need to:
The classpath of kyuubi spark sql engine with delta lake supported consists of
kyuubi-spark-sql-engine-1.9.0-SNAPSHOT_2.12.jar, the engine jar deployed with kyuubi distributions
a copy of spark distribution
delta-core & delta-storage, which can be found in the Maven Central
In order to make the delta packages visible for the runtime classpath of engines, we can use one of these methods:
Put the delta packages into
Please mind the compatibility of different Delta Lake and Spark versions, which can be confirmed on the page of delta release notes.
To activate functionality of delta lake, we can set the following configurations:
Delta Lake Operations#
As for end-users, who only use a pure SQL interface, there aren’t much differences between using a delta table and a regular hive table. Unless you are going to use some advanced features, but they are still SQL, just more syntax added.
CREATE A TABLE as a example,
CREATE TABLE IF NOT EXISTS kyuubi_delta ( id INT, name STRING, org STRING, url STRING, start TIMESTAMP ) USING DELTA;