Integration with Hive Metastore#

In this section, you will learn how to configure Kyuubi to interact with Hive Metastore.

  • A common Hive metastore server could be set at Kyuubi server side

  • Individual Hive metastore servers could be used for end users to set

Requirements#

So the whole thing here is to let Spark applications use this copy of Hive configuration to start a Hive metastore client for their own to talk to the Hive metastore server.

Default Behavior#

By default, Kyuubi launches Spark SQL engines pointing to a dummy embedded Apache Derby-based metastore for each application, and this metadata can only be seen by one user at a time, e.g.

bin/kyuubi-beeline -u 'jdbc:kyuubi://localhost:10009/' -n kentyao
Connecting to jdbc:kyuubi://localhost:10009/
Connected to: Spark SQL (version 3.4.2)
Driver: Kyuubi Project Hive JDBC Client (version 1.9.0)
Beeline version 1.9.0 by Apache Kyuubi
0: jdbc:kyuubi://localhost:10009/> show databases;
2020-11-16 23:50:50.388 INFO operation.ExecuteStatement:
           Spark application name: kyuubi_kentyao_spark_2020-11-16T15:50:08.968Z
                 application ID:  local-1605541809797
                 application web UI: http://192.168.1.14:60165
                 master: local[*]
                 deploy mode: client
                 version: 3.4.2
           Start time: 2020-11-16T15:50:09.123Z
           User: kentyao
2020-11-16 23:50:50.404 INFO metastore.HiveMetaStore: 2: get_databases: *
2020-11-16 23:50:50.404 INFO HiveMetaStore.audit: ugi=kentyao	ip=unknown-ip-addr	cmd=get_databases: *
2020-11-16 23:50:50.423 INFO operation.ExecuteStatement: Processing kentyao's query[8453e657-c1c4-4391-8406-ab4747a66c45]: RUNNING_STATE -> FINISHED_STATE, statement: show databases, time taken: 0.035 seconds
+------------+
| namespace  |
+------------+
| default    |
+------------+
1 row selected (0.122 seconds)
0: jdbc:kyuubi://localhost:10009/> show tables;
2020-11-16 23:50:52.957 INFO operation.ExecuteStatement:
           Spark application name: kyuubi_kentyao_spark_2020-11-16T15:50:08.968Z
                 application ID:  local-1605541809797
                 application web UI: http://192.168.1.14:60165
                 master: local[*]
                 deploy mode: client
                 version: 3.4.2
           Start time: 2020-11-16T15:50:09.123Z
           User: kentyao
2020-11-16 23:50:52.968 INFO metastore.HiveMetaStore: 2: get_database: default
2020-11-16 23:50:52.968 INFO HiveMetaStore.audit: ugi=kentyao	ip=unknown-ip-addr	cmd=get_database: default
2020-11-16 23:50:52.970 INFO metastore.HiveMetaStore: 2: get_database: default
2020-11-16 23:50:52.970 INFO HiveMetaStore.audit: ugi=kentyao	ip=unknown-ip-addr	cmd=get_database: default
2020-11-16 23:50:52.972 INFO metastore.HiveMetaStore: 2: get_tables: db=default pat=*
2020-11-16 23:50:52.972 INFO HiveMetaStore.audit: ugi=kentyao	ip=unknown-ip-addr	cmd=get_tables: db=default pat=*
2020-11-16 23:50:52.986 INFO operation.ExecuteStatement: Processing kentyao's query[ff902582-ba29-433b-b70a-c25ead1353a8]: RUNNING_STATE -> FINISHED_STATE, statement: show tables, time taken: 0.03 seconds
+-----------+------------+--------------+
| database  | tableName  | isTemporary  |
+-----------+------------+--------------+
+-----------+------------+--------------+
No rows selected (0.04 seconds)

Using this mode for experimental purposes only.

In a real production environment, we always have a communal standalone metadata store, to manage the metadata of persistent relational entities, e.g. databases, tables, columns, partitions, for fast access. Usually, Hive metastore as the de facto.

Activate Configurations#

Via kyuubi-defaults.conf#

In $KYUUBI_HOME/conf/kyuubi-defaults.conf, all Hive primitive configurations, e.g. hive.metastore.uris, and the Spark derivatives, which are prefixed with spark.hive. or spark.hadoop., e.g spark.hive.metastore.uris or spark.hadoop.hive.metastore.uris, will be loaded as Hive primitives by the Hive client inside the Spark application.

Kyuubi will take these configurations as system wide defaults for all applications it launches.

Via hive-site.xml#

Place your copy of hive-site.xml into $SPARK_HOME/conf, every single Spark application will automatically load this config file to its classpath.

This version of configuration has lower priority than those in $KYUUBI_HOME/conf/kyuubi-defaults.conf.

Via JDBC Connection URL#

We can pass Hive primitives or Spark derivatives directly in the JDBC connection URL, e.g.

jdbc:kyuubi://localhost:10009/;#hive.metastore.uris=thrift://localhost:9083

This will override the defaults in $SPARK_HOME/conf/hive-site.xml and $KYUUBI_HOME/conf/kyuubi-defaults.conf for each user account.

With this feature, end users are possible to visit different Hive metastore server instance. Similarly, this works for other services like HDFS, YARN too.

Limitation: As most Hive configurations are final and unmodifiable in Spark at runtime, this only takes effect during instantiating the Spark applications and will be ignored when reusing an existing application. So, keep this in our mind.

!!!THIS WORKS ONLY ONCE!!!

!!!THIS WORKS ONLY ONCE!!!

!!!THIS WORKS ONLY ONCE!!!

Via SET syntax#

Most Hive configurations are final and unmodifiable in Spark at runtime, so keep this in our mind.

!!!THIS WON’T WORK!!!

!!!THIS WON’T WORK!!!

!!!THIS WON’T WORK!!!

Version Compatibility#

If backward compatibility is guaranteed by Hive versioning, we can always use a lower version Hive metastore client to communicate with the higher version Hive metastore server.

For example, Spark 3.0 was released with a built-in Hive client (2.3.7), so, ideally, the version of server should >= 2.3.x.

If you do have a legacy Hive metastore server that cannot be easily upgraded, and you may face the issue by default like this,

Caused by: org.apache.thrift.TApplicationException: Invalid method name: 'get_table_req'
	at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:79)
	at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.recv_get_table_req(ThriftHiveMetastore.java:1567)
	at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.get_table_req(ThriftHiveMetastore.java:1554)
	at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.getTable(HiveMetaStoreClient.java:1350)
	at org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient.getTable(SessionHiveMetaStoreClient.java:127)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.invoke(RetryingMetaStoreClient.java:173)
	at com.sun.proxy.$Proxy37.getTable(Unknown Source)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.apache.hadoop.hive.metastore.HiveMetaStoreClient$SynchronizedHandler.invoke(HiveMetaStoreClient.java:2336)
	at com.sun.proxy.$Proxy37.getTable(Unknown Source)
	at org.apache.hadoop.hive.ql.metadata.Hive.getTable(Hive.java:1274)
	... 93 more

To prevent this problem, we can use Spark’s Interacting with Different Versions of Hive Metastore.

Further Readings#