The Spark SQL Engine Configuration Guide

Kyuubi aims to bring Spark to end-users who need not qualify with Spark or something else related to the big data area. End-users can write SQL queries through JDBC against Kyuubi and nothing more. The Kyuubi server-side or the corresponding engines could do most of the optimization. On the other hand, we don’t wholly restrict end-users to special handling of specific cases to benefit from the following documentations. Even if you don’t use Kyuubi, as a simple Spark user, I’m sure you’ll find the next articles instructive.

• How To Use Spark Dynamic Resource Allocation (DRA) in Kyuubi
  • The Basics of Dynamic Resource Allocation
  • How to Enable Dynamic Resource Allocation
  • Sizing for engines w/ Dynamic Resource Allocation
  • Resource Allocation Policy
  • Best Practices for Applying DRA to Kyuubi
  • References
• How To Use Spark Adaptive Query Execution (AQE) in Kyuubi
  • The Basics of AQE
  • Best Practices for Applying AQE to Kyuubi
  • Spark Known issues
  • References
• Solution for Big Result Sets
  • Incremental collection
  • Use in single connections
  • Change incremental collection mode in session