Question 1

What makes TabbyDB different from standard Apache Spark?

Accepted Answer

TabbyDB is a specialized fork of Apache Spark designed to optimize complex queries. It significantly reduces compilation time and memory usage for queries with nested joins, complex case statements, and large query trees. Unlike stock Spark, TabbyDB fixes these issues at the root — without disabling optimizer rules or requiring any code changes.

Question 2

Can TabbyDB handle extremely large and complex query trees?

Accepted Answer

Yes. The Project node capping algorithm prevents tree size from growing unboundedly — the root cause of most OutOfMemory failures in complex programmatic query patterns. Queries that previously failed with OOM now complete in minutes.

Question 3

I ran TPC-DS on TabbyDB and see the same performance as stock Spark. Why?

Accepted Answer

The TPC-DS toolkit defaults to partitioned tables. To see the difference, pass the partitioning flag as false and modify data generation to sort locally on the date column while writing partition splits. With this setup, data generation is 6-7x faster and benchmark results show approximately 13% better performance at 1TB-2TB scale.

Question 4

If TabbyDB on non-partitioned tables matches stock Spark on partitioned tables, why should I use TabbyDB?

Accepted Answer

Three reasons. First: Spark's Dynamic Partition Pruning only works when the join key IS the partition column — not all real-world queries do this. Second: partitioning data is expensive; creating a 1TB partitioned TPC-DS dataset took 6+ hours vs 40 minutes for non-partitioned. Third: TPC-DS queries don't stress compile time — real DataFrame API queries can spend far more time compiling than executing. TabbyDB eliminates that cost.

Question 5

Do I lose any functionality compared to stock Spark?

Accepted Answer

No. TabbyDB is fully compatible with stock Spark. It adds performance improvements without removing or compromising any Spark feature, functionality, or runtime behavior. For workloads that don't hit TabbyDB's target bottlenecks, results are identical to stock Spark.

Question 6

Is TabbyDB compatible with existing Spark DataFrame APIs?

Accepted Answer

100% compatible. Use your existing code unchanged. TabbyDB guarantees full compatibility with open-source Spark — now and in future releases.

Frequently Asked Questions