Home Original page

feat(datafusion): DataFusion metrics query layer by alexanderbianchi · Pull Request #6276 · quickwit-oss/quickwit

and others added 7 commits

April 7, 2026 13:36
Introduces a generic DataFusion execution layer with a pluggable
QuickwitDataSource trait. No data-source-specific code.

- QuickwitDataSource trait + DataSourceContributions (contribution-return pattern)
- DataFusionSessionBuilder with shared RuntimeEnv, check_invariants
- QuickwitSchemaProvider backed by DataFusion MemorySchemaProvider for DDL tables
- QuickwitWorkerSessionBuilder + build_quickwit_worker for distributed execution
- QuickwitWorkerResolver, QuickwitTaskEstimator
- QuickwitObjectStore: quickwit_storage::Storage → object_store::ObjectStore bridge
- DataFusionService::execute_sql (streaming Arrow IPC responses)

Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
Implements QuickwitDataSource for the parquet metrics pipeline from PR quickwit-oss#6237.

- MetricsDataSource: production (metastore-backed) and test (SimpleIndexResolver)
- MetricsTableProvider: filter pushdown with CAST-unwrapping fix for timestamp
- MetastoreSplitProvider: converts MetricsSplitQuery → ListMetricsSplitsQuery
- MetastoreIndexResolver: resolves index URI → QuickwitObjectStore per query
- MetricsSplitQuery + extract_split_filters: predicate extraction for split pruning
- MetricsTableProviderFactory: CREATE EXTERNAL TABLE … STORED AS metrics support
- test_utils: make_batch, TestSplitProvider, MetricsTestbed for integration tests

Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
Routes Substrait ReadRel nodes to registered QuickwitDataSource implementations.
Standard NamedTable reads resolve via MetricsDataSource::try_consume_read_rel.
ExtensionTable reads (custom protos) can be handled by downstream callers.

- QuickwitSubstraitConsumer implementing datafusion-substrait SubstraitConsumer
- execute_substrait_plan / execute_substrait_plan_streaming entry points
- DataFusionService::execute_substrait (bytes) and execute_substrait_json (dev path)
- session.rs: DataFusionSessionBuilder::execute_substrait convenience method

Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
…tegration tests

- Add datafusion.proto with DataFusionService (ExecuteSubstrait, ExecuteSql RPCs)
- Generate codegen and mod.rs for the new proto service
- Wire DataFusionService and WorkerService into quickwit-serve gRPC layer
- Add DataFusionServiceGrpcImpl handler
- Auto-create otel-metrics-v0_9 index on startup alongside logs/traces
- Add metrics_datafusion_tests: in-process SQL + Substrait over parquet splits
- Add metrics_distributed_tests: multi-node distributed execution
- Add rollup_substrait.json fixture for Substrait plan testing

Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
Remove extra schema argument from ParquetWriter::new; the API only accepts
a ParquetWriterConfig. Remove unused ParquetSchema import.

Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
…test

- Remove erroneous ParquetSchema argument from ParquetWriter::new calls
  in integration tests (API takes only ParquetWriterConfig)
- Mark test_rest_ingest_then_in_process_query as #[ignore] until the
  /ingest-metrics REST endpoint is wired in quickwit-serve

Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>

@alexanderbianchi alexanderbianchi changed the title feat(datafusion): wire DataFusion gRPC service + integration tests feat(datafusion): DataFusion metrics query layer

Apr 7, 2026

@alexanderbianchi @claude 

Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>

mattmkim

alexanderbianchi

alexanderbianchi

@alexanderbianchi @claude 

- fix: CAST unwrapping in classify_filter — reuse predicate::column_name
  so time-range predicates are correctly classified as Inexact and
  passed to scan(); previously CAST-wrapped filters were silently dropped
- fix: declare parquet sort order (metric_name, timestamp_secs ASC) on
  FileScanConfig so DataFusion avoids redundant sort operators
- fix: get_opts now respects GetOptions.range — dispatches to get_slice
  for Bounded/Suffix ranges instead of always downloading the full file
- fix: to_object_store_error propagates file path on NotFound
- fix: register_for_worker made a no-op; lazy scan-path registration is
  sufficient and avoids O(indexes) metastore RPCs per worker task;
  removes stale comment claiming a non-existent object-store cache
- fix: extract is_index_not_found helper, removing duplicated downcast
  block from try_consume_read_rel and create_default_table_provider
- fix: sort before dedup in QuickwitSchemaProvider::table_names
- fix: empty searcher pool returns Ok(vec![]) for local execution fallback
- fix: remove dead builder methods with_udf_batch, with_codec_applier,
  with_physical_optimizer_rule from DataSourceContributions
- feat: add tracing spans to execute_substrait and execute_sql
- feat: wire 4 GiB memory limit on DataFusionSessionBuilder in serve
- refactor: extract stream_to_receiver helper in gRPC handler

Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>