Expose subgraph logs via subgraph GraphQL by fordN · Pull Request #6278 · graphprotocol/graph-node
Introduces the foundation for the log store system with: - LogStore trait for querying logs from backends - LogLevel enum with FromStr trait implementation - LogEntry and LogQuery types for structured log data - LogStoreFactory for creating backend instances - NoOpLogStore as default (disabled) implementation
Implements three log storage backends for querying logs: - FileLogStore: Streams JSON Lines files with bounded memory usage - ElasticsearchLogStore: Queries Elasticsearch indices with full-text search - LokiLogStore: Queries Grafana Loki using LogQL All backends implement the LogStore trait and support: - Filtering by log level, timestamp range, and text search - Pagination via first/skip parameters - Returning structured LogEntry objects Dependencies added: reqwest, serde_json for HTTP clients.
Implements slog drains for capturing and writing logs: - FileDrain: Writes logs to JSON Lines files (one file per subgraph) - LokiDrain: Writes logs to Grafana Loki via HTTP push API Both drains: - Capture structured log entries with metadata (module, line, column) - Format logs with timestamp, level, text, and arguments - Use efficient serialization with custom KVSerializers
Adds a configuration layer for selecting and configuring log backends: - LogStoreConfig enum with variants: Disabled, File, Elasticsearch, Loki - LogConfigProvider for loading config from environment variables and CLI args - Unified GRAPH_LOG_STORE_* environment variable naming - CLI arguments with --log-store-backend and backend-specific options - Configuration precedence: CLI args > env vars > defaults - Deprecation warnings for old config variables Supported configuration: - Backend selection (disabled, file, elasticsearch, loki) - File: directory, max size, retention days - Elasticsearch: endpoint, credentials, index, timeout - Loki: endpoint, tenant ID
Refactors LoggerFactory to use LogStoreConfig instead of elastic-only: - Replaced elastic_config with log_store_config parameter - Build ElasticLoggingConfig on-demand from LogStoreConfig::Elasticsearch - Support all log drain types (File, Loki, Elasticsearch) - Maintain backward compatibility with existing elastic configuration This enables the factory to create drains for any configured backend while preserving the existing component logger patterns.
Adds GraphQL API for querying subgraph logs: Schema types: - LogLevel enum (CRITICAL, ERROR, WARNING, INFO, DEBUG) - _Log_ type with id, timestamp, level, text, arguments, meta - _LogArgument_ type for structured key-value pairs - _LogMeta_ type for source location (module, line, column) Query field (_logs) with filters: - level: Filter by log level - from/to: Timestamp range (ISO 8601) - search: Text search in log messages - first/skip: Pagination (max 1000, skip max 10000)
Integrates _logs query into the GraphQL execution pipeline: Execution layer: - Execute _logs queries via log_store.query_logs() - Convert LogEntry results to GraphQL response objects - Handle log store errors gracefully Query parsing: - Recognize _logs as special query field - Build LogQuery from GraphQL arguments - Pass log_store to execution context Service wiring: - Create log store from configuration in launcher - Provide log store to GraphQL runner - Use NoOpLogStore in test environments This completes the read path from GraphQL query to log storage backend.
Adds comprehensive integration test for _logs query: Test implementation: - Deploys logs-query subgraph and waits for sync - Triggers contract events to generate logs - Queries _logs field with various filters - Verifies log entries are returned correctly - Tests filtering by level and text search
- Create graph/src/log/common.rs for common log drain functionality - SimpleKVSerializer: Concatenates KV pairs to strings - VecKVSerializer: Collects KV pairs into Vec<(String, String)> - HashMapKVSerializer: Collects KV pairs into HashMap - LogMeta: Shared metadata structure (module, line, column) - LogEntryBuilder: Builder for common log entry fields - level_to_str(): Converts slog::Level to string - create_async_logger(): Consistent async logger creation - Updated FileDrain, LokiDrain, and ElasticDrain to use the log common utilities
- include _logs in the set of special fields that bypass indexing error shortcutting when subgraph failed - add integration test to ensure _log queries return logs after subgraph failed
Use map_while instead of filter_map on lines() iterator to properly handle read errors, and add missing orderDirection argument to the _logs field in mock introspection JSON.
- Replace level: String with level: Level in ElasticLog, FileLogDocument, and LokiLogDocument - Add shared serialize_log_level to common.rs that serializes Level as lowercase - Remove level_to_str() and level_str()
- Add [log_store] section to graph-node.toml as the sole configuration path for log store backends - Remove env var (GRAPH_LOG_STORE_*) and CLI arg (--log-store-*) config paths, LogStoreConfigProvider, and env var helper utilities
- Use q::Pos::default() instead of Pos::default() in api.rs - Add log_store field to Config constructors in tests - Pass NoOpLogStore to GraphQlRunner in gnd test runner - Update Cargo.lock
Switch from --postgres-url CLI args to --config with a generated TOML file, so the [log_store] section is available for the logs-query test.
The --ethereum-rpc CLI mode defaults to archive,traces features, but the generated TOML config had features = []. This caused subgraphs that depend on those capabilities to fail during indexing.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters. Learn more about bidirectional Unicode characters