Home Original page

xtdb/dev at main · xtdb/xtdb

Developing XTDB2

The top-level project ties all the other projects together for convenience whilst working within this repo. All of the below commands should be run in the root of the XT2 repo.

Links

Prerequisites

You will need at least JDK 21 installed.

Optional: AI Coding Assistant Tools

For AI-assisted development with REPL integration, install clojure-mcp-light tools via bbin:

# Install clj-nrepl-eval for REPL evaluation from command line
bbin install https://github.com/bhauman/clojure-mcp-light.git --tag v0.2.1 \
  --as clj-nrepl-eval --main-opts '["-m" "clojure-mcp-light.nrepl-eval"]'

# Optional: Install paren-repair hook for Claude Code
bbin install https://github.com/bhauman/clojure-mcp-light.git --tag v0.2.1

Getting started

XT2 uses Gradle - a JVM build tool that supports multi-module, polyglot projects. You do not need to install Gradle to develop XT2 - there is a Gradle wrapper in the repo.

  • Java classes are (re-)compiled automatically when you (re-)start a REPL or run tests

  • Start a REPL with ./gradlew :clojureRepl (potentially requiring ./gradlew clean :clojureRepl)

    • -PreplPort=7888 if you need the REPL on a specific port (e.g. for Clojure MCP)

  • Once you’ve connected to the REPL, in the user namespace, run:

    • (dev) to require and go to the dev namespace.

    • (go) to start up the dev node

    • (halt) to stop it

    • (reset) to stop it, reload changed namespaces, and restart it

    • if you’re using Emacs/CIDER, cider-ns-refresh will do all this for you - C-c M-n M-r, , s x in Spacemacs, , r r in Doom.

    • Conjure users can use ConjureRefresh, see the docs for bindings

    • see Integrant REPL for more details.

  • You should now have a running XTDB node under dev/node - you can verify this by calling (xt/status node) (in the dev namespace).

  • Most of the time, you shouldn’t need to bounce the REPL, but:

    • if you add a module, or change any of the dependencies of any of the modules, that’ll require a REPL bounce.

    • if you change any of the Java classes, that’ll require a REPL bounce

    • otherwise, (dev/reset) (or just (reset) if you’re already in the dev ns) should be sufficient.

    • Please don’t put any more side-effecting top-level code in dev namespaces - you’ll break this reload ability and make me sad.

    • Run git config core.hooksPath .githooks to add xtdb hooks to your local repo.

Custom JRE (jlink)

XTDB builds a custom minimal JRE via jlink to slim down Docker images and local task execution. All Test, JavaExec, and clojureRepl tasks automatically use this custom JRE.

If you hit missing-module errors (e.g. ClassNotFoundException for something in the JDK):

  • Escape hatch: add -PfullJdk to your Gradle command — e.g. ./gradlew :clojureRepl -PfullJdk

  • Fix forward: add the missing module to the jlinkModules list in build-logic/jlink/build.gradle.kts and rebuild.

XTDB 'playground'

XTDB nodes can be started in 'playground' mode - either through ./gradlew :run, clj -m xtdb.main playground or the standalone Docker image, or by running the playground-config ir/set-prep in the dev namespace.

In playground mode, to begin with, we only start the pgwire server, without an associated XTDB node. Then, whenever a new connection is initiated, we create a new isolated in-memory node for every distinct 'database' specified in the connection parameters.

This is particularly useful for non-JVM testing - see the /lang README for more details.

Attaching a debugger to the Clojure REPL

  1. Add -PdebugJvm to the Gradle command - e.g. ./gradlew :clojureRepl -PdebugJvm - you should see Listening for transport dt_socket at address: 5005.

    1. Optionally, add -PnoLocalsClearing if Clojure’s locals-clearing is getting in the way of you debugging (i.e. you’re seeing a lot of null local variables in the debugger)

  2. Connect the debugger - in IntelliJ, 'Run' → 'Attach to Process' (or Ctrl-Alt-5).

Testing

  • Test all with ./gradlew test; ./gradlew integration-test for longer tests

  • Property-based testing with ./gradlew property-test

    • Run with custom iterations: ./gradlew property-test -Piterations=500 (defaults to 100)

    • Uses test.check generators to test with randomized data including composite types

  • Some tests have external dependencies which require docker-compose:

    • docker-compose up (docker-compose up <kafka> etc for individual containers),

    • ./gradlew kafka-test

    • docker-compose down

Running specific tests

Most Clojure tests live in the root module (under /src/test/clojure) so that they’re available by default in the root REPL under the :test task.

Note

 For Clojure namespaces, replace dashes with underscores.
Run a test namespace
./gradlew :test --tests 'xtdb.api_test*'
Run tests matching a wildcard keyword
./gradlew :test --tests '*expression*'
Run a specific test
./gradlew :test --tests '**can-manually-specify-system-time-47**
Run a specific property test
./gradlew :property-test --tests '**update-deduplication**' -Piterations=10
Run a specific integration test
./gradlew :integration-test --tests '**test-on-disk-joining**'

Auth for testing cloud based resources

S3 (ensure all of the below are done on a consistent region):

  • Need to ensure the s3-stack is setup via CloudFormation

    • See the README for more info here.

  • Need to log in to the AWS CLI - using aws configure sso

  • Need to assume the role on the CLI - aws sts assume-role --role-arn <ARN of XTDBIamRole> --role-session-name <session-name> --profile <SSO profile>

  • Set AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY & AWS_SESSION_TOKEN from the output of the above, then set AWS_REGION=<region for the cloudformation>

  • Start the Gradle REPL and connect to it, to have all of the AWS creds available.

Azure

  • Ensure the Azure stack, azure-resource-manager/azure-stack.json stack is setup via the Azure deployment manager.

    • See the README for more info here.

  • Ensure a user/app registration is created with the create xtdb-custom-role.

  • Create an access token for said user/app registration.

  • Set AZURE_CLIENT_ID & AZURE_CLIENT_SECRET from the created access token.

  • Set AZURE_TENANT_ID based on the tenant id on which you created the user/app registration/

  • Set AZURE_SUBSCRIPTION_ID based on whichever subscription you created the stack on.

  • Start the Gradle REPL and connect to it, to have all of the Azure creds available.

Google Cloud

  • Ensure the Google Cloud deployment, cloud-deployment-manager/xtdb-object-store-stack.jinja, is setup on the XTDB google cloud account.

    • See the README for more info here.

  • Ensure a Service Account has been created for tests.

    • Ensure the Service Account has the XTDB Custom Role created by the deployment above.

  • Create a private key for the service account, saving a copy of the JSON credential file locally.

  • Authenticate as the service account, using gcloud auth activate-service-account <example-service-account@domain.com> --key-file <private-key.json>

  • Start the Gradle REPL and connect to it, to have all of the google cloud creds available.

Profiling

To attach YourKit:

  • Install YourKit (it’s on the AUR, for Arch folks)

  • ./gradlew :clojureRepl -Pyourkit

  • You might also want -ParrowUnsafeMemoryAccess which turns off bounds checking.

    This assumes YourKit is installed under /opt/yourkit (as it does from the AUR) - feel free to adapt the property (or even use its value) if you have it installed elsewhere.

Monitoring local dev

The monitoring/ directory contains a Docker Compose stack for local observability:

  • Prometheus - metrics collection (scrapes :8081/metrics)

  • Grafana - visualization with preloaded dashboards (http://localhost:3000, login admin/admin)

  • Tempo - distributed tracing backend for OpenTelemetry traces

Starting it from root:

docker-compose -f ./monitoring/docker-compose.yml up -d

The dev node includes a tracer (disabled by default). To enable tracing, set :tracer {:enabled? true} in your node config - traces will be sent to Tempo and viewable in Grafana’s Explore view.

See ../monitoring/README.adoc for full configuration details, including how to export dashboards and scrape multiple nodes.

Using Metabase locally

Metabase is included in the root docker-compose.yml for local development.

  1. Start Metabase:

    docker-compose up metabase

  2. Open http://localhost:3001 and complete the initial setup (create your admin username and password).

  3. Add XTDB as a database connection:

    • Database type: PostgreSQL

    • Display name: XTDB (or your preference)

    • Host: 172.17.0.1 (Docker bridge network, to reach your host machine)

    • Port: 5432 (or whichever port your dev node’s pgwire server is on)

    • Database name: xtdb

    • Username: xtdb

    • Password: (leave blank)

  4. Click "Connect database" - you should now be able to query XTDB from Metabase.

Note

 Your settings and login are persisted in a Docker volume, so they’ll be retained when you restart Metabase. To clear all Metabase data and start fresh, stop the container and run docker volume rm xtdb2_metabase-data.

Releasing XT2

Tooling

A couple of ./gradlew tools:

Reading an Arrow file

These tools output an Arrow file in EDN format

  • ./gradlew -q :readArrowFile -Pfile=<file>

  • ./gradlew -q :readArrowStreamFile -Pfile=<file> if it’s in 'stream IPC' format.

  • Pipe to a file: ./gradlew -q :readArrowFile -Pfile=<file> > output.edn

Reading a hash trie file

  • ./gradlew -q :readHashTrieFile -Pfile=<file>

  • Pipe to a file: ./gradlew -q :readHashTrieFile -Pfile=<file> > output.edn

Reading a table-block file

  • ./gradlew -q :readTableBlockFile -Pfile=<file>

  • Pipe to a file: ./gradlew -q :readTableBlockFile -Pfile=<file> > output.edn

Reading crash logs

Process crash logs into EDN format:

  1. Run: ./dev/read-crash-log-folder.sh <crash-log-dir>

  2. EDN files will be output to <crash-log-dir>/edn/

Example: ./dev/read-crash-log-folder.sh /path/to/my-crash-logs outputs to /path/to/my-crash-logs/edn/

Comments

Comments that merely restate the code add visual noise without value. Assume your reader is a senior developer familiar with XTDB and its codebase.

Don’t write:

  • Comments that repeat the function/variable name

  • Docstrings that describe obvious behaviour

  • Step-by-step narration of what code does

Do write:

  • Rationale for counter-intuitive choices ("This is for performance, because…​")

  • Non-obvious constraints or invariants ("this might be mutated by…​")

  • Links to issues or external context ("see #1234", "per RFC 7231 §6.5.1")

  • Warnings about subtle behaviour that could trip up future developers

Errors

Use xtdb.error (aliased as err) for throwing errors — not raw Java exceptions. We’re migrating the codebase to this convention; new code should always use it.

Each error takes a namespaced keyword as its error code (e.g. ::my-error-code) and a human-readable message. Choose the appropriate anomaly category:

  • err/incorrect — bad input or configuration (caller’s fault)

  • err/unsupported — feature not (yet) implemented

  • err/fault — internal error (our fault)

See api/src/main/clojure/xtdb/error.clj for the full set of categories and their semantics.

Git

see GIT.adoc for details on git practices in XTDB.