Hello everyone, this is the tracking update for the paimon rust 0.1.0 release.

Goal

Before we outline the tasks for the 0.1.0 release, let me clarify the project's goal:

Developing a complete implementation of Paimon in pure Rust.

• Users can read/write paimon table like they do in java API.
• Users can read/write paimon table in arrow format.
• Native support for DataFusion
  • But also enable users to implement their own query engines based on this project.
• Native WASM support (a.k.a paimon-wasm)
• Native Python binding based on rust core (a.k.a paimon-py)
• Hive catalog support

Tasks

This will be our initial release, and I aim to include basic read support in it.

• Spec: Implement types that needed by paimon.
  • feat(spec): Add schema types #5
  • Datatypes
  • spec: Implement Snapshot #6
  • spec: Implement Manifest List #14
  • spec: Implement Manifest #7
  • spec: Implement Datafile #10
  • Global Index
  • Data File Index
• Catalog
  • feat(catalog): add catalog API #62
  • catalog: Implement the filesystem catalog #70 (leave hive catalog in next release)
• Arrow Integration:
  • reader: Allow reading paimon data into arrow format. #71
• Basic Read Process: No schema evolution, no merging, no deletion.
  • scan: Implement scan to generate data split from the Snapshot #72 (filter & project push down are not required in v0.1)
• IO: Integrate with Apache OpenDAL for IO.
• Release Utils: Impelment scripts to help generate and verify ASF releases.

After all those tasks, I expect users can read an existing paimon table from storage services.