Cloud support deltalake by afterthought · Pull Request #280 · roapi/roapi
This PR pulls in updates from delta-rs that fixes: (missing support for cloud storage) mentioned here: https://roapi.github.io/docs/config/dataset-formats/delta.html. I had to remove the custom implemented object store registry. This functionality interferes with how delta-rs is implemented. I also do not think it is necessary any longer to have this custom implementation. From what I can tell it was only done as a convenience/workaround. My change registers object stores as tables are loaded for any table provider that has a URI. Hopefully this look OK to the maintainers.
I'm new to Rust but managed to use the linters/formatters to get the checks to pass. However, if I've made any blunders, please let me know.
I'm excited about this PR and it is working well for me in my testing. We have a deltalake on Azure and I was able to register several tables and execute streaming queries that were complex including joins and aggregations. It was enough to get a good sense of where the limits will be for our architecture.