Hi @ndrluis! I have been learning about Iceberg recently, so it is really cool that you are tackling this.
Regarding the focus of Explorer, I think it is a performant (and potentially distributed) DataFrame engine first with support for multiple backends. The multiple backends is a feature but not the priority. That’s my take though, other members of the Explorer team may have a different perspective.
What is next on my roadmap is to have something like Explorer.BatchFrame which is a collection of dataframes, potentially on several nodes, and then we execute the operations on those frames concurrently. The process to do so is:
- Get the location of all .parquet, .arrow, etc files
- Spawn several nodes, each pointing to one the locations above
- Create a BatchFrame
- ??? (do all work)
- Collect or write the results back
With this in mind, where to do you think Iceberg would enter? I believe it can help with step 1 but, what about step 5? Would you want to write the results back in Iceberg format too?
