> Are there examples of project that do this? I'd be very much interested in looking into this.
Nope, I don't believe there are. Unfortunately they don't seem like they're interested in exporting their logical plans to substrait, so there's no obvious way forward.
> DataFusion is primarily a batch OLAP system, so we should be able to support hybrid workloads as well. And definitely agree with you re: Polars dev exp. That is something we are aiming for with our forthcoming Python sdk.
Ah, since this is the case, it might also make sense to tap into the datafusion python bindings which recently got a massive overhaul to have a more similar dev ex as polars (though the docs are still quite a bit behind).
I'm looking forward to seeing what the result will be! I know Ibis also is an option, but with my little bit of playing around with it, I've found it's just the lowest common denominator and doesn't provide as nice of an experience as directly using polars (or whatever query engine api is provided).
Nope, I don't believe there are. Unfortunately they don't seem like they're interested in exporting their logical plans to substrait, so there's no obvious way forward.
> DataFusion is primarily a batch OLAP system, so we should be able to support hybrid workloads as well. And definitely agree with you re: Polars dev exp. That is something we are aiming for with our forthcoming Python sdk.
Ah, since this is the case, it might also make sense to tap into the datafusion python bindings which recently got a massive overhaul to have a more similar dev ex as polars (though the docs are still quite a bit behind).
I'm looking forward to seeing what the result will be! I know Ibis also is an option, but with my little bit of playing around with it, I've found it's just the lowest common denominator and doesn't provide as nice of an experience as directly using polars (or whatever query engine api is provided).