We see this trend as well. And AWS Security Lake goes exactly there.
Right now, we‘re working on OCSF normalization in our pipelines to drop structured security telemetry in the right format where you need it. Like a security ETL layer.
We considered ClickHouse and DuckDB but struggled with making the execution engine multi-schema, e.g., more jq-like but still on top of data frames. So we started with a custom catalog and engine on top of Parquet and Feather that we will later factor into a plugin to transpile our query language (TQL) to SQL. The custom language because security people are not data engineers.
Right now, we‘re working on OCSF normalization in our pipelines to drop structured security telemetry in the right format where you need it. Like a security ETL layer.
We considered ClickHouse and DuckDB but struggled with making the execution engine multi-schema, e.g., more jq-like but still on top of data frames. So we started with a custom catalog and engine on top of Parquet and Feather that we will later factor into a plugin to transpile our query language (TQL) to SQL. The custom language because security people are not data engineers.
https://docs.tenzir.com