We have, and I won’t lie, it’s kinda a pain. The reality we discovered is that async flows are second-class citizens in otel tracing.
We essentially propagate the context manually between APIs and lambdas, through HTTP headers, SNS/SQS and even storage. We also ended up pulling Tempo’s Parquet files (we’re self-hosting the grafana stack) into Redash to be able to do real analysis. We got a lot of great insights and were able to do a lot of tuning that would have been impossible otherwise, but it was quite an investment. Would love to know if there is anything out there that would have made this less painful.
We essentially propagate the context manually between APIs and lambdas, through HTTP headers, SNS/SQS and even storage. We also ended up pulling Tempo’s Parquet files (we’re self-hosting the grafana stack) into Redash to be able to do real analysis. We got a lot of great insights and were able to do a lot of tuning that would have been impossible otherwise, but it was quite an investment. Would love to know if there is anything out there that would have made this less painful.