I mean I'm running TensorRT-LLM on a basket of spot vendors at NVFP4 with auction convexity math and Clickhouse Keeper and custom passthrough.
I need more tokens not less because the available weight models aren't quite as strong, but I roofline sm_100 and sm_120 for a living: I get a factor of 2 on the spot arb, a factor of 2 on the utilization, and a factor of 4-16 on the quant.
I need more tokens not less because the available weight models aren't quite as strong, but I roofline sm_100 and sm_120 for a living: I get a factor of 2 on the spot arb, a factor of 2 on the utilization, and a factor of 4-16 on the quant.
I come out ahead.