If anyone from cloudflare comes here - it's not possible to create D1 databases on the fly and interact them because databases must be mentioned in the worker bindings.
Try Durable Objects. D1 is actually just a thin layer over Durable Objects. In the past D1 provided a lot of DX benefits like better observability, but those are increasingly being merged back into DO directly.
What is a Durable Object? It's just a Worker that has a name, so you can route messages specifically to it from other Workers. Each one also has its own SQLite database attached. In fact, the SQLite database is local, so you can query it synchronously (no awaits), which makes a lot of stuff faster and easier. You can easily create millions of Durable Objects.
Thank you! That's great and it is possible but... With some limitations.
The idea is from sign up form to a D1 Database that can be accessed from the worker itself.
That's not possible without updating worker bindings like you showed and further - there is an upper limit of 5000 bindings per worker and just 5000 users then becomes the upper limit although D1 allows 50,000 databases easily with further possible by requesting a limit increase.
Transactions are supported in Durable Objects. In fact, with DO you are interacting with the SQLite database locally and synchronously, so transactions are essentially free with no possibility of conflicts and no worry about blocking other queries.
Extensions are easy to enable, file a bug on https://github.com/cloudflare/workerd . (Though this one might be trickier than most as we might have to do some build engineering.)
I'm always a little hesitant to use D1 due to some of these constraints. I know I may not ever hit 10GB for some of my side projects so I just neglect sharding, but also it unsettles me that it's a hard cap.
This hampers the per user databases workflow.
Would be awesome if a fix lands.