Yes, I think your instinct to look at win32 APIs is a good one. Many structures in riched are public in headers, and of course you could reference Wine's implementation[1].
That is neat, thanks. I didn't check RICHED itself but when i was toying around with an HTML editor a few years ago (see my other post) i did check out an HTML editor API that Microsoft had (i don't remember its name but it was used by, e.g., Joel Spolsky's CityDesk - IIRC the control itself was documented in the MSDN CDs that came with Visual Studio 6), from where i got the idea of using a two node pointers and a subrange index to represent a selection.
[1] https://source.winehq.org/git/wine.git/tree/HEAD:/dlls/riche...