It also seems gameable: for every big question of societal importance that people care about for its own sake, have a thousand random little questions where the outcome is dead obvious and can be predicted trivially. Would you know anything talking about weighing questions to account for this?
There's calibration, but you can also just see contests where you pit the community aggregate against individual forecasters and see who wins. The Metaculus aggregate is really dominant in this contest of predicting outcomes in 2023, for example. See this: https://www.astralcodexten.com/p/who-predicted-2023
Trivial questions wouldn't result in a good histogram where a probability of 30% actually results in something happening roughly 1 in 3 times. Trivial would mean questions where the community forecast is 1% or 99%.
Those are not the vast majority of questions on the site. It would be very boring if the site was 70% questions where the answer is obviously yes or obviously no.
Additionally, many questions require that you give a distributional forecast, in effect giving you 25/50/75th percentile outcomes for questions such as "how much will Bitcoin be with at the end of 2024?"
Who would be gaming the system here anyway, the site? Individual users?
I'd also include under 'trivial' things like "Will this 6-sided die roll a 1?", or really any other well-understood i.i.d. process whose distribution of outcomes never changes under reasonable circumstances. Not just things that are 0.1% or 99.9%.
> Who would be gaming the system here anyway, the site? Individual users?
Cynically speaking, users would be incentivized to ask and answer more trivial questions to pad out their predictive accuracy, so they can advertise that accuracy elsewhere for social clout.
