I wonder if they'll let us search flagged and dead posts and comments.

I remember reading some insightful exchanges back in the day that got flagged because of being a controversial topic that other users didn't like.

No way to find them now, even knowing some keywords and approximate month and year.

We had to exclude [dead] and eventually even just [flagged] posts from the public API because many third-party clients and sites were displaying them as if they were regular posts. For the ever-fragile HN ecosystem, that is catastrophic. We would get angry emails saying "how can your expletive site possibly condone such expletive expletive comments as <link>" ... and then it would turn out that <link> was a post by some account that had been banned for years.

It's fine if users turn 'showdead' on in their profile to read everything—just please remember that by doing that, you're subscribing to various bottoms of various barrels. But it's definitely not ok when people browse HN with some app we have nothing to do with, run into horrible things, understandably are outraged and then forever have their view of HN imprinted.

IMO this issue is existential for HN. We've spent years and so much energy trying to find a balance between internet openness and human decency, a task which oscillates between barely-possible and simply-doomed, so the idea that anybody anywhere sees anything labeled "Hacker News" that pours all the toxic waste back into the commons is physically painful to me. Much as I dislike the idea of restricting anyone's curiosity about the entire corpus of what gets posted, I don't see what choice we have.

I figured as such. Can I ask that you change the website itself to make them visible to logged out accounts? I understand exactly why you did this but I feel like if you showed them collapsed by default on the single-comment page and you have to actively click on a "banned" to expand them you really are out of line when you complain about how Hacker News hosts horrible content or whatever.

You mean for [dead] comments? Sorry, but strong no. Logged-in-with-showdead-turned-on has proven to be the correct height for that gate. Anyone who wants to can easily clear it, but the small amount of effort and information required means that most people become core community members before turning it on. If we lowered it, naive-casual readers would (through no fault of their own) misunderstand what they were looking at and the dynamic I just described would kick in.

The longer I've worked on HN the more I've come to appreciate PG's design of this critical aspect of the site. No content is hidden from users who want to see it*, but the worst is (mostly) cordoned off so it doesn't destroy the community. Banned users can continue to post, but their comments are autokilled, so they're cordoned off by default.

We're often asked: why allow banned users to continue to post? The answer is that if we didn't, they'd just create new accounts, and then they'd be posting with unbanned accounts until we caught them and banned them again: a strictly worse situation. This is one aspect of PG's design that took me years to appreciate and got me thinking it might even be optimal.

The one major change we made to the original design was adding 'vouching' (https://news.ycombinator.com/item?id=10298512), which lets the community transfer cordoned-off posts back to the commons if there's nothing wrong with them. That bit has worked out really well.

* (Except for [deleted] posts. If you see [deleted] it always means that either the author deleted it or asked us to do that for them.)

Thank you dang, I very much appreciate you taking the time to explain.

I was going to suggest maybe the public API could have a "showdead" flag too but I guess that too easily enables the problem you're trying to prevent? As in an enterprising app developer could turn the "showdead" tap to "yes" with every request and then the waste gushes out once more.

I can appreciate that concern and see it even with flagging / dead / killed posts and submissions.

I've had my own concerns about HN's moderation, both excesses and insufficiencies. When I've done occasional polls about what people's issues are about HN I'm very often pointed to comments which now show as flagged. I'd found a few which hadn't been flagged and forwarded those to dang, who (admittedly long after the fact) flagged them. As dang's said many time, moderators don't see everything, most moderation is by members, and mods step in relatively rarely.

Based on Whaly's 2021 analysis and looking at dang's own comment post history (via Algolia), HN nets roughly 4 million comments/year and 400k submissions, with about 150k active members. Over his ten years as moderator dang's averaged about 20 comments per day, though there's a great deal more moderation occurring (some automated, some member-based, some manual but not noted with comments, which tend to be reserved for established accounts).

My read is that HN mostly tends toward its stated goals and, frankly, good-netizen behaviour. It does have a pronounced status-quo bias, though it seems to be self-aware on this point. I've a few further concerns I'm still thinking through.

The problem with an overly-open archive is that this makes possible misconstrued assertions about what HN does or doesn't tolerate. An open-access archive and third-party apps which don't reflect moderation actions, say, a third-party app which explicitly only showed flagged, killed, and/or dead posts, comments, and users, would paint a distinctly different picture of HN, and one which would greatly harm the reputation of the site.

There are some ... possible ways around this. HN uses sequentially-numbered IDs for posts and comments (both are treated the same so far as I can tell). UserIDs seem to have an internal representation which is similar (I've seen, for example, names which change over time), but the internal representation doesn't seem to be publicly exposed. If you want to find my own content you'd do it with "UserID=dredmorbius" and not by some numeric identifier.

But the numeric content ID means that a determined scraper could walk (sequentially or randomly) through the entire database, pull out every post and comment, and then glue those back together. That's somewhat north of 40 million items presently.... (There are benefits to using sparse, random / arbitrary UUIDs for systems.)

Whaly analysis (2021 data): <https://whaly.io/posts/hacker-news-2021-retrospective>

Here is the list of recent flagged/dead stories, ordered by rating, that they managed to accumulate: https://play.clickhouse.com/play?user=play#U0VMRUNUIGlkLCB0a...

Here is the list of recent flagged/dead stories: https://play.clickhouse.com/play?user=play#U0VMRUNUIGlkLCB0a...

Is that playground open source?

Yes, the backend is https://github.com/ClickHouse/ClickHouse/ It has no front-end (everything is served by the database itself).

The HTML page: https://github.com/ClickHouse/ClickHouse/blob/master/program... (or press "view source" in the browser).

The dataset: https://github.com/ClickHouse/ClickHouse/issues/29693

It's possible to favourite such items. They'll not be visible unless you're logged in, but at least you can go through your fave list and find them.

Or, of course, bookmark them yourself for later reference.

I've run into this issue myself, though dang's reply (still being edited as I write this) does hit on some valid points.

They're using the firebase API, https://github.com/devflowinc/trieve-hn-discovery/tree/main/... so no showdead, I think.

contributes to the echo chamber