Here's a interesting paper I found that attempts to categorize the mechanisms by which Goodhart's Law operates in the real world. The variants are separated into Causal and Non-causal mechanisms.
Thanks for posting that. I was going to say -- it's interesting to think about the reasons why Goodhart's Law might hold if it does.
I've always assumed the problem is that the metric is always influenced by other, nontarget variables that become more causally important when the metric becomes a proxy target. So, for example, "gaming the metric" becomes important (in a percent variance sense) after the metric becomes a target. I think the paper's adversarial scenario is closest to this maybe.
They discuss some other factors that seem more relevant to individual cases at any moment in time than an explanation for why a metric's utility might decline over time. In that sense the paper seems to be more about Goodheart-like phenomena in general.
It would be interesting to demonstrate Goodheart's law conclusively with real data in some domains.
Likes: Same as above. I'm impressed by the overall comfort of the keyboard, as I can use it all day with no joint pain or stress. It's hard to imagine a more comfortable and healthy keyboard for the long term.
Dislikes: The lack of a numpad is troublesome sometimes, but it makes sense for a keyboard with such a radical commitment to ergonomics. Gaming with it can be challenging, requiring me to pull out a different keyboard. Also, this might sound really petty, but the sound is less satisfying compared to my previous Fission MX Cherry Brown keyboard. The clacks reverberate through the large hollow chassis with a lower pitch. I even put a sock inside the keyboard to dampen the sound, which was only partly effective.
I’m a recent graduate of Lambda School’s data science program, where I learned to build and deploy machine learning models. I’m currently doing contract work but would like to find full-time employment somewhere that will broaden my exposure to software engineering practices. My recent projects include a classifier for non-standard tax forms, a movie recommender, and a knowledge discovery tool for private journals. If you need an ML engineer who is willing to tackle every aspect of a project, including client-facing work, I’m your guy.
This looks fantastic. One bit of feedback: the lack of a Paypal option for donating almost stopped me from donating. The fees might be worth the security guarantee.
a short film: https://maximecb.github.io/Turing-Drawings/#7,3,5,2,0,6,2,1,...
long time to reach NESS (if it does at all): https://maximecb.github.io/Turing-Drawings/#7,3,3,1,0,0,2,3,...
Cool glyphs: https://maximecb.github.io/Turing-Drawings/#4,4,0,2,1,3,1,0,...