I wrote this and I’m happy to answer any questions about savings, infrastructure, how we built and tested this in our environment, or open sourcing it with AWS.
I love the visual approaches used to explain these concepts. Words and math hurt my brain, but when accompanied by charts and diagrams, my brain hurts much less.
I'm also interested in the details on how this works in practice. I know that there was a front page post a few weeks ago about how Cursor worked, and there was a short blurb about how sets of security prompts told the LLM to not do things like hard code API keys, but nothing on the training side.
This is great. pip-tools` is so valuable right now in helping mitigate these dependency tree issues. I'd love to see some form of support in core Python. I really hope this becomes pushed out, similar to how `pip` actually has a dependency resolver now. Relying on running `pip freeze` to create a quasi-lock is a horrible pattern for enterprise environments and for packages. I'm really looking forward to how this turns out, even though it's still in a proposal-type phase. `
Pip has had a dependency resolver since 2020, it just had some issues initially (in large part because of the terrible state of metadata in the ecosystem). There are plenty of other problems with Pip, of course, and not having an obvious way to save and reproduce a package resolution is one of them.
Pip doesn't aim to manage packages in the first place, however; expecting to lock the entire environment is out of its wheelhouse (or should I say .whl-house?).
For Paper, rather than implementing anything like Pip's "dry-run" option (which is nowhere near as "dry" as you'd expect, but that's a separate issue...), I'm planning to have a separate command to output PEP 751 lockfiles representing an individual resolution, i.e. the packages that would be added to the environment.
Hah, yes. I used to work at a company that sold a popular DNA product. I built part of the GDPR data deletion pipeline. During my last week there, I submitted a request to delete my data from the systems. The final integration test!
I liked Scala and was excited about learning and using it when it was picking up steam many years back. Unfortunately, it felt like there wasn't as much traction, and Scala has been siloed into some niche domains like Spark. It's a bummer since it's so nice to write with.
I feel like Java has calmed down a bit and is more focused and thriving in its niche with enterprise systems. I recall the days when we believed that Java and the JVM could solve all of our problems.
I worked a lot with Spring and Spring Boot a while back and I can't help but smile when I think back to the discussions where a room full of us were arguing about beans.
Neat idea! If you can, someone figure out how to get Nextdoor animal-posters to join and participate, this would be really useful. Last time I saw a dog on the street, I pulled over to catch the dog, re-installed Nextdoor, made a post, and brought the dog to the local shelter. It's hard to beat not only the neighborhood aspect of Nextdoor, but also just the volume and mental "this is where it goes" vibe. What's not hard to beat about Nextdoor is the NIMBY and triggering comments so you're halfway there, hah!
Repo is here: https://github.com/aws-samples/sample-spot-balancer-spark-ek...