The bottleneck for most model training sizes is VRAM, and since each 4090 has 24 GB VRAM, that's 96 GB VRAM total. The article mentions that it can train LLMs from scratch up to 1 billion hyperparameters, which tracks.
Nowadays that's not a lot: a single H100 that you can now rent has 80 GB VRAM, and doesn't have the technical overhead of handling work across GPUs.
You should be able to train/full-fine-tune (i.e. full weight updates, not LoRA) a much larger model with 96GB of VRAM. I generally have been able to do a full fine-tune (which is equivalent to training a model from scratch) of 34B parameter models at full bf16 using 8XA100 servers (640GB of VRAM) if I enable gradient checkpointing, meaning a 96GB VRAM box should be able to handle models of up to 5B parameters. Of course if you use LoRA, you should be able to go much larger than this, depending on your rank.
Is there a reason you used hyperparameters rather than parameters? I was going to politely correct the terminology but you seem to be in AI for some time so either it was a mistype or I am misunderstanding what you are referencing.
People who are making quick social media posts while taking a casual walk outside on websites that don't make it easy to edit posts and are not expecting to be nitpicked about it.
Overall, it's something I've seen very often on social media and less technical articles about LLMs. OpenAI would fall into the "almost" category.
It's okay to say that you mistyped or whatever, while taking a casual walk outside on websites that don't make it easy to edit posts and are not expected to be nitpicked about it. Throwing in that everyone uses them interchangeably, however, is just profoundly wrong on every level.
I wasn't nitpicking. It is a HUGE differentiation, and I pointed it out specifically because people pick up on terminology so people who might not know better will go forward and just drop in the more super duper hyperparameter, not realizing that it makes them look like they don't know what they're talking about. As I said in the other post, no one who knows anything uses them interchangeably. It is just completely wrong.
Again, I've heard and used the terminology "model hyperparameter" in place of "model parameter", and I've also heard "model parameter" in place of "model hyperparameter" because not every human interaction is a paper on arXiv and the terms are obviously very similar. The context of the term is what matters in the end (as demonstrated by other comments following my correct intent), and society will not crumble if using either term incorrectly in casual conversation. No one intentionally uses the wrong term, but as jokingly said in another comment "when you get really deep into model training, it can seem like there are a billion hyperparameters you have to worry about."
I appreciate being corrected, but you are the one who asked for my opinion based on my extensive time in AI, you can choose to believe it or not.
I doubt the RAM is added up. I think that’s only a feature reserved for their NVLinked HPC series cards. In fact, without nvlink, I don’t see how you’d connect them together to compute a single task in a performant and efficient way.
How long does training a 1B or 500M model take approximately on the 4-GPU setup? Or does that dramatically depend on the training data? I didn’t see that info on your pages.
They've mixed up with Sam Bankman-Fried, not sure how that affects the point they were intending to make, but I think they both have.. mixed reputations. (Only one is currently in prison though...)
That's pre-publication review, not scientific peer review. Special interests try to conflate the two, to bypass peer review and transform science into a religion.
Peer review properly refers to the general process of science advancing by scientists reviewing each other's published work.
Publishing a work is the middle, not the end of the research.
Yes, but even that isn’t simple, I think. They’d not want to land on top of it, so they’d have to push it out from the lander or have it propel itself away from the lander. If they push it out and it doesn’t have a way to stabilize itself, keeping the lens pointing upwards then will require tight control over that push.
So, I guessed (see below) you’d need power to make the sat orient itself.
“EagleCam will be spring ejected from the Nova-C class lander Odysseus about 30 meters above the lunar surface during the final descent. It will take three images a second from each of its three cameras (a total of nine images a second), capturing its six-second freefall to the surface and Odysseus’ descent and soft landing. About an hour after landing, our team will receive the five images of our choosing. During descent, Dr. Henderson and I will be timing events in landing sequence to match to image numbers to choose the first five images we bring back to Earth. Once we have those images, I will post them directly to @eraueaglecam on Instagram. Shortly after that, they will also be available on @spacetechnologieslab on Instagram and @SpaceTechLab on X (formerly Twitter).”
So, it isn’t a 360 camera, and they’re making 50-ish images and hoping for the best. Doesn’t look like the sat has rockets or that they’re trying to make it possible to make more photos after impact on the moon.
If my guesses/intuition is right we won’t see the actual touchdown (still cool to have anything, of course), but corrections welcome.
Wouldn’t be any better. You’d need 4 to be able to reliably land with one pointed out of the regolith. That’s probably pushing it in terms of mass. 3 wouldn’t be any better than 2 though.
I, for one, am pretty sure the article is wrong there. What I've heard from geologists is that Yellowstone's magma chambers simply are not molten enough to erupt.
Right, "up to 20%", whereas the threshold for "barely eruptible" is in the 50% range, according to this article: https://www.pnas.org/doi/10.1073/pnas.1617105113 Unless the rules are way different for supereruptions, which I guess we have to consider, but probably not 30% different.
Yeah I don't really think a meaningfully larger amount people are traveling because Airbnb is an app that exist, so demand from tourism should stay flat, no?
Apartments in residential buildings are well.. apartments, not hotels. If you're a tourist, go to a hotel, I don't want a new set of loud obnoxius neighbors partying every few days next door to my apartment (and a shared wall in between). The apartment that should be rented out at cheaper monthly rate is now getitng rented out to lud tourists per daily rates, and that is a bad thing for locals living in that city. Some areas are already destroyed by airbnbs, so banning airbnbs (and all the other short-term rentals of residential apartments) should be banned.
An apartment building is owned collectively and with that comes a set of rules. In most places you aren't allowed to be a nuisance, even though you own a share of the house you don't get to dictate or do whatever you want just because you own something. Your property is just a part of a larger property, respect others or get the fuck out.
It's a pretty simple arrangement that one gets into when they purchase an apartment.
If you host a few guests a year, for a few days, and they are your friends or people who you might trust, no issue whatsoever.
A never ending cycle of tourists staying for 2-10 days as your neighbours? That's definitely a nuisance over time, very improbable that churning through 50-100 groups of different people per year won't create issues to neighbours.
If you don't see how it could be an issue I think only if you lived in a touristic place, neighbour to AirBnBs, to actually understand. I say that not to provoke you but because I feel it's hard to empathise when it's not your day-to-day life. Just this year I experienced that when staying at a friend's place in Lisbon, I shared it on another HN thread:
> As an anecdote: a month ago I stayed a few days (5-7) at a friend's place I was visiting who lives in Lisbon, just on his floor there are 4 AirBnBs (owned by the same person). Not only it was a nuisance with noise for most of the days I was there it was also a nuisance to have drunk British girls banging on your door at 02.00 in the night when they don't remember the fucking apartment they are supposedly going to. My friend mentioned it's not uncommon for that to happen, or to have a gag of people show up to a party in one of the apartments. Other people living in the building have complained to AirBnB, to the police, to the housing association, nothing really happens.
So sounds like the bug is lax enforcement of noise ordinances? Not sure why it makes a difference whether the drunk and noisy houseguest is a short-term renter or a friend of the owner.
A friend of the owner implies the owner is a neighbour, someone who you can eventually talk about the issue in the house. It makes a difference as there's a greater degree of social bond (and consequently shame).
Are you being obtuse on purpose to not give up on a flawed argument?
I live within walking distance dozens of short-term rentals in a very small town.
The difference between having dozens of neighbors with families, children, and strong ties to the community and not having those neighbors and community connections is in no way artificial.
Some reasons I care: schools that decline with population loss, the lack of available/affordable rental homes for local residents, the effects on local retail and service businesses, the tendency for hurried visitors to drive at unsafe speeds while staring at their phone for directions.
Problems that you do not experience remain real problems.
So basically instead of thinking about how you can better accommodate visitor demand and respect property owners’ rights, you would prefer to regulate your “small town feel” back into existence.
Because it's an apartment building with shared space, shared resources and shared investments. If your neighbors plot is a residential plot, he cannot build a pig farm there or a nuclear power plant. If your neighbors apartment is an apartment, he cannot have a shooting range, a bar, a strip club or a hotel there. There's a difference between occasional guests and airBnB, the same as there is a difference between a friend who helps you fix your car vs someone who regulary fixes cars for money.
As long as the property owner registers their short term rental with the local government and pays their taxes, and as long as their guests are respectful and don’t cause any trouble, why do you even care?
It's a limited resource being misused. It is in the interest of the locals for tourists to stay in hotels and apartments to be available for long(er) term rent for the locals. Barcelona was one of the first cities to ban short-term rentals (under 31 days), and hopefully not the last.
It’s only “limited” because housing has been made artificially scarce by local governments operating in the interests of property owners by restricting new builds in order to keep housing prices high. In regions and countries that have allowed enough housing to be built to actually meet demand there is much less hand-wringing about short-term rentals.
I think I get your perspective (but also you should really understand that your arguments are just going unidirectional instead of building a healthy dialog), but it made me wonder what really went wrong (or changed, depending on your perspective) from Hegel's utopian egalitarianism to how communism was implemented in practice a century later. I guess people just tend to "adapt" legal and governance theories built with perfectly good intentions to their advantage and then things go awry.
I mostly agree that there is probably some self-interest at play when they ask for regulation. Though the fact that one of the existing firms in the industry wants regulation doesn't invalidate all the legitimate reason for regulation. For example, existing car manufacturers might benefit significantly from all the safety requirements for cars as they create a barrier to entry for new players, but I doubt that many people want to abolish all the testing and certification cars have to go through.
Please don't complain about tangential annoyances—e.g. article or website formats, name collisions, or back-button breakage. They're too common to be interesting.
The properly GDPR compliant cookie banners allow you to itemize certain items on the website's TOS that you may choose to accept or not accept. Website TOS are very useful for the company operating the website and the cookie rules allow you as a visitor to get some of that usefulness back.
Browsers like Firefox could hypothetically kill cookies in their browser tomorrow but doesn't. Or at least make a big stink about it. Do you think they should? Do you think we'd be better off as a society?
I think when you click "accept", you also accept to things like fingerprinting and storing a fingerprinted identity on their server, as well as perhaps supercookies, that allow your ISP to track you.
Browsers can't tell if a cookie is a generic setting ("chose Yes/No on a banner") or a uniquely identifiable one; and they can't tell if a cookie is functionally required (ID for a logged-in session) or not (ID to track random visitors).
The distinction is legal, not technical; so it has to be enforced by legal, not technical means.
Firefox COULD default to cookies off (with an in menu widget to force them on for non-automatic handling), and if any forum submission happens _ask_ if the end user wants to accept the site's cookies.
Looking at a typical site, a reasonable user might want to accept one (or perhaps a couple) of many dozens of cookies a site attempts to set. Choosing it manually per site per cookie is difficult but perhaps theoretically possible, however even that still requires cooperation from the site to honestly identify that this one is the cookie which is functionally required, and these fifty are for ad tracking, and ensuring that cooperation still requires legal means and can't be done with purely technical ones.
Furthermore, there is the important distinction about multiple uses of the same data. There are uniquely identifiable cookies that are functionally required for one purpose but the site may want to use it for other purposes as well (e.g. share that data with heir "trusted partners" for targeted advertising) for which user may reasonably want to refuse permission, so a browser accepting a cookie doesn't imply such permission and something extra is required.
