Yes you can! Well, mostly, depends on how pedantic you are with definitions: you can transplant layers but not weights, which in common parlance are conceptually similar. But usually it isn’t a good idea for a few reasons.
There’s a really fascinating example[1] where a guy identifies a particular set of layers and transplants them. Overgeneralizing, early layers are encoders and the later layers are decoders and in the middle some blocks seem to do specific things or tasks related “reasoning”. So you can actually create a FrankenLLM and it sometimes works.
This needs architectures to be roughly similar however and internal representations to be consistent-ish so for “stealing” it’s not really a thing (other practical concerns aside)
I'm not 100% sure it's not possible. If (I don't know) it's possible to freeze the temperature of the model so it's deterministic, and if you could make a map of produced words back to tokens (via HMM probably), then you can probably alter a minimal input and observe the output to model it. If you perform waves of such minimal alterations, you can expect to be able to locate the distance where each alteration impact the model (the idea being that a small alteration on output is likely due to the last layers of the models, and a small alteration is likely due to the deeper layer). Once you've located most of the last layer(s?) weights, you can try to solve for them. With a hundreds of billions weights model, the last layers will likely be so huge that it's probably unfeasible technically, but it's theoretically possible.
You can do things like that - one example is averaging weights between related models - but not with Anthropic's models, because outsiders don't have access to the weights.
