I don't know about the US, but here in the UK voters turn up and discover that someone has voted "on their behalf" merely by accident - the poll staff aren't perfect and sometimes simply mark off the wrong person's name as having voted. This is common enough that, as far as I'm aware, it doesn't lead to any kind of serious investigation. (Apparently, there was even one UK council election in Barnes, Richmond-upon-Thames back in 1976 where an ineligible couple voted, were marked off as being a similarly-named couple who later turned up to vote, and correcting this actually changed the result. This correction was only possible because unlike in the US we don't have a fully secret ballot - every ballot paper is serialized and traceable to the person who cast it. I don't think anyone was charged with anything over this. Probably wouldn't even have been investigated if the affected voter didn't see the close result and kick up a fuss.)
At least in my jurisdiction, it's not simply a matter of the poll worker "marking off" your name: they find your name in the book, then they push the book across the table to you so you can make your own signature in the box next to your name.
If someone is attempting to commit in-person voter fraud, they'll need to sign the name that matches the line they put it in. If they're just mistakenly signing on the wrong line, that will, first of all, probably be obvious to both them and the poll worker immediately, and even if not, it will probably be obvious to anyone coming along behind and checking if there's a discrepancy. (Unless the signature is well and truly illegible, and if that's the case, they can check it against the signature from the previous election for that person.)
