So the GPS functionality is meant to communicate back the location over the internet? If all you need is offline music and GPS with maps, an old smartphone without a SIM card is the best option. If you need internet connectivity, an old smartphone is probably still the best option, but with a SIM card. Just lock it down if you're worried about various apps.

Another option would be to buy one of the dedicated kid s̶p̶y̶i̶n̶g̶ tracking devices and tape it to an iPod.