Can't this be on a case by case basis? After all, the British stole from so many places in the world - what if they returned stolen items to countries (India for example) that are interested and capable of safeguarding artifacts and didn't return to those countries where they might get destroyed (Afghanistan etc)?
What about examples where there was nothing like theft involved and no colonisation - the Pergamon Altar in the amazing Pergamon Museum in Berlin is a good example...