The point is Google respects publishers' desire not to be indexed. Padmapper does not. Robots.txt is simply a common method for conveying that message. It's not like Padmapper could argue they didn't know CL was unhappy; they got a certified letter!
This is just semantics. Google respects publishers who do not want their sites listed.
The point is Google respects publishers' desire not to be indexed. Padmapper does not. Robots.txt is simply a common method for conveying that message. It's not like Padmapper could argue they didn't know CL was unhappy; they got a certified letter!
This is just semantics. Google respects publishers who do not want their sites listed.