Lens material absolutely matters. Glasses suffer from chromatic aberration, and this causes visible rainbows away from the center of the lens. A material with a higher Abbe number will perform better.

For common lens materials, polycarbonate has a low Abbe number, and Trivex is a widely available alternative with a higher Abbe number. You can find tables online for common lens materials.