The _absence_ of visual imagery is binary: you cannot see images at all or, to whatever extent, you can. Those who do have any mental imagery at all, however, fall on a scale. There are numerous studies of certain real downsides to aphantasia, notably tied to episodic memory, which don't seem to be present in those simply with diminished visual imagery.