All balls wear down over time, just from colliding with other balls
And for the cue ball, being hit every shot with the chalked cue tip may be the biggest factor.I think that's a factor, but the cloth can be a larger factor if it is not cleaned well. Chalk is ground up sand. Chalk in the cloth turns the cloth into sandpaper.
pj
chgo