If you leave it alone, it will eventually get worked out of there from use. If it bugs you, you probably should take it to a cue repair guy, they can clean up a wrap very well without any damage. Someone cleaned a Joss for me that had 10+ years of use and the grip looked and felt like it was brand new. Granted it was Mike Webb so it's based one one of the top cue guys in the country, but I'm sure even a middle of the road repair guy can do the job well.
I don't like trying to clean out an linen wrap as you tend to make it rough and uneven if you try to clean it much more than a wipe of a cloth.