Yea, I have hear this before, so as using lighter fluid to clean your shaft.
Lots of people don't like abrasive, but inorder for you to smooth the shaft, I on't know any other way but to smooth it with light abrasive. The idea isto remove as little diameter of the shaft as possible.
I like a packof cue smooth. Or I use 1500grit than 2000grit and if you want to go further, go 2500grit to 3000grit as well. That will smooth it alot.
If you want to use an agent to smooth the shaft, use a wax. Light coat of cue wax (or can use butcher's wax or car wax with carnaubu) buff and polish with cotten or microfiber cloth and use a piece of leather to burnish again.
This process makes it very smooth and it also seals the shaft and keeps alot of dirt out of it.
Hope this helps,
Chino