Many builders put at least a sealer on the shaft, which will raise the grain and they would subsequently sand smooth.
Also for those that do sand their shafts, I recommend a backer of cork or rubber of some kind, even Styrofoam. I see a lot of guys just use their fingers, and what happens is...