If you use sandpaper that is too rough, the "sand" will make a slight bevel around the edge of the ferrule and that's not good.
So, use 150 grit or lighter so this doesn't happen.
I can't imagine trying to free-hand it. The Rapid Tip Sander works but some of those (knock-offs it is claimed) are not perpendicular, apparently, so buy carefully.
Jeff Livingston