Unless I'm missing something, the easiest to turn down round things is always on a lathe.
Chuck it up with a collet, indicate to within a thou if possible.
Use a sharp bit. A really sharp one.
Take slow passes and keep dry fitting till you get the right fit .
FWIW, I hate them.