Whenever in the process allows you or is most likely to bore the straightest hole you can.
In my experience, I bore holes when the object is as thick as it's going to be in round shape. I won't core a square, but as soon as wood is round, I bore it and plug it.
I imagine boring a close to final size shaft would be a nightmare if you aren't dialed in to bore the PERFECT concentric hole, relative to the axis of the piece. Sometimes when I core out forearms and handles, if I get lazy, I'll have the drill wander micro amounts and by the end of the piece, it's off center a couple thousandths. When I am working with 1 3/8" round, this obviously matters very little when the tapering cuts out any imperfections, then the piece is centered on the core. But if I was to core a finish tapered anything and get lazy with my guide hole or my feeding of the drill, there's a good chance I am gonna blow out a collar or nose.