Make the collet in that chuck and then mark the collet and the chuck so you put the collet in the same every time. Turn a piece of wood round to fit the collet and check the runout of the wood. Should be very close to zero runout as boring the collet to the chuck should make the collet zero. If you rotate the collet it will be out from zero up to .003 or twice what the bore is off. Always put the collet in the same way in reference to the jaws or exactly the position it was bored.
Or just go back and rebore the chuck but not at .001 per pass but do .0005 or less a pass and do it several times at this setting. The crossfeed that the toolholder fits on have several thousands of play and the tool will push off a bit because of this. This is fine for wood but you are boring metal and this is not a metal lathe. Go slow!!!