I pause probably long than 95% of people on this site both at CB and at the end of the backswing. I've never had an issue with what you describe. A pause if anything promotes a good straight stroke, and helps with a players timing. Others may not pause as long as me or others but as long as there is a pause at some point, it only aids the stroke.My backhand stroke tends to move towards my body, instead of straight back
IMO this is due to long pause of tip at CB, and it is natural muscle movement after a pause! Try to reduce your pause time after pre-warmup strokes so your muscles memory stays active to the pre-warmup movement. If you cannot, slow your back swing considerably, or lengthen your bridge. Could use TOI.
To avoid your back hand going off path do as you said and slow the backswing. Also visualize the straight line you are cueing on as you pull back and keep the cue in your vision whilst doing so. Doesn't have to be a focus point , you may focus on the CB or OB but as long as you notice it and consciously pay attention to pulling the cue straight back along the shot line it will eventually change your stroke and it will become natural.