I had (and still have a bit) a very similar issue, mine came from not seeing the center of the cueball properly due to how I lined up over it and how may eyes saw the ball and cue alignment. The fact that I basically leanered on my own from books and videos did not help as I had no-one to tell me I was doing something odd.
After 10 years, the swipe to the right kept messing me up, I had a tough time using left spin as I would either end up with NO spin or a lot less than what I want because my cue was moving to the right almost every shot. Another issue was with a straight on hit on the object ball, I would often hit off center and either miss by a tiny bit, or when I was trying to draw straight back, I would go to the left or right of the line and mess up the shot.
I fixed this in two ways, on my own though trial and error. I moved my head about 1/4 to 1/2 inch to the right over my cue and also bent in my wrist a tiny bit more towards my body. Those two things have given me a much straighter stroke. By moving in the wrist, it's realy my knuckles that moved a tiny bit inward more, straighting out the wrist to hand alignment.
This may help you out a bit also, but you also need to really focus on bringing the cue straight foward though the ball, I found I had to do that for a long time before it became more natural. I can now draw straight back when I focus on the shot and make sure I line up in the new way.
Just some things that worked for me to cure the same issue as you seem to have.