Keep in mind that unless you are playing on an isolated table all the time then there is constant movement and distractions going all around anyway. One would hope that pretty much every player has been able to train themselves at this point to tune all that out.
I personally have no issue if you are in my line of sight moving, walking by, getting a drink or what have you. I would not even notice it to begin with, even if you are standing over the pocket. My sight line / aim line has nothing to do with you, it is between me and that damn pocket that better be catching my ball.
Understanding that not everyone has that, I do try to keep still but I am not going inconvenience myself for your sake and get up from an assign seating arrangement.