from the rules at onepocket.org
technically you are the one to move it back or leave it where it is
he can shoot it if he wants to
i bolded some points for emphasis
'''''''''''''''''''''''''''
6.1.1 Restoring a position: There is no restoration option when any disturbed ball has had an effect upon the shot; in that case the balls must be played from where they lie. Once it is acknowledged that a ball or more was disturbed with no effect on the shot, then the opponent must be given the option to restore the position or leave the balls as they lie before play is resumed. If the opponent elects restoration, it shall be as near as possible to the original positioning according to both players, with the opponent responsible for final approval. It is a foul upon the shooter to restore a ball without permission from the opponent. However, if the shooter was notified, yet resumes play without the opponent’s consent to the final disposition of a disturbed ball, then it is a breach of the restoration
option, and a serious foul on the shooter. ref: 6.6.2 Serious fouls