Put it back. It is not a foul on the player who moved it since he as acting as the referee. If the ball is hanging over a pocket making it nearly impossible to get the balls out, you should consider marking it and removing it.
Similarly, if the break ball is very close to the rack but not in it, you should mark its position and remove it before trying to rack.