As you said, most peoples instinct is to remove it. I leave it on when practicing, and as a result, make no real effort to remove it in a game. A referee can do it, or an opponent if they feel so inclined. There are occasions where I will move it for a particular shot, but it annoys the hell out...