Just use your pants pockets. That's what I do.
If my keys are in my right front pocket, I must be shooting at the right hole. If not, check the left.
Can't find your keys in either pocket? Say "Sorry, I'd like to leave now" and ask your opponent or a sympathetic bystander to call a cab for you...