Agreed. It's not the ref's job to decide what the rules "really" mean. He's to enforce the rules as they are, and he made the correct call.
What no one needs, least of all the ref, is the introduction of subjectivity into the job, and then have to deal with complaints/accusations of favoring...