Then it goes back to the shooter for his opinion (since the referee didn't hear...) and the TD has the right to take audience opinion as well (which would possibly bring into his hands the spectator's clear record of 2 ball called).
Sorry, but I disagree with you completely that a referee or tournament director should ever rely on the audience's opinion of what happened.