I feel you. There are guys on my league team I don't want to play with, and they're actually fairly nice guys and straight shooters too. It's awkward.
The weaselly way out is to say you decided to get off the current team and form a new team. Don't tell him the new team is going to be everyone on the old team except Tony. Eventually he will figure it out but by then he will have his own new team.
Second weaselly way out: Say that someone (not you) has told you in private they cannot stand Tony, and wants him off the team, and someone else backed it, and you decided to go along with it rather than break up the winning combination. Refuse to name names. That way you're not the bad guy, you're just breaking the news to him.
Third weaselly way out: Have someone else say the whole team voted and it was 3 to 1, with you being the 1. Could backfire though if he asks you to join him in a new team and split from the guys who voted him off the island.
The honest way out is to just tell him you guys want to try something new, with a 5th guy who has heavy firepower, and don't all gang up at once to tell him, just you break the news. "Hey tony, I've got some bad news... we're gonna try someone else in your spot on the team... there's a guy who's a strong B (or whatever) who looks like he could take us all the way. Sorry man."
Then he'll piss on your wedding cake.