Fargo Ratings vs WPA Rankings?

mikepage

mikepage

AzB Silver Member
Silver Member
In many respects this is an apples-to-oranges comparison, because Fargo Ratings are designed to predict the outcome between two players whether those players show up for events or not, and WPA Rankings are designed in part to encourage participation of the world's best in particular events by rewarding high finishes in those events.

But we can do an interesting test. Which between Fargo Ratings and WPA Rankings does a better job predicting the outcome of matches between top players?

The next WPA event is a Eurotour event, the Dynamic Italy Open, Feb 25-27

Stage 1 is double-elimination groups that advance 32 players to a stage 2 elimination phase.

So Stage 2 is 31 matches in total
Round of 32--16 matches
Round of 16 - 8 matches
round 8 --4 matches
semifinals - 2 matches
Finals -- 1 match

The challenge is predicting the winners of more of those 31 matches.

The prediction can be by who is higher on the WPA ranking list
or by who has a higher Fargo Rating

For instance, if Albin Ouschan Jr (WPA 3, FR779) plays Francisco Sanchez Ruiz (WPA 104, FR782), WPA predicts a win for Albin, and Fargo Ratings predict a win for Francisco.

Which do YOU think will do better at predicting those 31 match outcomes?
 
mikepage said:
In many respects this is an apples-to-oranges comparison, because Fargo Ratings are designed to predict the outcome between two players whether those players show up for events or not, and WPA Rankings are designed in part to encourage participation of the world's best in particular events by rewarding high finishes in those events.

But we can do an interesting test. Which between Fargo Ratings and WPA Rankings does a better job predicting the outcome of matches between top players?

The next WPA event is a Eurotour event, the Dynamic Italy Open, Feb 25-27

Stage 1 is double-elimination groups that advance 32 players to a stage 2 elimination phase.

So Stage 2 is 31 matches in total
Round of 32--16 matches
Round of 16 - 8 matches
round 8 --4 matches
semifinals - 2 matches
Finals -- 1 match

The challenge is predicting the winners of more of those 31 matches.

The prediction can be by who is higher on the WPA ranking list
or by who has a higher Fargo Rating

For instance, if Albin Ouschan Jr (WPA 3, FR779) plays Francisco Sanchez Ruiz (WPA 104, FR782), WPA predicts a win for Albin, and Fargo Ratings predict a win for Francisco.

Which do YOU think will do better at predicting those 31 match outcomes?
Click to expand...

Fargo Rate and I'll take even money on it.
Jason
 
I'll say Fargorate.

I think it would be interesting to approach the predictions 2 ways.

1. Make the predictions all the way through the finals from the original 32 player bracket (like a March Madness bracket), giving extra points for correct predictions in the rounds of 16-finals.

2. Make the predictions on each match as it occurs, regardless of who the model predicted in the previous round.
 
CallShotCowboy said:
What a great experiment...

I wonder how well did the FargoRate worked in predicting this year's Mosconi Cup winner?

http://www.azbilliards.com/news/stories/12365-fargorate-predicts-usa-mosconi-cup-victory/
Click to expand...

touchee!

And I suspect you know this. But for the benefit of others, I will point out that the effort here is to find a sufficient number of matches so that the difference in the approaches will show up in the statistics.

I would like to do it for ALL the matches in the tournament, including the stage 1 matches. But I suspect a lot of the participants won't be on the 500-or-so-strong WPA list.

The Mosconi cup is a fantastic event--drama of the highest order. A key part of that drama, though is that results hinge on which way a handful of key shots turn...
 
mikepage said:
In many respects this is an apples-to-oranges comparison, because Fargo Ratings are designed to predict the outcome between two players whether those players show up for events or not, and WPA Rankings are designed in part to encourage participation of the world's best in particular events by rewarding high finishes in those events.

But we can do an interesting test. Which between Fargo Ratings and WPA Rankings does a better job predicting the outcome of matches between top players?

The next WPA event is a Eurotour event, the Dynamic Italy Open, Feb 25-27

Stage 1 is double-elimination groups that advance 32 players to a stage 2 elimination phase.

So Stage 2 is 31 matches in total
Round of 32--16 matches
Round of 16 - 8 matches
round 8 --4 matches
semifinals - 2 matches
Finals -- 1 match

The challenge is predicting the winners of more of those 31 matches.

The prediction can be by who is higher on the WPA ranking list
or by who has a higher Fargo Rating

For instance, if Albin Ouschan Jr (WPA 3, FR779) plays Francisco Sanchez Ruiz (WPA 104, FR782), WPA predicts a win for Albin, and Fargo Ratings predict a win for Francisco.

Which do YOU think will do better at predicting those 31 match outcomes?
Click to expand...

I am going to think that for now WPA rankings will probably win because the cream generally rises to the top. The FR for the 104th ranked player is slightly higher but the experience and confidence and name-intimidation has to be on Albin's side. These are the intangibles that come into play in any given single matchup.

I suspect that it will be similar for most of the match where the FargoRate is close. For any which show a 50-100 point disparity I think that the FR will accurately predict the winner.
 
mikepage said:
For instance, if Albin Ouschan Jr (WPA 3, FR779) plays Francisco Sanchez Ruiz (WPA 104, FR782), WPA predicts a win for Albin, and Fargo Ratings predict a win for Francisco.
Click to expand...

Does it really? You are convinced that a difference of 3 is great enough to choose a winner. I would think that the difference would be less than a single ball in a match. So FargoRate should not make a prediction of a winner.

In fact, when I have FargoRates compute a fair match, they come out even, all the way up to 13-13. So your program is predicting an even match.

Thank you kindly.
 
Corwyn_8 said:
Does it really? You are convinced that a difference of 3 is great enough to choose a winner. I would think that the difference would be less than a single ball in a match. So FargoRate should not make a prediction of a winner.

In fact, when I have FargoRates compute a fair match, they come out even, all the way up to 13-13. So your program is predicting an even match.

Thank you kindly.
Click to expand...

I think you miss my point. I was just clarifying how the little experiment is performed.

If the WPA #52 player is matched up against the WPA #53 player, then WPA predicts a win for the first one. Nobody in his or her right mind would say this is really a meaningful difference. And of course the WPA is really saying this appears to be an even matchup. Fargo ratings a few points apart say the same thing--an even matchup.

But if we are going to take 31 matches and see who predicts the match winner more frequently, then that is is going to have to include matches that either of us would say is close to a toss up. I was just clarifying how those matches are tallied.
 
JB Cases said:
I am going to think that for now WPA rankings will probably win because the cream generally rises to the top. [...]
Click to expand...

You've got action with the first person to respond in this thread.

But you are my friend John. So I'm going to say do not take this because you will not like it...
 
mikepage said:
I think you miss my point.
Click to expand...

I think you missed mine.

All those ratings should be considered to have error bars. And you know better than I how big those error bars should be. Anything within them should be classified as 'too close to call'. Pretending to have knowledge that you don't in fact have, is only going to lead to people mistrusting you, and your rating system. You [the rating system] already have a credibility problem [that is, people are not believing in it]; don't make it worse.

Thank you kindly.
 
Last edited:
mikepage said:
I think you miss my point.
If the WPA #52 player is matched up against the WPA #53 player, then WPA predicts a win for the first one.
Click to expand...

Does the WPA really try to predict a winner, or are they just ranking players based on their past performance?

I think Bob Jewett summed it up pretty nicely when he compared a ranking system vs. a rating system in one of his past articles in BD.
 
JB Cases said:
I am going to think that for now WPA rankings will probably win because the cream generally rises to the top.
Click to expand...

What does this even mean? Or more to the point, why do you think WPA's data and ranking system take this into account better than FargoRate's data and ranking system.

The FR for the 104th ranked player is slightly higher but the experience and confidence and name-intimidation has to be on Albin's side.
Click to expand...

Why do you think those things were not enough to cause Albin to win more frequently as opposed to Francisco in their respective games, thus causing his rating to be higher?

These are the intangibles that come into play in any given single matchup.
Click to expand...

Why are those intangibles not reflected in the outcome of matches? If they were, then they would be reflected in ratings.

Thank you kindly.
 
CallShotCowboy said:
Does the WPA really try to predict a winner, or are they just ranking players based on their past performance?
Click to expand...

They are BOTH based on past performance. When we invent a rating system based on FUTURE performance, gamblers are going to be very unhappy.

Thank you kindly.
 
Corwyn_8 said:
I think you missed mine.

All those ratings should be considered to have error bars. And you know better than I how big those error bars should be. Anything within them should be classified as 'too close to call'. Pretending to have knowledge that you don't in fact have, is only going to lead to people mistrusting you, and your rating system. You already have a credibility problem; don't make it worse.

Thank you kindly.
Click to expand...

I don't think he has a credibility problem, unless having the most credible rating system is a problem.

You're right that it might be better if close Fargo ratings don't predict a winner, but I'm not sure what that margin of error would be for Fargo, and we definitely don't know what it would be for the WPA rankings. Therefore, the fairest way to compare them would be to look at which rating (WPA or Fargo) is higher and then look at who won, and see which one predicted correctly more often. It's basically a statistical test called the sign test, and I think it's a legit test in this situation. I'd be happy to run it when the data are in if you're skeptical.
 
BRussell said:
I don't think he has a credibility problem, unless having the most credible rating system is a problem.

You're right that it might be better if close Fargo ratings don't predict a winner, but I'm not sure what that margin of error would be for Fargo, and we definitely don't know what it would be for the WPA rankings. Therefore, the fairest way to compare them would be to look at which rating (WPA or Fargo) is higher and then look at who won, and see which one predicted correctly more often. It's basically a statistical test called the sign test, and I think it's a legit test in this situation. I'd be happy to run it when the data are in if you're skeptical.
Click to expand...

Imo, Fargo would win that challenge. It's based on performances between individuals, down to the game count. People will play over or under their number, but after a while it becomes increasingly difficult to play regularly outside of the norm. Keep in mind, that the number separation is a reflection of relative skill, too.
 
BRussell said:
I don't think he has a credibility problem, unless having the most credible rating system is a problem.
Click to expand...

Have you been reading the forum threads? There are a quite a number of people who are questioning the ratings, including the person I was responding to in my original post.

I'd be happy to run it when the data are in if you're skeptical.
Click to expand...

Sure, that would be great to see.

Thank you kindly.
 
You must log in or register to reply here.
Back
Top