Billiard University (BU) playing-ability-exam scores and ratings

[JoeW]

Now that the exams have been out a while and some of you have tried them, do you have any suggestions for possible changes in the future?

What do you think should be added?

What do you think should be removed?

Are there any important skills missing?

Are the exams too long?

If you think something should be added, what would you remove so the exams don't get longer?

What do you like or dislike about the scoring system?

Do you think the BU rating is an accurate measure of your playing ability?

Thanks in advance for any input or suggestions you might have.

Best regards,
Dave

Congratulations on an impressive piece of work Dave. I read most of the materials and have downloaded Exam 1. This is an impressive project and a major undertaking that could make a significant contribution to the pool playing world. I wish you all the best for a truly bright future with Billiards University. I look forward to completing the tests. However, before I get further into it I have a few suggestions.

1. Ask your colleagues to submit 25 – 30 sets of scores for people with a range of skills. You have a sufficiently large cadre of “professors” who could contribute to this project.
2. Collect demographic data such as age, sex, current rankings from various sources. Perhaps the amount of money or awards won in the last year, etc.
3. I would pay particular attention to table size and characteristics.

With a 300 or so data sets you would be ready for an item analysis that could allow you to refine the exams. See http://www.personality-project.org/r/book/chapter8.pdf with specific reference to ability testing. The Progressive Drills are a variant of Likkert scaling in the ability domain and would require some special attention during the analysis based on the underlying assumptions for distribution theory. In psychology we are quite pragmatic, if it looks like a normal distribution and it acts like a normal distribution then we treat it as one until such time that some other distribution is needed. A professor at your university with a specialty in psychometrics or multivariate statistics could be of much use In test creation. While many people have little respect for psychology in general you may be pleasantly surprised with the depth and range of their knowledge

I would probably use discriminant function analysis to create equations that would allow you to weight for different table sizes and characteristics. This could be useful in the next version. Table manufacturer, table length, pocket size, type of cloth may or may not be useful variables for equating across conditions.

From what I have read (in this thread) it would appear that there may be some issues with gain (or change) scores over time and practice. Training and evaluation are confounded and you might want to think about this issue as it relates to reliability and generalizability of the findings.

I suspect that your current examination procedure is sufficient for “awarding” various degrees. However, you may need some built in recommendations for the amount of training that is recommended to obtain one's high score. Some floor and ceiling that could be recommended. I would include the amount of time spent with the exams prior to posting a grade in the item analysis.

BTW you should seek financial funding . You have certainly done your home work here and it would be worth while to see if unrestricted funding is available.

While I realize that Billiards University is currently about the degree granting process, I suggest that even now you might want to consider some some solicited e-books monographs, seminars, and similar activities. These types of activities would not require a great deal of time on your part, especially if you enlisted your colleagues participation. Obviously, I think that you are really unto something here.

 

[dr_dave]

FWIW Dr. Dave...

I administered the fundamentals test to 4 people of varying abilities. The result were in line with my thoughts on their respective skill levels.

Subject A is a fella who used to play a bit in his younger days but has not played in a number of years. He scored a 13.

Subject B is a gal who is a 62 in USAPL and plays quite regularly. I would rank her about a C- or so. She scored a 26.

These were on a 9 footer diamond with pockets >4.5" . I have not measured the pockets yet but estimate them to be either 4 5/8" to 4 3/4" at the points.

Subject C is most likely a B- to C+ skill level. He scored a 52.

Subject D is a solid B to B+. He could even peak into A- on occasion. He scored a 68 but he performed much faster than he normally plays. Might have been bored with it. I thought he would have scored a little higher. Was hoping he would hit the 80's. He started with 10, 9, 10 then fell off on the next two with a 5 and a 4.

C and D tests were on a Diamond 7' table with 4.5" pockets

Thank you for sharing this information. It sounds like the BU Rating System did a decent job of rating these people into the categories you expected, based on the BU Ratings Comparison Chart.

I have taken the fundamentals test a second time on my Diamond 9 footer and improved all the way to 62!

I apologize for no videos but time is short right now. I will post up a video asap.

I look forward to seeing your total score and videos.

I have seen a lot of interest in what I am doing by bystanders while administering these tests. I anticipate a lot more random results in the future. We'll see

Ken,

I notice from your signature that you are a PBIA instructor. If you would also like to be a BU instructor, check out the BU Instructor How-To Guide.

BTW, if you convince some of those bystanders or students to apply for a BU Diploma, you can generate some additional income. For more info, see How to Make Money as a BU Instructor.

Best regards,
Dave

 

[dr_dave]

Congratulations on an impressive piece of work Dave. I read most of the materials and have downloaded Exam 1. This is an impressive project and a major undertaking that could make a significant contribution to the pool playing world. I wish you all the best for a truly bright future with Billiards University.

Joe,

Thank you very much for your kind and supportive words. I appreciate it.

I know you had tried a similar thing in the past with your Pool Quotient ability test, so I know that you are aware of how much effort this sort of thing takes.

I look forward to completing the tests.

Thank you for participating. I look forward to seeing your scores.

However, before I get further into it I have a few suggestions.

1. Ask your colleagues to submit 25 – 30 sets of scores for people with a range of skills. You have a sufficiently large cadre of “professors” who could contribute to this project.

I've run a fairly large number of tests myself with players over a fairly wide range of abilities. The BU professors have also done some testing. I don't have any of this data posted, but it was used to create the BU scoring and rating system that is currently in place.

2. Collect demographic data such as age, sex, current rankings from various sources. Perhaps the amount of money or awards won in the last year, etc.

We had already planned to collect some self-assessment player-ability information (see the last page of the diploma application form), but I also just added the demographic data, as you suggested. Good idea. Data is good.

3. I would pay particular attention to table size and characteristics.

I wanted to collect additional info (e.g., the angles of the pocket walls, and info for both corner and side pockets), but we decided to keep things as simple as possible (without requirement too much effort or special measuring devices), while still providing some context for the score. Again, you can see the data we collect on the diploma application form (see the first page).

With a 300 or so data sets you would be ready for an item analysis that could allow you to refine the exams. See http://www.personality-project.org/r/book/chapter8.pdf with specific reference to ability testing. The Progressive Drills are a variant of Likkert scaling in the ability domain and would require some special attention during the analysis based on the underlying assumptions for distribution theory. In psychology we are quite pragmatic, if it looks like a normal distribution and it acts like a normal distribution then we treat it as one until such time that some other distribution is needed. A professor at your university with a specialty in psychometrics or multivariate statistics could be of much use In test creation. While many people have little respect for psychology in general you may be pleasantly surprised with the depth and range of their knowledge

I would probably use discriminant function analysis to create equations that would allow you to weight for different table sizes and characteristics. This could be useful in the next version. Table manufacturer, table length, pocket size, type of cloth may or may not be useful variables for equating across conditions.

From what I have read (in this thread) it would appear that there may be some issues with gain (or change) scores over time and practice. Training and evaluation are confounded and you might want to think about this issue as it relates to reliability and generalizability of the findings.

I suspect that your current examination procedure is sufficient for “awarding” various degrees. However, you may need some built in recommendations for the amount of training that is recommended to obtain one's high score. Some floor and ceiling that could be recommended. I would include the amount of time spent with the exams prior to posting a grade in the item analysis.

Thanks for all of the input and suggestions. When I'm ready for a sophisticated analysis after we get lots of reliable data, I might contact you for some advice and help (if you are willing).

BTW you should seek financial funding . You have certainly done your home work here and it would be worth while to see if unrestricted funding is available.

That's a good idea ... when I can find some time and motivation for this sort of thing.

While I realize that Billiards University is currently about the degree granting process, I suggest that even now you might want to consider some some solicited e-books monographs, seminars, and similar activities. These types of activities would not require a great deal of time on your part, especially if you enlisted your colleagues participation. Obviously, I think that you are really unto something here.

We currently plan to offer the annual BU Summer School Boot Camp, and we already have lots of focused learning resources available, but we plan to do a lot more along these lines as things progress. For example, I am hoping all of the BU professors will host BU-theme events all around the country (and the world) eventually.

Again, thank you so much for you well-though-out ideas and positive feedback.

Best regards,
Dean Dave

 

[dr_dave]

FYI to those using the BU Exams, we just posted a spreadsheet on the BU website to automate exam scoring and player rating. It also helps keep historical records to track your improvement over time. If you are interested, here's the direct link.

It's also now available along with all of the other documents on the BU Exam Resources page.

Enjoy,
Dave

PS: The original version of this spreadsheet was created by Patrick Johnson, a BU Instructor. For those who don't know, PJ is currently banned from AZB (for a really long time) because he wasn't a good boy (apparently). Regardless, I appreciate his spreadsheet skills and willingness to share with others.

 

[BRussell]

Hey dr_JoeW, fellow psychologist dr_brussell here. (And ribbing dr_dave a bit. )

 

[dr_dave]

Hey dr_JoeW, fellow psychologist dr_brussell here. (And ribbing dr_dave a bit. )

It's "Dean Dave" to you psycho PhDs.

Catch you later,
Dave

PS: My first wife was a PhD in psychology, hence the slightly passive aggressive behavior. Sorry.

 

[iusedtoberich]

...snip....
If they saw me putting stickers on their poor equipment, they would have a fit. They're funny that way. The owner and mngr of one room, and the mngr of the other room know nothing about pool. He almost barred one guy for laying his cue on the table! (they lightened up on that one now, thank goodness!)

Funny, I can relate to this. I was doing drills years ago in a player's room, but the owner's wife was a banger and knew nothing about the game. She saw me putting the donuts on the table and almost had a heart attack. I had to show her it was ok for the cloth, and left no residue behind.

 

[iusedtoberich]

I've never seen YouTube videos take this long to process either. Does anybody know why this is happening? Is it because these posters are new Youtube users, requiring the videos to be checked more thoroughly?

Thanks,
Dave

I've spent the last 6 months fooling around with 3 different cameras, different codecs, youtube and vimeo accounts, editing software, 2 different computers, etc, all to be able to quickly and easily record pool practice for both local computer archiving, and uploading to youtube/vimeo. I'm still not there.

Video has an extremely tough learning curve, is what I've concluded from this!

 

[dr_dave]

I've spent the last 6 months fooling around with 3 different cameras, different codecs, youtube and vimeo accounts, editing software, 2 different computers, etc, all to be able to quickly and easily record pool practice for both local computer archiving, and uploading to youtube/vimeo. I'm still not there.

Well this is your chance to finally get "there."

Keep it simple ... stick with one camera, don't worry about codecs, use YouTube, don't edit the video. It doesn't need to be perfect.

And if you can't or don't want to post videos, please share your scores anyway.

Good luck,
Dave

 

[iusedtoberich]

As far as having the data, don't discount the fact that a number of prominent instructors had a hand in this being what it is. I would say they have a lot of "data" to go off of.

Actually, I did do a fairly large number of tests with players over varying levels of playing ability. Several of the BU founding professors also did some tests. The BU ratings seemed to be appropriate for all the players we tested.

However, we do plan to tweak the scoring and rating system over time as necessary as more data comes in. On the last page of the diploma application form, we are collecting data to help us better correlate the BU scores and ratings to actual levels of play.

Also, I was hoping more AZB members would try the exams, post their scores, and comment on how well they think the scoring and rating system worked to characterize their level of play. Hopefully, some of you will do this. The more data and feedback we have, the better.



I think that is common. That's why one should practice and keep records while working on the exams, and video record all of the attempts if a camera is available. That way, the person can submit their best performances for the official diploma application.


Are you telling me you have all of these opinions about the BU exams and rating system, and you haven't even tried the exams yet? I suspect you're not the only person in this category. Would it help if I asked nicer? Would you please try the exams ... pretty, pretty please?


The camera angles make it look worse than it actually is. My view of the OB is always well beneath the top rim, but I appreciate the advice. I've tried my contacts playing pool, and I don't like them. I've also thought about getting Lasik.

Thanks for your input and suggestions,
Dave

I know you have many good instructors helping devise the tests. But what were the actual scores and data used to make the determination that scores xyz = Touring pro, and score ijk=semi pro. Thats what I was getting at. Did you have "touring pro's" such as Stevie Moore, Corey, Hunter, Rodney, etc, do the test? And did you have "shortstops/Open" like maybe a Matt Krah or Eddie Abraham (local to me) or the top local guys in your area do the test to come up with a "semi-pro" rating? Thats what I mean.

I agree that the better the player, the higher the rating, in general. What I still believe is a hypothesis on your part is where the numbers correspond with known players ability. If you shared the tested data and showed how you formed your conclusion, I might change this opinion. But what you have publicly shared thus far is a hypothesis, with no supported [public] data. I know I'm being anal, but that is because I'm holding you to a higher standard of the Scientific Method than the "tribal knowledge" that is what we have in most pool discussions.

Regarding the glasses, you are definitely better off than most people that wear "normal" glasses, I believe due to your head position in your stance. The tell tale signs of an issue:

1. adjusting your frame before shots
2. flexing your face muscles/nose/eyebrows/forehead while down on the shot in order to move the glasses higher subconsciously while shooting.

You do #1 on several shots, and your forehead muscles look a bit tight on some, but not all shots. Definitely not as severe as most people with glasses, but noticeable if you know what to look for.

I know this cause I had 4 pairs of glasses made just for pool, and see others with glasses having the exact same issues. I'm too scared for contacts or lasik messing up my eyes, so I wear pool glasses. They really improved what I can see while down on the shot. And before I got them, I didn't think I had any issue whatsoever. But as soon as I got them, everything looked different, and it even afforded me more flexibility with my stance.

Anyway, obviously you can do whatever you like with regard to glasses, just sharing some info

Since you asked pretty please with a cherry on top, I'm looking forward to doing the tests. I plan on doing a video, with no practice on the drills. See what I score. Then practice them a bit and try it again later. I put myself at a high C player, that when I'm at my best, I get into the B range. I base this on 15 years of gambling with all level players in the Philladelphia region, which has players that range from D to Corey.

I'll try to get the test done sooner rather than later. I'm eager to try it

 

[iusedtoberich]

...snip...

Also more weight should be put on potting in the fundamental section. Cut down to 2 shots each at the targets and 10 more pots. You can miss that target paper by a couple inches and still run out 99% of the time but when you miss a ball it's game over for you.



JC

I have some input on this. Again, I'm an intermediate player C+range to B- on my better days.

Years ago I wanted to really see what was the cause of ending my runouts. So what I did was devise a drill where I attempted to run out 10 times each a random spread of 2 balls, 3 balls, 4 balls, 5 balls, 6 balls, 7 balls, 8 balls, and finally 9 balls. I recorded for each attempt (80 attempts) what happened to end the run:

Run out, missed shot, or scratch.

If it was a missed shot, what was the cause. (for example easy shot I should have made 8 out of 10 times, or a hard shot that I got bad on and wasn't likely to make).

My data showed overwhelmingly that the cause of ended runs was due to poor control of the CB leading to very hard or impossible shots or scratches. This showed up exponentially more when more balls were on the table. I'd end up snookered, and forced to kick. Or I'd end up with a near impossible shot that a pro wouldn't even be favored to make 70% of the time.

Another set of evidence I have is I'm a gambler. At my level (C+ to B-), I've beaten the local open level players for thousands of dollars (I'm not exaggerating), with them giving me the handspan. This means I can basically move the CB about a 6" radius any shot. With my overall game level and shot making ability, this handicap allowed me to play at literally a Pro level, as I beat all these guys with this spot.

Those are my experiences that give me great evidence to believe that position play is by far the most important aspect of our great game. Your mileage may vary

 

[iusedtoberich]

FYI to those using the BU Exams, we just posted a spreadsheet on the BU website to automate exam scoring and player rating. It also helps keep historical records to track your improvement over time. If you are interested, here's the direct link.

It's also now available along with all of the other documents on the BU Exam Resources page.

Enjoy,
Dave

PS: The original version of this spreadsheet was created by Patrick Johnson, a BU Instructor. For those who don't know, PJ is currently banned from AZB (for a really long time) because he wasn't a good boy (apparently). Regardless, I appreciate his spreadsheet skills and willingness to share with others.

Thanks PJ for the spreadsheet. I checked it out and found it works fairly well. I plan on using it for my testing, and can offer any feedback on the spreadsheet after using it more.

 

[iusedtoberich]

Whew, I'm all caught up on the thread Thanks to Dr Dave and the other instructors for setting this up and everyone else for contributing to the discussion. I have enjoyed it thus far. I might be able to do a trial run myself tomorrow...

 

[Pathetic Shark]

PS: The original version of this spreadsheet was created by Patrick Johnson, a BU Instructor. For those who don't know, PJ is currently banned from AZB (for a really long time) because he wasn't a good boy (apparently). Regardless, I appreciate his spreadsheet skills and willingness to share with others.

He really went off on 'tilt' apparently, and committed forum suicide. I know this because some of the forum's uber right wing freedom lovers have said so.

 

[justadub]

He really went off on 'tilt' apparently, and committed forum suicide. I know this because some of the forum's uber right wing freedom lovers have said so.

...................................................

 

[Autist]

Hello.

Dr Dave,

I wanted to thank you for all the hard work you have done towards billiards community. You are awesome. We are lucky to have you, really.

Billiard University is simply awesome.

I am a lower level player who has been playing for about 3 years, not so much during the last year, tho. And because you wanted some input from lower level players, here is my contribution. Only a simple version of what I have done so far, but I intend to videotape myself soon, and maybe a good friend of mine who is a higher level player compared to me.

First of all, I did the Exam I.
I did it on a 8 foot table at my work. I had been working for 10 hours, had a suite and a tie, and a house cue. I only scored 43.
The next day, I went to our regular pool hall with a friend, played with my own cue and on a 9 foot table. I scored 57, which was very surprising. My friend, who is atleast 2 levels better player than me(on a scale of 1-10), scored actually lower than me. I believe that this shows how much it matters when you have taken the exam before.
For the record, my honest opinion is that playing against ghost, 9 ball, 9 foot table, I can probably win a maximum of ~20 games out of 100 games.

Oh, and to add, I have always played on a 9 foot table. Comfy dress code, my own cue, and rested+familiar table probably resulted in a better score.

The table was a Brunswick Metro with what you would call bucket pockets, but the shelf is pretty deep and some have said the pockets are cut "funny", which makes it spit out balls hit from rail in a higher speed much more frequently than I have experienced on any other table.

Sorry that I don't have more input at the moment, but I really do intent to do the full exam on a 9 foot table and record it, as soon as I get a small break from work.

Cheers, and thank you again!

- Margus R.

 

[BilliardsAbout]

Thank you, Dr. Dave, for all the hard work and effort. It's a student's delight to see all the fine instructors who are now the esteemed and tenured faculty at BU. I'm also pleased that position skills are the emphasis above shot making on exams, as that is what is needed to take most students to the next level. Soon BU will help all those PSOL (Pool for Speakers of Other Language) students out there...

I've already had students ask me about taking exams and about the DVDs for training. I'm available in Florida/Georgia and also to travel further afield to proctor exams for a diploma/video students/assist you in choices to score as high as possible on examinations.

Suggestions for a BU mascot, fight song and colors? So far, the colors look to be blue and orange, which I'm comfortable with from my alma mater, UF.

 

[JoeW]

Hey dr_JoeW, fellow psychologist dr_brussell here. (And ribbing dr_dave a bit. )

Welcome to the forums. There are many well educated people here (check out Mike Page) and a few psychs who are out of the closet :thumbup: I was surprised to find so many computer programmers. Seems like college where most of the guys at the computer shop also rode motorcycles (Zen and the art of motorcycles and all that). One day I found the president of our university (who had asthma) discussing the virtues of Harley's and Hondas, he thought his Honda was far superior - strange world. Oh yeah, My 650 Yamaha was a much better bike

And engineers are over represented but then we all know they are OCD cases

On another note.

Dave, as evidenced by the previous post (# 217) I think it would be good to include previous pool playing experience in you demographic data collection.

1. How long have you been playing pool for more then a two hours a week?
2. In the last six months, how may hours do you usually play in a week?
3. How many hours have you spent with Exam X before submitting your score?

I think these will be significant variables in any analysis you conduct. I am sure that you realize that partial data sets could be used to determine the relationships of these types of variables to the examination process.

I would expect that prior experience and length of time training to take the exam are related to exam score. You could derive a table for players recommending the optimal amount of time a player should spend with the exam before submitting a score. Heterogeneity in scores is good for validity but reliability should rule the day for a commercially acceptable product..

Warm regards,
Joe

 

[Eric.]

Well, Dave, I will say this... it does look like you've improved a decent amount since a few years ago, when you were touting the Pool instructor/billiards authority role, writing instructional articles and... you couldn't play better than rank novice level. I guess you no longer need to duck us when asked to put up a video of yourself playing, lol. Looking at your old videos, I found it laughable to have you pontificate when you hadn't even achieved a modest amateur playing level, yet.

I feel that a score of 135 shouldn't equate to an "A" level player. If you scored 135, but were weak in your safety play and you were very weak in banking...that is not an A player. An "A" player can execute fairly well, at all aspects of the game. an A player lacks the level of consistency of better players, though.

To give some constructive feedback, i feel that your exams, while good and comprehensive, need one more thing; the "put-it-all-together" test. You should add in the Joe Tucker 10 ball ghost rating test as your "final exam". Why? Well, drills are good for demonstrating ability and building skill thru repetition, BUT... doing the same exact set-up drills over and over only shows that you've mastered that particular set up shot. After shooting that set up shot a few dozen times, you should have it down pretty good. Thing is, how many times will you get that exact shot in a match? More likely, it will be a slight variation of it. That's why the Joe Tucker 10 ball drill adds in some "real life" situations due to the open break and randomness of ball layouts. It forces the student to adjust their play a bit, when they don't get the exact layout of the old, familiar drill shots.

Long story, longer...I like the program that your group put together. I feel that the ratings are a lil high i.e. an "A" should be more like a "B", but i know this is subjective. Again, if you added the Joe Tucker 10 Ball Ghost rating drill as your final exam, i think it would be a more thorough test. Perhaps, you can find a way to incorporate the JT 10 Ball scores into your final tallies.


Eric

 

[dr_dave]

I know you have many good instructors helping devise the tests. But what were the actual scores and data used to make the determination that scores xyz = Touring pro, and score ijk=semi pro. Thats what I was getting at. Did you have "touring pro's" such as Stevie Moore, Corey, Hunter, Rodney, etc, do the test?

No. I still haven't tested anybody at the top end of the rating scale, so I don't know how good the higher score ranges are.

And did you have "shortstops/Open" like maybe a Matt Krah or Eddie Abraham (local to me) or the top local guys in your area do the test to come up with a "semi-pro" rating?

The only one so far is Gerry Williams (who claims he is close to the "semi-pro" level), but I have a couple of top-player (semi-pro and pro) tests planned for the near future.

Also, hopefully, CJ and other top players on AZB will post their results and videos.

When I get some data in on top players, I might need to tweak the upper-end score ranges or interpretations slightly, but I don't know yet.

I agree that the better the player, the higher the rating, in general. What I still believe is a hypothesis on your part is where the numbers correspond with known players ability. If you shared the tested data and showed how you formed your conclusion, I might change this opinion. But what you have publicly shared thus far is a hypothesis, with no supported [public] data. I know I'm being anal, but that is because I'm holding you to a higher standard of the Scientific Method than the "tribal knowledge" that is what we have in most pool discussions.

We have seen lots of "data" in this thread already ... all of which seems to support the BU Ratings fairly well. Every player I have tested and known personally has also fallen where they should (although, I did make quite a few tweaks in the original score ranges based on the results of early testing). Regardless, the BU Rating score ranges are now set. And I plan to collect much more data as people apply for official diplomas (see the last page of the diploma application form). This data will be used to correlate to the more-traditional player rating categories. This won't change the BU Rating system, but it might change how I list the comparisons and interpretations in the BU Rating Comparison Chart.

Regards,
Dave