I guess there has to be some way to tell the algorithm how difficult the shot is
There actually is a way to tell the algorithm how difficult a shot is - you do it by playing the shot and posting your result.
Let me explain.
Every drill in Cueist has a difficulty rating. Initially, these ratings were calculated using our physics engine. To simplify it a bit, once we set up a shot, the algorithm would ‘play it out’ and calculate what we call the permissible error. Things like: if I hit this shot with a little bit of left English, will it still go, and how much is too much. Factors such as top or bottom spin, left or right English, speed, shot direction and cue elevation all feed into that initial rating.
Those ratings were pretty good - honestly better than I expected - but they weren’t perfect.
To improve this, we added another system on top. Every time a player completes a drill and posts the result, that result is used to adjust the drill’s difficulty.
So if people consistently fail a drill, that’s a strong signal that it was rated too low, and the system will gradually increase it. If a drill turns out to be easier than expected, the opposite happens.
There are a lot of drills in Cueist, so this is still an ongoing process. But players who’ve been with us from the start have already noticed that the ratings are becoming more and more accurate over time.