Brushing doesn't work at all. It just grinds the chalk and dirt into the cloth, and it ends up underneath the cloth.
First, vacuum good. There are a lot of alternatives.
Second, damp wipe with wash cloth, to get the rest of the surface chalk and dirt out. That also helps to keep the cloth nice and tight.
Third, wipe the rails and the pockets. Do you realize how much chalk and dirt gets there?
Fourth, get on your hands and knees. No, I'm not suggesting anything bad. Get on your hands and knees, and damp wipe all the chalk and dirt off the rails underneath the table. If not, it gets on the balls, then gets on the table, then gets on your hands, then gets on your shaft.
Clean everything.
All the best,
WW