I've tried 3 different brushes. One had nylon bristles, worked ok-ish but not great. 2 different brands (hell, they are all unbranded so who knows who makes them) of horse hair. They worked but shed like hell, especially at first.
The Simonis X1 works quite well for taking chalk dust off the cloth, but won't touch any actual dirt or pet hair. It's expensive but it does a good job for manually removing chalk dust.
I found an old fabric lint brush in a thrift store, the ones they used before the sticky rollers. Like
this. I'm not endorsing that site, could be a total scam, just had a picture similar to the type I use. This works VERY well at pet hair or stubborn little lint and small pieces of crud. It's not a thing you would do the entire job with, but makes a great last pass.
As philly said, a vacuum cleaner works well. You don't want one with a rotating brush, you want to use the attachments for cleaning upholstery. The hand held Orecks work well and you can find them for a reasonable price used. Just be sure that the vacuum isn't super powerful or you can stretch the cloth. Don't just use the circular tube on the hose, use the attachments. The Oreck also has a nice crack tool that can get under the rails.
Ideally you would use a brush and vacuum once in a while, but the vacuum is so much easier than a brush that I'd rather use that.