The game is hard enough with spotlessly clean balls. Hitting object balls and getting chalk on them just it makes it even harder. Really shouldn't do it. Pet peeve for me, i never do it.
Mark Wilson gives lessons and uses the object balls for practice. Bring it up with him