Those bearings are dandy for farm equipment or off road vehicles, but for fine or semi-fine woodwork? Those set screws are intended to hold your shaft key in place or bore a bit into the side of the shaft. If you screw them into the plastic collet, you may end up with a ding in the collet, and little else. If put in softly to avoid drilling the plastic, they may back themselves out.
Are you using one pillow block bearing or two? If you are using two, they should be self centering, although using a cue shaft to self center them could be risky, as it may take some force to get them out of their seating. If they self center, and the collets give a snug fit, that may be the best you can do with this system, and it may well be enough.
Your stated torque is consistent with about 0.33 HP at 1000 rpm, which is about what some prepackaged cue lathes come with.
Suggest you buy some half inch dowel and cheap tips to practice on.