I use a #00 all the way up to a #6.
Depends on what I'm drilling the center for.
I've cut 1.5 squares to round between centers with a #00 on both ends with minimal pressure and have never had problem.
I suspect you may have forgotten to snug something up.