Sure - cleaning the ferrule will help. But using chalk that is too deep and using the incorrect “drilling” method of chalking WILL matter. Sounds like you don’t understand the correct method of chalking, because drilling can scratch and will discolor the ferrule.
Good realistic post.
During the excessive/addictive play yrs, my play shafts (3) turned pure blue from of course, embedded MC & 30+ hrs of match play....weekly for decades.
I never/ever tried to remove/clean the blue/off, from within the grain along with my natural skin oil.
I felt, excessive cleaning too remove the blue, would open up the pores of the wood, too much....plus, why remove your hand oils....Oil helps cure/preserve wood.
I did wash my hands allot, and always put on hand cream excessively a day before an event, and always driving to one.
I never cleaned my ferrules, it's not something ''I'' ever look at when shooting, and it's not a distraction. I could care less.
If I did, I wasn't looking at the shot right, we all know what that does.
Have a great 4th.