I would leave it on if I were you.
There is no difference in speed of the Cue Ball on a break shot with phenolic vs leather. Even very soft tips.
I think Dr. Dave did a test on this and the difference between a phenolic and a super soft tip was maybe 0.5%.