Some updates:
So the issue seems to be that i was not on a high enough RPM when switching towards the finer polishing compounds.. *d'oh*
My lathe is on the verge of death & didnt want to push it, but i guess i have no choice now, but to max it.
Results could be better, will continue working...