Use a dry Magic Eraser. I use em on my shafts to clean em up so I'm sure it'll clean up a ferrule on an OB shaft. I've heard of ppl saying to wet it, but I use it dry and it works just fine. I just used one to clean up a shaft I've had for 12 years or more for the first time and it cleaned it up really good.
*edit* I need to remind myself to buy some more of those. When you use em dry they wear out pretty darn quick.