I am assuming we are talking about (for a lack of words) that little "demple" in the middle of the ferrules that can sometimes be left after facing. If so I aggree with Fred, to make sure the tool is centering well, with the center of the ferrule.
Sometimes even having runnout in the shaft when cutting can cause simular problems, as well as not facing as well.
Also be sure that the tool your using is best suited for a facing operation. I have a few different types I use, so you can use other tools not made specifically for facing, but some require setting the tool post at the correct angle for the best cut.
Make sure your crosslide is going all the way through and not stopping right where the demple starts. that ones usually pretty obvious, but can cause the same thing.
If all else fails, It really depends on how thin the material that's left is, as to options on knocking it off. Sometimes if thin enough depending on what material, they will pop right of with ease, and can be done with your finger nail, but for the sake of keeping the oils from your hands off the glueing surface, I would opt for the tip of a utility knife.
sanding- altough there's ways of doing It without harming the face, would'nt be My first choise, or I would just use a rapid top sander. sometimes you can roll the sandpaper up and try to just knock the center off, but usually does'nt knock the demple off completely without getting into the good part, and compromising the rest of the face.
If it continues to be a problem for you, then you could take a very small center drill, and drill just deep & wide enough so when you face it won't be a problem. I stress that you don't won't to over do it though, just enough to keep the demple from showing up.
Hope this helps, Good Luck Greg