Ivory is rather stiff, so might want to balance it with a bit softer tip. Go with medium or medium soft and use a layered tip to lessen the stretching load onto the tip (mushrooming means stress to the ferrule).
Also Ivory is a natural material, so it's imperfect. You might want to protect it a bit more by using a Kamui clear (for the pad below the leather) or use a carbon fiber pad below the tip (like OB does).
Cheers!