There are may reasons for you to carry ID at all times. As an example, if you go out alone and have a serious health incident (stroke, heart attack, seizure, etc.) and are rushed to a hospital, how will they identify you? If they can't identify you, how will they pull your medical records and history in tying to determine what course of treatment is safe? How will they notify your loved ones that you've been hospitalized? In that scenario, failing to carry ID could cost you your life.
As already noted in this thread, some places card you no matter what you look like. Several years back, I was asked for proof of age when I ordered a drink in the bar at the airport in Charlotte, NC. The thing is that I was 58!