Sounds like he conceded to me, and if someone else heard him say that, I would say that it was right to pick them up and rerack. If no one else heard it, I would for sure double check with him first to make sure that's what he meant. Should you have to do that, no, but some people are that way.
I had a situation happen to me last year where it looked as though my opponent fouled. he went over to the table where they were sitting at and I was already getting up as I was positive he fouled. I then, before picking the ball up, looked at him and said, ball in hand right? He just looked at me like I wasn't there and never said yes or no. I picked up the cue ball and all hell breaks loose with him saying what are you doing, there was no foul. Some pool players are just jerks, no two ways about it.