In the end neither is mentally challenging, it is the middle that can get interesting! When asking which is the mentally toughest of the two I would pick pool, one pocket in particular. You have an infinite number of moves, so does your opponent. A really strong opponent will surprise you over and over taking you out of your game.
Chess you may have three or four positions you are trying to develop simultaneously while watching a few more situations your opponent is trying to develop. Most situations, you are both working on. It is largely a matter of timing, one leads the other follows. The leader generally comes out best unless the follower can make a move with multiple potentials, seizing the lead.
Playing one pocket with a stranger I somewhat absentmindedly let him outmove me quite badly in a game. Two shots on my part reversed that. That is so unlikely in chess that you simply concede.
The freewheeling potential of pool makes it harder. The restrictions of chess make it harder. Oddly enough after a few years of playing a lot of chess I found checkers to be harder than either one! I was usually playing chess three to five moves ahead, every piece on the board. I beat far better players simply because they would get five to seven moves ahead and fall into considering logical moves. I played an illogical game even if it cost me a point or two here and there to be able to keep them a little off balance. The issue with checkers, now I was playing endgame almost from the first move! That was a brain cruncher to put the way I played chess or pool to shame. The very limitations of checkers makes it an extremely difficult game to play.
One thing of interest, pool and chess started off as war games, teaching future military leaders strategy. I suspect checkers did too but I don't know that for sure.
Hu