I've actually played quite a bit of golf, and at first, I shared the same opinion as you do, but after hittings thousands of balls, playing almost daily for some period of time, I've come to the following conclusion. Golf is infinitely more difficult *physically*. However, I think there are more shot possibilities on a pool table. To play good golf, you really have to master a physical skill, which is why only 2%, or so, of the golfers out there can break 100. If you aren't a very good golfer, it means you struggle to hit the ball. On the other hand, there is pool. Most players, after playing for any length of time, can pocket balls fairly well. However, pocketing balls isn't enough to be all that good of a player. So, in contrast to golf, if you aren't a very good pool player, chances are it isn't because of a physical limitation. It is probably because you make mistakes in judgement, pattern selection, position, etc.
Think of this, if you could absolutely *master* 15 shots in golf, and you could hit the ball straight 90% of the time (which not even the pros do, but I'm speaking hypothetically here), you would be a HELL of a player. Probably PGA tour quality. However, try being a good shotmaker on a pool table with only 15 shots in your arsenal, and you'll go broke if you try to take your game out on the road.
So, to sum up my opinion here, I think if you take just the two games, not including a persons mechanics or anything else, just the games themselves, I think pool is the more difficult of the two. I don't know if "difficult" is the right word, but I do think that pool is the more "involved" of the two.