That's pool, man. Your physical shaping matters very little when dealing with the matters of mental stability and toughness. Your mind is telling you , "Hey man! I gotta perform here! Let's do this!" and it's making your body react in a certain way because of the natural chemicals released by your brain. That is what happens when we become emotional about something. You must learn to control your mind.
Remedy #1: Martial arts / meditation is the best way to learn how to control this. As you progress, you will slowly become more and more confident in your ability to focus and block out distractions.
Remedy #2: Play some $5 a game, or even $2-3 a game on a regular basis with different people. Make sure you adjust weight accordingly. Take a comfortable amount of money aside for this month, let's say $100 and decide that's your gambling money. If you lose it, wait until next month. If you win, keep that in a separate fund labeled your 'bank roll'. Use that as a goal to see how large you can get your bank roll to be. Over time, you will see that your desire to 'beat the other guy' will overcome your 'fear of losing'.
Just a side note: Ultimately in the martial arts, it's not about fighting. It's about overcoming the fears which strangle your potential to find true happiness in your life.