Ulnar compression...
I'll echo some of the others here and state it's most definitely a problem with your Ulnar nerve. I rode rigid bikes for years and started developing the exact symptoms you've mentioned. I got some nice gel padded gloves and forgot about it until... it got worse, way worse. I play bass guitar professionally and started to really notice the pain and numbness when playing about six years ago. I thought it was CTS and tried every type of instrument position, stretching exercise, anti-inflamitory med, massage, etc... Nothing made it any better.
I finally wound up at a neurologists office. The diagnosis was Tardy Ulnar Palsey. Basically, my Ulnar nerve had been permanently damaged and I required surgery. To "drive home" the extent of the damage my "degree of loss of nerve conductivity" on a scale from 1-10 (one being the least amount of damage) I was at an 8. The muscles in my left hand had started to atrophy and are visibly different to this day. I'll add as well that it only hurt "sometimes" and being the "tough-guy" that I think myself to be led me not to bring it to anybodys attention as I didn't want to "complain". Big mistake on my part! Surgery stopped the problem from getting any worse but it also removed mtbing from my life for the past five years. It also took me months to be able to play music again.
My suggestion to you would be to bring it up to your doctor now before it gets any worse. The Specialized BG gloves will definitely help (I use them) but they will not fix the problem. Other things that might help would be to run your fork a little plusher (I went with the MX Pro w/ETA btw), tire pressure a little lower, shorter stem, riser bars, and use your abdomen and lower back to support your weight on the bike. I still feel you should consult a physician though and make sure you haven't already done irreparable damage. Like I said, mine only hurt "sometimes" and that was with permanent damage already done.