It all depends what works for you. I'm almost exactly the same size as you (but with added weight). I ride a Large SJ FSR Expert. The riding I do is tight single track. Oftentimes they're dirt bike paths. I'd feel a bit uneasy with a XL between my legs.
On short road trips the bike does seem a bit short between the saddle and bars, but that goes away when things get tight in the woods.
On short road trips the bike does seem a bit short between the saddle and bars, but that goes away when things get tight in the woods.