If you are planning on racing, you can build a lightweight competitive hardtail for much less than a FS bike. To go 22lbs or so for a hardtail is pretty easy, with a chinese carbon frame you can do this easily for around $2K with a suspension fork and smart choices (lightweight saddle, seatpost off of amazon, etc), maybe a smidge more for a better fork. To do the same with a quality FS frame that will be light, efficient, work well, etc, will cost more like 5-6K at a minimum. So the entry-cost is a lot higher with the FS bike. Most people do not go "all out" and spend that kind of money on an FS bike. Sure, some do, but most people end up with a more "middle of the road" bike, which makes a bigger chasm between the hardtail and FS bike, since the FS now weighs significantly more, probably has heavier wheels, may not come with the best shock that would allow for better pedaling or suspension, and so on. Said bike is not a *bad* bike, but if your goal is simply to go fast, it's probably not the fastest, but it probably will be more comfortable and handle much better over rough terrain/inspire confidence.
No XC race is won on the downhills. A master-downhiller may come in a few seconds ahead on a specific downhill, but a marginally more fit climber will make that up easily and more on the climbs and level stuff. FS bikes are great and there are some that are absolutely worthy of XC racing. The entry cost is high though if you want to outfit them as competitively as a hardtail. You can go amazingly fast by "holding on for dear life".
And I agree with that's been said, a smooth dirt trail can even seem to favor rigid, whereas a very rough one will usually favor more suspension. Just a suspension fork can make a world of difference though on something that is a little rough for rigid but doesn't really require FS.