Until this year, my only wheeled transportation was a fat bike. Last year, I did 2500 miles. For every 10 miles of single track i'd ride, I'd have 12-20 miles of pavement to cover. It's doable and really not that bad, but you need to make some compromises to make it work. I suspect that one of the main reasons the naysayers are saying it will suck, is that many of these guys ride their fat bikes with pretty low tire pressure, anywhere from 5-9 psi. At those pressures, it certainly does suck, as it creates a lot of friction and rolling resistance. The tires will exhibit a lot of self steer and it will be tiresome to pedal. However, increasing the pressure to 12-14 psi will largely alleviate these issues- I've taken roadies on my fat bike before- it's still a chore to pedal, but it can be done.
The problem then, becomes what to do when you get to the trails: you can leave them high and your bike becomes essentially a rigid 29er, with wide heavy wheels. you still have gobs of traction, albeit not as much with lower pressures and it kinda beats the heck out of you. Or, you can lower the pressure for the trails for even more traction and a cushy ride. To get home, you either need to suck it up in ride with semi flat tires over pavement, which sucks balls, or you can pump the tires back up, but even with the best mini pumps, this also sucks donkey balls. a third compromise, is to get a suspension fork and ride with about 10-12 psi all the time. I mostly do this, all though occasionally, I will lower the pressure on the rear, especially if i'm riding over rough terrain. the big upshot to riding a fat bike, is that it will ultimately make you hella strong. I've done metric centuries on mine and when I switch to my road or SSCX bikes, they feel like rockets!