when should you call out LBS on their mistake?

Yes they should have caught it but since the cause of the problem is outside of the normal troubleshooting procedure I'd let it slide once. When the bike came back with the same problem that should have been a clue that they needed to give it a much more thorough working over.
I've had two somewhat similiar problems. The shop where I bought my full squish is mostly a roadie shop and though I don't regret buying the bike (got an 08 on closeout for a damn good price) I'm done with them otherwise. They don't know there way around a mountain bike past the basics. Talked all kinds of trash about disc brakes and while yes I agree that they aren't necessary here in Florida for 99% of the riding that most people do that's a big jump from saying that they're a waste and not worthwhile. They're definitely a worthwhile upgrade if you're going to be riding in non-pristine conditions. The XT V's on my hardrock or the Avid Shorties on my cross bike work just fine but if the rim is wet or muddy they take a rotation or two to clean the rim off enough to grip whereas the Juicies on my HiFi just work immediately every time. That's worth it to me. Here's the issues that have made me say goodbye to that shop:
1) Rear disc dragged excessively whenever I'd hit a puddle. It was like I just dumped three or four cogs whenever the rear rotor got splashed. Took it in and they try to tell me I ****ed it up by bumping the brake lever when I changed the tires to a tread more appropriate for the terrain. A) I had the bike in a stand when I dropped the wheels. B) It dragged before I changed the tire. That incident deprived me of my bike for a weekend while they bled the brakes and changed the pads. Doesn't seem to drag anymore but the rear doesn't modulate nearly as well either. Charged me for new pads and $20 labor which they told me was a great deal because the tech spent hours bleeding the rear brake without the proper bleed kit. Doesn't sound like a deal to me. Why would a shop not have a bleed kit for Juicies? It's not a high dollar item and it's pretty darn useful. I'm probably going to order a kit and re-bleed the rear to see if I can get the modulation back. If that doesn't work or the dragging comes back it's going to a different shop that knows discs better.
2) The chain was too long which was giving me massive chain suck and chain slap issues, yet they try tell me that it was fine and I made a mistake by shortening it. The way I see it, if it can still hit big-big with the suspension fully collapsed and before that it had no chain tension at all for half the cassette when in the inner ring then I don't think I've made a mistake shortening it. Yeah the rear derailleur is stretched to within a tiny margin of it's maximum extension like that but I don't think it's likely that I'll find myself in big-big in any situation that would use that much travel. The chain suck caused by their supposed 'correct length' chain CRASHED me, TWICE, because the crank arms freezing solid from a jammed chain tends to throw my balance all to hell and the chainstay on my new bike is all scarred up now. I think I'd rather have the shorter chain that has been totally trouble-free since I shortened it than their length which they claim will last longer. I guess the chain would last a really long time if I were to crash again and break myself or the bike.