I had the exact same experience with a
Trek 3500 about 5 months ago. I came back to mountain biking from a 17 year lay off (no suspension fork on my '91 Bridgestone MB-4) and went to my LBS to get an inexpensive name brand bike. I didn't want to spend a ton of money only to realize I wasn't going to use the bike much (there was also the issue of convincing my wife to spend more on a bike. . .). I got a new sales clerk at the LBS who pulled out a
Trek 3500 and didn't tell me squat about what I was looking at despite my questions. He "fit" me for the 18" (I'm 5'9" with a fairly short inseam) by having me sit on the 16 and then the 18. I told him the 18 felt too big but he said I'd get used to it. I never did. It always felt too big and I was in constant fear that I was going to rack myself due to a too high stand-over.
That left me with a bad taste in my mouth for my LBS (I've had other "bike snob" issues with them since). I also realized that the components on the 3500 weren't going to stand up well to moderately aggresive trail riding (you get what you pay for, although I loved the black and green color scheme). I sold it for a slight loss on Craigslist and bought a slightly more expensive 17" bike online. I spent some time comparing the geometry of the bikes on paper first, though, to make sure the new bike was shorter in critical areas (but not too short).
The 17" bike feels much better. And, since I feel comfortable with the frame and have decided I will continue to ride, I've spent money upgrading the components to my own liking (it's also easier to fly my rather pricey build-up under the radar). Of course, since I didn't get the chance to try out the online bike first, there was some luck involved with the fit. I also like to do my own work on the bike, which made the online route less daunting.
What's the point? The 3500 may be the wrong bike for you in general because you are between sizes. You might want to ask your LBS to let you test ride a comparable bike from another manufacturer to see if you can find a better match. If that doesn't work, you might want to consider returning the bike and going to another shop or the online route. If you consider online, make sure to read a few of the online bike purchase threads here first, though, so you get a sense of the pros and cons.