Definatly would have to say course is important... A good course will make or break a race, if it's good they will come back, if the course is crap then Next time around no one will be there

Also no point making a course too difficult, you want tough, but not something that is unrideable to all those that aren't Elite riders.
Organization, Atmosphere, Facilities: are all Important... It means you can take the S/O or the family with you and they wont be too bored. Also showers and such are a great help to riders 12 or 24hrs of dust can be quiet disgusting. Organization is usefull, it helps to know what is going on, a well run race with a decent course are usually those that are remembered and talked about.
Cost: This is always important, not everyone can afford to fork out say $400 just on race entry... Simple fact is not many will pay that much for race entry.
Time of Year: Here in Australia we take it as it comes... At an 8hr during Feb (summer) it was an all round mud fest, the weekend after at a 6hr it was hot and dry conditions. Obviously if it snows, or is rather cold during winter or very hot in summer, that may not be the best time to hold the race.
Famous Racers: Really who gives a **** if I'm passing or being passed, Josh Street, Tinker, or who ever else....
Hope that helps some
-Cul