Given the same number of teeth on the chainrings, it'll be slightly harder to pedal. You'll have a little more clearance over obstacles, but you say no jumping/dropoffs, so maybe that's not much of an issue.
Usually cranks that small are used for people with short legs (at least in an XC scenario - dunno about DH, where the clearance may be paramount.)
Usually cranks that small are used for people with short legs (at least in an XC scenario - dunno about DH, where the clearance may be paramount.)