Nothing wrong with listening to some music if you want to, I think most people only get annoyed when someone is blasting music over a speaker rather than using headphones (which you aren’t doing). Will the music distract you? Maybe, but only you can answer that for yourself. I prefer the quiet and listening to my wheels on the trail, so I don’t listen to music. I also listen for rattlesnakes, so music would be a distraction in that regards. I think complacency on home trails, when we don’t give our full focus to the trail because we’ve ridden that trail numerous times, is probably more hazardous than having music play in the background. At least it seems like I crash most often on trails I’ve ridden numerous times and aren’t fully focused on for whatever reason that day.