After the birth of my first child, I took eight years off until 2006. I went from about 3k miles a year to about 50. I'm still not able to find the time to ride like I used to, but I do get one or two rides a week in for about 400 miles a season.
I found the first year was the most difficult. I was starting from nothing aerobically speaking. It took me about 6 weeks before I could even have fun on my bike like I used to. (Gotta go fast for more then 200 yards to have fun.

)
The second year I could tell I had an improved starting point after the winter layoff. I was only slightly faster over my 14 mile course I use as my fitness gauge by the end of the summer. (Speed on test course = 14 mph)
This year I'm only part way into the summer and am already 1 mph faster over the course I gauge my fitness with. The big difference this summer is I've been riding my SS bike almost exclusively on a tight short course with lots of acceleration required. I think the added work of the SS has forced me into better fitness. (Current speed on test course = 15 mph)
For comparison on similar, though slighly less hilly, course when I was 19 I'd average around 17-18 mph. As much as I'd like to, I doubt I'll ever reach that level of fitness again. I just don't have enough time to train and I'm almost 20 years older.