The sag could miraculously increase a few minutes after you left home...

Last winter I went out into temperatures up to -10°C. This year I only went trekking overnight when it suddenly turned -15°C whereas the weather report still said -5°C. Generally, it´s not so much a problem for the bike, although the suspension might feel a little odd and slow.
When the ground is frozen stiff in these temperatures, you´ll get the best grip ever during the year. It´s just plain fun to ride like this.