Not in my experience on a long wheelbase (touring style) Schwinn circa mid 80's with SKS fenders and the steel Wald rear rack and a DIY aluminum front Porteur rack. It's the load that makes the ride different. So I switched out the old 27" (not quite round) wheels for 700C's and cassette and 35 mm instead of 28 mm tires. Not as sprightly when empty but way more comfortable and better handling loaded.
I suspect that if you went the full touring racks and 70+ pund loads and the frame was not built to take such loads, you'd be in trouble.
YMMV.