Another part of the problem that took a while to figure out is that for some reason I need my seat lower when riding flats. Not just a tiny bit from the difference in the height of the shoes and pedals, but lower than I've ever been used to. It's taken some getting used to.
Most people place their foot closer to the middle of their foot on flats as opposed to more under the ball of the foot with clipless. That can make a big difference for seat height.