Have you loc-tite'd it? Did you use a torque wrench on it to reinstall?
My current generation swingarm has been trouble free. The first install/removal had the wedge stick on and spin out with the axle, but from then on I have used loc-tite on the treads and grease on the wedge. IMO this is about the least fussy of any pivot hardware I have used.