If all-out speed is your goal, then this approach will have diminishing returns. How do I know? I used to ride a 45 pound DH bike. Yes, it was stable in the rough. However, even in a bike park where it is predominantly steep and, well, downhill, the direction changes are what kill you on those heavy bikes. When I ditched the Monster T fork and replaced it with a 3 pound lighter Marz. 66 RC2X my times immediately got faster. That brought the bike down to around 42 pounds. Additionally, I also found that the ridiculously burly Intense 2.7 tires, when replaced with lighter casing 2.5s also sped me up considerably. Fast forward a decade and I found myself on a 35 pound enduro bike, with better geometry, suspension that was just as good, and guess what, my times got remarkably better.
I've ridden my dad's E-bike too. I would guess it weighs around 45 pounds as well. Great geometry and suspension. Guess who wins on the downhills when we ride together?
I come from a moto background as well. Apples to oranges. Yes, a motocross/off-road dirt bike can plow through chunder like nothing else, yet it has the brakes and traction to shut it all down, and then immediately rocket out of corners via 50+ horsepower. On tight, technical mountain bike trails, a nimble enduro bike or even modern DH bike can often negotiate that kind of terrain faster. I know there's trails I ride where I wouldn't want to be on a moto at all -- it is far too tight for such a big machine.