I did some more tweaks, and I've gotten it working with scconfigmgr modern driver management.
So before applying the correct upgrade image, it downloads and stages the proper driver pack for the computer model, and during the upgrade, it will also install the current drivers for the model.
Pretty slick solution
-Jannis