The HMC703 RF input pin can handle 8GHz signals but the internal prescaler is limited to 4G. This means we have to enable a divide-by 2 in front of the prescaler by setting Reg08h[17]=1. Now the PLL operating frequencies are set to half the VCO frequency:
F0=7900MHz/2=3950MHz
F1=8100MHz/2=4050MHz
With a 50MHz reference this requires N=79 (0x4F) for F0 and N=81 (0x51) for F1. I've made these changes to the attached register file. Your version was essentially correct but you did not enable the internal divide-by 2.