"uses the unused receiver" means that Tx Quad cal uses portions of the AD9364 receiver to carry out the calibration. No, there are not additional receivers on AD9364.
The issue you are seeing is most likely due to the part configuration. Do you "initiate a calibration outside of this function it fails to converge." as described in your initial post?
Tx Quad cal generates two test tones, ftest and 2xftest. The bandwidth you are setting, for example at 10MHz, is the RF Bandwidth. Baseband bandwidth is only half of that.
Does Tx Quad Cal pass if you initialize and configure the part with a lower bandwidth than 10MHz (i.e. 1MHz) using ad9361_internal_init function?