Thank you for your answer.
I'm using the Agilent E8257D signal generator as my external LO, which has pretty low phase noise (better than -150dBc/Hz at 5MHz offset at carrier frequencies up to 10GHz). I set the external LO freq to x2 the desired Rx frequency and the level was set to 0 dBm.
I will measure the Tx RFPLL phase noise as you suggested to see what kind of phase noise it has.
I wonder - maybe the issue I'm seeing is caused by some kind of saturation / compression of the receiver chain components ?
Best regards,
Oleg C.