Are you seeing these delays regardless from the frequency step size?
There are in fact some more things related to frequency changes.
By default if you tune TX more than 100MHz apart it does a QUAL cal automatically - it's an option and can be turned off. Also if you cross some RX frequency boundaries - it'll swap the GAIN table - which are lot's of SPI transactions.
If you don't like that we need to add an option to force a specific GAIN table always.
-Michael