Hello Oliver,
Perhaps you could try throttling down the AD8475's bandwidth with a cap directly across the ADC differential input (after the 47 ohm resistors). 0.1uF might be a good starting point. The AD8475 is built for 4 MS/s @ 18 bit operation, so it may allow through some high-frequency noise that your ADC can't handle. The LTC2440 has a neat resolution vs. speed tradeoff feature, but ADCs are not always 100% resistant to out-of-band noise.
Best regards,
Bob