Hello Tarzan,
Over what frequency range?
The noise and distortion introduced by the driver may not be the most important selection criteria if there is already higher noise and distortion in your single-ended signal. But for now, let's assume you can get a perfect noiseless sine wave as your single-ended input.
For low frequencies, the distortion should be similar. A 1:1 comparison can be made there because they both specify distortion at 10kHz and 2Vp-p, where AD8476 has the slight edge, but both are in the 120dB to 130dB range for HD2 and HD3. However, the ADA4940 has a much wider bandwidth and higher slew rate, and will therefore have better distortion at high frequencies.
I believe that you will find that you can get lower noise with ADA4940. AD8476 is designed for very low power and high accuracy, hence the relatively large 10k integrated resistors which help reduce loading power, but also add noise as you noticed. These resistors are trimmed to a very tight tolerance so the AD8476 can specify very good gain accuracy and common-mode rejection ratio. ADA4940 allows you to choose smaller resistors and reduce noise at the cost of higher current to drive those resistors. The noise of those resistors as well as the current noise introduced must be added to the data sheet specification to get the full number.
I think you will also find that our DiffAmpCalc tool will help you. It includes both of these products, and will calculate noise and power and estimate distortion for your signal frequency and circuit bandwidth.
I hope this helps.
Best regards,
Scott