Hi Andoni,
I have simulated the AD9517 IBIS model in ADS but not in hyperlynx. I would expect the IBIS model to work fine with ac coupling capacitors. One issue may be if the ac coupling cap is an ideal model and you have them placed before the termination resistor. The modeling software will likely see that there is no dc path to ground and show that there is an error.
Can you paste a screen shot of your simulation configuration?
What issues are you seeing at 2.8GHz? The model should work at 2.8GHz as well. I would not expect the waveform to look square at this speed.
Regards,
Kyle