The AD9984A does not have a color space converter, it can not convert from RGB to YCbCr. Since the input is VGA and VGA is RGB the AD9984 will output RGB data. The FPGA must convert RGB444 to YCbCr422. The color issue is because the input is RGB444 which you are mapping to 422, probably just mapping G to Y and dropping R & B completely.
analog VGA --> AD9984 --> RGB444 (24-bit) --> FPGA --> YCbCr 422 (16-bit) --> ADV7511 --> monitor