Hello Andrej,
If I understand correctly you are talking about my ADTRX1 and problem with unwanted side band rejection. 74HC4053...4066 are not new technology and there is big difference between IC from different producers. I had situation that in the same circuit rejection was going from 25-60 dB(MIL IC). I suggest you to improve linearity in AF band decreasing C84,29,30,31 from 47NF to 22 nF or 10nF. This solution will increase useful bandwidth and problems will be reduced.
VY 73/72 Tasa YU1LM/QRP