I really don't think so. I think we are looking at people of the highest moral character. We have to trust their own judgment and decision-making. There seems to have been an attempt to politicize the courts. We have to be careful. The courts are not there to basically do what politicians want. Parliament is sovereign. Parliament makes the laws. All the judges do is interpret the laws.
With respect to recusing yourself, if it's so outlandish, then of course there should be objections made, but generally speaking, the members of the Supreme Court are not eunuchs. They have opinions and views. Especially if it's something that was said before they even became... I think it's even more unlikely that they should be tempered or affected by that.