Sure, but if they're not debating that topic, why should it be an issue? Not that I think such a person belongs on the panel, but the reason they're not invited should be, as you said, that they have few or no qualifications to discuss either ethics or AI.
The fact that they don't agree with the others on LGBT rights should only be an issue if they insist on trying to espouse their unrelated viewpoints using the panel, in which case by all means kick them off.
Because you can’t look at these beliefs in a vacuum. Someone who’s against LGBTQ rights isn’t against it for the hell of it, they have some underlying reason such as religious ideology. I’m not saying there’s anything wrong with religion, but when you believe that a man and a man together is unnatural and a sin against nature, then I have to wonder what other backwards regressive medieval beliefs they hold as well.
(Note that I'm not trying to straw-man you here, because I know you're only talking about the specific instance of this panel, which I've already agree was a farce to begin with. However,...) I still think it's harmful to society as a whole to try to use someone's views on any given topic as an excuse to refuse to work with them on something unrelated.
Say, for instance, that you're asked by a supervisor to collaborate with a Muslim living in a middle eastern country on writing a piece of software. For the sake of argument, say that this particular person believed that the laws recently enacted in Brunei were in line with the commands of Islam, and therefore just. (For the sake of argument only--obviously not everybody in those categories is like this at all--but clearly at least some must be or we wouldn't have situations like this recent one.) Would it be right to refuse to work with this person on writing say, some python code, when the topic of "sharia law" is never going to be a factor? Furthermore, by your argument, would you be willing to trust this person's judgement on software architecture when they have exhibited a, in your view, extreme lack of good judgement in another field?
Second, you can even take this argument to a higher level. You admit that the opposing party has a reason to believe what they believe, and I'd imagine that you'd even grant that they probably believe that their reasons for their particular position are good reasons. Leaving aside whether the positions themselves are good or not, how do you evaluate whether someone's reasons for holding a particular position are good? After all, a religious person may believe that one day an omniscient being will call them to answer for why they failed to keep holy commands in their life. To the holder of such a view, that's a pretty good reason to do some twisted stuff. If anything, (if their religion is right) such a reason for them holding their particular position (fear of eternal condemnation) may even be better to them than your reasons for your position are to you!
The fact that they don't agree with the others on LGBT rights should only be an issue if they insist on trying to espouse their unrelated viewpoints using the panel, in which case by all means kick them off.