It's worse.
The same argument can be used to attack the legitimacy of judges, civilian control of the military and much else.

We don't require our judges to have both a criminal record and have been a victim before we allow them to judge, right?

The argument that only insiders are entitled to an opinion furthermore doesn't stand the most basic plausibility tests.
Think of Wall St, for example. Would you want to hear from a Wall St banker that you should shut up because you've got no clue, and only long-time Wall St bankers should be allowed to exercise oversight over Wall St?

I have no respect for the "only insiders' opinions count" defence. It has a terrible track record, is implausible and impractical.

There's honour among thieves, that's why the "insiders only" defence is so popular - and crappy.