I imagine the people with illegal starlink terminals are fairly tech savy and can use custom protocols. Living in Iran they probably already have a lot of experience with vpns and lower level protocols to evade censorship.
Illegal just means there’s a black market. You pay some guy in cash and he gives you a smuggled Starlink terminal. Neither buyer or seller is likely to be anymore sophisticated here than someone trading in any other smuggled goods (cigarettes, alcohol, drugs, etc).