Protesters on the right often claim to be defending ‘our women and girls’ from asylum seekers. But where is the real threat?