I don't think the issue is that people who are trans use a bathroom which doesn't match with their initial biological sex, because I think almost no one cares about that. It's more the general tendencies of dissolving women only spaces making it more accessible to all kinds of men, (> not transwomen in particular), which is weird when the issue of sexual violence in society isn't completely resolved yet. You could also have shared bathrooms in addition to separate bathrooms to give another option for trans people, but the main line of action seems to be to make all bathrooms unisex, even if some people are not happy with it (and obviously not for transphobic reasons).