Non-binary people? Yes sex is a physical distinction. Or at least a chromosome distinction. Hermaphroditism is rare and arguably is a defect. The way a person's body unfolds isn't always predictable, but if they are XX or XY they are defined as male or female. They can be gay or opposite fem/masc than might seem appropriate for their sex or be uncomfortable with the genitals that they got, but these are all different continuums outside of sex. People are beyond 3 dimensional. No reason to try to pack everything between male and female.