I don't think it's such a cut and dry thing. I mean I can be physically attracted to someone who I would say is objectively pretty plain. And I'm not talking about getting to know someone better and gradually thinking they're better looking than they are. I mean immediate physical attraction. I think it's more about what sort of energy the person is producing and whether you are attracted or repulsed by it.
Within reason though.