I can't say I've ever heard someone claim that the Mazda3 is only for women. I've heard this about the Miata, but it just doesn't make sense. MANY very male car enthusiasts drive both Mazdas. In fact, one reason I personally bought an alternative to the Miata is that it's such an obvious choice for anyone seeking a great small sports car. Sometimes it seems like I'm the only auto writer who hasn't owned a Miata. And many auto writers suggest the Mazda3 to anyone seeking a compact car, irrespective of their gender.
To anyone I'd suggest buying whatever you personally like, and ignore what some other people seem to think of the car. No auto writer I know thinks of cars in terms of owner stereotypes. They just don't have a basis in reality.
If you want a sedan that's quick, efficient, not pretty enough to risk claims from clueless people that it's a woman's car, and relatively inexpensive, I'd suggest a Volkswagen Jetta GLI.