It's possible for both of your realities to be true. The reality is that oppression isn't a binary: people get shit on for being black, for being a woman, for being poor, etc. Intersectionality is the study of the oppression and privilege at the meeting of various groups - the oppression faced by a poor white woman is different than the oppression faced by a trans-black woman, etc.