Yes, as a non-native, I agree that it is not very common (almost never) for me to mix them up because I prononunce them differently.
Another reason non-natives don't do this mistake might be that usually they are thought the written language first, so schools concentrate on grammatical issues more.
Adding another datapoint, as a non-native speaker myself, I believe I've rarely (if ever) mixed them up.
As the parent commenter said, I've also noticed in the past that this mix up occurs usually among native English speakers.
I've noticed it with other languages too, that native speakers often mix up homophones, probably because as a native speaker you write as you would speak and either you don't care or don't even think about it.
Another reason non-natives don't do this mistake might be that usually they are thought the written language first, so schools concentrate on grammatical issues more.