Also, on reflection, I think the firefox window size is half of what I reported - I'm not sure what OSX does to its screenshots, but using SHIFT-CTRL-CMD-3, it's reporting about half that size. Not sure if that makes a difference to your changes (likely not).
OSX, Firefox, browser size 3326×2830 pixels. Only happens at or around that particular size - resizing the window fixes the issue.