A core tenant of many-o-religions is faith, and if "the world" has counter-factuals, those are tests against that faith. Tossing away elements and breaking faith likely ends in exclusion from the group. And that is too much for many.