Understanding the factors that promote, disrupt, or shape the nature of cooperation is one of the main tasks of evolutionary biology. Here, we focus on attitudes and beliefs supportive of in-group favoritism and strict adherence to moral consensus, collectively known as ideological rigidity, that have been linked with both ends of the political spectrum. The presence among the political right and the left is likely to make ideological rigidity a major determinant of the political discourse with an important social function. To better understand this function, we equip the indirect reciprocity framework--widely used to explain evaluation-mediated social cooperation--with multiple stylized value systems, each corresponding to the different degree of ideological rigidity. By running game theoretical simulations, we observe the competitive evolution of these systems, map conditions that lead to more ideologically rigid societies, and identify potentially disastrous outcomes. In particular, we uncover that barriers to cooperation aid ideological rigidity. The society may even polarize to the extent where social parasites overrun the population and cause the complete collapse of the social structure. These results have implications for lawmakers globally, warning against restrictive or protectionist policies.