Chirality in shape and motility can evolve rapidly in microbes and cancer cells. To determine how chirality affects cell fitness, we developed a model of chiral growth in compact aggregates such as microbial colonies and solid tumors. Our model recapitulates previous experimental findings and shows that mutant cells can invade by increasing their chirality or switching their handedness. The invasion results either in a takeover or stable coexistence between the mutant and the ancestor depending on their relative chirality. For large chiralities, the coexistence is accompanied by strong intermixing between the cells, while spatial segregation occurs otherwise. We show that the competition within the aggregate is mediated by bulges in regions where the cells with different chiralities meet. The two-way coupling between aggregate shape and natural selection is described by the chiral Kardar-Parisi-Zhang equation coupled to the Burgers equation with multiplicative noise. We solve for the key features of this theory to explain the origin of selection on chirality. Overall, our work suggests that chirality could be an important ecological trait that mediates competition, invasion, and spatial structure in cellular populations.