The existence of long-lived reservoirs of latently infected CD4+ T cells is the major barrier to curing HIV, and has been extensively studied in this light. However, the effect of these reservoirs on the evolutionary dynamics of the virus has received little attention. Here, we present a within-host quasispecies model that incorporates a long-lived reservoir, which we then nest into an epidemiological model of HIV dynamics. For biologically plausible parameter values, we find that the presence of a latent reservoir can severely delay evolutionary dynamics within a single host, with longer delays associated with larger relative reservoir sizes and/or homeostatic proliferation of cells within the reservoir. These delays can fundamentally change the dynamics of the virus at the epidemiological scale. In particular, the delay in within-host evolutionary dynamics can be sufficient for the virus to evolve intermediate viral loads consistent with maximising transmission, as is observed, and not the very high viral loads that previous models have predicted, an effect that can be further enhanced if viruses similar to those that initiate infection are preferentially transmitted. These results depend strongly on within-host characteristics such as the relative reservoir size, with the evolution of intermediate viral loads observed only when the within-host dynamics are sufficiently delayed. In conclusion, we argue that the latent reservoir has important, and hitherto under-appreciated, roles in both within- and between-host viral evolution.