The dynamic nature of contact patterns creates diverse temporal structures. In particular, empirical studies have shown that contact patterns follow heterogeneous inter-event time intervals, meaning that periods of high activity are followed by long periods of inactivity. To investigate the impact of these heterogeneities in the spread of infection from a theoretical perspective, we propose a stochastic model to generate temporal networks where vertices make instantaneous contacts following heterogeneous inter-event intervals, and may leave and enter the system. We study how these properties affect the prevalence of an infection and estimate R(0), the number of secondary infections of an infectious individual in a completely susceptible population, by modeling simulated infections (SI and SIR) that co-evolve with the network structure. We find that heterogeneous contact patterns cause earlier and larger epidemics in the SIR model in comparison to homogeneous scenarios for a vast range of parameter values, while smaller epidemics may happen in some combinations of parameters. In the case of SI and heterogeneous patterns, the epidemics develop faster in the earlier stages followed by a slowdown in the asymptotic limit. For increasing vertex turnover rates, heterogeneous patterns generally cause higher prevalence in comparison to homogeneous scenarios with the same average inter-event interval. We find that [Formula: see text] is generally higher for heterogeneous patterns, except for sufficiently large infection duration and transmission probability.