Most organisms live in ever-changing environments, and have to cope with a range of different conditions. Often, the set of biological traits that are needed to grow, reproduce, and survive varies between conditions. As a consequence, organisms have evolved sensory systems to detect environmental signals, and to modify the expression of biological traits in response. However, there are limits to the ability of such plastic responses to cope with changing environments. Sometimes, environmental shifts might occur suddenly, and without preceding signals, so that organisms might not have time to react. Other times, signals might be unreliable, causing organisms to prepare themselves for changes that then do not occur. Here, we focus on such unreliable signals that indicate the onset of adverse conditions. We use analytical and individual-based models to investigate the evolution of simple rules that organisms use to decide whether or not to switch to a protective state. We find evolutionary transitions towards organisms that use a combination of random switching and switching in response to the signal. We also observe that, in spatially heterogeneous environments, selection on the switching strategy depends on the composition of the population, and on population size. These results are in line with recent experiments that showed that many unicellular organisms can attain different phenotypic states in a probabilistic manner, and lead to testable predictions about how this could help organisms cope with unreliable signals.