We discuss the failure of the Markov approximation in the description of atom-surface fluctuation-induced interactions, both at equilibrium (Casimir-Polder forces) and out-of-equilibrium (quantum friction). Using general theoretical arguments, we show that the Markov approximation can lead to erroneous predictions of such phenomena with regard to both strength and functional dependencies on system parameters. In particular, we show that the long-time power-law tails of temporal correlations, and the corresponding low-frequency behavior of two-time dipole correlations, neglected in the Markovian limit, dramatically affect the prediction of the force. Our findings highlight the importance of non-Markovian effects in dispersion interactions.