In this paper we study the role of surface plasmon modes in the Casimir effect. First, we write the Casimir energy as a sum over the modes of a real cavity. We may identify two sorts of modes, two evanescent surface plasmon modes and propagative modes. As one of the surface plasmon modes becomes propagative for some choice of parameters we adopt an adiabatic mode definition where we follow this mode into the propagative sector and count it together with the surface plasmon contribution, calling this contribution ``plasmonic''. The remaining modes are propagative cavity modes, which we call ``photonic''. The Casimir energy contains two main contributions, one from the plasmonic, one from the photonic modes. Surprisingly, we find that the plasmonic contribution to the Casimir energy becomes repulsive for intermediate and large mirror separations. Alternatively, we discuss the common surface plasmon definition, with includes only evanescent waves, where this effect is not found. We show that, in contrast to an intuitive expectation, for both definitions the Casimir energy is the sum of two very large contributions which nearly cancel each other. The contribution of surface plasmons to the Casimir energy plays a fundamental role not only at short but also at large distances.