We derive an expression for the collective Casimir-Polder interaction of a trapped gas of condensed bosons with a plane surface through the coupling of the condensate atoms with the electromagnetic field. A systematic perturbation theory is developed based on a diagrammatic expansion of the electromagnetic self-energy. In the leading order, the result for the interaction energy is proportional to the number of atoms in the condensate mode. At this order, atom-atom interactions and recoil effects lead to corrections compared to the single-atom theory, through shifts of the atomic transition energies. We also discuss the impact of the spatial delocalization of the condensate mode.