It has been a long-standing goal to bring massive objects into a superposition of different locations in real space, not only to confirm quantum theory in new regimes, but also to explore the interface with gravity. The main challenge is usually thought to arise from forces or scattering due to environmental fields and particles that decohere the large object's wave function into a statistical mixture. We unveil a decoherence channel which cannot be eliminated by improved isolation from the environment. It originates from sound waves within the object, which are excited as part of any splitting process and carry partial "Welcher Weg" information. This puts stringent constraints on future spatial superpositions of large objects.