We discuss the coherent splitting and recombining of a nanoparticle in a mesoscopic ``closed-loop'' Stern-Gerlach interferometer in which the observable is the spin of a single impurity embedded in the particle. This spin, when interacting with a pulsed magnetic gradient, generates the force on the particle. We calculate the internal decoherence which arises as the displaced impurity excites internal degrees of freedom (phonons) that may provide Welcher Weg information and preclude interference. We estimate the constraints this decoherence channel puts on future interference experiments with massive objects. We find that for a wide range of masses, forces and temperatures, phonons do not inhibit Stern-Gerlach interferometry with micro-scale objects. However, phonons do constitute a fundamental limit on the splitting of larger macroscopic objects if the applied force induces phonons.