The limitations for the coherent manipulation of neutral atoms with fabricated solid state devices, so-called `atom chips', are addressed. Specifically, we examine the dominant decoherence mechanism, which is due to the magnetic noise originating from the surface of the atom chip. It is shown that the contribution of fluctuations in the chip wires at the shot noise level is not negligible. We estimate the coherence times and discuss ways to increase them. Our main conclusion is that future advances should allow for coherence times as long as 1 second, a few micron away from the surface.