This paper considers the problem of inferring an unknown network of dynamical systems driven by unknown, intrinsic, noise inputs. Equivalently we seek to identify direct causal dependencies among manifest variables only from observations of these variables. For linear, time-invariant systems of minimal order, we characterise under what conditions this problem is well posed. We first show that if the transfer matrix from the inputs to manifest states is minimum phase, this problem has a unique solution irrespective of the network topology. This is equivalent to there being only one valid spectral factor (up to a choice of signs of the inputs) of the output spectral density. If the assumption of phase-minimality is relaxed, we show that the problem is characterised by a single Algebraic Riccati Equation (ARE), of dimension determined by the number of latent states. The number of solutions to this ARE is an upper bound on the number of solutions for the network. We give necessary and sufficient conditions for any two dynamical networks to have equal output spectral density, which can be used to construct all equivalent networks. Extensive simulations quantify the number of solutions for a range of problem sizes. For a slightly simpler case, we also provide an algorithm to construct all equivalent networks from the output spectral density.