Ecological theory predicts that the high local diversity observed in tropical forests is maintained by negative density-dependent interactions within and between closely related plant species. By using long-term data on tree growth and survival for coexisting Inga (Fabaceae, Mimosoideae) congeners, we tested two mechanisms thought to underlie negative density dependence (NDD): competition for resources and attack by herbivores. We quantified the similarity of neighbors in terms of key ecological traits that mediate these interactions, as well as the similarity of herbivore communities. We show that phytochemical similarity and shared herbivore communities are associated with decreased growth and survival at the sapling stage, a key bottleneck in the life cycle of tropical trees. None of the traits associated with resource acquisition affect plant performance, indicating that competition between neighbors may not shape local tree diversity. These results suggest that herbivore pressure is the primary mechanism driving NDD at the sapling stage.