Based on provincial panel data on the textile industry in China, this paper calculates the total-factor energy efficiency of this industry as the dependent variable. Additionally, based on linear panel analysis of the relationship between industrial agglomeration and energy efficiency, in-depth analysis of the industry is performed at different industrial agglomeration levels. The paper identifies different impacts of industrial agglomeration on energy efficiency, uses the threshold regression model to extend the research to a nonlinear framework, and constructs a double threshold regression model in which the threshold of the textile industry agglomeration level serves as the threshold variable. The results show, first, a threshold effect occurs when industrial agglomeration affects total-factor energy efficiency. Second, a significant positive correlation exists among the degree of economic development, energy prices, research and development investment (R&D), enterprise scale, and total factor energy efficiency of the textile industry. Third, a non-linear relationship exists between industrial agglomeration and energy efficiency in the industry. When industrial agglomeration is low, promoting it improves energy efficiency. However, when industrial agglomeration reaches a certain level, agglomeration and energy efficiency show a negative relationship. Finally, based on the empirical results, ways of improving energy efficiency in the industry are suggested.