In the exploration of new infrared nonlinear optical (IR NLO) materials, element substitution has been developed as an effective way to adjust the structural features and material performances. A series of new IR NLO materials have been discovered in the I-Ba-M-IV-Q system (I = Li, Na, Cu, and Ag; M-IV = Si, Ge, and Sn; Q = S and Se), and they undergo interesting structural transformation with different element substitution except Li analogues. Herein, we have successfully synthesized three selenides with different space groups (Ag2BaSiSe4: I (4) over bar 2m; Ag2BaGeSe4 and Ag2BaSnSe4: I222) in the above system and studied their properties through experimental and theoretical methods. Remarkably, the detailed analysis on the structural changes and properties comparison was also systematically investigated in the I-Ba-M-IV-Q system and the results indicate that the distortion degrees of different IQ(4) tetrahedra play the critical role to cause the structural transformation with the M or Q elements substitution. More importantly, we have also found that the structural changes have the close relationship with the distance d(I-I) between adjacent I cations in the I2BaSnSe4 system, which makes the four-membered rings formed by edge-sharing BaSe8 units change from the square to rhombus with the increase of d(I-I). The properties comparisons (band gap and NLO effect) in this system have been also systematically studied.