Demand Response (DR) is a Smart Grid technology aiming to provide demand regulation for dynamic pricing and ancillary services to the grid. Thermostatically controlled loads (TCLs) are among those with the highest potential for DR. Some of the challenges in modelling TCLs is the various factors that affect their duty cycle, mainly human behaviour and external conditions, as well as heterogeneity of TCLs (load parameters). These add an element of stochasticity, with detrimental impact on the aggregated level. Most models developed so far use Wiener processes to represent this behaviour, which in aggregated models, such as those based on Coupled Fokker-Planck Equations (CFPE), have a negligible effect as "white noise". One of the main challenges is modelling the effect of external factors on the state of TCLs' aggregated population and their impact in heterogeneity during operation. Here we show the importance of those factors as well as their detrimental effect in heterogeneity using cold loads as a case study. A bottom up detailed model has been developed starting from thermal modelling to include these factors, real world data was used as input for realistic results. Based on those we found that the duty cycle of some TCLs in the population can change significantly and thus the state of the TCLs' population as a whole. Subsequently, the accuracy of aggregation models assuming relative homogeneity and based on small stochasticity (i.e. Wiener process with typical variance 0.01) is questionable. We anticipate similar realistic models to be used for real world applications and aggregation methods based on them, especially for cold loads and similar TCLs, where external factors and heterogeneity in time are significant. DR control frameworks for TCLs should also be designed with that behaviour in mind and the developed bottom up model can be used to evaluate their accuracy. (C) 2018 The Authors. Published by Elsevier Ltd.