Whole sky imagers are commonly used for forecasting irradiance available for solar energy production, but validation of the forecast models used is difficult due to sparse reference data. We document the use of Large Eddy Simulations (LES) and a 3D Radiative Transfer Model to produce virtual clouds, sky images, and radiation measurements, which permit comprehensive validation of the sky imager forecast. We then use this virtual testbed to investigate the primary sources of sky imager forecast error on a cumulus cloud scene. The largest source of nowcast (0-min-ahead forecast) errors is the converging-ray geometry implied by use of a camera, while longer-term forecasts suffer from overly-simplistic assumptions about cloud evolution. We expect to use these findings to focus future algorithm development, and the virtual testbed to evaluate our progress.