In this paper, we approach supervisory control as an online decision problem. In particular, we introduce "calibrated forecasts" as a mechanism for controller selection in supervisory control. The forecasted quantity is a candidate controller's performance level, or reward, over finite implementation horizon. Controller selection is based on using the controller with the maximum calibrated forecast of the reward. The proposed supervisor does not perform a pre-routed search of candidate controllers and does not require the presence of exogenous inputs for excitation or identification. Assuming the existence of a stabilizing controller within the set of candidate controllers, we show that under the proposed supervisory controller, the output of the system remains bounded for any bounded disturbance, even if the disturbance is chosen in an adversarial manner. The use of calibrated forecasts enables one to establish overall performance guarantees for the supervisory scheme even though non-stabilizing controllers may be persistently selected by the supervisor because of the effects of initial conditions, exogenous disturbances, or random selection. The main results are obtained for a general class of system dynamics and specialized to linear systems.