Consider the problem of developing a controller for general (nonlinear and stochastic) systems where the equations governing the system are unknown. Using discrete-time measurements, this paper presents an approach for estimating a controller without building or assuming a model for the system (including such general models as differential/difference equations, neural networks, fuzzy logic rules, etc.), Such an approach has potential advantages in accommodating complex systems with possibly time-varying dynamics. Since control requires some mapping, taking system information, and producing control actions, the controller is constructed through use of a function approximator (FA) such as a neural network or polynomial (no FA is used for the unmodeled system equations), Creating the controller involves the estimation of the unknown parameters within the FA. However, since no functional form is being assumed for the system equations, the gradient of the loss function for use in standard optimization algorithms is not available. Therefore, this paper considers the use of the simultaneous perturbation stochastic approximation algorithm, which requires only system measurements (not a system model), Related to this, a convergence result for stochastic approximation algorithms with time-varying objective functions and feedback is established. It is shown that this algorithm can greatly enhance the efficiency over more standard stochastic approximation algorithms based on finite-difference gradient approximations.