Direct adaptive robust state and output feedback controllers are proposed for the output tracking control of a class of uncertain systems. The proposed controllers incorporate a variable structure radial basis function (RBF) network to approximate unknown system dynamics, where the RBF network can determine its structure on-line dynamically. Radial basis functions can be added or removed to ensure the desired tracking accuracy and to prevent the network redundancy simultaneously. The closed-loop systems driven by the direct adaptive robust controllers are characterized by the guaranteed transient and steady-state tracking performance. The performance of the proposed output feedback controller is illustrated with numerical simulations.