Density functional theory is, perhaps, the most popular and convenient tool in computational chemistry. DFT methods allow solving different chemical tasks with a good balance of accuracy and computational time. Dozens of existing functionals cover a majority of possible systems, and the development of new ones is still ongoing. However, despite the existence of different databases with accurate quantum-chemical data, the functional design remains a complicated and time-demanding task. Here, we propose a novel approach for simplifying and accelerating this process. The approach is based on a Bayesian search with stochastic sub-sampling that allows considering the 'history' of fitting steps, reduces the computational time for each step, and avoids overfitting to training data. Besides the general testing of the approach efficiency, we also showed an example of training specialized DFT functionals, outperforming the popular ones. The approach is presented as a free code with built-in analysis tools. Using the code with an appropriate reference database can help in constructing a DFT approximation for a highly specialized task.