Many deep models have been recently proposed for anomaly detection. However, no comprehensive and rigorous comparison of different models exists. We present a comparison of selected generative deep models and classical anomaly detection methods on an extensive number of non-image benchmark datasets. We provide  statistical comparison of the selected models, in many configurations, architectures and hyperparamaters. We arrive to conclusion that performance of the generative models is determined by the process of selection of their hyperparameters. Specifically, performance of the deep generative models deteriorates with decreasing amount of anomalous samples used in hyperparameter selection.