The heterogeneity of schizophrenia makes it difficult to discover reliable imaging biomarkers, and most existing fMRI-based classification methods fail to combine temporal coherence between brain regions and temporal dynamics of brain activity. Therefore, we proposed a unified Hybrid Deep Learning Framework that effectively integrates temporal Coherence and Dynamics (HDLFCD) to classify psychiatric disorders by combining C-RNN, DNN and SVM. An attention module was also introduced into the C-RNN model to improve classification accuracy and interpretability without increasing the computation complexity. An accuracy of 85% was achieved in a large multi-site WRI dataset with 542 healthy controls and 558 schizophrenia patients, in which striatum, dorsolateral prefrontal cortex and cerebellum were identified as the most group-discriminative brain regions by the attention module. Note that the proposed framework is an end-to-end general module, which not only shows high superiority in combining multiple sources of information, but also can be easily applied to integrate other multimodal data.