In order to achieve accurate computer-aided diagnosis of Alzheimer’s disease (AD)and mild cognitive impairment (MCI) patients, this paper proposes a multimodal Alzheimer’s multi-class diagnostic framework (MAMDF) that uses an asymmetric cross-attention mechanism for multimodal fusion to better reveal the relationship between clinical data and medical imaging data. Moreover, to address the two MCI subtypes that are rarely mentioned in previous computer-aided diagnosis work, we combined frequency-domain transformers and Transformers to propose a novel deep feature extraction module for feature fusion. This method captures the internal connections of fused features and obtains richer multimodal joint representations, thus improving the diagnostic performance of the model on the two MCI subtypes. Experimental results on the ADNI dataset show that the proposed model achieves higher accuracy and F1 scores, compared with similar works. Thus the model can more effectively handle multimodal data fusion and mine the deep feature relationships between different modal medical data, thereby better integrating and analyzing the multimodal information of AD patients.