A fuzzy inference network with multimodal feature knowledge embedding and its application in medical diagnosis

A fuzzy radial basis inference network with multimodality prior feature knowledge embedding is proposed for multimodality feature fusion and small sample set classification. This model mainly consists of a multi-channel modal feature input layer, a fuzzy radial basis neuron network (FRBN) layer, a modal category feature aggregation layer, a fuzzy rule layer, and a T-S fuzzy classifier. For the differences in spatiotemporal distribution, information granularity, and semantic representation of different modal features, FRBNs is used to embed multimodality diversity prior category feature knowledge, as well as to process the semantic and spatiotemporal information consistency of input modal features. Based on fuzzy computing logic, inference rules are established and multimodality features are fused layer by layer. Specifically, for classification tasks, high-order semantic features of each modality are extracted separately, and a similarity measurement function for modal features is constructed. Then, the fuzzy dynamic C-means clustering algorithm is used to select representative sample features from subsets of different categories in each modality, in order to implicitly represent prior category knowledge. They are used as the kernel center of FRBN to achieve the embedding of diversity prior feature knowledge. In the modality category feature aggregation layer, the diversity feature information of each modality category subclass is aggregated towards the category and generates a non-convex class interface. Fuzzy multiplication operation is adopted to establish fuzzy reasoning rules for classification tasks, and multimodality classification is implemented based on T-S fuzzy classifier. The proposed method can embed and utilize prior feature knowledge of each modality in a mechanism, impose structural and data constraints on the model, reduce the requirement for the completeness of the sample set, and maintain semantic consistency between modalities and within modalities. Applying the proposed method to medical diagnosis, four types of heart disease classification were performed using multimodality medical data such as echocardiogram, multi-lead electrocardiogram, myocardial enzyme examination, and clinical symptoms, with an accuracy rate of 83.37%. This verifies the effectiveness and application value of the proposed method.