DGM: deep graph clustering with mincut for analysis of single-cell transcriptomics

Single-cell RNA sequencing (scRNA-seq) is pivotal for deciphering cellular heterogeneity in biomedical research, from developmental biology to cancer studies. However, accurate cell type identification via clustering remains challenging. Current graph-based methods are limited by ignoring higher-order topologies. We present Deep Graph MinCut (DGM), an end-to-end deep learning framework designed to overcome these limitations. Its core innovation is the joint optimization of three objectives: (1) a higher-order MinCut loss for capturing complex cellular communities; (2) an inter-cluster orthogonality loss to enforce clear separation between cell populations; and (3) a reconstruction constraint to maintain global graph structure.The model jointly optimizes autoencoder denoising, graph learning, and cluster assignment. Evaluated on ten public scRNA-seq datasets, DGM achieved average improvements of 2.03% in NMI and 3.11% in ARI, outperforming baseline methods. Crucially, downstream biological analysis demonstrated alignment with known marker genes and developmental trajectories, verifying its ability to produce biologically meaningful and interpretable clusters. DGM provides a robust and biologically interpretable framework for cell type identification, with potential for applications in cancer research and developmental biology. By addressing key computational challenges, DGM generates reliable outcomes that may contribute to downstream research, such as biomarker discovery or characterization of tumor microenvironment heterogeneity.