Hybrid Spatially Regularized NMF and Marker-Guided Benchmarking: A Unified Framework for Interpretable Spatial Transcriptomics in Cancer

"Poster presented at TMC AI Summit 2026"Background and Problem— High-resolution spatialtranscriptomics (ST) platforms, such as 10x Xenium, have revo-lutionized the characterization of the Tumor Microenvironment(TME) by preserving single-cell spatial coordinates. However,the resulting high-dimensional data pose significant challengesfor representation learning. Standard dimensionality reduction(DR) techniques, such as PCA and NMF, typically treat cellsas independent samples (“a-spatial” assumption). In complexsolid tumors like cholangiocarcinoma, this leads to fragmentedclustering and biological incoherence, hindering the identificationof subtle cell states. There remains a critical gap in rigorouslybenchmarking DR strategies and developing scalable algorithmsthat effectively balance geometric compactness with biologicalfidelity.Methods— We present a unified study utilizing a large-scale 10x Genomics Xenium dataset (480-gene panel) derivedfrom 40 tumor microarray (TMA) cores across 25 intrahepaticcholangiocarcinoma patients. Following rigorous quality control,doublet removal, and log-normalization, 191,125 high-quality,spatially resolved cells were retained from an initial pool of212,000. Our approach is two-fold:1) Rigorous Benchmarking: We systematically evaluated sixDR frameworks—including Linear (PCA, NMF), Non-linear (VAE, AE), and Hybrid embeddings—via Pareto-front analysis across varying latent dimensions (k = 5−40).To quantify performance, we introduced two novel biologi-cal metrics: Cluster Marker Coherence (CMC) and MarkerExclusion Rate (MER), alongside a post-hoc reassignmentalgorithm to correct biologically implausible labels.2) Algorithmic Innovation (hSNMF): Leveraging insightsfrom our benchmark, we developed Hybrid Spatial NMF(hSNMF). This novel framework extends NMF by in-corporating a dual “contact-radius” graph (rc = 20μm,rr = 80μm) to enforce local spatial smoothness on latentfactors prior to clustering, effectively bridging the gapbetween transcriptomic similarity and physical proximity.Results—Our benchmarking revealed fundamental trade-offsin existing methods: while PCA maximized geometric separation(Silhouette > 0.15), it suffered from high marker exclusion rates.Standard NMF excelled at biological signal recovery but lackedspatial cohesion. The proposed hSNMF framework outperformedall non-spatial and spatial baselines. By jointly optimizing spatialcontiguity and molecular coherence, hSNMF achieved superiorspatial compactness (CHAOS < 0.002) and high spatial autocor-relation (Moran’s I > 0.94) while maintaining distinct clusterseparability (Silhouette > 0.22).Clinical Impact—This work establishes a validated computa-tional roadmap for dissecting heterogeneity in cholangiocarci-noma. By replacing “black-box” embeddings with interpretable,spatially regularized factors, hSNMF enables the precise mappingof continuous tissue structures and cell-cell interactions. Thisscalable framework directly supports the discovery of spatiallydistinct tumor niches, facilitating the development of targetedtherapies for complex solid tumors.Index Terms—Spatial Transcriptomics, Generative AI, Dimen-sionality Reduction, Cholangiocarcinoma, Xenium.