Joint Rate Splitting and Beamforming Design for RSMA-RIS-Assisted ISAC System

In conventional integrated sensing and communication (ISAC) systems, the multiplicative fading of radar signals and space division multiple access (SDMA) restrict system performance. This work proposes a novel rate splitting multiple access (RSMA)-reconfigurable intelligent surface (RIS)-assisted ISAC system to improve the sensing performance, where RIS is deployed to simultaneously assist target detection and RSMA-aided multi-user communication for a dual-functional base station (BS). The objective of this work is to maximize the SNR of the target detection by optimizing the rate splitting, active beamforming at BS, and passive beamforming at the RIS. To solve this non-convex problem, an iterative algorithm is proposed exploiting majorization-minimization (MM), the approximation of the difference of two convex functions (DC), and the penalty method. Finally, simulation results verify the effectiveness of the proposed algorithm and demonstrate a clear sensing gain of the proposed RSMA-RIS-assisted ISAC scheme, approaching the upper bound, due to the special transmission mechanism of rate splitting in RSMA, and the cascaded passive beamforming of order O(N4) from RIS, respectively.