Improving the accuracy of competency based hiring decision models by generating job competency feature distributions using diffusion models

The limited historical selection records for high-skilled talent positions lead to limitations in the accuracy of traditional selection models due to the difficulty of joint distribution estimation of competency features under small-sample conditions. This paper proposes a competency feature distribution generation framework based on a Denoising Diffusion Probabilistic Model (DDPM) for data augmentation and model optimization in high-skilled talent selection. Based on four dimensions of competency indicators—knowledge, skills, abilities, and work style—extracted from the Occupational Information Network (O*NET) database, an initial training set is constructed using a small number of real selection samples. Gaussian diffusion is applied to the continuous latent variables produced by the VAE encoder, with discrete feature reconstruction delegated to the decoder’s grouped Softmax outputs under cross-entropy loss. After encoding the job semantic embedding vectors using a pre-trained language model, a classifier-free guidance (CFG) mechanism is injected into the reverse denoising process, carrying the job-level semantic condition through the denoising iterations of each sampled trajectory. To address the curse of dimensionality in high-dimensional sparse scenarios, a Variational Autoencoder (VAE) latent space compression module is introduced. This module maps the original competency features to a low-dimensional continuous latent space before performing diffusion and generation, effectively mitigating the distribution shift of high-dimensional data. The generated data undergoes dual quality screening using Wasserstein distance and pairwise Spearman correlation coefficients, is then mixed with the original scarce samples, and input into an XGBoost (eXtreme Gradient Boosting) classifier to complete the training of the selected model. Experimental results show that, under standard experimental settings, the proposed framework achieves an AUC (Area Under the Curve) of 0.9100, a Macro-F1 score of 0.8700, and a Hit@10 hit rate of 83.6% in the high-skilled job selection task. These figures represent improvements of 0.0309, 0.0677, and 4.4% points, respectively, compared to the strongest baseline, TabSyn. At a training size of ten records the framework reaches an AUC of 0.823, a subset containing two to three positive records and characterized by a widening spread across random seeds. All differences pass the Wilcoxon signed-rank test ( p < 0.05). This framework provides a systematic technical path for high-fidelity generation and pattern-consistent decision modeling using competency data whose sample size is insufficient relative to feature dimensionality in the selection of high-skilled personnel, where selection performance metrics reflect model consistency with historical hiring decisions rather than direct prediction of candidates’ true competency levels. The main experiments validate the framework on a medium-scale training set, while an additional robustness sub-experiment verifies performance under extreme sample scarcity.