Does categorizing scale scores with cutoff points affect hypothesis-testing results?

Abstract Background The aim of this study is to evaluate the hypothesis test results after categorizing the scale scores with cut-off points and to assess whether similar results would be obtained in that best represent the categories. Methods This cross-sectional study was conducted between March 15 and 20, 2023 via the Lime Survey. The questionnaire included questions about the sociodemographic and life characteristics of the participants and the Beck Depression Inventory II (BDI-II). Four groups (minimal, mild, moderate, severe depression) were formed using the cutoff points. Data analysis was performed with all participants and referred to as the conventional analysis group. Then, six subanalysis groups were determined to best represent the groups formed according to the BDI-II. In each BDI-II category, six subanalysis groups were created, including those between Q1–Q3 (IQR group), including those within ± 1 std, including those between 5p–95p (90% of the sample), including those between 2.5p–97.5p (95% of the sample). In addition, 100 different samples were randomly selected containing 50% of each group. Results Of the 1950 participants, 84.7% (n = 1652) were female and 15.3% (n = 298) were male. In terms of depression, it was observed that the significance varied in the analysis groups for sex (p = 0.039), medication use (p = 0.009) and age (p = 0.010) variables. However, these variables were not significant in some of the subanalysis groups. On the other hand, a p < 0.001 value was obtained for income, physical activity, health perception, body shape perception, life satisfaction, and quality of life variables in terms of depression in the conventional analysis group, and it was seen that the significance continued in all subanalysis groups. Conclusions Our findings showed that variables with p < 0.001 in the conventional analysis group maintained their significance in the other analysis groups. In addition, as the p value got closer to 0.05, we observed that the significance changed according to different cutoff points in the analysis groups. In addition, 50% randomly selected samples support these results. At the end of our study, we reached results that support the necessity of secondary tests in the evaluation of scales. Although further studies are needed, we anticipate that our study will shed light on other studies.