Cautions and Considerations in Artificial Intelligence Implementation for Child Abuse: Lessons from Japan

In Japan, child abuse is becoming increasingly prevalent, with the number of cases handled by child guidance centers tripling over the past decade to over 200,000 per year.1Ministry of Health, Labour and Welfare. Further promotion of child abuse prevention measures [in Japanese]. 2022https://www.niph.go.jp/h-crisis/wp-content/uploads/2022/09/20220906104112_content_11900000_000982089.pdf Accessed March 19, 2024Google Scholar The government has suggested the development and use of artificial intelligence (AI) tools as a means to improve the personnel shortage on these centers, and plans to operate them nationwide beginning in FY2024.1Ministry of Health, Labour and Welfare. Further promotion of child abuse prevention measures [in Japanese]. 2022https://www.niph.go.jp/h-crisis/wp-content/uploads/2022/09/20220906104112_content_11900000_000982089.pdf Accessed March 19, 2024Google Scholar However, in June 2023, in a municipality that had implemented the system earlier, tragedy struck when a four-year-old girl, for whom the AI estimated a 39% need for protection, died from domestic abuse without receiving protection.2The Japan Times. Girl dies in Mie after AI suggested not taking her into custody. 2023 https://www.japantimes.co.jp/news/2023/07/12/national/ai-assessment-mie-girl-dies/ Accessed March 19, 2024Google Scholar Details about the AI used remain undisclosed, but reportedly, it bases its decisions on 21 factors, including the site of the injury, suspected sexual abuse and difficulty in contacting parents and child during instruction by child guidance centers.3Tokai Television. A father arrested for beating his 7-year-old son after lifting temporary custody... The reality of the use of "AI judgment" by child guidance centers and its merits and demerits [in Japanese]. 2021 https://www.tokai-tv.com/tokainews/feature/article_20211204_14009 Accessed March 19, 2024Google Scholar In this case, the child guidance center had not visually verified the girl's safety for nearly a year after she stopped attending the daycare center.2The Japan Times. Girl dies in Mie after AI suggested not taking her into custody. 2023 https://www.japantimes.co.jp/news/2023/07/12/national/ai-assessment-mie-girl-dies/ Accessed March 19, 2024Google Scholar An AI-based high-risk approach may miss important non-data-based information and is not a sufficient solution to personnel shortages. AI assessments must be utilized cautiously, understanding their limitations, even when intended only as a decision-making factor. The government should advice to pair this data-driven approach with a commitment to securing necessary human resources. Moreover, when risk prediction modeling is used in areas such as child welfare, it is crucial to establish evaluation procedures pre- and post-implementation.4Drake B, Jonson-Reid M, Ocampo MG, Morrison M, Dvalishvili DD. A practical framework for considering the use of predictive risk modeling in child welfare. Ann Am Acad Political Soc Sci 2020;692(1):162–181. doi: 10.1177/0002716220978200.Google Scholar For instance, in the medical field, the introduction of new technologies necessitates the disclosure of clinical trial data, publication of peer-reviewed research papers and post-market verification. Thus far, no system has been established to verify AI to determine child abuse in Japan. Therefore, the problem in this particular case is that it is not possible to externally verify whether the 39% figure given by the AI was a reasonable estimation. Furthermore, despite this sensitive area, the ethical standards for AI in decision-making in child abuse were also not clear in Japan. After the tragedy, the developer of this AI began working with experts on ethical standards for AI in this area5AiCAN. Joint Research Agreement with the University of Tokyo on Ethical Standards for Decision-Making AI in Response to Child Abuse [in Japanese]. 2023 https://www.aican-inc.com/news/20231002-01/ Accessed March 19, 2024Google Scholar, but even as ethical standards were being developed, AI continued to be used in the field, a contradiction. In conclusion, in this case, there should have been a system in place to verify the accuracy of the AI estimation, and there should have been more open scientific evaluation and discussion about the introduction of AI in advance. Child maltreatment is a major public health issue, and the use of AI in the field of digital health has the potential to assist in its prevention. However, it should be introduced and used only after sufficient scientific verification, especially when estimating the need for child protective services.