Trust in Generative AI: A Necessary Warning for Research and IT Security

We are currently witnessing a rapid, unexpected, and widespread deployment of Generative Artificial Intelligence (Generative AI) systems. Their astonishing capabilities are leading quickly to their proliferation across professional, academic, and decision-making environments. This raises a fundamental question: can we truly trust them? This paper examines the risks associated with blind trust in AI, particularly for non-expert users. Based on interactions experienced in real time with multiple freely available Generative AI systems, we identify serious and potentially dangerous recurring issues. These include misleading failure modes such as data hallucination, code-ghosting, sycophancy, reference fabrication, non-terminating interaction loops, and Slopsquatting (Package Hallucination Exploitation). We also point to the critical yet often overlooked issue of data confidentiality when sensitive information is submitted voluntarily to remote AI servers. The implications of these findings are critical for researchers, decision-makers, and AI developers, as blind trust in these systems can lead to severe academic, professional, and security consequences. In parallel, we briefly consider the historical perspectives of embodied AI, Automation, and Human-Machine Interaction systems (HMIS) where security and safety are of paramount importance. This paper argues that Generative AI must be understood as it is designed: a probability engine, not a truth engine. Thus, human expertise must remain the final arbiter and bear responsibility for the outcomes of their work, including tasks voluntarily delegated to AI agents. We conclude with practical recommendations for the safe and critical use of Generative AI systems. These systems need to be redesigned with built-in self-control mechanisms, for instance, inspired by or analogous to anticipatory systems. They must enforce strict control over the information delivered to users. They must also protect private data and limit chaotic behavior when Generative AI agents reach their operational limits.