Speaking to machines: A critical review of prompt engineering as a 21st-century competency, automatic optimization, and human–AI collaboration

Prompt engineering now includes instruction design, scaffolding of reasoning, integration of tools and orchestration of multiple agents. This review traces its progression from zero-shot and few-shot prompting to chain-of-thought prompting, retrieval-augmented generation, automatic prompt optimization and autonomous agent frameworks. We explore two opposing trends: a lowering of the technical floor by automation, e.g. via tools like DSPy, and a raising of the competency ceiling for designing, evaluating and controlling AI systems, as a result of agentic complexity. Structured prompting generally improves task performance, but the size of the effect varies greatly with model scale, task type and evaluation protocol. We place prompt engineering in the context of existing competency frameworks (P21, ATC21S, DigComp) and introduce the conceptual, literature-based framework of the Agentic Prompting Competency Stack (APCS) that has explicit boundary conditions and testable propositions, not an empirically proven taxonomy. A conflict-of-evidence analysis indicates that prompting is strong in structured reasoning and rapid prototyping but requires human verification and guardrails in high-stakes domains. We list the following research gaps: standard evaluation protocols, longitudinal research on literacy-acquisition systems, causal attribution of gains achieved by agentic systems, security benchmarks for agentic systems, cross-linguistic validity, and equitable access to advanced tools. This review summarizes when, why and by how much both automatic and manual optimization can help to improve performance, as well as the analysis of human–AI collaboration in agentic settings, including the emergence of a skill divide and a move for democratization of AI.