A Dead Call Cannot Leak: Liveness-Guarded Measurement of Canary Leakage from Shared Agent Skill Pools

The shared agent skill pool, in which a procedure distilled from one user’s task is offered to other users, raises a measurement question before it raises a defense question: how do you reliably tell whether a confidential value, carried under a never-reveal instruction, survives into a second user’s hands? We show that the obvious instrument is broken by default, fix it in three lines, and use the fixed instrument to measure the deployed defense. The instrument. An adversarial extraction red-team that does not check whether its target is alive does not fail loudly when its calls die: it silently reports a fake 0%, because a call that errors returns no text and a response with no text cannot contain the secret it was probing for. Our own first multi-model run hit exactly this: a decommissioned target returned a confident, clean-looking 0%. The error is directional: it only ever removes recoveries, so an uninstrumented red-team systematically publishes the safe-looking artifact. An extraction rate is therefore uninterpretable without a liveness denominator; enforcing one costs three lines (a pre-flight-and-post-run liveness guard; a dead call is not a refusal). We pair it with deterministic canary ground-truthing and a released second-annotator κ pipeline for the judge increment. The finding: the defense fails, and its worst failure is off the final answer. With the instrument in place, the standard sharing-time defense, an explicit never-reveal instruction on a flagged confidential value, does not contain it: Llama-3.1-8B-Instruct returns a planted canary from 17 of 20 skills (85% on Featherless; three-run mean 83%) despite the note, zero control false-positives, and it fails identically on 20 skills distilled by a real third-party pipeline (recovered at 100% on the model we can measure without serving errors). The more consequential failure is where the secret surfaces. In two reasoning models the canary appears in almost none of the final answers but in most or all cases once the visible chain-of-thought is counted (gpt-oss-20b 0% answer vs 87% reasoning-inclusive; DeepSeek-R1-70B 3% vs 100%; three-run means): a model whose answer reads clean can leak the secret completely in a reasoning trace that deployments routinely log or display, so a final-answer audit is structurally blind to it. Containment tracks alignment more than scale, scoped honestly. A 22-model census that separates the two axes within a single provider (the rate is serving-stack-dependent, so we never pool across providers) finds that size never reduces leakage in any of the three families measured, while alignment does move it, and its single largest and cleanest move comes from a capability-positive intervention: adding safety-tuning to gemma-2-9b cuts leakage from 100% to 65% (a 35-point drop, three runs, sd 0), which a “the model is merely degraded” account cannot explain. We are deliberate about how clean the rest of the dissociation is. The within-weights abliteration contrast (instruct vs. the same weights with the refusal direction surgically removed) is directional, not capability-neutral, and on a 100-canary real-document panel it does not reproduce (instruct 90% vs. abliterated 87%, paired p=0.33): abliteration’s capability cost bites on longer documents. The alignment ordering that does hold on real documents is the permissive-finetune contrast on a shared base (Llama-3.1-70B instruct 87% vs. Hermes-3 97% and Hermes-4 99%; paired p=1.2×10⁻⁴ and 5×10⁻⁶; the two finetunes statistically indistinguishable from each other). So we report the dissociation as a robust-but-scoped trend, holding on the synthetic census (the abliteration gradient and the gemma safety-tuning move) and on the real-document finetune panel, not as a clean single-isolator result. Two scope notes: recovery is exact-marker-deterministic (the paraphrase judge added zero beyond the markers, so the single-operator judge changed no headline rate), and the leak is recall of a present value, not reconstruction (0 of 100 secrets survive ablation of their mention). We test the never-reveal instruction on a present-but-flagged value, not entity-scrubbing-then-reconstruction, a distinct complementary threat.