Answers are grounded in this paper and cite their sources.
Machine learning models, while very powerful, have their operation obfuscated behind millions of parameters. This obfuscation can make deriving a human meaningful process from a machine learning model very difficult. However, while the intermediate states of a machine learning model are similarly obfuscated, using probing, we can start to explore looking at possible structure in those intermediate states. Large language models are a prime example of this obfuscation, and probing can begin to allow novel experimentation to be performed.
No takes yet. Share an insight, caveat, or question.
Ashlen A. Plasek (2023) studied this question.
Synapse has enriched 10 related papers. Consider them for comparative context: