BPS2026 – In-context adaptation of vision language models (VLMs) for few-shot cell detection in microscopy

Foundation vision-language models (VLMs) excel on natural images, but their utility for biomedical microscopy remains underexplored. We study how in-context learning adapts state-of-the-art VLMs to few-shot object detection when large annotated data sets are unavailable, as in the case of microscopic images. We introduce the microscopic object detection (Micro-OD) benchmark, a collection of 252 images, curated explicitly for in-context learning, with bounding-box annotations spanning 11 cell types across four sources, including two in-lab expert-annotated sets. We systematically evaluate eight VLMs under few-shot conditions and compare variants with and without implicit test-time reasoning tokens. We further implement a hybrid few-shot object detection (FSOD) pipeline that combines a detection head with a VLM-based few-shot classifier that boosts the few-shot performance of recent VLMs on our benchmark. Across data sets, we see that zero-shot performance is weak due to the domain gap, but few-shot support consistently improves detection, with marginal gains after 6 shots. We observe that models with reasoning tokens are more effective for end-to-end localization, whereas simpler variants are preferable for classification of pre-localized crops. Our results highlight in-context adaptation as a practical path in microscopy, and our benchmark provides a reproducible testbed for advancing open-vocabulary detection in biomedical imaging.