MBARI's Video Annotation and Reference System

The amount of video information used for scientific purposes is increasing as new ocean observing systems are deployed. To date, underwater video has primarily been collected by SCUBA divers, remotely operated vehicles (ROV) and human-occupied submersibles. However, large quantities of video information are likely to be collected by other platforms in the future, such as cabled observatory systems (for example, the VENUS, MARS and NEPTUNE projects) and autonomous underwater vehicles (AUV). This video information is extremely useful for science but locating events or objects of interest from hundreds or thousands of hours of video requires a system for effectively managing video annotation information. The Monterey Bay Aquarium Research Institute (MBARI) uses high-resolution video equipment to record over 300 remotely operated vehicle (ROV) dives per year. Over the past sixteen years, 16,000 videotapes (equates to over 13,500 hours) have been archived and managed as a centralized institutional resource. MBARI has developed a software and hardware system, Video Annotation and Reference System (VARS), to facilitate the creation, storage, and retrieval of video annotations based on ROV dive tapes. The VARS components reference a knowledge database of over 3500 biological, geological and technical terms. This hierarchical information allows for consistent and rapid classification, description, and complex querying of objects observed on video. MBARI currently has over 1.8 million annotations stored in the VARS system. These annotations have been used in over 200 professional publications and presentations, in addition to numerous education and outreach projects. These publications span multiple sub-disciplines within biology, geology and chemistry. An operational VARS suite consists of a relational database, a videotape recorder (VTR) that supports the Sony 9-pin protocol over RS-422, and the three VARS applications: Knowledgebase, Annotation, and Query. The relational database is used as a centralized repository for storing the knowledgebase data and annotations. The VARS applications interact directly with this database and many VARS users can access the database simultaneously. Each VARS application is responsible for different aspects of the information stored in the database. The Knowledgebase application is used for managing the controlled vocabulary for creating and searching for annotations. Annotations and images captured from the video are added to the relational database through the Annotation application. The Query application provides simple, convenient access to the annotations, video images, and related ancillary data. The VARS applications use a shared API for interacting with the relational database. This API could be adopted by other software that makes use of video annotations, such as automated event detection applications. In 2005, VARS was released as open-source software. VARS was designed so that other institutions could easily adopt it. Written in Java, VARS can be run on a variety of operating systems. In addition, VARS is not tied to a particular database; groups interested in deploying VARS can use one of a number of relational database systems. The following paper will give an overview of the capabilities and technical aspects of the VARS system. More information about VARS as well as a VARS demonstration application can be obtained at http://vars.sourceforge.net