The Video Annotation and Reference System (VARS) is a software interface and database system that provides tools for describing, cataloging, retrieving, and viewing the visual, descriptive, and quantitative data associated with MBARI’s deep-sea video archives.

MBARI uses high-resolution video equipment to record hundreds of remotely operated vehicle (ROV) dives per year. Since 1988, more than 23,000 hours of videotape have been archived, annotated, and maintained as a centralized institutional resource. This video library contains detailed footage of the biological, chemical, geological, and physical aspects of the Monterey Bay submarine canyon and other areas including the Pacific Northwest, Northern California, Hawaii, and the Gulf of California. MBARI has developed the Video Annotation and Reference System (VARS), consisting of software to facilitate the creation, storage, and retrieval of video annotation records from the ROV dive tapes.

As shown in the flowchart, video is collected by cameras on MBARI ROVs and recorded to tape in the shipboard control rooms. Using the VARS annotation application, researchers take frame grabs and enter observations as annotations during the dives. Annotations are created and constrained using concepts which have been entered into the knowledge database and approved by the knowledge administrator. When the ships return to shore, the tapes are reviewed and annotated in more detail by MBARI video lab personnel, also using VARS. Nightly processes merge video observations with ancillary data collected while the ROV is deployed so that for each video observation metadata is available and easily accessible. These ancillary data include latitude, longitude, depth, temperature, oxygen concentration, salinity, transmittance, and altitude. All of this information goes into the annotation database, and the annotated tapes are stored in the video library. The annotations and images are accessible through the VARS query interface.

A more recent development within the VARS software family has been the development of two additional VARS annotation and query applications. As more scientific imaging platforms are developed by MBARI and elsewhere, including still imaging and video systems on ROVs, AUVs, and stationary platforms, the types of data streams needing analysis have grown. To accommodate this growing need, VARS has been modified so that users can now analyze digital video files and still images. Two additional versions of the query and annotation applications leverage the VARS knowledgebase but allow for distinct analysis of these different data sources. More information about these additional VARS applications is explained below and within the respective installation and user guides.

Publications that use VARS data

The VARS database provides invaluable qualitative and quantitative data that have been used in hundreds of peer-reviewed publications. Some examples can be viewed here: VARS Publications.

VARS Knowledge database

The core of the VARS system is a knowledge database of over 4,000 biological, geological and technical terms used to describe deep-sea research conducted by the institute. This database is comprised of objects, or concepts, which are identifiable things that we observe — for example, species (Atolla wyvillei), geologic features (rock outcrop, gas vent, scarp), sampling devices and other equipment (suction sampler, larval incubation chamber), and marine debris (plastic bag, tire). The knowledgebase also contains associations, which serve as descriptors — such as colors, sizes, behaviors, sample numbers — that can be associated with the objects being observed.

Hierarchical relationships between objects are preserved within the knowledgebase allowing for consistent, rapid classification and description and complex querying of objects observed on video. For example, the hierarchical nature of the knowledgebase allows a user to search for all descendants of a particular taxon that has been observed in MBARI’s video archives (e.g., all species of Sebastes (rockfish)). Additionally, the knowledgebase allows for the easy entry of descriptive information for each concept (species geographic and depth ranges, references, physical characteristics), thus the knowledgebase also serves as a marine identification reference guide available via the web which we call the Deep-Sea Guide.

Documentation and instructions for VARS Knowledgebase Interface.

VARS annotation application

The annotation application references the knowledgebase and allows researchers on MBARI ships and in the lab to make annotations of observations from ROV video and still images from time-lapse cameras, the Benthic Rover, and Autonomous Underwater Vehicles (AUV’s). VARS annotation entries are constrained by the knowledgebase such that only concepts that have been entered into the knowledgebase and approved by the knowledgebase manager can be entered into the database. Associations that enhance video observations can be added and include such things as shape, position, color, behavior, sample number, etc.

VARS for tape: This version allows for analysis of video recorded onto videotape. This version uses RS-422 protocol for deck control and a video capture card for video frame grabbing.

VARS for video: This version allows for analysis of digital video files. Common video containers are supported including QuickTime (.mov) and MPEG 4 (.mp4). The software has been developed with plugin architecture in mind so that additional video containers and codecs can be supported. Frame grabbing from the video file is supported.

VARS for images: This version of VARS allows for analysis of still images and video frame grabs. It allows the user to enumerate and identify objects by clicking on every object visible within the image. Image survey area, object position, and object or size can be calculated if certain camera parameters are known or if calibrated lasers are used. Note that survey pre-planning is imperative or image calculations may not be possible after data has been collected.

Documentation and instructions for VARS Annotation Application.

VARS query interface

The VARS query allows for easy searching of the observations stored in MBARI’s video annotation database. Complex queries can be made by constraining for concept, temporal, spatial, or physical parameters (for example: species, season, location, or depth). The VARS query references the knowledgebase. Query results are delivered as a table and include the concept and associations, image links, physical and ancillary data, dive information, and tape information. The query results can be saved as tabular text data, Google Keyhole Markup Language (for geospatial viewing), or as images (where they exist).

VARS for tape – This version searches the video tape database and includes video recorded from MBARI ROVs Ventana, Tiburon, and Doc Ricketts.

VARS for video – This version searches the digital video file database and includes video from fly-away ROVs and AUVs.

VARS for images – This version searches the digital still image database and that includes digital still images from a variety of camera platforms including ROVs, AUVs, stereo imagers, stationary camera platforms, time-lapse cameras, etc.

Documentation and instructions for VARS query interface.

Data characteristics disclaimer

Though they have proven to have great inherent and inferred scientific value, these data have not always been collected using procedures which render them consistent or quantitative. Biases have been introduced by observation techniques, camera settings, mission planning, and changing science terminology, among other factors. In no way should these data be viewed or analyzed as random, structured, or systematic observations. As with all databases, understanding how data were collected over time is necessary prior to using them in any analysis

For help accessing or using the VARS applications, email MBARI’s Video Lab team.

For reporting bugs or technical issues with VARS applications, email Brian Schlining (VARS software engineer).

To request or share questions regarding images, video, and related data for research and educational purposes, email Nancy Jacobsen Stout (Video Lab manager).