Interactive Multimedia

Interactive video search (also known as video browsing or exploratory video search) is a method of interactive information retrieval in video content that is not limited to the typical query-and-browse results approach employed by traditional retrieval tools. Instead, exploratory video search tools are highly interactive and try to integrate the user during several stages of the search process (through iterative human-computer interaction). This way they provide both visual browsing facilities (e.g., smart navigation features, content overviews and structuring means) as well as query features that allow users to translate their imaginations to automatic content retrieval (e.g., query by filtering, query by sketch, similarity search, etc.). Exploratory video search therefore supports content search situations where users cannot formulate a concrete query or simply want to inspect the video in order to see its content.

This kind of modern video browsing supports two content search scenarios: directed and undirected search. In the first scenario, users have a clear information need and want to find a specific target segment in the video (e.g., the weather forecast in a news show); such a search is also known as known-item search or target search. In the second scenario, users have no concrete search goal but want to explore the content in order to learn or find something interesting (e.g., a violent scene in surveillance videos); such a search scenario is known as exploratory search.

Over the years many tools for interactive video search and exploration have been proposed in the literature (see [1],[2] for a survey), and it has been shown that these tools can effectively help users find desired content in videos (e.g., [3],[4]). Some of these tools combine sophisticated content analysis methods controlled by the user for their personal needs. Some others provide rather simple content navigation features but give the users more interactivity to allow them to effectively take advantage of their knowledge about the content and the content structure. Interestingly, it has been shown that tools of the latter kind can even outperform tools of the first kind for some search tasks [5].

Slides of a tutorial on Interactive Video Search, presented at the ACM International Conference on Multimedia 2015 (ACM MM’15) in Brisbane, Australia, can be found here:


The Video Browser Showdown (aka Video Search Showcase) is an annual interactive video retrieval competition, held at the International MultiMedia Modeling (MMM) conference, where participating teams try to solve ad-hoc known-item search tasks in a shared data set of a few hundreds hours of video content as quickly as possible. The purpose of this search competition, which I am co-organizing, is to advance interactive video search tools to the next level. All the details about the VBS competition can be found this website:

Mobile Video Browsing

Im highly interested on the interaction with mobile multimedia systems, I have been a mobile app developer for several years, and I have co-organized international workshops on that topic too (e.g., Mobile Multimedia Computing 2014, Mobile Multimedia Computing 2015, and Mobile Multimedia Computing 2016, all three in conjunction with the IEEE International Conference on Multimedia & Expo.

Furthermore, I have recently conducted a research project that focused on interactive video search on mobile devices, the Next Generation Video Browsing (NGVB) project, an FWF funded translational research project (TRP 273-N15) that ran for three years. In the NGVB project we investigated how to design better tools for content-based search in videos, in particular when used on tablet devices, and how these search tools could benefit from the powerful features provided by tablets (e.g., multi-touch interaction and high computing power). We focused on improving video content navigation and visualization of the video structure through 3D models and color-sorted arrangement of images. To this end, we performed several initial user studies in order to find out how users search with common video player software and used the results for the design of novel search features. In order to evaluate these novel search features we developed software tools with new interaction models and search features and performed comparative user studies with them. The results of these studies show that the newly proposed features and interfaces provide significantly better search performance and are easier to use for the vast majority of tested users.

The research work of the NGVB project brought forward several interesting results and I would like to thank all involved people (in particular Dr. Marco A. Hudelist and Dr. Claudiu Cobarzan). The figure below shows an overview of research prototypes (i.e., interfaces) for video and image browsing that evolved from the project.


More details can be found on our project website (including related research papers).





opencvpaper2 opencvpaper tabletglobestackofrings

scrubbingwheel ringpaperflickpapervntpaper

mediastatisticsthumbbrowser 3dtabletinterfaces 3dinterfaces