Medical Multimedia Systems

An exciting and challenging research topic I am working on is content understanding in videos from surgical procedures and interaction with medical videos. The rational behind is the fact that these days many surgeons record videos during endoscopic/microscopic surgeries and inspections and archive them for later use, since these videos contain highly important information that can be used later for:

  • revisit specific moments of the surgery for retrospective analysis
  • explanations to patients
  • discussions among surgeons
  • reviews of technical skills
  • teaching and training

While this valuable video data enables completely new use scenarios it also bears new challenges, such as managing the┬ástorage of a medical video archive – that increases on a daily basis – as well as automatic content analysis (e.g., phase recognition, instrument detection, surgical action classification, etc.) in order to make it efficiently accessible. A broad overview of this special field of medical multimedia (including its challenges and opportunities, as well as different aspects and specialities) can be found in the slides of our tutorial, presented recently at the ACM Multimedia 2018 conference in Seoul, South Korea:

Over the last few years we at ITEC have put much effort into research on this challenging new topic. Among other problems we focused on:

  • compression optimization of endoscopic videos
  • content segmentation of endoscopic videos
  • relevance filtering of content from medical endoscopy
  • keyframe detection in endoscopic videos
  • content classification of surgical actions and anatomic structures
  • content similarity search for endoscopic video segments
  • visualization of endoscopic video archives
  • content annotation of endoscopic videos
  • deep learning surgical actions
  • automatic smoke detection
  • retrieval of video segments with specific┬ádescriptors

Here are a few papers that give more details.

Domain Specific Video CompressionHierarchical Visualization of Endoscopic KeyframesPhase Segmentation Through Instrument Detectiongynlearning
endotabletannotationendotmultibrowserGradient-Based SignaturesSupporting Surgical Quality Assessment

Additional information can be found at