Text Segmentation and Text Recognition in Digital Videos

Abstract

There is no doubt that video is an increasingly important modern information medium. Setting free its complete potential and usefulness requires efficient content-based indexing and access. One powerful high-level index for retrieval is the text contained in videos. This index can be built by detecting, extracting and recognizing such text. The index enables the user to submit sophisticated queries such as a listing of all movies featuring John Wayne or produced by Steven Spielberg. Or it can be used to jump to news stories about a specific topic, since captions in newscasts often provide a condensation of the underlying news story. For example, one can search for the term "Financial News" to get the financial news of the day. The index can also be used to record the broadcast time and date of commercials, helping the people who check for their clients whether their commercial has been broadcasted at the arranged time on the arranged television channel. Many other useful high-level applications are imaginable if text can be recognized automatically and reliably in digital video.

Unlike other systems using text for the indexing of videos we do not take advantage of close-captions that might be transmitted on some television channels; rather, our system extracts the text from the video itself. The algorithms we propose make use of typical characteristics of text in videos to enable and enhance segmentation and recognition performance. We also demonstrate their suitability for indexing and retrieval by our video retrieval application.

Video Examples

First Approach
Some video samples of our (older) text segmentation algorithms for grayscale videos (SPIE 2666):

video type	orignal movie	segmented movie
moving text, stationary scene	example1 (1508 KB)	example1 (737 KB)
moving text, moving scene	example1 (898 KB)	example1 (1508 KB)
stationary text, moving scene	example1 (1036 KB)	example1 (593 KB)

Second Approach
Some video samples of our (older) segmentation algorithms for color videos (ACM Multimedia 1996):

video type original video segmented video

closing sequence with
title and credits (1.9 MB) (2.2 MB)

closing sequence with
title and credits (3.6 MB) (2.3 MB)

commercial (717 KB) (327 KB)

New Approach
Now some video samples of our new segmentation algorithms for color videos (TR-98-009):

video type original and segmented video

closing sequence with
title and credits Demo Video (1.9 MB)

closing sequence with
title and credits (2.4 MB)

commercial (1.8 MB)

Source Code

Text segmentation:

for grayscale videos (359 KB) (SPIE 2666):
for color videos (279 KB) (ACM Multimedia 1996)
for color videos (see general MoCA download page) (TR-98-009)

Text Recognition:

for color videos (ACM Multimedia 1996)

Publications

Rainer Lienhart and Frank Stuber, Automatic text recognition in digital videos, in Image and Video Processing IV 1996, Proc. SPIE 2666-20 (1996). [Abstract] [PS : 242KB]; also Technical Report TR-95-036, December 1995.
Rainer Lienhart, Automatic Text Recognition for Video Indexing, Proc. ACM Multimedia 96, Bosten, MA, Nov. 1996, pp. 11-20. [Abstract] [PS : 631KB]; also Technical Report TR-96-006 (gzipped 631723 bytes), April 1996.
Rainer Lienhart and Wolfgang Effelsberg. Automatic Text Segmentation and Text Recognition for Video Indexing. Technical Report TR-98-009, Praktische Informatik IV, University of Mannheim, Mai 1998. [Abstract] [PS : 453 KB] [PDF : KB]