DETAILED DESCRIPTION

[0021] In the following description, like reference numbers are used to identify like elements. Furthermore, the drawings are intended to illustrate major features of exemplary embodiments in a diagrammatic manner. The drawings are not intended to depict every feature of actual embodiments nor relative dimensions of the depicted elements, and are not drawn to scale.

[0022] The embodiments described below feature systems and methods of generating annotated video files from original video files, which may or may not have been previously annotated. The annotated video files include embedded information enabling the rendering of at least one video summary that is contained in the annotated video file and includes digital content summarizing at least a portion of the original video file. In this way, the video summaries are is always accessible to a rendering system because the resulting annotated video files contain the contents of both the original video file and the video summaries. Users may therefore quickly and efficiently browse through a collection of annotated video files without risk that the video summaries will become disassociated from the corresponding video files, regardless of the way in which the video files are transmitted from one rendering system to another.

[0023] As used herein, “video summary” refers to any digital content that summarizes (i.e., represents, symbolizes, or brings to mind) the content of an associated sequence of video frames of an original video file. The digital content of a video summary may be in the form of one or more of text, audio, graphics, animated graphics, and full-motion video. For example, in some implementations, a video summary may include one or more images representative of original video file content and digital audio content synchronized to the one or more representative images.

[0024] I. System Overview

[0025] Referring to FIG. 1, in one embodiment, a system for generating and rendering annotated video files includes a video file annotating engine 10 and a video file rendering engine 12. Both of these engines may be configured to operate on any suitable electronic device, including a computer (e.g., desktop, laptop and handheld computer), a video camera, or any other suitable video capturing, video editing, or video viewing system (e.g., an entertainment box, such as a video recorder or player, which is connected to a television).

[0026] In a computer-based implementation, both video file annotating engine 10 and video file rendering engine 12 may be implemented as one or more respective software modules operating on a computer 30. Computer 30 includes a processing unit 32, a system memory 34, and a system bus 36 that couples processing unit 32 to the various components of computer 30. Processing unit 32 may include one or more processors, each of which may be in the form of any one of various commercially available processors. System memory 34 may include a read only memory (ROM) that stores a basic input/output system (BIOS) containing start-up routines for computer 30 and a random access memory (RAM). System bus 36 may be a memory bus, a peripheral bus or a local bus, and may be compatible with any of a variety of bus protocols, including PCI, VESA, Microchannel, ISA, and EISA. Computer 30 also includes a persistent storage memory 38 (e.g., a hard drive, a floppy drive 126, a CD ROM drive, magnetic tape drives, flash memory devices, and digital video disks) that is connected to system bus 36 and contains one or more computer-readable media disks that provide non-volatile or persistent storage for data, data structures and computer-executable instructions. A user may interact (e.g., enter commands or data) with computer 30 using one or more input devices 40 (e.g., a keyboard, a computer mouse, a microphone, joystick, and touch pad). Information may be presented through a graphical user interface (GUI) that is displayed to the user on a display monitor 42, which is controlled by a display controller 44. Computer 30 also may include peripheral output devices, such as speakers and a printer. One or more remote computers may be connected to computer 30 through a network interface card (NIC) 46.

[0027] As shown in FIG. 1, system memory 34 also stores video file annotating engine 10, video rendering engine 12, a GUI driver 48, and one or more original and annotated video files 50. In some implementations, video file annotating engine 10 interfaces with the GUI driver 48, the original video files, and the user input 40 to control the generation and rendering of annotated video files. Video file rendering engine 12 interfaces with the GUI driver 48 and the annotated video files to control the video browsing and rendering experience presented to the user on display monitor 42. The original and annotated video files in the collection to be rendered and browsed may be stored locally in persistent storage memory 38 or stored remotely and accessed through NIC 46, or both.

[0028] II. Generating Annotated Video Files

[0029] Referring to FIG. 2, in some embodiments, an annotated video file may be generated as follows. Video file annotating engine 10 obtains an original video file (step 60). The original video file may correspond to any compressed (e.g., MPEG or Motion JPEG) or uncompressed digital video file, including video clips, home movies, and commercial movies. Video file annotating engine 10 also obtains information that enables at least one video summary to be rendered (step 61). Video file annotating engine 10 annotates the original video file by embedding the video summary rendering information in the original video file (step 62).

[0030] Referring to FIGS. 3A and 3B, in some embodiments, video summary rendering information 64 is embedded in the header 66 of an original video file 68 (FIG. 3A). In other embodiments, video summary rendering information 70, 72, 74 is embedded at different respective locations (e.g., locations preceding each shot) of an original video file 76 separated by video content of the original video file 76 (FIG. 3B). In some of these embodiments, pointers 78, 80 to the locations of the other video summary rendering information 72, 74 may be embedded in the header of the original video file 76, as shown in FIG. 3B.

[0031] In some implementations, the video summary rendering information that is embedded in the original video file corresponds to the video summary itself. As mentioned above, a video summary is any digital content (e.g., text, audio, graphics, animated graphics, and full-motion video) that summarizes (i.e., represents, symbolizes, or brings to mind) the content of the associated sequence of video frames of the original video file. Accordingly, in these implementations, the digital content of the video summaries are embedded in the original video files. In some implementations, a video summary may be derived from the original video file (e.g., keyframes of the original video file, short segments of the original video file, or an audio clip from the original video file). In other implementations, a video summary may be obtained from sources other than the original video file yet still be representative of the original video file (e.g., a trailer of a commercial motion picture, an audio or video clip, or a textual description of the original video).

[0032] Referring to FIGS. 4 and 5, in some embodiments, a video summary may be derived automatically from the original video file based on an analysis of the contents of the original video file. For example, in some implementations, video file annotating engine 10 may perform shot boundary detection, keyframe selection, and face detection and tracking using known video processing techniques. Shot boundary detection is used to identify discontinuities between different shots (e.g., shots 1, 2, 3 in FIG. 4), each of which corresponds to a sequence of frames that is recorded contiguously and represents a continuing action in time and space. Keyframe selection involves selecting keyframes that represent the content of each shot. In some implementations, the first frame in each shot that passes an image blur test is selected as a representative keyframe (e.g., keyframes 82, 84, 86 corresponding to frame numbers 1, 219, 393). In addition to the first frames of each shot, frames containing detected faces (e.g., frames 88, 89, 90, 92, 94) also are selected as representative keyframes; the selected frames may correspond to the first substantially blur-free frame in a contiguous sequence of frames in a shot containing the same face. Face tracking is used to associate frames containing the same person within a continuous video shot.

[0033] In some embodiments, the keyframes of each shot are organized into a hierarchy to allow a user to browse video summaries at multiple levels of detail. For example, in the illustrated embodiment, the first level of detail corresponds to the first keyframes 82, 84, 86 of each shot. The next level of detail corresponds to all of the keyframes of each shot arranged in chronological order. Other known hierarchical representations also may be used.

[0034] Referring to FIG. 5, in some embodiments, the video summary rendering information that is embedded in the original video file corresponds to pointers to frames of the original video file. In the illustrated embodiment, the pointers correspond to keyframe numbers of the representative keyframes that are identified by the automatic summarization process described above in connection with FIG. 4. In other embodiments, the pointers may correspond to rendering (or playback) times. In the illustrated embodiments, the pointers and hierarchical information are stored in an XML (extensible Markup Language) data structure 96 that may be embedded in the header or other location of the original video file. In hierarchical level 1 of data structure 96, keyframes 82, 84, 86 (see FIG. 4) respectively correspond to images kf-001.jpg, kf-0219.jpg, kf-0393.jpg. In hierarchical level 2, keyframes 82, 84, 86 correspond to images kf-001.jpg, kf-0219.jpg, kf-0393.jpg, respectively, as before; in addition, keyframes 88, 89, 90, 92, 94 (see FIG. 4) respectively correspond to images kf-0143.jpg, kf-0420.jpg, kf-0550.jpg, kf-0602.jpg, kf-0699.jpg. As explained above, each of keyframes 88, 89, 90, 92, 94 corresponds to images containing one or more detected faces. The video frame number range of each keyframe image is specified by respective “begin” and “end” tags.

[0035] Referring back to FIG. 2, after the video file has been annotated, video file annotating engine 10 stores the annotated video file (step 98). For example, the annotated video file may be stored in persistent storage memory 38 (FIG. 1).

[0036] III. Rendering Annotated Video Files

[0037] Referring to FIG. 6, in some embodiments, an annotated video file may be rendered by video file rendering engine 12 as follows. Video file rendering engine 12 obtains a video file that has been annotated in one or more of the ways described above (step 100). Video file rendering engine 12 identifies video summary rendering information that is embedded in the annotated video file (step 102). As explained above, the video summary rendering information may correspond to one or more video summaries that are embedded in the header or in other locations of the annotated video file. Alternatively, the video summary rendering information may correspond to one or more pointers to locations where respective video summaries are embedded in the annotated video file. Based on the video summary rendering information, video file rendering engine 12 enables a user to browse the summaries embedded in the annotated video file (step 104). Video file rendering engine 12 initially may render video summaries at the lowest (i.e., most coarse) level of detail. For example, in some implementations, video file rendering engine 12 initially may present to the user the first keyframe of each shot. If the user requests summaries to be presented at a greater level of detail, video file rendering engine 12 may render the video summaries at a greater level of detail (e.g., render all of the keyframes of each shot).

[0038] In some implementations, while the user is browsing video summaries, the user may select a particular summary (e.g., keyframe) as corresponding to the starting point for rendering the original video file. In response, video file rendering engine 12 renders the original video file beginning at the point corresponding to the video summary selected by the user (step 106).

[0039] IV. Conclusion

[0040] Other embodiments are within the scope of the claims.

[0041] The systems and methods described herein are not limited to any particular hardware or software configuration, but rather they may be implemented in any computing or processing environment, including in digital electronic circuitry or in computer hardware, firmware, or software.