This invention relates to the fields of communications and computer arts. Although the invention is not limited to the specific embodiments described below, these embodiments relate to wireless communications, data encoding and decoding, and particularly to user interaction with images or video (broadcast or on-demand).
There is a growing demand for interactive video services. Until now, this experience has been, at best, slow, clumsy, and not very interactive. For example, “instant voting” implementations simply encourage viewers to send text messages to numbers in order to vote for their favorite television personalities.
Unless they require specialized hardware, existing systems are neither instant nor truly interactive. In existing systems that do not have specialized hardware, the basic problem is that the display system is physically separate from any system that could provide the interactive user experience. Further, most display systems do not provide any mechanism for viewer input and are designed to provide images and video to the viewer, not to provide information about the images and video. For example, if the viewer's interactive device is a mobile communication device, such as a mobile phone, the device does not have any method of discovering the context in which it is operating, such as which program the viewer is watching. Instead, the viewer/mobile device user must supply this intelligence, greatly reducing the convenience and usability. There is a clear need for a system that can work with existing image and video delivery systems, viewing systems, and interactive communication devices to provide a truly interactive experience.
The invention consists of a complete system for delivering an interactive image or video experience. The system encodes information into the images or video. The viewer of these images or video can decode the information from the images and immediately interact with the images using almost any computing device, but the system is particularly well suited to mobile communication devices, for example, mobile camera phones with internet access.
For example, by allowing the viewer's mobile device to extract specific information from the image, this invention greatly simplifies the interactive experience. With this invention, the viewer simply uses their mobile device to acquire the images or video containing the encoded information. Depending on what system is used to distribute the images or video, the viewer may receive these images or video directly onto their mobile device, or the viewer may need to aim the mobile device in the direction of the external display that is showing the images or video so that software in the mobile device can use the device's camera to capture the images or video. In either case, the mobile device software then quickly extracts and decodes the encoded information. The decoded information may, for example, include the identity of the image or video program being viewed and phone numbers to call or text message.
Knowing the identity of the display makes it possible for software in the mobile device to display information and provide interaction unique to the images or video being viewed. For example, the interactive device can provide additional information about the program or even individual video elements, e.g. horses in a race, poker hands, athletes, favorite entertainers, etc., allow the viewer to vote by selecting and clicking a box, or even allow the user to quickly change their device settings.
This invention's utility is not limited to broadcast media. It is also useful for recorded images or video and even static or printed images.
The nature, objects, and advantages of the invention will become more apparent to those skilled in the art after considering the following detailed description in connection with the accompanying drawings, in which like reference numerals designate like parts throughout, and wherein:
FIG. 1 provides an overview of an example of a complete system where the distribution system delivers the image(s) or video to an external display system rather than directly to the viewer's interactive device. In this case, the information extraction and decoding system captures the image(s) or video from the external display system.
FIG. 2 provides an overview of an example of a complete system where the distribution system delivers the images or video directly to the viewer's interactive device.
FIG. 3 provides a flowchart for the encoding and distribution processes.
FIG. 4 shows the process of capturing an image/images or video from an external display.
FIG. 5 provides a flowchart for the extraction, decoding, and user interaction processes.
FIG. 6 provides an example of display mapping and transformation.
FIGS. 1-6 illustrate examples of various apparatus, method, and article of manufacture aspects of the present invention. For ease of explanation, but without any limitation intended, these examples are described in the context of existing digital signal processing and existing image and video distribution and display apparatuses.
The invention consists of several components, as shown in FIGS. 1 and 2:
An interactive response subsystem ( 102 and 202 )
An image processing and encoding subsystem ( 101 and 201 )
A distribution subsystem ( 105 and 205 )
A display subsystem ( 106 )
An information extraction and decoding subsystem ( 108 and 208 )
An interactive user interface subsystem ( 109 and 209 )
The interactive response subsystem ( 102 and 202 ) manages the interaction with image viewers. It is responsible for:
The image processing and encoding subsystem ( 101 and 201 ) processes images or video frames and encodes the information into the images. The information can come from any system, including the interactive response system, or the operator can enter the information manually using the image processing and encoding system's operator interface.
The distribution subsystem ( 105 and 205 ) distributes the images or image frames to the end users who will view the images on a display system. Any existing system will work, for example: video via terrestrial wireless, satellite, or cable broadcast; via internet download or broadcast; or via physical media, for example, DVD, CD, flash memory, or even printed images. The distribution subsystem may distribute the images or video containing encoded information directly to the viewer's information extraction and decoding subsystem, as shown in FIG. 2, or it may send the images or video to a display system from which the viewer must capture the images or video, as shown in FIG. 1.
The display subsystem ( 106 ) displays the images or image frames to the viewer. Any existing system will work, for example, television, printed images, computer-driven monitors, video playback systems, etc. The display subsystem is not required if the images or video can be sent directly to the device as shown in FIG. 2.
The information extraction and decoding subsystem ( 108 and 110 in FIG. 1; 208 and 210 in FIG. 2). Any programmable system that is capable of basic image processing can fill this role. If this subsystem will be used to capture images or video from external display systems (rather than, or in addition to, receiving the images or video directly), it should include a digital camera. As an example, a mobile camera phone can be used as the embodiment of this subsystem. This system performs several tasks:
The interactive user interface subsystem ( 109 and 108 in FIG. 1; 208 and 209 in FIG. 2) allows the user to interact with the image or video. Using the information encoded in the images or video frames, this subsystem presents information and choices to the user. This subsystem may use information cached locally on the device, or it may query the interactive response system for the information. The interactive user interface subsystem may also report user selections to the interactive response subsystem or other systems and subsystems. This subsystem may send reports via any number of methods, for example, the internet, SMS (short messaging system), instant messaging, or email. Usually, this subsystem runs on the same hardware platform as the information extraction and decoding system, for example, a mobile camera phone.
The processing flows for an example embodiment are shown in FIGS. 3 through 6. FIG. 3 shows the encoding and distribution processes. FIG. 5 shows the extraction, decoding, and user interaction processes. In the case where the viewer's device must capture the image(s) or video from an external display rather than receiving the image(s) or video directly from the distribution system, FIG. 4 provides additional detailed information on an example embodiment of the image capture process. FIG. 6 provides an example of how image elements or regions are mapped to interactive display elements.
In these example embodiments, the information extraction and decoding subsystem and the interactive user interface subsystem are implemented in a mobile camera phone or video device, but these subsystems can be implemented in any programmable computing device with basic user interface and image processing capabilities.
The encoding and distribution process is shown in FIG. 3. This process involves the image processing and encoding ( 301 ), interactive response ( 302 ), and distribution ( 305 ) subsystems.
The process flow diagrams shown in FIG. 5 describe the extraction, decoding, and user interaction processes performed by the information extraction and decoding subsystem ( 502 ). FIG. 4 shows an example of the image capture process in the case where the display subsystem ( 400 ) is external to the information extraction and decoding system ( 502 ). FIG. 6 shows an example how regions or elements ( 604 ) of an image ( 601 ) are mapped to interactive elements ( 605 ) in the display of the viewer's interactive device ( 603 ).
Details of the example embodiment are as follows:
The system may be implemented as shown in FIG. 1 or 2 . For example, the image processing subsystem ( 101 , 201 ) is configured with specialized software to process video and image data. Here, the term “software” is used broadly and comprises, for example, a machine readable language construct that specifies an operation and identifies operands (instructions), application programs, algorithms, software configuration data, multimedia data, video data, and audio data. These data may reside in any type of storage unit using any type of data storage media. In various embodiments, the software may comprise or emulate lines of compiled “C-type” language, “Java-type” interpreted or pre-compiled language, source code, object code, executable machine code, executable programs, data banks, or other types of commonly known data.
The image processing subsystem ( 101 , 201 ) may, for example, be a standard personal computer, a personal computer with specialized video and image processing hardware and software, or a specialized, computer-based image and video processing system.
The interactive response subsystem ( 102 , 202 ) may be implemented on different hardware and software computing platforms. Platform selection depends on many factors, including: the number of transactions and viewer requests, size of the image/identifier/overlay database, response time, etc.
The invention does not require a specialized distribution subsystem ( 105 , 205 ), so it can use almost any existing or readily available distribution system.
The display subsystem ( 106 ) is an optional component of the invention. The invention works with almost any existing or readily available display subsystem.
The information extraction and decoding subsystem ( 108 , 208 ) and the interactive user interface system ( 109 , 209 ) can be implemented using almost any programmable computing platform that is capable of basic image processing and user interaction, but are ideally suited for implementation in mobile phones and video devices. Optional platform features, such as digital cameras and high speed internet access, can dramatically enhance the user experience, but are not required.
Despite the specific foregoing descriptions, ordinarily skilled artisans having the benefit of this disclosure will recognize that the apparatus, method, and article of manufacture discussed above may be implemented in an apparatus, system, method, or article of manufacture of different construction without departing from the scope of the invention. Similarly, parallel methods may be developed.
For example, without departing from the scope of the invention, future embodiments may combine or improve components and functions for the sake of more efficient and/or accurate processing. Other possible enhancements include the addition of error detection and recovery methods. The embodiment of the entire system or individual components may need to be adapted to meet higher throughput, capacity, and reliability requirements.