20100161786 | QOS CONTROL METHOD FOR ISCSI | June, 2010 | Cao et al. |
20040073648 | Network calculator system and management device | April, 2004 | Tanino et al. |
20020129164 | Advanced path checker | September, 2002 | Van Der et al. |
20160344789 | WIRELESS MEDIA STREAMING SYSTEM | November, 2016 | Watts et al. |
20080189383 | DISTRIBUTED CACHE BETWEEN SERVERS OF A NETWORK | August, 2008 | Schuh et al. |
20020138641 | Targeted multimedia proxy server (tmps) | September, 2002 | Taylor et al. |
20030055982 | Communications control method, relaying method and relaying device | March, 2003 | Noro et al. |
20070174395 | Systems, methods, and media for communication with database client users | July, 2007 | Bostick et al. |
20080082665 | METHOD AND APPARATUS FOR DEPLOYING SERVERS | April, 2008 | Dague et al. |
20060075091 | System and method for historical presence map | April, 2006 | Beyda et al. |
20120303792 | METHOD AND APPARATUS FOR PROVIDING RECOMMENDATIONS WITHIN CONTEXT-BASED BOUNDARIES | November, 2012 | Sathish |
[0001] Applicant(s) hereby claims the benefit of the following provisional patent applications:
[0002] provisional patent application serial no. 60/177,397, titled “VIRTUAL SET ON THE INTERNET,” filed Jan. 21, 2000, attorney docket no. 38903-007;
[0003] provisional patent application serial no. 60/177,394, titled “MEDIA ENGINE,” filed Jan. 21, 2000, attorney docket no. 38903-004;
[0004] provisional patent application serial no. 60/177,396, titled “TAP METHOD OF ENCODING AND DECODING INTERNET TRANSMISSIONS,” filed Jan. 21, 2000, attorney docket no. 38903-006;
[0005] provisional patent application serial no. 60/177,395, titled “SCALABILITY OF A MEDIA ENGINE,” filed Jan. 21, 2000, attorney docket no. 38903-005;
[0006] provisional patent application serial no. 60/177,398, titled “CONNECTION MANAGEMENT,” filed Jan. 21, 2000, attorney docket no. 38903-008;
[0007] provisional patent application serial no. 60/177,399, titled “LOOPING DATA RETRIEVAL MECHANISM,” filed Jan. 21, 2000, attorney docket no. 38903-009;
[0008] provisional patent application serial no. 60/182,434, titled “MOTION CAPTURE ACROSS THE INTERNET,” filed Feb. 15, 2000, attorney docket no. 38903-010; and
[0009] provisional patent application serial no. 60/204,386, titled “AUTOMATIC IPSEC TUNNEL ADMINISTRATION,” filed May 10, 2000, attorney docket no. 38903-014.
[0010] Each of the above listed applications is incorporated by reference herein in its entirety.
[0011] A portion of the disclosure of this patent document contains material which is subject to copyright protection. The copyright owner has no objection to the facsimile reproduction by anyone of the patent document or the patent disclosure, as it appears in the Patent and Trademark Office patent files or records, but otherwise reserves all copyright rights whatsoever.
[0012] This application is related to the following commonly owned patent applications, filed concurrently herewith, each of which applications is hereby incorporated by reference herein in its entirety:
[0013] application serial no.______, titled “METHOD AND SYSTEM FOR DISTRIBUTING VIDEO USING A VIRTUAL SET,” attorney docket no. 4700/2;
[0014] application serial no.______, titled “SYSTEM AND METHOD FOR ACCOUNTING FOR VARIATIONS IN CLIENT CAPABILITIES IN THE DISTRIBUTION OF A MEDIA PRESENTATION,” attorney docket no. 4700/4;
[0015] application serial no.______, titled “SYSTEM AND METHOD FOR MANAGING CONNECTIONS TO SERVERS DELIVERING MULTIMEDIA CONTENT,” attorney docket no. 4700/6; and
[0016] application serial no.______, titled “SYSTEM AND METHOD FOR RECEIVING PACKET DATA MULTICAST IN SEQUENTIAL LOOPING FASHION,” attorney docket no. 4700/7.
[0017] The invention disclosed herein relates generally to techniques for distributing multimedia content across computer networks. More particularly, the present invention relates to improved systems and methods for efficiently dividing processing between servers distributing content and clients playing back the received content, thereby allowing a richer experience and maximizing processing power of both clients and servers.
[0018] Over the past decade, processing power available to both producers and consumers of multimedia content has increased exponentially. Approximately a decade ago, the transient and persistent memory available to personal computers was measured in kilobytes (8 bits=1 byte, 1024 bytes=1 kilobyte) and processing speed was typically in the range of 2 to 16 megahertz. Due to the high cost of personal computers, many institutions opted to utilize “dumb” terminals, which lack all but the most rudimentary processing power, connected to large and prohibitively expensive mainframe computers that “simultaneously” distributed the use of their processing cycles with multiple clients.
[0019] Today, transient and persistent memory is typically measured in megabytes and gigabytes, respectively (1,048,576 bytes=1 megabyte, 1,073,741,824 bytes=1 gigabyte). Processor speeds have similarly increased, modern processors based on the x86 instruction set are available at speeds up to 1.5 gigahertz (approximately 1000 megahertz=1 gigahertz). Indeed, processing and storage capacity have increased to the point where personal computers, configured with minimal hardware and software modifications, fulfill roles such as data warehousing, serving, and transformation, tasks that in the past were typically reserved for mainframe computers. Perhaps most importantly, as the power of personal computers has increased, the average cost of ownership has fallen dramatically, providing significant computing power to average consumers.
[0020] The past decade has also seen the widespread proliferation of computer networks. With the development of the Internet in the late 1960's followed by a series of inventions in the fields of networking hardware and software, the foundation was set for the rise of networked and distributed computing. Once personal computing power advanced to the point where relatively high speed data communication became available from the desktop, a domino effect was set in motion whereby consumers demanded increased network services, which in turn spurred the need for more powerful personal computing devices. This also stimulated the industry for Internet Service providers or ISPs, which provide network services to consumers.
[0021] Computer networks transfer data according to a variety of protocols, such as UDP (User Datagram Protocol) and TCP (Transport Control Protocol). According to the UDP protocol, the sending computer collects data into an array of memory referred to as a packet. IP address and port information is added to the head of the packet. The address is a numeric identifier that uniquely identifies a computer that is the intended recipient of the packet. A port is a numeric identifier that uniquely identifies a communications connection on the recipient device.
[0022] Once the data packet is addressed, it is transmitted from the sending device across a network via a hardware network adapter, where intermediary computers (e.g., routers) relay the packet to the appropriate port on the device with the appropriate unique IP address. When data is transmitted according to the UDP protocol, however, no attempt is made to inform the sender that the data has successfully arrived at the destination device. Moreover, there is neither feedback from the recipient regarding the quality of the transmission nor any guarantee that subsequent data sent out sequentially by the transmitting device will be received in the same sequence by the recipient.
[0023] According to the Transmission Control Protocol, or TCP, data is sent using UDP packets, but there is an underlying “handshake” between sender and recipient that ensures a suitable communications connection is available. Furthermore, additional data is added to each packet identifying its order in an overall transmission. After each packet is received, the receiving device transmits acknowledgment of the receipt to the sending device. This allows the sender to verify that each packet of data sent has been received, in the order it was sent, to the receiving device. Both the UDP and TCP protocols have their uses. For most purposes, the use of one protocol over the other is determined by the temporal nature of the data. Data can be viewed as being divided into two types, transient or persistent, based on the amount of time that the data is useful.
[0024] Transient data is data that is useful for relatively short periods of time. For example, a television transmits a video signal consisting of 30 frames of imagery each second. Thus, each frame is useful for {fraction (1/30)}
[0025] UDP is useful for the transmission of transient data, where the sender does not need to be delayed verifying the receipt of each packet of data. In the above example, a television broadcaster would incur an enormous amount of overhead if it were required to verify that each frame of video transmitted has been successfully received by each of the millions of televisions tuned into the signal. Indeed, it is inconsequential to the individual television viewer that one or even a handful of frames have been dropped out of an entire transmission. TCP, conversely, is useful for the transmission of persistent data where the failure to receive every packet transmitted is of great consequence.
[0026] One of the reasons that the Internet is a successful medium for transmitting data is because the storage of information regarding identity and location of devices connected to it is decentralized. Knowledge regarding where a device resides on a particular part of the network is distributed over a plurality of sources across the world. A connection between to remotely located devices can traverse a variety of paths such that if one path becomes unavailable, another route is utilized.
[0027] The most simplistic path is between two devices located within a common Local Area Network, or LAN. Because these devices are located on a common network, they are said to reside in the same subnet. Each network on the Internet is uniquely identified with a numeric address. Each device within a network, in turn, is identified by an IP address that is comprised of a subnet address coupled with a unique device ID. According to version four of this standard (“IPv4”) an IP address is a 32-bit number that is represented by four “dot” separated values in the range from 0 through 255, e.g., 123.32.65.72. Each device is further configured with a subnet mask. The mask determines which bits of a device's IP address represent the subnet and which represent the device's ID. For example, a device with an IP address of 123.32.65.72 and a subnet mask of 255.255.255.0 has a subnet address of 123.32.65 and an ID of 72.
[0028] Each packet of data sent by a device, whether it is formatted according to the UDP or TCP protocols, has a header data field. The header is an array of bytes at the beginning of a packet that describe the data's destination, its origin, its size, etc. When a sender and recipient are both located within the same subnet, the recipient device's network hardware examines network traffic for packets tagged with its address. When a packet addressed to the recipient is identified, the network hardware will pass the received data off to the operating system's network services software for processing.
[0029] When a sender and recipient are located in different subnets, data is relayed from the originating subnet to the destination subnet primarily through the use of routers. While other physical transport methodologies are available, e.g., circuit switched transmission systems such as ATM (Asynchronous Transfer Mode), the majority of computer networks utilize packet switched hardware such as routers. A router is a device that interconnects two networks and contains multiple network hardware connections. Each network connection is associated with, and provides a connection to, a distinct subnet.
[0030] Two tasks are performed when a packet, destined for a subnet that is different from the subnet it is currently in, reaches a router within the current subnet. First, the router will examine the subnets that it is connected to via its network hardware. If the router is connected to the packet's destination subnet, it forwards the packet to the router in the appropriate subnet. If the router is not directly connected to the packet's destination subnet, it will query other routers available on its existing connections to determine if any of them are directly connected to the destination subnet. When a router directly connected to the destination subnet is discovered, the packet will be forwarded to it. Where a router connected to the destination subnet is not found, however, the router will propagate the packet to a top level router that is strategically placed to allow access, either directly or through other top level routers, to the entire Internet. These top level routers are currently maintained by a registration authority under government oversight.
[0031] The transmission method described above is referred to as the unicast method of transmission, whereby a sender establishes a unique connection with each recipient. By utilizing a unicast model, the specific address of the receiving machine is placed in the packet header. Routers detect this address and forward the packet so that it ultimately reaches its intended recipient. This method, however, is not the most efficient means for distributing information simultaneously to multiple recipients. The transmission method that best facilitates broadcasting to many recipients simultaneously is multicasting.
[0032] Multicasting relies on the use of specialized routers referred to as multicast routers. These routers look only for data packets addressed to devices in the range of 224.0.0.0 through 239.255.255.255. This address range has been specifically set aside for the purpose of facilitating multicast transmissions. Multicast routers retain an index of devices that wish to receive packets addressed to ports in this address range. Recipients wishing to receive multicast packets “subscribe” to a specific IP address and port within the multicast address space. The multicast routers respond to the subscription request and proceed to forward packets destined to the particular multicast address to clients who have subscribed to receive them.
[0033] Under the multicast model, the sender transmits packets to a single address, as opposed to the unicast model where the data is transmitted individually to each subscribing recipient. The multicast routers handle replication and distribution of packets to each subscribing client. The multicast model, like the broadcast model, can be conceptually viewed as a “one-to-many” connection and, therefore, must use the UDP protocol. UDP must be utilized because the TCP protocol requires a dialog between the sender and receiver that is not present in a multicast environment.
[0034] Clearly, there have been drastic improvements in the computer technology available to consumers of content and in the delivery systems for distributing such content. There have also been drastic improvements in bandwidth available for transmission of video and other multimedia, such as through the use of broadband systems such as digital subscriber lines and cable systems employing cable modems.
[0035] However, such improvements have not been properly leveraged to improve the quality and speed of video distribution. For example, existing multimedia distribution systems over the Internet offer the content to all or to classes of clients on the same basis, much as is done in traditional analog systems such as television broadcasting. As a result, these systems fail to take account of the wide variability in processing, storage and other technical capabilities among personal computers.
[0036] There is thus a need for a system and method that takes account of such client variability, and that furthermore distributes responsibilities for video distribution and presentation among various components in a computer network to more effectively and efficiently leverage the capabilities of each part of the network and improve overall performance.
[0037] It is an object of the present invention to solve the problems described above relating to existing content delivery systems.
[0038] It is another object of the present invention to alter the traditional model of distribution of video and other multimedia content to take advantage of each particular client's capabilities.
[0039] It is another object of the present invention to make the distribution of video and other multimedia content more efficient and effective through a better distribution of tasks.
[0040] The above and other objects are achieved by a system and method that allows the efficient distribution of rich media to clients by maximizing the use of available bandwidth and client processing capabilities. Rich media refers generally to multiple types of digital media that are directly sensed by a viewer, including video, audio, text, graphics, and the combination of these and other media. In accordance with the invention, a rich media presentation is divided into discrete components, and a producer specifies how a presentation is to be assembled and where resources needed for the presentation are to be found. This information is packaged into a data structure and sent to clients. Clients use this data structure to retrieve the necessary resources for the presentation.
[0041] This modularization of the presentation provides numerous advantages. For example, producers are able to prioritize the particular resources that form part of the ultimate presentation according to their importance in the presentation. It also allows clients to retrieve the resources most suitable for their capabilities, including processing power, graphics production speed, and bandwidth. A benchmarker routine running on the client helps identify these capabilities just before retrieval of the presentation components, to more closely assess the conditions under which the client will retrieve, assemble and present the desired show.
[0042] In preferred embodiments, the client device works in a highly autonomous manner, thereby allowing the server to use multicast techniques to distribute data to many clients simultaneously. This autonomy allows each client the ability to display received rich media in the most effective way allowed by each individual client's hardware profile.
[0043] Some of the objects of the present invention are method for preparing a multimedia presentation for transmission to a client over a network. The method involves allowing a producer of the presentation to identify elements of software which process data representing resources used in the presentation, specify connections between two or more elements, wherein a connection represents a flow of data from one element to another element, and specify a plurality of resources to be processed by the identified elements and location data indicating the locations of the resources on one or more servers connectable to the network. The method further involves generating a set of presentation data structures, including the identified elements, connections, and resources, for transmission to a client. This enables the client to reproduce the multimedia presentation from the presentation data structures by retrieving at least some of the resources and processing the retrieved resources with the identified elements in accordance with the specified connections.
[0044] The set of presentation data structures may include a show graph representing a plurality of identified elements and the connections extending therebetween. The presentation data structures may also include a table of contents listing all or some of the specified resources and corresponding locations. The producer may also be able to replace or swap a first identified element with a second element while retaining for the second element any connection and resources associated with the first element.
[0045] As another advantage of the present invention, producers may further specify an encoder to tap a signal on a selected connection and encode the data flowing at the selected connection. The producer may associate the specified encoder with a given set of client processing capabilities or available bandwidth for transmission to a client. Thus, if the server receives data indicating the processing capabilities of a client requesting the presentation or available bandwidth for transmission of the presentation to the client, such as from a benchmarking program running on the client, an agent on the server can traverse a series of identified elements via their connections until a tapped encoder is located being associated with the client's processing capabilities or available bandwidth.
[0046] The data flowing at the tapped connection, e.g., the data output from the element at the start of the connection, is then encoded using the specified encoder, and made available for transmission to the client. When the client receives the presentation data structures, it executes an agent which traverses the series of identified elements via their connections until a tapped decoder is located being associated with the client's processing capabilities or available bandwidth. The agent initiates this decoder to decode the data at the tapped connection.
[0047] Objects of the present invention are also achieved by a system for delivering a multimedia presentation from a server to a client. The system contains a presentation authoring tool for use by a producer of the presentation to identify software elements for processing data representing resources used in the presentation, specify connections between two or more elements representing a flow of data from one element to another element, specify a plurality of resources to be processed by the identified elements and location data indicating the locations of the resources on one or more servers connectable to the network, and specify an encoder to tap a signal on a selected connection and encode the data flowing at the selected connection. The system also contains an agent for traversing a series of identified elements via their connections until a tapped encoder is located and encoding the data at the tapped connection using the specified encoder.
[0048] The invention is illustrated in the figures of the accompanying drawings which are meant to be exemplary and not limiting, in which like references are intended to refer to like or corresponding parts, and in which:
[0049]
[0050]
[0051]
[0052]
[0053]
[0054]
[0055]
[0056]
[0057]
[0058]
[0059]
[0060] Embodiments of the present invention are now described with reference to the drawings in FIGS.
[0061] The data stream
[0062] As described more fully below, the breakdown of the show into discrete components allows for a number of enhancements to reduce bandwidth, increase efficiency, and improve the ultimate quality of the show on the client computers. Different renderers process the different media types. Producers adjust settings within the show. Clients use renderers differently in accordance with client processing capabilities or available bandwidth.
[0063]
[0064] Referring to
[0065] In accordance with the invention, the Media Server
[0066] The Packetized Data Source Structure
[0067] One or more Looping Data Senders
[0068] The operation of the Packetized Data Source Structure and Looping Data Senders are described in greater detail in commonly owned patent application Ser. No.______ filed on even date herewith and titled “SYSTEM AND METHOD FOR RECEIVING PACKET DATA MULTICAST IN SEQUENTIAL LOOPING FASHION,” which has been incorporated herein by reference. Among other things, the use of this delivery mechanism in the show distribution system described herein enlarges the number of connections that can be made to receive a show at any one time.
[0069] The server
[0070] The Media Player
[0071] The Media Player
[0072] Client device
[0073] When a client
[0074] The Connection Manager
[0075] Producers of multimedia content use the Show Graph Authoring Tool
[0076] The Show Graph
[0077] The Show Graph Authoring Tool
[0078] The Show Graph Authoring Tool
[0079] The Show Graph
[0080] Client systems vary widely in terms of their computational and graphics power. When a producer generates a Show Graph
[0081] The Media Server
[0082] Providing pointers or links to a plurality of identical Resources of varying resolutions or detail allows the system to provide for potentially unlimited scalability. The client
[0083] The Benchmarker
[0084] One side effect of this process is that measurements of the amount of time a process takes using a real world clock may be different than the actual amount of CPU time the process needs. Averaging minimizes these inconsistencies and allows a more accurate appraisal of the capabilities of a client
[0085] The client
[0086] As described above, Resources
[0087] In preferred embodiments, Media Servers
[0088] Agents
[0089] In accordance with preferred embodiments of the invention, the Gather Agent
[0090] A Scatter Agent
[0091] One embodiment of a process using the system of
[0092] Once all child paths have been traversed, the Gather Agent's data buffer is delivered to the client, step
[0093] FIGS.
[0094] Elements are arranged within the Show Graph Authoring Tool. The Producer places Taps within the Show Graph to determine the most advantageous point to analyze and transmit the data, creating Shows of different quality for clients with different capabilities.
[0095] The low bandwidth Tap
[0096]
[0097] Placement of Taps
[0098] One embodiment of a process using the system of FIGS.
[0099] Once the Agent arrives at the specified Tap, data contained within its buffer is encoded, the Agent deactivates, and traverses the Show Graph until arriving at the Show Node, step
[0100] The encoded data buffer is transmitted from the server, step
[0101] Data in the Scatter Agent's buffer is decoded by the Tap, step
[0102] Resources
[0103]
[0104] The Benchmarker calculates final CPU processing and graphics fill time by performing a weighted averaging of transformation and fill times, step
[0105] While the invention has been described and illustrated in connection with preferred embodiments, many variations and modifications as will be evident to those skilled in this art may be made without departing from the spirit and scope of the invention, and the invention is thus not limited to the precise details of methodology or construction set forth above as such variations and modifications are intended to be included within the scope of the invention.