User-centered Convergence    
Today and Tomorrow

1/4/12

CoTV (Coactive TV) includes a variety of advanced interactive Web services that are synchronized with TV viewing.  Such services provide "enhancements" to the video, including related content, social Web functions, etc.  Coactivity of video and Web services can occur on a single screen, but is often far more powerful when done across multiple screens.

• Many advanced CoTV technologies are described on this Web site, and more fully in Reisman patent filings, including many that are still far beyond current offerings.
• The introduction of the iPad finally brought wide realization of the value of multi-screen CoTV, with the iPad (and/or iPhone) as a TV "companion device." Numerous CoTV "companion" apps emerged beginning in 2009, and are now coming from a full range of companies.  CoTV (and coactive media, more broadly) generalizes this functionality with extensive development of architectures for "Multi-Machine User Interfaces" (MMUIs).
• One key aspect of CoTV that is beginning to be recognized by some providers is the need for ubiquity -- CoTV should work for any program on any channel (or Internet source), without need to open a program-specific app (and to first download it).
• Advanced features not yet widely recognized include flexible targeting of enhancement content as services to any screen, alternative enhancement channels from any independent source, link-and-pause features, and media concierge services (as described further below).
• The essential concept of CoTV is that we, as users, want to browse media of any kind, from any source, on any desired device, in any combination, all fully hyperlinked. (We may want to restrict this for convenience and simplicity at certain times, but this full functionality must be there when we want it.)
• CoTV also comes from a user-centric perspective of open services (as seen in the Web), and -- while supportive of the value provided by content providers, curators, and other industry stakeholders -- looks beyond proprietary boundaries and walled gardens to the full range of open hypermedia services to come.

These features are summarized further below, and many are described in this Web site.  Much fuller descriptions are contained in extensive Reisman patent filings dating from 2002. The first two of those patents were granted in 2011, with more pending.

CoTV Yesterday

While many rich features of interactive TV date from decades ago, in many respects the vision has been limited by the barriers between the TV and Internet industries, and the blinders of industry thinking about TV.

• 1-Screen: Most concepts of ITV had been oriented to the single TV screen, the inherently limiting experience of interactivity with the "10 foot interface" and the idea that TV viewers always want a "lean-back" experience (not the "lean forward" "2-foot interface" experience of PC, phone, or tablet).
• 2-Screen: Early experiments in 2-screen ITV such as ABCs Enhanced TV (dating from 1999) were conceived as experiments to try ITV out when wide deployment of 1-screen ITV infrastructures were not available, and were expected to fade away when 1-screen became widespread.
• Trivia:  Essentially all early ITV trials provided very simplistic enhancement content, even on 2-screen PC services not constrained by the limitations of the single-screen. It seems TV producers were afraid of losing eyeballs, and saw little value in going beyond brain-dead enhancements like polls and trivia in small snippets.
• Commerce: The driving force for ITV was recognized to be advertising, and "T commerce," but in the limited form of "buying Jennifer Anniston's sweater" or submitting a "Request For Information," not the full richness of Web shopping.

CoTV Today -- CoTV 1.0

The increasing prevalence of "media multitasking" (simultaneous use of TV and the Web) on laptops and smartphones began to change perceptions, and 2-screen ITV began to be seen as desirable in itself. Users were creating their own manual ITV experiences by finding relevant Web services on their own.  That set the stage for the emergence of CoTV 1.0, which was then kick-started by the iPad.  One indication of CoTV crossing the chasm into mainstream attention was the survey by Katherine Boehret of the influential Mossberg/Wall Street Journal/All Things D team on 12/20/11.

• Social TV: The emergence of the Social Web added to the growth of media mutitasking.  People were not just looking up actors, sports statistics, and other related content, but tweeting and posting on Facebook. (Spot411 was introduced in 2009 as an iPhone app to automatically synchronize Social Web activity with TV viewing.)
• ACR (Automated Content Recognition): Based on audio or video recognition, ACR was applied as a way to synchronize enhancements to TV or other video, without direct access to the video distribution infrastructure (closed closed cable or similar systems). ACR gained support from major players like Nielsen.
• iPad: The introduction of the iPad in January 2010 changed the game, as it became apparent that it was a nearly ideal second screen device. The term "companion app" came in vogue, and numerous CoTV companion apps began to appear, both from program providers and independents.  IntoNow offered a universal ACR-based companion app in early 2011, and was soon bought by Yahoo
• System Operators:  The incumbent distributors also bought into it. Major cable services and other providers offered companion apps in 2010, and began work on 2nd-screen enhancements. They also began work on exposing EBIF triggers to enable content identification and synchronization via the TV distribution infrastructure.

Many of the basic concepts of CoTV are now gaining wide acceptance, but the full power of this technology is still emerging, and even the basic principles are still often ignored.

• Ubiquity:  One key limitation is that CoTV services must be ubiquitous and universally applicable to a critical mass of programs to have wide appeal. Most users cannot be expected to download and launch a different app for each program or network. The whole power of coactivity is to be automatic, and to work for any program with little or no setup. Programmers and networks may want special skins and content, but it is not sensible for them to own the basic platform for coactivity. Services like IntoNow point the way.
• Always-on: A related limitation is that synchronization should be automatic (unless turned off), and not requite a manual synchronization action every time a user tunes to a new program. CoTV should follow the user as he channel surfs or navigates across DVR, VOD, or Internet TV content. Enhancements should be unobtrusively available, ready to be offered and seen, even when the user has not asked in advance for them.
• The power of the Web:  A deeper failure of the imagination is that many of the offering continue the mind-set of simplistic enhancements. Rich CoTV services will provide the full power of the Web, in all dimensions of richness, depth, and variety. Web providers learned a decade ago to offer rich experiences, and to make their gardens hold users with their appeal, not with impenetrable walls. CoTV providers need to do the same. Zeebox is one of the few offerings to point to the broader potential. Another hint of this is in Related Content Database (RCDB), a backend service for associating diverse content relating to any program. See this crude mock-up of a rich CoTV interface -- one that shows the richness that can be found in conventional Web services.
• Commerce: Commerce has not been forgotten, but coactive commerce experiences still remain rather limited.

CoTV Tomorrow -- CoTV 2.0

The following are features described in the Reisman patents that remain advanced, and point to powerful opportunities yet to be realized.

• Selectable, Alternative "Enhancement Channels:" Alternative/indie sync content suppliers should be empowered and exploited to offer rich CoTV experiences. 
  • These should be accessible in the form of alternative "enhancement channels"  (ECs) – user selectable channels of enhancements to a given program – such as director commentary, critics, editorial/political spin, humor, etc.  (For example users might select among one or more of liberal, conservative, or humorous spins on news. Film viewers might select film commentary from the director, various critics, friends, or other commentators. Sports viewers might select from commentaries spun neutrally or to either side.)
  • ECs might be be sourced from (and branded by) the programmer/distributor of a given program and/or ad, or from any number of independent third parties (including user-generated content), and might be segregated on that basis.
  • Social TV can be thought of as a case of one or more ECs, each relating to specific social network services and/or subgroups.
  • EC selector settings might be persistent across programs and sessions, or specific to a given viewing situation.
  • A basic sketch of a companion screen showing multiple enhancement channels is given in a figure from the patent disclosure.
• Screen targeting: Flexible targeting of enhancements to any screen (TV or companion) and at varying levels of richness/intrusiveness – such as depending on enhancement type and user mood (lean forward/lean back), controllable by producers and/or users.
  • Current CoTV services are locked in to a specific screen, whether TV or companion, but that is a provider and technology centered design, not compatible with the fact that users seek different modes of interaction at different times, and that the distinction of TV and Internet content is not inherent.
  • Users should be given the capability to target any content to any screen. It is now clear that sometimes we want our video on TV, sometimes on a tablet, and sometimes on a phone. The same is true for related Internet content, including HTML text, multimedia, Social Web services, transaction services, etc.
  • Targeting choices may depend on whether a user wants to lean back or forward (what level of interaction intensity), share enhancements with others in the room or let them watch without obstruction, etc.
  • Content authors and programmers might also use the same flexibility to author once, display anywhere, and to suggest desired targeting (which users may or may not be able to override).
  • In concept this is similar to the current HTML targeting capabilities that allow HTML pages resulting from a link to reuse a current window, open a new tab, open a new browser window, etc.  Just as a "target=" parameter can be used by an author to control that, or controlled by a user in their browser (such as with a right click), similar "target=" parameters can be provided for CoTV enhancement content to cause an enhancement to appear on the same screen as the video, or on a second (or third) screen.
  • Such flexible targeting might also control alternative styles on any selected display, such as frames of text, reduced video images, tickers/zippers, text overlays, etc. on a video screen, or the usual range of sidebars, pop-ups, frames, etc. on a Web screen.
• Flexible session-shifting: Related to screen targeting is the ability to move all or any portion of a session from one device to another. This may relate to the video, the enhancements or both.
  • The simplest case is video session-shifting. Such capabilities have been announced by distributors, such as to shift a program being viewed on a TV to a mobile device, so the viewer can continue viewing when they must leave home. Desirable features include control of time-shifting, such as for live viewing, pause and resume (with or without a short repeat), etc.
  • Another common case is to shift all enhancements (but not the video). This can be useful to shift to or from lean-back to lean-forward mode, or to make enhancement interactions private on a personal screen or public on a shared TV screen.
  • Advanced cases are to shift selected classes of enhancements but not others.
• Link-and-pause (and sync bookmarks): Users should be able to enable sync to be frozen on either screen -- such as to pause video while viewing an enhancement (link-and-pause), or to hold enhancements for later review (bookmarks).  This is explained more fully on the page linked above, and a detailed example is also provided.
• Full hypermedia browsing:  The full functionality of CoTV emerges in a context of full hypermedia browsing, where all media types can be fully interlinked, in a manner that is fully consistent. Hotspots can serve as link anchors whether in text, image or video, and targets can be of any media type -- rich combinations of hypermedia browsing and navigation across devices, including "Web as program guide," Web to video, and video to Web.
• TVC (TV Context): To support the features described here, communication of TV context can be facilitated with a simple context parameter that identifies the program and the time position of the viewer within the program. Web services architectures can be based on this TVC parameter to allow any site to serve content that is synced to a program.
  • Any Web site could be programmed to check for a TVC parameter to decide if it wants to serve relevant content, or to ignore it and operate conventionally. Thus CoTV enhancements can be provided by any and all Web sites.  Commerce sites can provide a home page to some users, and a special landing page to viewers with a TVC that indicates they are watching a specific TV ad.
  • Such TVC parameters can be fully resolved formats that specifically identify a program (or ad) by Digital Object Identifiers or any other standard ID code, or unresolved forms that present a channel number, time zone, and service provider identifier, to allow reverse lookups in a program guide (or ad tracking service), or other methods to resolve the actual program identity.
• Full Coactive Internet commerce and advertising: This is the elephant in the room. It is secondary in the consumer value proposition, but central to the business case. Having Web services that ubiquitously sync to TV ads is the Holy Grail.
  • Anytime a consumer sees an ad, they can have access to a companion mini-site or other Web service (often on a second device) that knows the ad, the offer, and the viewer.
  • Not just personalized advertising (like Visible World), but advertising with a closed loop of interaction. TV ads meet direct marketing.
  • Not just siloed "trans-media," "multi-screen" advertising channels, any one or more of which a viewer might happen to encounter or not, but full, automatic synchronization for the viewer, driven from the video.
  • And, when ads are well targeted and intelligently interactive, they become not a nuisance to the consumer, but a benefit.
  • More can be seen in a somewhat old, but still timely, Web page on CoTV ads.
  • As a by-product, ubiquitous TV context tracking will provide rich ratings data on programs, eventually more accurate than current ratings data, and with rich detail on engagement.
• Third-party linking rights/fees:  Related to the above features are questions of rights to link from video content by independent parties, and of fees relating to the value of such links, when provided as a service for use with relevant content/services/commerce. Methods can be applied to monitor such links and to effect charges in any desired direction.

Other opportunities and issues from the perspective of rich CoTV:

• "Over The Top" vs. Incumbents: As consumers mix Internet video sources ("Over The Top," OTT) with use of incumbent,closed TV services, they will demand consistent support for coactivity. Details of EBIF triggers versus ACR should be hidden -- technology should support the user, not get in their way. Incumbents should use their power to provide superior services and compete on that basis, but trying to compete by blocking alternatives is likely to be self-defeating. Reasonable interoperability is essential to maximizing revenues in the long run, and to building the critical mass of usage that will give incumbents a healthy slice of a large pie (if they get their heads out of the sand).
• The dumb question of smart TV: The emergence of connected TVs has added to the opportunities to offer effective CoTV services, but the question of "smart TV" has distracted prople. Connected TVs need not be smart, as long as they are connected to smarts, whether in a head-end, an Internet cloud, and/or another device. Some smarts in the TV may be desirable, but what is smart today will be dumb and very limiting in a few years. Consumers will not want to buy a TV every few years, just to get current intelligence features that can be provided in better ways.  Too smart is dumb.
• Media Concierge services ("Context is King"): Exploiting the Web as program guide, rich media concierge services can be built to cross media boundaries, and provide discovery services, recommenders, and guides, that apply knowledge of users, their social networks, their media providers, and other sources in an integrated, holistic service. Think of TV Guide + EPGs + Netflix Queue + iTunes + Genius/Pandora + Social Web + MetaCritic/RottenTomatoes/IMDB, in an integrated, open, mashup. As others have observed, it may well be that providing such context will increasingly be more profitable than providing the content. This is more fully explained in a 2005 blog post.

Richard Reisman -- Bio
Consulting/About