jeff watson

“…what does it mean for a cybertext to be experienced via a mobile device? Traditional forms specify the syntagm because the possibilities of the real world are unquantifiable. Text-messaging, however, blurs the boundaries between a hermetic narrative space and the unpredictable logics of the real world. A cybertext on a mobile device is neither specifying a syntagm nor a complete paradigm. Instead, it elicits the specific creativity of the human consciousness interacting with its environment, temporarily organizing how reality, whatever its subjective nature, whatever preexisting simulacra exist, might be experienced.”

– Knifeandfork – Subversive (Mobile) Storytelling

I’m interested in the ways that augmented reality can be used to extend storytelling and interaction into the real world. The literary and gameplay potentials presented by this nascent technology seem limitless. That said, we have a fair distance to go before our world starts looking like the one depicted in Denno Coil. One of the biggest stumbling blocks I’ve encountered is the issue of precise positioning. Without knowing the user’s exact location and orientation, an AR system can’t properly overlay/position objects. Most of the AR apps we’ve seen thus far depend on glyphs to accomplish this task; others use carefully pre-positioned wifi routers or Bluetooth nodes to triangulate the user’s location. The problem with these solutions is that — while they make for decent demos — they don’t really scale. If we’re going to tell stories using AR, I suspect that we’ll be looking for solutions that break free of the need for pre-set glyphs, routers or other equipment.

This is where image recognition comes in. Projects like Microsoft’s Photosynth illustrate the capacity of image databases to define 3D space. More recently, AR researchers have started to use image recognition/mapping metaphors to create fluid “glyph-free” applications. The team at the University of Graz’s Christian Doppler Laboratory have just posted some exciting new videos of their work in this field.

These videos hint at the kind of seamlessness of interaction we can expect from AR in the near future.

More: Handheld Augmented Reality at the Christian Doppler Laboratory

Overview

What if there was a hidden world layered atop visible reality that could only be seen by certain people? What if you were one of those people?

Viz is a casual augmented reality game that enables players to discover and embed virtual objects — or “vobjects” — in physical space by looking through the camera viewfinders of their mobile phones.

By discovering and embedding vobjects, players earn points, advancing their positions on local, national and international leaderboards. These points also allow players to unlock special add-ons, tricks and bonuses for their vobjects (see “Variations and Extensions,” below).

Pervasive & Persistent Social Gaming

Players who install the Viz application on their phone can expect an ambient “always on” play experience that exists within the flow of their everyday lives. Players engage with Viz when they have spare moments in public spaces. The application icon, which sits inobtrusively in the notification area of the player’s phone (see figures below), works in the background and changes in appearance depending on the proximity of vobjects (i.e., a quick glance at your phone’s desktop tells you all you need to know). If the player wants to score points by locating these objects, they can view GPS information and activate the “scanner” tool by simply clicking on the application icon. The game is meant to accompany players as they move through space, providing active and mobile individuals with a persistent low-intensity creative social gaming outlet.

Players

Because of its dependency on user-created content, Viz requires a sizable player base in the areas where it will be played. Like a certain social networking service or other open-ended collaborative production games such as SF0, Viz will begin in specific neighborhoods and cities before expanding to national or international distribution. The game will be released in limited numbers to geographically-constrained player groups, beginning with student beta-testers at urban post-secondary educational institutions. The targeted distribution of the game will expand from there, first to tech subcultures in relevant urban centers, then to various segments of early adopters and tastemakers across the Web. These initial user groups will “seed” the environments of their respective urban areas, ensuring that secondary waves of users have no trouble discovering and interacting with active vobjects.

Objective

The objective of the game is to accumulate points by a) discovering vobjects other players have embedded in the physical world; and, b) having vobjects that you have embedded discovered by others.

Setup/Interface

Viz has two main “active” modes: scanning and embedding. Players toggle between these modes via a simple button interface. The game also has a persistent “passive” mode. This mode informs players of nearby vobjects via inobtrusive notification bar icons on their phone’s desktop. A “library” mode gives the player access to game statistics and other information.

The following illustrations show how Viz looks on the Android operating system.

Figure 1: Passive mode

Subtle alerts, similar to those used to denote wi-fi range/signal strength, appear in the notification bar when players are within 500 yards of a vobject (see icon in upper right corner). By clicking on the alert, players can activate the Viz interface.

• The application can also be launched by clicking on its icon in the applications drawer.
• The rest of the user’s phone works the same as always: the game tries hard not to be intrusive.

Figure 2: Viz Main Screen

Viz’s main screen changes in various ways according to the location and context of the player.

• If the player is outdoors (pictured), a simple compass-like feature will point in the direction of the nearest vobject.
• If the player is indoors and within range of one or more vobjects, the main screen will provide the player with additional information about the number and nature of any vobjects in the immediate area.
• The tabs at the bottom of the main screen toggle between modes and settings.
• Hardware controls enable the user to exit the app at any time.

Figure 3: Map mode

Viz’s map mode shows the location of nearby vobjects (red pins) relative to the player (blue pin). By consulting this map, players can move to within scanning range of a vobject.

• Clicking on a vobject causes a text balloon to pop up indicating whether it has been captured by the player. If the player has already captured the vobject (vobjects can only be captured once — see “Rules,” below), they will be able to click on the balloon and view relevant comments and statistics. If the player has yet to capture the vobject, the balloon will merely say, “Unknown.”
• Players can opt-in to share their location with other players. Map mode will display any participating players as yellow pins.
• Players can zoom in and out and pan the map as in other applications.

Figure 4: Scan mode

Once players have reached the vobject’s GPS location, they can scan the immediate area for the vobject itself by switching Viz into scanning mode. Scanning mode uses the phone’s built-in camera functionality to turn the screen into a kind of window through which the player can look. The player finds the vobject by panning the camera around, examining visual features of their surroundings until they find the feature that contains the vobject. This feature is an image stored in the Viz database and streamed to the phone based on the player’s present location. Viz “locks on” to features by comparing the original embedded image to the image currently visible through the player’s phone. This effect is achieved through the use of image recognition and orientation systems like those recently developed by Nokia or Microsoft. Once players have locked on to a vobject, they can press the “Capture” button (the only button on the scanning screen) to add it to their library and collect points.

• Vibration feedback and various visual cues let the player know when they pan past a vobject.
• Image ghosting helps players to lock in the image.
• A standard dialog box pops up when the player successfully captures a vobject, informing them about how many points they have scored. Players can click on “more” to view a separate info screen containing statistics and comments about the vobject. This screen also contains a submission form enabling players to add their own comments. Once a player has captured a vobject, this info screen is accessible from their Library (see below).
• The “Capture” button appears grayed-out until the image is framed within the approximate boundaries of the original (embedded) image.

Figure 5: Embed mode

Once players have accumulated enough points to embed a vobject, they may do so by switching Viz into embed mode. In this mode, players can point their phone’s camera at an object in the physical world to which they wish to attach a vobject. By pressing the “Embed” button, players will expend points and attach a vobject linked to their profile to the physical object, building or landscape they see in their viewfinder. This vobject now becomes something for other players to find (and therefore a new way for the embedding player to earn points).

• Embedding a vobject is a simple point-and-shoot operation.
• Players can consult an “Image Legibility” reading while in embed mode to make sure that the vobject they are embedding will be easily capturable by anyone who finds it.
• The embed screen is only available to players who have enough points to embed a vobject. If a player attempts to switch into embed mode when they do not have enough points to embed a vobject, an alert box (not pictured) will inform them that they need to earn more points before they can complete this action.
• If a player has purchased or unlocked a power-up — for example, a bonus modifier of 10 points/scan — they will be asked if they would like to use this power-up on the vobject they are embedding. This process is discussed below in both “Rules” and “Variations and Extensions.”

Figure 6: The Library

Library mode enables players to view lists of the vobjects they have captured, the vobjects they have embedded, and the points the have earned. The screen also contains a button that launches a Settings window (not pictured) wherein players can adjust profile information.

• Two simple buttons allow players to toggle between lists of vobjects they have captured and vobjects they have embedded.
• Clicking on an individual vobject in a list opens a dialog containing additional information about that vobject, including how many times it has been scanned, when and where it was embedded, and who embedded it.
• Profile information can be modified in the “Settings” screen (not pictured).
• All lists are scrollable to conserve screen space.

Procedures/Controls

Read Map To locate vobjects, players will activate the GPS map function of Viz to help them get within range.

Capture Vobject Once players are near a vobject, they will scan the area using Viz’s camera function until they “see” the vobject superimposed over real space. They will then press the “Capture” button to collect the vobject and score points.

Embed Vobject To embed vobjects, players must first accumulate enough points to do so. Once they have the requisite points, players can activate Viz’s embedding tool. This tool enables players to embed a vobject in a particular location by pointing their phone’s camera at a real-world object, building or landscape. Once the player has selected the view to which they want to attach their object, they will press the “embed” button. Doing so subtracts points from the player’s account.

Check stats Players can read statistics about their own usage, how many points they have, and how many points they earn from embedded vobjects by switching to library mode.

Review Vobjects Library mode also enables players to view lists of the vobjects they have captured or embedded; by clicking on the vobject identifiers (GPS coordinates or city/postal code designators), players can view additional information about each vobject.

Unlock Power-ups Players can use points to unlock power-ups which modify the way the vobjects they embed tally scores and affect other players. The power-ups button in library mode accesses a list (not pictured) of power-ups, organized by price. Power-ups that players have purchased are displayed in this list with a star next to them.

Use Power-ups Players can attach a power-up to a vobject as they embed it in a physical place. If a player has any power-ups to use, they will be given the option to use them at the time of embedding.

Modify profile settings Players can change their username, home location and other information in a standard profile/settings dialog. This profile can also be modified via an online web interface.

Flow/Game States

See attached flow chart (pictured below).

Rules

Beyond the rules implied and specified above, readers should be aware of the following scoring rules and constraints:

• Whenever a vobject is discovered, points are awarded to both the discoverer and the embedder.
• Objects that are discovered on a regular basis score fewer points than objects that are only viewed occasionally. Players can thus adopt a variety of strategies for accumulating points: for example, player A might embed objects in obvious public spaces (e.g. tourist sites, popular cafes, etc) in order to get a large volume of low-value discoveries, while player B might embed fewer objects in more hard-to-find locations in order to get a small volume of high-value discoveries.
• Any objects that are not found within a period of thirty days will disappear. This constraint prevents players from embedding objects in overly obscure locations and adds an element of gambling to certain placement strategies — e.g. players must balance the risk of their object not being found at all against the potential gain of it being found only once or twice.
• Embedding vobjects (and unlocking extra functionality) “costs” points, motivating players to earn as much as possible.
• Vobjects may only be scanned once. Players may not scan their own Vobjects.

Variations

Power-ups

When embedding, players can opt to attach “power-ups” to their vobjects. These power-ups have a variety of effects on scoring and gameplay. Below are two proposed power-ups and their effects:

• Trap The Trap power-up causes the player who scans the vobject to lose 500 points. The embedder of the vobject receives these points in addition to those they collect for the vobject being scanned.
• Trade When a vobject imbued with the Trade power-up is scanned, the player who scans it becomes the owner. In exchange, the former owner of the vobject receives one of the scanning player’s vobjects (selected at random). If the scanning player has no embedded vobjects in play, the Trade power-up is lost.

Other

Finding and Leaving Traces

Viz is an inherently social game. Players get to experience the thrill of discovering secret objects left behind by other people, often in scenic or humorous locations. Interacting with these objects reveals additional layers of sociality: for example, upon discovering a vobject, players may choose to attach a voice or text “comment” to the item, which can be then read (and responded to) by the vobject’s embedder and any subsequent discoverers. Importantly, however, players don’t need to comment on vobjects in order to experience the social dimension of Viz — even if players ignore all but the most basic affordances of the game, they will engage in the fundamentally social activity of finding and leaving traces of passage. In this sense, embedded vobjects are reminiscent of the Inuit inuksuks (or inuksuit) found in the far North:

An inuksuk… is a man-made stone landmark or cairn, used by the Inuit, Inupiat, Kalaallit, Yupik, and other peoples of the Arctic region of North America, from Alaska to Greenland…

The inuksuk may have been used for navigation, as a point of reference, a marker for hunting grounds, or as a food cache. The word inuksuk means “something which acts for or performs the function of a person.” The word comes from the morphemes inuk (“person”) and -suk (“ersatz” or “substitute”)… (Wikipedia: Inuksuk)

Inspiration

This idea sprung from two sources. First, a whiteboard session with my colleagues Amanda Tasse and Lauren Fenton got me thinking about the issues involved in making an augmented reality game with a social dimension (our original idea was a kind of “truth or dare” casual mobile app wherein players could discover other players in proximate space via GPS or other location/context-sensing technologies and challenge them to do humiliating and/or dangerous things). This brainstorming session raised a few intriguing questions: What kinds of ambient AR gaming experiences would players reasonably want to layer atop their “real” lives? What motivation would they have to participate? And, crucially, what kinds of game mechanics can we create with AR that simply can’t be created any other way (ie, how can we avoid simply using AR as a gimmick)?

The second source of inspiration is largely a consequence of the first. In an effort to answer some of the questions raised during the whiteboard session with Amanda and Lauren, I started looking around the web for existing AR games and apps. I found plenty of cool examples. But the most thought-provoking discovery occurred when I came across Denno- Coil, a recent Japanese science fiction series about kids living in a world wherein AR has been seemlessly integrated into everyone’s lives via special eyeglasses and visors. The story and setting conceits of Denno Coil are a goldmine for AR developers. Here are a few notes I clipped from the Wikipedia article on the series [Accessed 20 September 2009] as I was developing my own game concept:

• [Users] access the virtual world through Internet-connected visors called denno- eyeglasses. This allows them to see virtual reality superimposed on objective reality. To visually confirm something as virtual, the children often lift their glasses from their eyes. The visors also work in conjunction with futuristic ear monitors placed behind the ear, which allows them to hear sounds from the virtual environment.
• Software tools are visually represented as tactile tools (e.g. a fishing rod) which occupy 3D virtual space and must be manipulated by hand. Metatags, which can be used to damage or enhance virtual objects with certain properties, are visually represented as o-fuda.
• Virtual objects such as pets cannot be recalled or reset; when a pet runs away, it must be chased and caught in 3D space. Virtual objects and pets are also susceptible to a form of “death” by data corruption or deletion.
• Fumie Hashimoto handles a software spray can loaded with so-called “black bug spray”.
• A virus on the run…leaves behind a (virtually) tangible, evaporating black trail of raw “denno- substance”.

See also: Serial Experiments Lain

Adrian Hon’s company, Six to Start, won Best in Show at this week’s SXSW Web Awards for their project, Telling Stories. In an interview with Marshall Kirkpatrick, Hon talks up the nascent potential of ubicomp storytelling:

"Soon people will realize that there is no ‘mobile internet’ – there is only the Internet," he says. "And stories are everywhere." Hon says web content today is like the early days of TV, when all anyone could think to do was broadcast actors from the theater in the new medium. But new types of media enable fundamentally new types of content and experiences.

For example, we’re just beginning to learn how to leverage the web’s social connections, Hon says. He points to the first iteration of "urban games" as something rudimentary that won’t last: groups of people organizing online to meet in person dressed, let’s say, as Pac-man characters, running through city streets and posting videos of their adventures on YouTube. "Those games ask people to get up and do something they don’t really want to do," Hon says.

Instead, he believes that the future of interactive story telling will be pervasive – it will be available throughout your typical day. Walking to work, even while at work.

"I have no idea what we can produce in this medium," he said, "but I think it’s going to be like turning the whole world into Disney Land." (readwriteweb)

Echoes of Spook Country…

H/T Scott Fisher

The last gestural interface demo I saw involved about a dozen precision cameras, several computer systems, two technicians and a lot of money. This system, developed by Pattie Maes and Pranav Mistry at the MIT Media Lab, costs only $350, is portable/wearable, works on any surface, and does the whole Minority Report thing about as well as I’ve ever seen…

More info: MIT Fluid Interfaces Group.

The goal of the Fluid Interfaces research group is to radically rethink the human-machine interactive experience. By designing interfaces that are more immersive, more intelligent, and more interactive we are changing the human-machine relationship and creating systems that are more responsive to people’s needs and actions, and that become true “accessories” for expanding our minds. (MIT Media Lab)

UPDATE: More info about this project can be found at Pranav Mistry’s site: http://www.pranavmistry.com/projects/sixthsense/

[The entirety of this document, including lists of deliverables and a preliminary timeline, can be downloaded in .pdf form here.]

Summary

This course of directed research seeks to identify and implement a range of context- and location-based storytelling techniques by leveraging ubiquitous computing technologies – including mobile communications devices, broadband wi-fi networks, real-time sensor systems and the Internet itself – to create a layered, pervasive and interactive story experience rooted in physical space.

The primary deliverable of this project will be a so-called Alternate Reality Game (ARG) centered on the newly-constructed School of Cinematic Arts (SCA) building at USC. Working in concert with a team of mobile storytelling investigators and technologists headed by professor Scott Fisher, I will coordinate the development of narrative content for a long-arc story experience intended to enrich the physical and virtual environment of the SCA with layers of mystery, playfulness and interactivity.

By implementing a small-scale location-based ARG such as the one proposed herein, I expect to glean a variety of quantitative and qualitative insights into the limitations and possibilities of this nascent narrative form. These insights will feed directly into my primary doctoral inquiry into the question of how an increasingly mobile, ubiquitous and interoperable communications infrastructure can enable new forms of computationally-mediated narrative, both in terms of traditional author-to-audience storytelling and emerging modes of collaborative networked expression and participation. Further, by creating and managing a small team of co-conspirators (see “Approach,” below), and by extending an invitation to participate in the project to the SCA community as a whole (both through the mechanics of the game itself and as a part of the development and pre-production process), I hope to broaden the level of interest in this kind of storytelling by inspiring others to investigate new avenues for the transmedial exploration of character and myth.

Background

Alternate Reality Games (ARGs) are a relatively new form of narrative gameplay – the earliest incarnations of the genre date to the late 1990s and early 2000s – that use the real world and all its myriad communications modalities as the surface upon which to layer story and interaction. As a spatially- and temporally-distributed storytelling form, ARGs deploy narrative across a wide range of expressive media, including physical spaces and artifacts, websites, game worlds, books and graphic novels, music, television and movies, online video, rumors, cell phone content and live performances. For the player-participants of an ARG, apprehending the story and the mechanics of the game is an active investigative or archaeological task, a process of uncovering mysteries and sifting through answers in an effort to find the right questions. As such, playing an ARG is fundamentally distinct from traditional game or story forms in which a “magic circle” of play or spectatorship ceremonially defines the boundaries between the fictional and the real. In an ARG, the line between “in-game” and “out-of-game” is intentionally blurred. Jane McGonigal, a leading researcher in the field, describes this approach as the “this is not a game” (TINAG) aesthetic. When this aesthetic is adhered to with a modicum of discipline, the ARG as a story medium becomes more akin to hoax-making than novel-writing (although, it should be said, an ARG could conceivably deploy a novel as a component of its overall storytelling strategy): like a good hoax, a successful ARG will conceal itself beneath layers of compelling real-world information. Players of ARGs thus often begin playing the game before they even know that a game is afoot. In the apotheosis of this form, by the time players realize that there is an intelligence guiding their investigations into the mysteries that have inexplicably infiltrated their lives, their desire to uncover the truth of the matter becomes irresistible, and players will pursue the mystery to the end of the line, oftentimes sharing their insights and solving game problems collectively via self-organized online interactions.

To date, the peculiar affordances of the ARG have been exploited primarily by media corporations such as Microsoft, Dreamworks and Sony Pictures in order to launch viral marketing campaigns for other products. For example, one of the most successful ARGs in recent memory began with a mysterious bee-keeping website that came to the attention of participants via a brief flash of text at the end of a movie trailer. A great deal of Internet buzz ensued as interested parties began to investigate the website and its provenance, revealing a strange series of what initially appeared to be distress signals from some kind of trapped or kidnapped individual. Increasingly large groups of players began working collectively online to solve the mystery, and as things got weirder and weirder, word-of-mouth drew more and more participants into the world of the story. In the end, this project, known as “ilovebees”, turned out to be a promotional initiative for the launch of Microsoft’s Halo 2.

While the origins and historical applications of the ARG are rooted in viral marketing, more recent iterations employing the TINAG aesthetic have sought to create so-called “self-monetizing” ARGs. For example, 42 Entertainment, the studio that produced ilovebees and several other seminal ARGs, recently partnered with a small press to publish Cathy’s Book, a book for young adults purporting to be the private diary of a missing teenage girl. By framing this publication with an invitation to readers to help locate the missing teenager, the designers were able to simultaneously steer their audience toward multiple story assets exterior to the book itself (e.g. websites mentioned in the book, phone lines accessible via numbers scrawled in the “diary’s” margins, and so on), offset the production costs of writing and producing the project as a whole, and draw new readers to discover (and purchase) the book itself via the creation of an online following. The book sold well, debuting at #7 on the New York Times Best Seller list for Children’s Books, and the online community continues to be active.

In addition to revealing more about the potential uses of the ARG for both viral marketing and self-monetization, a significant goal of the project proposed herein is to explore the capacity of ambient story and game play to create and shape communities of player-participants within the constraints of institutional space. Questions to be addressed include: can ARGs and other kinds of layered or ambient story/game systems help to foster a sense of community and camaraderie in work and study environments? Does the presence of a continuously-unfolding mystery “accelerate serendipity” by bringing like-minded individuals into physical and virtual association in order to solve the mysteries with which they have been confronted? How can an ARG stimulate the production and sharing of media by its participants? More broadly, can highly-mediated approaches to play and narrative that involve the deep and tangled integration of story-telling, story-consuming, and even story-producing, into the fabric of everyday life produce emotional and social effects of similar character to those produced by the novel or the narrative cinema? Addressing these questions through the deployment of a small-scale ARG in the SCA will provide key practical insights into the creation and management of such projects, while also revealing new data regarding the intersection between structured social play and networked computational systems. It is my belief that an inquiry into the nature of this intersection is essential as we enter an age of ubiquitous information technology wherein the respective agencies of authors, crowds and machines promise to collide in productive and unpredictable ways.

Approach

Time is of the essence in the successful development and deployment of an ARG, and this project is no exception. The opening ceremonies for the new SCA Building are scheduled for mid-March, 2009, meaning careful coordination of resources and a strict adherence to a workflow timeline will be essential. To this end, work has already begun in concert with Scott Fisher’s “The Future of the Story” (TFOTS) research group (comprised of Professor Fisher, Will Carter, Marientina Gotsis, Jen Stein and Hidefumi Yasuda) to determine the specific technological, budgetary and personnel requirements and availabilities of the project. By continuing to liaise with this group and its derivatives (such as Will Carter’s “Design and Technologies for Mobile Experiences” class, CTIN 405), I expect to develop a precise set of practical parameters within which the story and game mechanics can operate by mid-January 2009. Once these parameters have been defined, I will begin the process of structuring the story/game experience and creating the necessary narrative content.

To act as a force-multiplier for these efforts, I have assembled a small team (“The Story Group”)  of writers, designers and programmers from the ranks of the Graduate Program of the IMD. This team will work under my direction to develop and deploy the creative and technical assets  required of the project. Four of these team members will be assigned primary roles and will be paid as interns for their work; an additional cadre of four to six collaborators will work in association with this core group to develop and execute the ARG. Further, as the timeline progresses, “pre-game” elements – small mini-games and story segments deployed in and around the SCA – will act as “rabbit holes,” enticing early participants who themselves will shape the development of the final experience, which will “go live” during the opening ceremonies of the new SCA Building.

UPDATE (15 July 2009): A portion of this project was demonstrated to a small test group in mid-May; observations and comments from the demonstration are forthcoming (as components of a larger paper on Location-Based Ambient Storytelling); their appearance will be noted here.

Nokia researcher Jan Chipchase’s investigation into the ways we interact with technology has led him from the villages of Uganda to the insides of our pockets. He’s made some unexpected discoveries along the way. (ted.com)

Cross-media whiz Jim Monroe has recently been working as a consultant and project coordinator at OCAD’s Mobile Experience Lab, and it looks like he’s been in the groove. The Lab’s upcoming suite of projects designed for deployment on and around Queen Street in Toronto are inspiring and often quite humorous. My favorite is “Your News Box,” a newspaper box that contains a screen displaying a fictional newspaper’s front page, which passers-by can alter and customize using their cell phones.

Link: No Media Kings.