CLOUD MIRROR

I met Eric Gradman at a meeting of the recently-formed Transmedia LA group; his enthusiasm and sense of humor are as infectious in person as they are in his work. Gradman’s “uncomfortably augmented reality” project, CLOUD MIRROR, is currently on show at the Sundance festival.

The CLOUD MIRROR is an interactive augmented reality art installation… Live video captured by a camera and is re-projected on the wall behind the camera, functioning like a “magic mirror.” But the CLOUD MIRROR software alters the images on the way to the screen. It runs an algorithm that tracks faces from frame to frame and also examines each frame for 2D barcodes printed on attendee badges. By pairing each face with a badge, and each badge id with a database row, the CLOUD MIRROR can identify by name whoever is standing in front of the installation.

The CLOUD MIRROR then augments each frame, adding a thought bubble to each face in the image. The contents of that thought bubble are selected from a set of “tags” associated with that person. Tags come from various sources, including Facebook, Twitter, and SMS data.

When registering for the event, attendees were asked to optionally provide their Twitter name, Facebook profile ID, and to answer the question “Where is your favorite place in LA?” In the weeks leading up to the event, the CLOUD MIRROR software sent a friend request to any attendee that provided that information. The poor trusting souls who accepted this request had their personal profile gently data-mined. Specifically, the information captured was “Facebook updates,” “Twitter updates,” and “Facebook relationship status.”

CLOUD MIRROR also capitalized on peoples’ innate desire to embarrass their friends by allowing anyone to anonymously “graffiti” in a thought bubble by sending an SMS message to a special number containing the target’s unique badge ID. (monkeys and robots)

Update: Eric’s documentation from Sundance and his reflections on some of the privacy implications of the project.

Tuttuki Bako: 8-Bit AR?

ad59_tuttukibako_black
This pokey AR toy might not be as spectacular as something like LevelHead, but give it props for thinking, er, inside the box:
ad59_tuttukibako_black_screen

Here at ThinkGeek we’ve seen Attack ships on fire off the shoulder of Orion but never have we encountered anything quite like the Tuttuki Bako Virtual Finger Game. When we caught sight of this wacky electronic game on the Japanese blogs, we figured it must be a joke… but no it’s 100% real and fantastically crazy. Simply stick your finger in the hole and a virtual representation appears on the screen. Then you can use your virtual finger to play all kinds of cool mini games… from swinging a panda to having a karate fight with a tiny little man. It’s so odd yet so wonderful. Domo Arigato Japan! (ThinkGeek)

Via Architectradure

Image Mapping and Tracking for AR Apps

I’m interested in the ways that augmented reality can be used to extend storytelling and interaction into the real world. The literary and gameplay potentials presented by this nascent technology seem limitless. That said, we have a fair distance to go before our world starts looking like the one depicted in Denno Coil. One of the biggest stumbling blocks I’ve encountered is the issue of precise positioning. Without knowing the user’s exact location and orientation, an AR system can’t properly overlay/position objects. Most of the AR apps we’ve seen thus far depend on glyphs to accomplish this task; others use carefully pre-positioned wifi routers or Bluetooth nodes to triangulate the user’s location. The problem with these solutions is that — while they make for decent demos — they don’t really scale. If we’re going to tell stories using AR, I suspect that we’ll be looking for solutions that break free of the need for pre-set glyphs, routers or other equipment.

This is where image recognition comes in. Projects like Microsoft’s Photosynth illustrate the capacity of image databases to define 3D space. More recently, AR researchers have started to use image recognition/mapping metaphors to create fluid “glyph-free” applications. The team at the University of Graz’s Christian Doppler Laboratory have just posted some exciting new videos of their work in this field.

These videos hint at the kind of seamlessness of interaction we can expect from AR in the near future.

More: Handheld Augmented Reality at the Christian Doppler Laboratory

Viz

Overview

iphone camera cu
What if there was a hidden world layered atop visible reality that could only be seen by certain people? What if you were one of those people?

Viz is a casual augmented reality game that enables players to discover and embed virtual objects — or “vobjects” — in physical space by looking through the camera viewfinders of their mobile phones.

By discovering and embedding vobjects, players earn points, advancing their positions on local, national and international leaderboards. These points also allow players to unlock special add-ons, tricks and bonuses for their vobjects (see “Variations and Extensions,” below).

Pervasive & Persistent Social Gaming

Players who install the Viz application on their phone can expect an ambient “always on” play experience that exists within the flow of their everyday lives. Players engage with Viz when they have spare moments in public spaces. The application icon, which sits inobtrusively in the notification area of the player’s phone (see figures below), works in the background and changes in appearance depending on the proximity of vobjects (i.e., a quick glance at your phone’s desktop tells you all you need to know). If the player wants to score points by locating these objects, they can view GPS information and activate the “scanner” tool by simply clicking on the application icon. The game is meant to accompany players as they move through space, providing active and mobile individuals with a persistent low-intensity creative social gaming outlet.

Players

Because of its dependency on user-created content, Viz requires a sizable player base in the areas where it will be played. Like a certain social networking service or other open-ended collaborative production games such as SF0, Viz will begin in specific neighborhoods and cities before expanding to national or international distribution. The game will be released in limited numbers to geographically-constrained player groups, beginning with student beta-testers at urban post-secondary educational institutions. The targeted distribution of the game will expand from there, first to tech subcultures in relevant urban centers, then to various segments of early adopters and tastemakers across the Web. These initial user groups will “seed” the environments of their respective urban areas, ensuring that secondary waves of users have no trouble discovering and interacting with active vobjects.

Objective

The objective of the game is to accumulate points by a) discovering vobjects other players have embedded in the physical world; and, b) having vobjects that you have embedded discovered by others.

Setup/Interface

Viz has two main “active” modes: scanning and embedding. Players toggle between these modes via a simple button interface. The game also has a persistent “passive” mode. This mode informs players of nearby vobjects via inobtrusive notification bar icons on their phone’s desktop. A “library” mode gives the player access to game statistics and other information.

The following illustrations show how Viz looks on the Android operating system.

Figure 1: Passive mode

gui-main

Subtle alerts, similar to those used to denote wi-fi range/signal strength, appear in the notification bar when players are within 500 yards of a vobject (see icon in upper right corner). By clicking on the alert, players can activate the Viz interface.

  • The application can also be launched by clicking on its icon in the applications drawer.
  • The rest of the user’s phone works the same as always: the game tries hard not to be intrusive.

Figure 2: Viz Main Screen

gui-interface-compass

Viz’s main screen changes in various ways according to the location and context of the player.

  • If the player is outdoors (pictured), a simple compass-like feature will point in the direction of the nearest vobject.
  • If the player is indoors and within range of one or more vobjects, the main screen will provide the player with additional information about the number and nature of any vobjects in the immediate area.
  • The tabs at the bottom of the main screen toggle between modes and settings.
  • Hardware controls enable the user to exit the app at any time.

Figure 3: Map mode

gui-interface-mapmode

Viz’s map mode shows the location of nearby vobjects (red pins) relative to the player (blue pin). By consulting this map, players can move to within scanning range of a vobject.

  • Clicking on a vobject causes a text balloon to pop up indicating whether it has been captured by the player. If the player has already captured the vobject (vobjects can only be captured once — see “Rules,” below), they will be able to click on the balloon and view relevant comments and statistics. If the player has yet to capture the vobject, the balloon will merely say, “Unknown.”
  • Players can opt-in to share their location with other players. Map mode will display any participating players as yellow pins.
  • Players can zoom in and out and pan the map as in other applications.

Figure 4: Scan mode

gui-interface-scanmode

Once players have reached the vobject’s GPS location, they can scan the immediate area for the vobject itself by switching Viz into scanning mode. Scanning mode uses the phone’s built-in camera functionality to turn the screen into a kind of window through which the player can look. The player finds the vobject by panning the camera around, examining visual features of their surroundings until they find the feature that contains the vobject. This feature is an image stored in the Viz database and streamed to the phone based on the player’s present location. Viz “locks on” to features by comparing the original embedded image to the image currently visible through the player’s phone. This effect is achieved through the use of image recognition and orientation systems like those recently developed by Nokia or Microsoft. Once players have locked on to a vobject, they can press the “Capture” button (the only button on the scanning screen) to add it to their library and collect points.

  • Vibration feedback and various visual cues let the player know when they pan past a vobject.
  • Image ghosting helps players to lock in the image.
  • A standard dialog box pops up when the player successfully captures a vobject, informing them about how many points they have scored. Players can click on “more” to view a separate info screen containing statistics and comments about the vobject. This screen also contains a submission form enabling players to add their own comments. Once a player has captured a vobject, this info screen is accessible from their Library (see below).
  • The “Capture” button appears grayed-out until the image is framed within the approximate boundaries of the original (embedded) image.

Figure 5: Embed mode

gui-interface-embedmode

Once players have accumulated enough points to embed a vobject, they may do so by switching Viz into embed mode. In this mode, players can point their phone’s camera at an object in the physical world to which they wish to attach a vobject. By pressing the “Embed” button, players will expend points and attach a vobject linked to their profile to the physical object, building or landscape they see in their viewfinder. This vobject now becomes something for other players to find (and therefore a new way for the embedding player to earn points).

  • Embedding a vobject is a simple point-and-shoot operation.
  • Players can consult an “Image Legibility” reading while in embed mode to make sure that the vobject they are embedding will be easily capturable by anyone who finds it.
  • The embed screen is only available to players who have enough points to embed a vobject. If a player attempts to switch into embed mode when they do not have enough points to embed a vobject, an alert box (not pictured) will inform them that they need to earn more points before they can complete this action.
  • If a player has purchased or unlocked a power-up — for example, a bonus modifier of 10 points/scan — they will be asked if they would like to use this power-up on the vobject they are embedding. This process is discussed below in both “Rules” and “Variations and Extensions.”

Figure 6: The Library

gui-interface-librarymode

Library mode enables players to view lists of the vobjects they have captured, the vobjects they have embedded, and the points the have earned. The screen also contains a button that launches a Settings window (not pictured) wherein players can adjust profile information.

  • Two simple buttons allow players to toggle between lists of vobjects they have captured and vobjects they have embedded.
  • Clicking on an individual vobject in a list opens a dialog containing additional information about that vobject, including how many times it has been scanned, when and where it was embedded, and who embedded it.
  • Profile information can be modified in the “Settings” screen (not pictured).
  • All lists are scrollable to conserve screen space.

Procedures/Controls

Read Map To locate vobjects, players will activate the GPS map function of Viz to help them get within range.

Capture Vobject Once players are near a vobject, they will scan the area using Viz’s camera function until they “see” the vobject superimposed over real space. They will then press the “Capture” button to collect the vobject and score points.

Embed Vobject To embed vobjects, players must first accumulate enough points to do so. Once they have the requisite points, players can activate Viz’s embedding tool. This tool enables players to embed a vobject in a particular location by pointing their phone’s camera at a real-world object, building or landscape. Once the player has selected the view to which they want to attach their object, they will press the “embed” button. Doing so subtracts points from the player’s account.

Check stats Players can read statistics about their own usage, how many points they have, and how many points they earn from embedded vobjects by switching to library mode.

Review Vobjects Library mode also enables players to view lists of the vobjects they have captured or embedded; by clicking on the vobject identifiers (GPS coordinates or city/postal code designators), players can view additional information about each vobject.

Unlock Power-ups Players can use points to unlock power-ups which modify the way the vobjects they embed tally scores and affect other players. The power-ups button in library mode accesses a list (not pictured) of power-ups, organized by price. Power-ups that players have purchased are displayed in this list with a star next to them.

Use Power-ups Players can attach a power-up to a vobject as they embed it in a physical place. If a player has any power-ups to use, they will be given the option to use them at the time of embedding.

Modify profile settings Players can change their username, home location and other information in a standard profile/settings dialog. This profile can also be modified via an online web interface.

Flow/Game States

See attached flow chart (pictured below).

IMAP-541-Viz

Rules

Beyond the rules implied and specified above, readers should be aware of the following scoring rules and constraints:

  • Whenever a vobject is discovered, points are awarded to both the discoverer and the embedder.
  • Objects that are discovered on a regular basis score fewer points than objects that are only viewed occasionally. Players can thus adopt a variety of strategies for accumulating points: for example, player A might embed objects in obvious public spaces (e.g. tourist sites, popular cafes, etc) in order to get a large volume of low-value discoveries, while player B might embed fewer objects in more hard-to-find locations in order to get a small volume of high-value discoveries.
  • Any objects that are not found within a period of thirty days will disappear. This constraint prevents players from embedding objects in overly obscure locations and adds an element of gambling to certain placement strategies — e.g. players must balance the risk of their object not being found at all against the potential gain of it being found only once or twice.
  • Embedding vobjects (and unlocking extra functionality) “costs” points, motivating players to earn as much as possible.
  • Vobjects may only be scanned once. Players may not scan their own Vobjects.

Variations

Power-ups

When embedding, players can opt to attach “power-ups” to their vobjects. These power-ups have a variety of effects on scoring and gameplay. Below are two proposed power-ups and their effects:

  • Trap The Trap power-up causes the player who scans the vobject to lose 500 points. The embedder of the vobject receives these points in addition to those they collect for the vobject being scanned.
  • Trade When a vobject imbued with the Trade power-up is scanned, the player who scans it becomes the owner. In exchange, the former owner of the vobject receives one of the scanning player’s vobjects (selected at random). If the scanning player has no embedded vobjects in play, the Trade power-up is lost.

Other

Finding and Leaving Traces

Viz is an inherently social game. Players get to experience the thrill of discovering secret objects left behind by other people, often in scenic or humorous locations. Interacting with these objects reveals additional layers of sociality: for example, upon discovering a vobject, players may choose to attach a voice or text “comment” to the item, which can be then read (and responded to) by the vobject’s embedder and any subsequent discoverers. Importantly, however, players don’t need to comment on vobjects in order to experience the social dimension of Viz — even if players ignore all but the most basic affordances of the game, they will engage in the fundamentally social activity of finding and leaving traces of passage. In this sense, embedded vobjects are reminiscent of the Inuit inuksuks (or inuksuit) found in the far North:

An inuksuk… is a man-made stone landmark or cairn, used by the Inuit, Inupiat, Kalaallit, Yupik, and other peoples of the Arctic region of North America, from Alaska to Greenland…

The inuksuk may have been used for navigation, as a point of reference, a marker for hunting grounds, or as a food cache. The word inuksuk means “something which acts for or performs the function of a person.” The word comes from the morphemes inuk (“person”) and -suk (“ersatz” or “substitute”)… (Wikipedia: Inuksuk)

Inspiration

This idea sprung from two sources. First, a whiteboard session with my colleagues Amanda Tasse and Lauren Fenton got me thinking about the issues involved in making an augmented reality game with a social dimension (our original idea was a kind of “truth or dare” casual mobile app wherein players could discover other players in proximate space via GPS or other location/context-sensing technologies and challenge them to do humiliating and/or dangerous things). This brainstorming session raised a few intriguing questions: What kinds of ambient AR gaming experiences would players reasonably want to layer atop their “real” lives? What motivation would they have to participate? And, crucially, what kinds of game mechanics can we create with AR that simply can’t be created any other way (ie, how can we avoid simply using AR as a gimmick)?

The second source of inspiration is largely a consequence of the first. In an effort to answer some of the questions raised during the whiteboard session with Amanda and Lauren, I started looking around the web for existing AR games and apps. I found plenty of cool examples. But the most thought-provoking discovery occurred when I came across Denno- Coil, a recent Japanese science fiction series about kids living in a world wherein AR has been seemlessly integrated into everyone’s lives via special eyeglasses and visors. The story and setting conceits of Denno Coil are a goldmine for AR developers. Here are a few notes I clipped from the Wikipedia article on the series [Accessed 20 September 2009] as I was developing my own game concept:

  • [Users] access the virtual world through Internet-connected visors called denno- eyeglasses. This allows them to see virtual reality superimposed on objective reality. To visually confirm something as virtual, the children often lift their glasses from their eyes. The visors also work in conjunction with futuristic ear monitors placed behind the ear, which allows them to hear sounds from the virtual environment.
  • Software tools are visually represented as tactile tools (e.g. a fishing rod) which occupy 3D virtual space and must be manipulated by hand. Metatags, which can be used to damage or enhance virtual objects with certain properties, are visually represented as o-fuda.
  • Virtual objects such as pets cannot be recalled or reset; when a pet runs away, it must be chased and caught in 3D space. Virtual objects and pets are also susceptible to a form of “death” by data corruption or deletion.
  • Fumie Hashimoto handles a software spray can loaded with so-called “black bug spray”.
  • A virus on the run…leaves behind a (virtually) tangible, evaporating black trail of raw “denno- substance”.

See also: Serial Experiments Lain

Adrian Hon: Turning the Whole World into Disneyland

westorriesscreen2

Adrian Hon’s company, Six to Start, won Best in Show at this week’s SXSW Web Awards for their project, Telling Stories. In an interview with Marshall Kirkpatrick, Hon talks up the nascent potential of ubicomp storytelling:

"Soon people will realize that there is no ‘mobile internet’ – there is only the Internet," he says. "And stories are everywhere." Hon says web content today is like the early days of TV, when all anyone could think to do was broadcast actors from the theater in the new medium. But new types of media enable fundamentally new types of content and experiences.

For example, we’re just beginning to learn how to leverage the web’s social connections, Hon says. He points to the first iteration of "urban games" as something rudimentary that won’t last: groups of people organizing online to meet in person dressed, let’s say, as Pac-man characters, running through city streets and posting videos of their adventures on YouTube. "Those games ask people to get up and do something they don’t really want to do," Hon says.

Instead, he believes that the future of interactive story telling will be pervasive – it will be available throughout your typical day. Walking to work, even while at work.

"I have no idea what we can produce in this medium," he said, "but I think it’s going to be like turning the whole world into Disney Land." (readwriteweb)

Echoes of Spook Country

H/T Scott Fisher

MEML notes 6 March 2009

These notes were taken at a hectic mid-semester meeting of the MEML group. By this time, the three main streams of the Million Story Building project were well underway: the iPhone “building app,” the transmedial ARG prototype, and the “movietagger” system.

Each of these streams are discussed in these minutes. First, regarding the Building App, the issue of motivation once again rears its head due to the Apple-enforced iPhone limitation on background processing (ie, Apple won’t allow it, at least not at the time of writing). The problem, recorded here in ALL CAPS (“NO BACKGROUND PROCESSING”) boils down to this: since we can’t have the Building App persistently running in the background on every user’s phone, we need to somehow encourage users to take the time to fire up the application themselves — that is, to take the active step of “turning on” the system, instead of having it run passively in the background. This is where things like the ARG, the SCA History Project and the Provenence Project all come into play: by enriching the building with opportunities for discovery and interaction that can only be accessed via the Building App, we encourage users to actively participate…in theory, at least.

The second main issue discussed in at this meeting relates to the MovieTagger Project led by Andreas Kratky. For our demo of the system, we wanted users to be able to tag movie clips in the building’s database, and we wanted them to do so by controlling plasma screens with their iPhones. Here, we discuss what, if any, fixed data fields should be used for categorizing/tagging the clips. In the end, we settled on a more or less “freestyle” approach, similar to the folksonomies of popular social media sites like flickr, for the demo.

The last few bullets in these notes refer to some ideas we developed for the future of the Million Story Building project. One of these ideas, the “digital commons” (an augmented reality interface for adding info-bubbles to the main courtyard of the SCA Building), would later appear in our demo, albeit in mockup format.

TFOTS MEETING NOTES
March 06 2009

_____________________________

Peter ==

quartz thingy will read various xml containers and display content
thingy can also respond to bluetooth presence

to display: IMD stuff, movietagger stuff, SCA stuff, History Project

Jen ==

mockups
Scott: what happens when App goes into background/closed?

Will ==

no no no — NO BACKGROUND PROCESSING
find ways to keep it on — via MOTIVATION/Modal techniques

Jen ==

mockups pt 2
new icons appearing
messages appearing in “your media” AKA “HISTORY”
anomalies appearing in the app

Will ==

data server… logging tables for different users, tracking info — log ins, scans, etc
really basic content database
CODE references via Wordpress

Andreas ==

mySQL for the movietagger?

movietagger: what kinds of fields do we want to attach to these movies?
e.g. scene descriptions, dialogue lines versus one or two word tags
tag-based EDL using submitted tags
different kinds of keywords and tags: “interestingness algorithms”
stripping out useless info

buttons on phone — list of vids
what is the reward for this? shoe tagged things appearing on the other screens?

size and style of database questions, etc FREESTYLE

could Emily do some of this?

scott: figure out how much we’re going to have to support this, demo it pre-mobile…

Scott ==

statue as focus for “digital commons” and Archive
could you “send your stories” to Douglas Fairbanks

“letters to douglas fairbanks”

how do we force people to be at the statue when they submit, and what can they get back?

icon that appears only when you’re in front of the statue

what about voice?
tie-in with ARG
sending email

AR Toolkit stuff

BIM meeting monday morning

Hide ==

let’s setup some kind of iPhone app for ARToolkit so we don’t have to walk around with laptops

Jen ==

ambient music tags following around the people in the bldg
different sounds in the elevators?

The Making of ARf: Me, My Dog and iPhone

Proof-of-concept for an augmented reality pet running on an iPhone. Via Games Alfresco.