Transmedia and Education: Three Essential Readings

Henry Jenkins’ New Media Literacies class at USC has been a treasure-trove of readings and insights. Three recent articles covered in class — read alongside Jenkins’ own book, Convergence Culture, and his excellent MacArthur-funded New Media Literacies white paper — struck me as particularly essential for anyone who’s looking to build an understanding of what multimodal communication is and how transmedia relates to education, literacy and literature. Most of these readings can be found in various corners of the Web, but I’ve also posted them here (along with a brief gloss and anecdote) for those who are interested. They are:

Ito’s succinct article makes the case most directly: “technologies of the imagination populate even the most mundane corners of our lives,” (34) and, contrary to the fears of those who worry that new media threatens to compartmentalize and disembody, the media mix is in practice productive of a culture that is “extroverted and hypersocial, sociality augmented by a dense set of technologies, signifiers, and systems of exchange.” (32) Buckingham and Sefton-Green hammer the point home: skeptics ought to consider examples like the “striking contrast between the high levels of [multimodal reading, sociality and production] activity that characterize the Pokemon phenomenon and the passivity that increasingly suffuses our children’s schooling” (30); and who could disagree that banning such phenomena from the school yard would do anything other than increase their “forbidden appeal” and “prevent schools from building on the enthusiasms children possess”? (31)

Of course, we have a long way to go before these kinds of messages can establish a critical mass in institutional and creative practice. Last week, I attended an experimental literature conference and found that while many of the assembled authors and scholars were keen on experimenting with new media, few if any of them were open to a wholesale redefinition of what literature is/can be. (( …and here I’m thinking not in either/or terms but in both/and: the novel will always be around and will always be the best at doing those things that novels do best. But there are other kinds of literature lurking in the shadows, and that’s what I’m interested in here. )) The works presented would inevitably employ language — spoken or written — as their core expressive resource (unsurprising for a conference largely run and constituted by poets and English professors), which they would then back up with video, flash animations, sound effects, etc. The effect of this was to reduce any image, sound, interactive or procedural elements present in the works to subordinate “supporting” status, lending credence to the commonly-expressed concern that the use of new media “in” literature amounts to little more than gimmickry. As Kress argues, we need to not only shift our definition of text to include “any instance of communication in any mode or in any combination of modes, whether recorded or not,” (48), but also our concept of the role design plays in both reading and writing. “Design does not ask, ‘what was done before, how, for whom, with what?” Kress writes. Rather, Design asks, “What is needed now, in this one situation, with this configuration of purposes, aims, audience, and with these resources, and given my interests in this situation?” (49)

The easy analogy here is that of the early cinema, wherein fiction films were shot using the conventions of the proscenium drawn from the theatre. It was only after a thorough interrogation of the affordances of the camera and the film splicer that the cinema began to reveal itself as a space for artistic endeavor. That is, once filmmakers let go of the notion that the cinema should attempt to create the same experiences as earlier forms of narrative art, they found themselves liberated to discover the unique way of “speaking” that film affords. What complicates this analogy is that we now confront a dynamic multiplicity of media modes. Like Gardner’s multiply-intelligent children, not all authors are going to be able to work well across all media. But in an age of expanding definitions of words like “text,” “author,” and “reading,” creators of literature, as educators and thought leaders, need to ask themselves the questions Kress’s personified Design asks: “What is needed now…with these resources, and given my interests?” Intelligently using new media is not about adding bells and whistles or referencing the Web — rather, it’s about selecting the right mode (or modes) to express what it is you have to say.


Belgian experimental video game developer Tale of Tales has released its new game, Fatale:

Fatale is an interactive vignette in realtime 3D inspired by the story of Salome.

Explore a living tableau filled with references to the legendary tale and enjoy the moonlit serenity of a fatal night in the orient. Fatale offers an experimental play experience that stimulates the imagination and encourages multiple interpretations and personal associations. (Tale of Tales: Fatale)

Don’t Worry Vivian, the World is Really Real

I agree with Vivian Sobchack that “within the dominant cultural and techno-logic of the electronic there are those out there who prefer the simulated body and a virtual world…” and that these people are, well, basically nuts. The nanotech immortality fantasies of Ray Kurzweil et al notwithstanding, I think this perspective is increasingly a minority one, and I can’t accept that there are many serious people out there who would argue for a completely disembodied, brain-in-a-vat/brain-in-the-machine cyberfuture. Indeed, I believe such thoughts basically emerge from an early wave of naive techno-utopianism and catastrophe theory, and have been propagated largely via non-technical and somewhat dim-witted (vaguely new age, often times stoner burnout) quarters. It all smacks of mediocre 50s sci-fi and its derivatives in Scientology and other tech/alien cult manifestations, and yet Sobchack talks about it as if this motive is at the dark heart of our culture.

All this is to say that the whole argument in “The Scene of the Screen: Envisioning Photographic, Cinematic, and Electronic ‘Presence'” is a bit of a straw-man kind of thing. Yes, definitely — “an insubstantial electronic presence can ignore [all the] ills the flesh is heir to outside the image and the datascape,” and that would suck, but are we really moving deeper into the screen and disembodiment, full stop, end of story? Sobchack says yes, and I can understand how she got there.

At the time of her writing, the notion that we (or at least those of us who will be able to afford it) are all going to live our futures through our brainstems in an all-encompassing, “jacked-in” Virtual Reality was at its zenith, particularly in popular culture. The fear was that people would become so wrapped up in their disembodied virtual existences (foisted upon them by their robot overlords) that they would fail to notice that the were, phenomenologically speaking, sitting in a tub of goo far away from the site of their action. But while the Matrix films provided the culture with an outlet for various anxieties about technology, identity and control, they were, at bottom, kind of vacant, semi-obvious entertainments that recycled ideas and story figures that have been around since the 50s. Their cause was putatively a noble one, but the films themselves were little more than the pop finales to the initial gasps of fear and anxiety that accompanied the birth of “the electronic.”

I would argue that our future can’t be plotted on a phenomenological continuum that has “embodied” at one end and “disembodied” at the other. Blade Runner and the Matrix trilogy tell us more about the fears and conceits of the early 1980s and late 1990s than they do about the future we are actually confronting in the here and now. And yes, I know — sure, there’s something to be harvested there about the “techno-logic” of late capitalism and of course, yes, it’s all valuable, all of it. But Sobchack spends a lot of intellectual capital worrying about the phenomenological effects of a theoretical reality that increasingly bears little resemblence to what’s actually happening on the ground. No one outside of the most moronic and outmoded subcultures of misguided paranoid technoenthusiasm uses the words “meat” or “wetware” to refer to their bodies. Indeed, highly embodied cultural manifestations like DIY, networked public play and mobility are arguably emerging as the dominant paradigms. While screens continue to proliferate, they are arguably becoming less central to our “lifeworld” (while computation nonetheless continues its ascent behind the scenes). The once easily-drawn line between the Virtual and the Real is now revealed to be a grand fallacy, a product of the very fears and ignorances that produced Sobchack’s essay in the first place.

Put away Neuromancer and pick up Pattern Recognition or Spook Country. Blade Runner and La Jetee are great — so is Denno Coil.

Image Mapping and Tracking for AR Apps

I’m interested in the ways that augmented reality can be used to extend storytelling and interaction into the real world. The literary and gameplay potentials presented by this nascent technology seem limitless. That said, we have a fair distance to go before our world starts looking like the one depicted in Denno Coil. One of the biggest stumbling blocks I’ve encountered is the issue of precise positioning. Without knowing the user’s exact location and orientation, an AR system can’t properly overlay/position objects. Most of the AR apps we’ve seen thus far depend on glyphs to accomplish this task; others use carefully pre-positioned wifi routers or Bluetooth nodes to triangulate the user’s location. The problem with these solutions is that — while they make for decent demos — they don’t really scale. If we’re going to tell stories using AR, I suspect that we’ll be looking for solutions that break free of the need for pre-set glyphs, routers or other equipment.

This is where image recognition comes in. Projects like Microsoft’s Photosynth illustrate the capacity of image databases to define 3D space. More recently, AR researchers have started to use image recognition/mapping metaphors to create fluid “glyph-free” applications. The team at the University of Graz’s Christian Doppler Laboratory have just posted some exciting new videos of their work in this field.

These videos hint at the kind of seamlessness of interaction we can expect from AR in the near future.

More: Handheld Augmented Reality at the Christian Doppler Laboratory