I saw the Avengers (movie) last week – loved it. In one of the (many) commercials before the previews began, was an ad promoting the Marvel Augmented Reality (AR) application that would let the viewer interact with Avenger toys, posters, cutouts, etc. I’ve seen AR in action and it is cool – really cool. It can be a fun and engaging way to get people to interact with brands, but, as with any marketing effort, enabling people to interact on a different layer means taking the good with the bad. An excellent article in New Scientist by Catherine de Lange addresses “The hidden digital world uncloaked.”
It’s Sunday afternoon in London’s Trafalgar Square. Not far from the fountains, someone has left an enigmatic message on the steps: “Fine mist washes over my face,” it reads. “Gray skies seem bluer here.” Tourists wander by, but none of them can see the words. In fact, there are hidden messages painted all over the square. On the side of the National Gallery, somebody has scribbled a declaration in large letters: “Maria, I love you”. That graffiti has been there for more than a month without anybody cleaning it off. And inside a nearby restaurant, waiters are serving pizza, blissfully unaware of the message on the wall saying the food looks like pigeon droppings. The square is home to invisible images too – and even strange creatures. A yellow pufferfish, for example, is floating in the air above a metal sign, blowing bubbles.
Visit any major city, and you will find it teeming with similar messages and images. For the past few years, millions of people have been using location-aware smartphone apps to daub comments, ratings, images and videos on top of places, objects and, to a certain extent, even themselves. This digital graffiti is personal, informative, sometimes trivial and often subversive. And soon it will be thrust into much clearer view. Later this year, Google is expected to launch a set of “augmented reality” glasses that can overlay digital information onto the physical world. Apple is also rumoured to be investing in such technology. One of the upshots will be that the digital graffiti people have been creating for the past few years will become increasingly visible. It’s a development that is poised to disrupt business, challenge the law and transform how we navigate the world.
People have daubed messages and tags onto physical places since at least the 19th century, but they only began writing onto the world digitally in a big way in the past decade, when smartphones equipped with GPS location-tracking arrived. The first thing this technology allowed people to do was add reviews to restaurants and other spots they visited. Those with the app Yelp, for instance, can whip out their phone to reveal hundreds of comments left by fellow users attached to points of interest in the vicinity.
Then people began to tag the physical world as part of their online social life. Take the popular smartphone app Foursquare. It encourages people to “check in” as they arrive at a bar or other venue, sharing their location or leaving local advice for friends. The status updates on social networks like Facebook and Twitter can also be geotagged, meaning that people share not only what they are doing, but where. A smartphone can display a map of people tweeting nearby, for example.
Now apps are emerging that explicitly trumpet their ability to tag the physical world like graffiti. One comes from Caterina Fake, a co-founder of photo-sharing website Flickr. Her new app, Pinwheel, allows users to leave photos, videos and text at any location they want. Some might be memories, like the scene of a first kiss, others are tips like where to find the best doughnuts in the area. “All of these stories and information and gossip and history are actually already there,” says Fake. “It’s just never had a way of making itself known.”
At the moment, smartphone users see most of this tagging as pins on a map of their vicinity. But increasingly, it is becoming possible to see the digital notes that people leave behind superimposed over the world as graphics – or “augmented reality” (AR).
AR has come a long way since the term was coined in the 1990s by Tom Caudell and David Mizell at Boeing. They designed a heads-up display through which computer-generated graphics were incorporated into a person’s view of the physical world. It was initially seen as a tool for aiding people in manufacturing, but it soon became apparent that such devices could overlay any scene with any kind of information.
Achieving that vision of an augmented world has proven more difficult technically than many hoped, but smartphone technology has brought it a step closer. Location-tracking via GPS was the start, but when phones began to incorporate cameras and compasses a whole host of AR apps and games emerged. You peer at a scene through the device’s camera, and see graphics laid on top of it in real time. Yelp users, for instance, can see star ratings floating over the restaurants in front of them.
Integrated world
More recently, it has become much easier for individuals to create AR overlays themselves. An app called Aurasma enables people to add photos, videos and animations to anything they care to augment: buildings, signs, statues and so on. Aurasma claims to have more than 3 million users already. Central London, for example, features augmentations ranging from YouTube clips to animations of Batman.
To add one yourself, you first snap a photo of a scene, say a brick wall. Then you search your phone’s memory, the internet or the app’s library for a digital layer – perhaps a video you had recorded – and position it over the image of the wall using the touchscreen. People nearby who have the app will then know it is there because they can see local augmentations on a map. If they visit the same wall and look at it through their phone camera, Aurasma will display your layer – in real time. “Everyone uses the app for a different kind of thing,” says Tamara Roukaerts of Aurasma. “I take videos of my son and leave them around the house. I can see videos from a year ago in the place where they were taken: it’s a window on the past.”
The major downside with such AR apps is the need to view this digital world via a small screen. But there are signs that augmented reality is poised to enter much wider use. Earlier this year, Google announced it is developing a headset worn like glasses that displays video, graphics and text. Apple has also filed a series of AR-related patents, hinting that it may have a device of its own in the pipeline.
If such eyewear becomes widely available, it will allow for the seamless integration of the physical and digital world, with no need to log into individual apps to see what is around you. All the information would be displayed automatically: clouds of comments, images and other ephemera popping up as you walk down the street.
So if the era of seamless AR is at hand, what will be the consequences when the rest of the world sees all the digital tags people have created? For a start, it will have a disruptive effect on business and the law. It might be tempting to see digital tagging as harmless – it is certainly easier to avoid or remove than actual paint – but not all of it is benign.
Consider this case featuring the oil giant BP. In 2010, Mark Skwarek, an artist, released an AR app called “The leak in your home town” in response to the Deepwater Horizon oil spill. When users pointed their phone’s camera at any BP logo, it became augmented with a broken pipe spewing out oil. Naturally, BP was not happy. “It was cited by trademark lawyers as the first example of a logo hack,” says Skwarek.
In another of his projects, called protestAR, Skwarek managed to tag a place that was otherwise off limits. During the protests carried out by the Occupy movement in the US last year, many parts of New York City were cordoned off so people could not protest there – particularly around the New York Stock Exchange. So Skwarek staged an AR protest instead. Anyone visiting this part of the city, even today, can switch on their phones, and via an AR app called Layar, see the street teeming with images of protesters accompanied by raucous chants.
These augmentations might be seen by many as nothing more than mischievous stunts, and today you have to make an effort to view them. But they illustrate how digital tagging could be unwanted in the eyes of some people. It is conceivable that offensive digital imagery could be plastered onto a school or place of worship, for example, and in a seamlessly augmented world such tags may be harder to avoid. The internet reflects all aspects of human nature, including the darker side. If that unpleasantness became visibly attached to physical places, it will surely test our notions of freedom of expression.
According to James Gatto, a technology lawyer at legal firm Pillsbury, based in McLean, Virginia, there are no specific laws in place to stop people from placing digital tags wherever they want, as long as they are not defamatory, threatening or in breach of copyright.
Still, eventually the law may be forced to adapt, says Andy Miah, chair of Ethics and Emerging Technologies at the University of the West of Scotland in Paisley, UK. That might mean that building owners have a right to demand that app designers erase unwanted tags. “We might even ‘write protect’ buildings so only those with ownership of the property can add layers of content,” he says.
There are some signs that app designers could also put in place technical barriers to prevent people from augmenting certain things and areas. For example, if companies so wish, they can already stop people annotating their logo using the Aurasma app. Right now, though, it would be virtually impossible to bar the augmentation of a particular place from all apps – there are simply too many to block.
As digital tagging of physical things continues to converge with AR, another implication worth considering is what it could mean for all the data that is already “attached” to individuals. In one vision of an augmented world, a person’s online activity and other details could be visibly linked to them – their social network activity floating like a cloud above their head as they move around, for example.
Some people might not want that kind of link, but it could have its uses. After all, being able to visibly see a person’s online persona while at a conference or party, for instance, might help spark a conversation.
That potential might turn sour if people lose control of the details that others can see about them. Some of the issues that such technology raises were brought into focus earlier this year with a controversial smartphone app calledGirls Around Me. It wasn’t AR per se, but it did allow its users to identify people around them along with their digital annotations. In particular, the app told people about the “hot girls” in their vicinity, by using information from Foursquare. And if the women had not made their Facebook profiles private, it also allowed users to see their interests, photo albums and even telephone numbers.
Tagged people
Of course none of the women had volunteered any information for this purpose. So a woman might have checked into a bar nearby using her Foursquare app, oblivious to the fact that round the corner, a Girls Around Me user was browsing through her holiday snaps. None of this is illegal, but many people felt it crossed a line. Whether or not you agree, the furore it caused was enough for Foursquare to block the app from using its data. It was also pulled from Apple’s app store.
One day, we might be able to add AR tags to other people, and the tags would then follow them around for all to see. This might have its uses, butAmber Case, co-creator of location-based app Geoloqi has pointed out that such a development could also enable new forms of bullying. For example, a child might attach the equivalent of a “kick me” sign to a classmate, she says.
For better or worse, the convergence of these technologies is poised to change how we interact with one another, says Nilesh Zacharius, an online privacy and policy consultant based in New York. “Today, many of us assume that there exists a separation between our online and offline lives,” he says. “The advent of social media began to blur that divide, but augmented reality has the potential to shatter it completely.”
In the meantime, at every moment the world is filling up with digital annotation. We are already swimming in it, and soon a bigger question may be how to filter it down.
Back in Trafalgar Square, the tourists remain unaware of the tags and ephemera floating around them. By the time many of them visit again, the city will have revealed itself in all its multilayered madness.
# # #
We build strategies and everything that goes with them.
Some of the largest organizations in the world, including many in the mortgage and finance industries, trust us with the most important aspects of their business. From defining clients’ brands and identities to developing ongoing campaigns in a variety of media, we provide the communications and measurement tools to move them forward. Applying our experience and dedication to the media and the message, bloomfield knoble handles every detail of our clients’ strategic marketing initiatives.