Virtual Reality (Computing)
Virtual reality (VR), the use of computer modeling and simulation that enables a person to interact with an artificial three-dimensional (3-D) visual or other sensory environment. VR applications immerse the user in a computer-generated environment that simulates reality through the use of interactive devices, which send and receive information and are worn as goggles, headsets, gloves, or body suits. In a typical VR format, a user wearing a helmet with a stereoscopic screen views animated images of a simulated environment. The illusion of "being there" (telepresence) is effected by motion sensors that pick up the user's movements and adjust the view on the screen accordingly, usually in real time (the instant the user's movement takes place). Thus, a user can tour a simulated suite of rooms, experiencing changing viewpoints and perspectives that are convincingly related to his own head turnings and steps. Wearing data gloves equipped with force-feedback devices that provide the sensation of touch, the user can even pick up and manipulate objects that he sees in the virtual environment.
The term virtual reality was coined in 1987 by Jaron Lanier, whose research and engineering contributed a number of products to the nascent VR industry. A common thread linking early VR research and technology development in the United States was the role of the federal government, particularly the Department of Defense, the National Science Foundation, and the National Aeronautics and Space Administration (NASA). Projects funded by these agencies and pursued at university-based research laboratories yielded an extensive pool of talented personnel in fields such as computer graphics, simulation, and networked environments and established links between academic, military, and commercial work. The history of this technological development, and the social context in which it took place, is the subject of this [section].
Early work
Artists, performers, and entertainers have always been interested in techniques for creating imaginative worlds, setting narratives in fictional spaces, and deceiving the senses. Numerous precedents for the suspension of disbelief in an artificial world in artistic and entertainment media preceded virtual reality. Illusionary spaces created by paintings or views have been constructed for residences and public spaces since antiquity, culminating in the monumental panoramas of the 18th and 19th centuries. Panoramas blurred the visual boundaries between the two-dimensional images displaying the main scenes and the three-dimensional spaces from which these were viewed, creating an illusion of immersion in the events depicted. This image tradition stimulated the creation of a series of media—from futuristic theatre designs, stereopticons, and 3-D movies to IMAX movie theatres—over the course of the 20th century to achieve similar effects. For example, the Cinerama widescreen film format, originally called Vitarama when invented for the 1939 New York World's Fair by Fred Waller and Ralph Walker, originated in Waller's studies of vision and depth perception. Waller's work led him to focus on the importance of peripheral vision for immersion in an artificial environment, and his goal was to devise a projection technology that could duplicate the entire human field of vision. The Vitarama process used multiple cameras and projectors and an arc-shaped screen to create the illusion of immersion in the space perceived by a viewer. Though Vitarama was not a commercial hit until the mid-1950s (as Cinerama), the Army Air Corps successfully used the system during World War II for anti-aircraft training under the name Waller Flexible Gunnery Trainer—an example of the link between entertainment technology and military simulation that would later advance the development of virtual reality.
Sensory stimulation was a promising method for creating virtual environments before the use of computers. After the release of a promotional film called This Is Cinerama (1952), the cinematographer Morton Heilig became fascinated with Cinerama and 3-D movies. Like Waller, he studied human sensory signals and illusions, hoping to realize a cinema of the future.
By late 1960, Heilig had built an individual console with a variety of inputs—stereoscopic images, motion chair, audio, temperature changes, odours, and blown air—that he patented in 1962 as the Sensorama Simulator, designed to stimulate the senses of an individual to simulate an actual experience realistically.
During the work on Sensorama, he also designed the Telesphere Mask, a head-mounted stereoscopic 3-D TV display
that he patented in 1960. Although Heilig was unsuccessful in his efforts to market Sensorama, in the mid-1960s he extended the idea to a multiviewer theatre concept patented as the Experience Theater and a similar system called Thrillerama for the Walt Disney Company.
The seeds for virtual reality were planted in several computing fields during the 1950s and '60s, especially in 3-D interactive computer graphics and vehicle/flight simulation. Beginning in the late 1940s, Project Whirlwind, funded by the U.S. Navy, and its successor project, the SAGE (Semi-Automated Ground Environment) early-warning radar system, funded by the U.S. Air Force, first utilized cathode-ray tube (CRT) displays and input devices such as light pens (originally called light guns). By the time the SAGE system became operational in 1957, air force operators were routinely using these devices to display aircraft positions and manipulate related data.
During the 1950s, the popular cultural image of the computer was that of a calculating machine, an automated electronic brain capable of manipulating data at previously unimaginable speeds. The advent of more affordable second-generation (transistor) and third-generation (integrated circuit) computers emancipated the machines from this narrow view, and in doing so it shifted attention to ways in which computing could augment human potential rather than simply substituting for it in specialized domains conducive to number crunching. In 1960 Joseph Licklider, a professor at the Massachusetts Institute of Technology (MIT) specializing in psychoacoustics, posited a man-computer symbiosis
and applied psychological principles to human-computer interactions and interfaces. He argued that a partnership between computers and the human brain would surpass the capabilities of either alone. As founding director of the new Information Processing Techniques Office (IPTO) of the Defense Advanced Research Projects Agency (DARPA), Licklider was able to fund and encourage projects that aligned with his vision of human-computer interaction while also serving priorities for military systems, such as data visualization and command-and-control systems.
Another pioneer was electrical engineer and computer scientist Ivan Sutherland, who began his work in computer graphics at MIT's Lincoln Laboratory (where Whirlwind and SAGE had been developed). In 1963 Sutherland completed Sketchpad, a system for drawing interactively on a CRT display with a light pen and control board. Sutherland paid careful attention to the structure of data representation, which made his system useful for the interactive manipulation of images. In 1964 he was put in charge of IPTO, and from 1968 to 1976 he led the computer graphics program at the University of Utah, one of DARPA's premier research centres. In 1965 Sutherland outlined the characteristics of what he called the ultimate display
and speculated on how computer imagery could construct plausible and richly articulated virtual worlds. His notion of such a world began with visual representation and sensory input, but it did not end there; he also called for multiple modes of sensory input. DARPA sponsored work during the 1960s on output and input devices aligned with this vision, such as the Sketchpad III system by Timothy Johnson, which presented 3-D views of objects; Larry Roberts's Lincoln Wand, a system for drawing in three dimensions; and Douglas Engelbart's invention of a new input device, the computer mouse.
Within a few years, Sutherland contributed the technological artifact most often identified with virtual reality, the head-mounted 3-D computer display. In 1967 Bell Helicopter (now part of Textron Inc.) carried out tests in which a helicopter pilot wore a head-mounted display (HMD) that showed video from a servo-controlled infrared camera mounted beneath the helicopter. The camera moved with the pilot's head, both augmenting his night vision and providing a level of immersion sufficient for the pilot to equate his field of vision with the images from the camera. This kind of system would later be called augmented reality
because it enhanced a human capacity (vision) in the real world. When Sutherland left DARPA for Harvard University in 1966, he began work on a tethered display for computer images. This was an apparatus shaped to fit over the head, with goggles that displayed computer-generated graphical output. Because the display was too heavy to be borne comfortably, it was held in place by a suspension system. Two small CRT displays were mounted in the device, near the wearer's ears, and mirrors reflected the images to his eyes, creating a stereo 3-D visual environment that could be viewed comfortably at a short distance. The HMD also tracked where the wearer was looking so that correct images would be generated for his field of vision. The viewer's immersion in the displayed virtual space was intensified by the visual isolation of the HMD, yet other senses were not isolated to the same degree and the wearer could continue to walk around.
Virtual Reality in Education and Training
An important area of application for VR systems has always been training for real-life activities. The appeal of simulations is that they can provide training equal or nearly equal to practice with real systems, but at reduced cost and with greater safety. This is particularly the case for military training, and the first significant application of commercial simulators was pilot training during World War II. Flight simulators rely on visual and motion feedback to augment the sensation of flying while seated in a closed mechanical system on the ground. The Link Company, founded by former piano maker Edwin Link, began to construct the first prototype Link Trainers during the late 1920s, eventually settling on the blue box design acquired by the Army Air Corps in 1934. The first systems used motion feedback to increase familiarity with flight controls. Pilots trained by sitting in a simulated cockpit, which could be moved hydraulically in response to their actions. Later versions added a cyclorama scene painted on a wall outside the simulator to provide limited visual feedback. Not until the Celestial Navigation Trainer, commissioned by the British government in World War II, were projected film strips used in Link Trainers—still, these systems could only project what had been filmed along a correct flight or landing path, not generate new imagery based on a trainee's actions. By the 1960s, flight trainers were using film and closed-circuit television to enhance the visual experience of flying. The images could be distorted to generate flight paths that diverted slightly from what had been filmed; sometimes multiple cameras were used to provide different perspectives, or movable cameras were mounted over scale models to depict airports for simulated landings.
Inspired by the controls in the Link flight trainer, Sutherland suggested that such displays include multiple sensory outputs, force-feedback joysticks, muscle sensors, and eye trackers; a user would be fully immersed in the displayed environment and fly through concepts which never before had any visual representation.
In 1968 he moved to the University of Utah, where he and his colleague David Evans founded Evans & Sutherland Computer Corporation. The new company initially focused on the development of graphics applications, such as scene generators for flight simulator systems. These systems could render scenes at roughly 20 frames per second in the early 1970s, about the minimum frame rate for effective flight training. General Electric Company constructed the first flight simulators with built-in, real-time computer image generation, first for the Apollo program in the 1960s, then for the U.S. Navy in 1972. By the mid-1970s, these systems were capable of generating simple 3-D models with a few hundred polygon faces; they utilized raster graphics (collections of dots) and could model solid objects with textures to enhance the sense of realism (see computer graphics). By the late 1970s, military flight simulators were also incorporating head-mounted displays, such as McDonnell Douglas Corporation's VITAL helmet, primarily because they required much less space than a projected display. A sophisticated head tracker in the HMD followed a pilot's eye movements to match computer-generated images (CGI) with his view and handling of the flight controls.
Advances in flight simulators, human-computer interfaces, and augmented reality systems pointed to the possibility of immersive, real-time control systems, not only for research or training but also for improved performance. Since the 1960s, electrical engineer Thomas Furness had been working on visual displays and instrumentation in cockpits for the U.S. Air Force. By the late 1970s, he had begun development of virtual interfaces for flight control, and in 1982 he demonstrated the Visually Coupled Airborne Systems Simulator—better known as the Darth Vader helmet, for the armoured archvillain of the popular movie Star Wars. From 1986 to 1989, Furness directed the air force's Super Cockpit program. The essential idea of this project was that the capacity of human pilots to handle spatial information depended on these data being portrayed in a way that takes advantage of the human's natural perceptual mechanisms.
Applying the HMD to this goal, Furness designed a system that projected information such as computer-generated 3-D maps, forward-looking infrared and radar imagery, and avionics data into an immersive, 3-D virtual space that the pilot could view and hear in real time. The helmet's tracking system, voice-actuated controls, and sensors enabled the pilot to control the aircraft with gestures, utterances, and eye movements, translating immersion in a data-filled virtual space into control modalities. The more natural perceptual interface also reduced the complexity and number of controls in the cockpit. The Super Cockpit thus realized Licklider's vision of man-machine symbiosis by creating a virtual environment in which pilots flew through data. Beginning in 1987, British Aerospace (now part of BAE Systems) also used the HMD as the basis for a similar training simulator, known as the Virtual Cockpit, that incorporated head, hand, and eye tracking, as well as speech recognition.
Sutherland and Furness brought the notion of simulator technology from real-world imagery to virtual worlds that represented abstract models and data. In these systems, visual verisimilitude was less important than immersion and feedback that engaged all the senses in a meaningful way. This approach had important implications for medical and scientific research. Project GROPE, started in 1967 at the University of North Carolina by Frederick Brooks, was particularly noteworthy for the advancements it made possible in the study of molecular biology. Brooks sought to enhance perception and comprehension of the interaction of a drug molecule with its receptor site on a protein by creating a window into the virtual world of molecular docking forces. He combined wire-frame imagery to represent molecules and physical forces with haptic (tactile) feedback mediated through special hand-grip devices to arrange the virtual molecules into a minimum binding energy configuration. Scientists using this system felt their way around the represented forces like flight trainees learning the instruments in a Link cockpit, grasping
the physical situations depicted in the virtual world and hypothesizing new drugs based on their manipulations. During the 1990s, Brooks's laboratory extended the use of virtual reality to radiology and ultrasound imaging.
Virtual reality was extended to surgery through the technology of telepresence, the use of robotic devices controlled remotely through mediated sensory feedback to perform a task. The foundation for virtual surgery was the expansion during the 1970s and '80s of microsurgery and other less invasive forms of surgery. By the late 1980s, microcameras attached to endoscopic devices relayed images that could be shared among a group of surgeons looking at one or more monitors, often in diverse locations. In the early 1990s, a DARPA initiative funded research to develop telepresence workstations for surgical procedures. This was Sutherland's window into a virtual world,
with the added dimension of a level of sensory feedback that could match a surgeon's fine motor control and hand-eye coordination. The first telesurgery equipment was developed at SRI International in 1993; the first robotic surgery was performed in 1998 at the Broussais Hospital in Paris.
Entertainment
As virtual worlds became more detailed and immersive, people began to spend time in these spaces for entertainment, aesthetic inspiration, and socializing. Research that conceived of virtual places as fantasy spaces, focusing on the activity of the subject rather than replication of some real environment, was particularly conducive to entertainment. Beginning in 1969, Myron Krueger of the University of Wisconsin created a series of projects on the nature of human creativity in virtual environments, which he later called artificial reality. Much of Krueger's work, especially his VIDEOPLACE system, processed interactions between a participant's digitized image and computer-generated graphical objects. VIDEOPLACE could analyze and process the user's actions in the real world and translate them into interactions with the system's virtual objects in various preprogrammed ways. Different modes of interaction with names like finger painting
and digital drawing
suggest the aesthetic dimension of this system. VIDEOPLACE differed in several aspects from training and research simulations. In particular, the system reversed the emphasis from the user perceiving the computer's generated world to the computer perceiving the user's actions and converting these actions into compositions of objects and space within the virtual world. With the emphasis shifted to responsiveness and interaction, Krueger found that fidelity of representation became less important than the interactions between participants and the rapidity of response to images or other forms of sensory input.
The ability to manipulate virtual objects and not just see them is central to the presentation of compelling virtual worlds—hence the iconic significance of the data glove in the emergence of VR in commerce and popular culture. Data gloves relay a user's hand and finger movements to a VR system, which then translates the wearer's gestures into manipulations of virtual objects. The first data glove, developed in 1977 at the University of Illinois for a project funded by the National Endowment for the Arts, was called the Sayre Glove after one of the team members. In 1982 Thomas Zimmerman invented the first optical glove, and in 1983 Gary Grimes at Bell Laboratories constructed the Digital Data Entry Glove, the first glove with sufficient flexibility and tactile and inertial sensors to monitor hand position for a variety of applications, such as providing an alternative to keyboard input for data entry.
Zimmerman's glove would have the greatest impact. He had been thinking for years about constructing an interface device for musicians based on the common practice of playing air guitar
—in particular, a glove capable of tracking hand and finger movements could be used to control instruments such as electronic synthesizers. He patented an optical flex-sensing device (which used light-conducting fibres) in 1982, one year after Grimes patented his glove-based computer interface device. By then, Zimmerman was working at the Atari Research Center in Sunnyvale, California, along with Scott Fisher, Brenda Laurel, and other VR researchers who would be active during the 1980s and beyond. Jaron Lanier, another researcher at Atari, shared Zimmerman's interest in electronic music. Beginning in 1983, they worked together on improving the design of the data glove, and in 1985 they left Atari to start up VPL Research; its first commercial product was the VPL DataGlove.
By 1985, Fisher had also left Atari to join NASA's Ames Research Center at Moffett Field, California, as founding director of the Virtual Environment Workstation (VIEW) project. The VIEW project put together a package of objectives that summarized previous work on artificial environments, ranging from creation of multisensory and immersive virtual environment workstations
to telepresence and teleoperation applications. Influenced by a range of prior projects that included Sensorama, flight simulators, and arcade rides, and surprised by the expense of the air force's Darth Vader helmets, Fisher's group focused on building low-cost, personal simulation environments. While the objective of NASA was to develop telerobotics for automated space stations in future planetary exploration, the group also considered the workstation's use for entertainment, scientific, and educational purposes. The VIEW workstation, called the Virtual Visual Environment Display when completed in 1985, established a standard suite of VR technology that included a stereoscopic head-coupled display, head tracker, speech recognition, computer-generated imagery, data glove, and 3-D audio technology.
The VPL DataGlove was brought to market in 1987, and in October of that year it appeared on the cover of Scientific American. VPL also spawned a full-body, motion-tracking system called the DataSuit, a head-mounted display called the EyePhone, and a shared VR system for two people called RB2 (Reality Built for Two
). VPL declared June 7, 1989, Virtual Reality Day.
On that day, both VPL and Autodesk publicly demonstrated the first commercial VR systems. The Autodesk VR CAD (computer-aided design) system was based on VPL's RB2 technology but was scaled down for operation on personal computers. The marketing splash introduced Lanier's new term virtual reality as a realization of cyberspace,
a concept introduced in science fiction writer William Gibson's Neuromancer in 1984. Lanier, the dreadlocked chief executive officer of VPL, became the public celebrity of the new VR industry, while announcements by Autodesk and VPL let loose a torrent of enthusiasm, speculation, and marketing hype. Soon it seemed that VR was everywhere, from the Mattel/Nintendo PowerGlove (1989) to the HMD in the movie The Lawnmower Man (1992), the Nintendo VirtualBoy game system (1995), and the television series VR5 (1995).
Numerous VR companies were founded in the early 1990s, most of them in Silicon Valley, but by mid-decade most of the energy unleashed by the VPL and Autodesk marketing campaigns had dissipated. The VR configuration that took shape over a span of projects leading from Sutherland to Lanier—HMD, data gloves, multimodal sensory input, and so forth—failed to have a broad appeal as quickly as the enthusiasts had predicted. Instead, the most visible and successfully marketed products were location-based entertainment
systems rather than personal VR systems. These VR arcades and simulators, designed by teams from the game, movie, simulation, and theme park industries, combined the attributes of video games, amusement park rides, and highly immersive storytelling. Perhaps the most important of the early projects was Disneyland's Star Tours, an immersive flight simulator ride based on the Star Wars movie series and designed in collaboration with producer George Lucas's Industrial Light & Magic. Disney had long built themed rides utilizing advanced technology, such as animatronic characters—notably in Pirates of the Caribbean, an attraction originally installed at Disneyland in 1967. Star Tours utilized simulated motion and special-effects technology, mixing techniques learned from Hollywood films and military flight simulators with strong story lines and architectural elements that shaped the viewers' experience from the moment they entered the waiting line for the attraction. After the opening of Star Tours in 1987, Walt Disney Imagineering embarked on a series of projects to apply interactive technology and immersive environments to ride systems, including 3-D motion-picture photography used in Honey, I Shrunk the Audience (1995), the DisneyQuest indoor interactive theme park
(1998), and the multiplayer-gaming virtual world, Toontown Online (2001).
In 1990, Virtual World Entertainment opened the first BattleTech emporium in Chicago. Modeled loosely on the U.S. military's SIMNET system of networked training simulators, BattleTech centres put players in individual pods,
essentially cockpits that served as immersive, interactive consoles for both narrative and competitive game experiences. All the vehicles represented in the game were controlled by other players, each in his own pod and linked to a high-speed network set up for a simultaneous multiplayer experience. The player's immersion in the virtual world of the competition resulted from a combination of elements, including a carefully constructed story line, the physical architecture of the arcade space and pod, and the networked virtual environment. During the 1990s, BattleTech centres were constructed in other cities around the world, and the BattleTech franchise also expanded to home electronic games, books, toys, and television.
While the Disney and Virtual World Entertainment projects were the best-known instances of location-based VR entertainments, other important projects included Iwerks Entertainment's Turbo Tour and Turboride 3-D motion simulator theatres, first installed in San Francisco in 1992; motion-picture producer Steven Spielberg's Gameworks arcades, the first of which opened in 1997 as a joint project of Universal Studios, Sega Corporation, and Dreamworks SKG; many individual VR arcade rides, beginning with Sega Arcade's R360 gyroscope flight simulator, released in 1991; and, finally, Visions of Reality's VR arcades, the spectacular failure of which contributed to the bursting of the investment bubble for VR ventures in the mid-1990s.
Augmented reality
Augmented reality, in computer programming, a process of combining or augmenting video or photographic displays by overlaying the images with useful computer-generated data. The earliest applications of augmented reality were almost certainly the heads-up-displays
(HUDs) used in military airplanes and tanks, in which instrument panel-type information is projected onto the same cockpit canopy or viewfinder through which a crew member sees the external surroundings. Faster computer processors have made it feasible to combine such data displays with real-time video. Among the earliest and most prominent examples of this type of augmented reality, as first shown on the Fox Broadcasting Company's network in the mid-1990s, were the yellow first-down stripes superimposed on television images of American gridiron football fields and the virtual flight paths added to help television viewers track the paths of hockey pucks and golf balls.
Augmented reality is commonly used in electronic first-person shooter games to add environmental, health, and other information to players' viewpoints. (Various militaries have begun to experiment with adding similar overlays to real soldiers using personal head-mounted visors.) Augmented reality applications also have been developed for smartphones to display information such as building addresses, real estate signs, retail sales offers, and restaurant reviews on specific sites seen through the devices' viewfinder or electronic displays. Such information may be supplied using a global positioning system (GPS) linked to a commercial or open-source database. Some visionaries hope to take the next step of adding such informational displays to lightweight sunglasses, and proponents of social networking envision all sorts of personal identification tags being added to such displays.
SAGE's Bloopers
Whirlwind's design was heavily influenced by ENIAC; in turn, it laid the groundwork for the Semi-Automatic Ground Environment (SAGE), the vast computer system that ran the North American Air Defense Command (NORAD) from the 1950s until the 1980s. Four-storey "direction centres" were installed in twenty-seven command-and-control stations across the United States, and their twin terminals – one for operation, one for backup – included a light gun for designating targets (resembling the Nintendo "Zapper") and ashtrays integrated into the console. SAGE is best memorialised in the vast, paranoid aesthetic of Cold War computing systems, from Dr. Strangelove in 1964 to WarGames, the 1983 blockbuster that told the story of a computer intelligence unable to distinguish between reality and simulation, and famous for its concluding line: the only winning move is not to play.
In order to make such a complex system work, 7,000 IBM engineers were employed to write the largest single computer programme ever created, and 25,000 dedicated phone lines were laid to connect the various locations. Despite this, SAGE is best known for its bloopers: leaving trai[l]ing tapes running so that follow-on shifts mistook simulation data for actual missile attacks, or designating flocks of migrating birds as incoming Soviet bomber fleets.
Histories of computation projects typically write off such efforts as anachronistic failures, comparing them to modern bloat-ridden software projects and government IT initiatives that fall short of their much-vaunted goals and are superceded by subsequent, better-engineered systems before they're even completed, feeding a cycle of obsolescence and permanent revision. But what if these stories are the real history of computation: a litany of failures to distinguish between simulation and reality; a chronic failure to identify the conceptual chasm at the heart of computational thinking, of our construction of the world?
ENIAC's Shape-shift
We have been conditioned to believe that computers render the world clearer and more efficient, that they reduce complexity and facilitate better solutions to the problems that beset us, and that they expand our agency to address an ever-widening domain of experience. But what if this is not true at all? A close reading of computer history reveals an ever-increasing opacity allied to a concentration of power, and the retreat of that power into ever more narrow domains of experience. By reifying the concerns of the present in unquestionable architectures, computation freezes the problems of the immediate moment into abstract, intractable dilemmas; obsessing over the inherent limitations of a small class of mathematical and material conundrums rather than the broader questions of a truly democratic and egalitarian society.
By conflating approximation with simulation, the high priests of computational thinking replace the world with flawed models of itself; and in doing so, as the modellers, they assume control of the world.
Once it became obvious that SAGE was worse than useless at preventing a nuclear war, it shape-shifted, following an in-flight meeting between the president of American Airlines and an IBM salesman, into the Semi-Automated Business Research Environment (SABRE) – a multinational corporation for managing airline reservations. All the pieces were in place: the phone lines, the weather radar, the increasingly privatised processing power, and the ability to manage real-time data flows in an era of mass tourism and mass consumer spending. A machine designed to prevent commercial airlines from being accidentally shot down – a necessary component of any air defence system – pivoted to managing those same flights, buoyed by billions of dollars of defence spending. Today, SABRE connects more than 57,000 travel agents and millions of travellers with more than 400 airlines, 90,000 hotels, 30 car rental companies, 200 tour operators, and dozens of railways, ferries and cruise lines. A kernel of computational Cold War paranoia sits at the heart of billions of journeys made every year.
Aviation
Aviation will recur as a site where technology, scientific research, defence and security interests, and computation converge in a nexus of transparency/opacity and visibility/invisibility. One of the most extraordinary visualisations on the internet is that provided by real-time plane-tracking websites. Anyone can log on and see, at any time, thousands upon thousands of planes in the air, tracking from city to city, mobbing the Atlantic, coursing in great rivers of metal along international flight paths. It's possible to click on any one of the thousands of little plane icons and see its track, its make and model, the operator and flight number, its origin and destination, and its altitude, speed, and time of flight. Every plane broadcasts an ADS-B signal, which is picked up by a network of amateur flight trackers: [...] thousands of individuals who choose to set up local radio receivers and share their data online. The view of these flight trackers, like that of Google Earth and other satellite image services, is deeply seductive, to the point of eliciting an almost vertiginous thrill – a sublime for the digital age. The dream of every Cold War planner is now available to the general public on freely accessible websites. But this God's-eye view is illusory, as it also serves to block out and erase other private and state activities, from the private jets of oligarchs and politicians to covert surveillance flights and military manoeuvres. For everything that is shown, something is hidden.
GPS
In 1983, Ronald Reagan ordered that the then-encrypted Global Positioning System (GPS) be made available to civilians, following the shooting down of a Korean airliner that strayed into Russian airspace. Over time, GPS has come to anchor a huge number of contemporary applications and become another of the invisible, unquestioned signals that modulate everyday life – another of those things that, more or less, just works
. GPS enables the blue dot in the centre of the map that folds the entire planet around the individual. Its data directs car and truck journeys, locates ships, prevents planes flying into one another, dispatches taxis, tracks logistics inventories and calls in drone strikes. Essentially a vast, space-based clock, the time signal from GPS satellites regulates power grids and stock markets.
But our growing reliance on the system masks the fact that it can still be manipulated by those in control of its signals, including the United States government, which retains the ability to selectively deny positioning signals to any region it chooses.
In the summer of 2017, a series of reports from the Black Sea showed deliberate interference with GPS occurring across a wide area, with ships' navigation systems showing them tens of kilometres off their actual position. Many were relocated onshore, finding themselves virtually marooned in a Russian airbase – the suspected source of the spoofing effort. The Kremlin is surrounded by a similar field, as first discovered by players of Pokémon GO, who found their in-game characters teleported blocks away while trying to play the location-based game in the centre of Moscow. (Particularly enterprising players later turned this realisation to their advantage, using electromagnetic shielding and signal generators to collect points without leaving the house.) In other cases, workers whose labour is remotely monitored by GPS, such as long-distance lorry drivers, have simply jammed the signal to enable them to take breaks and unauthorised routes – throwing off other users along their paths. Each of these examples illustrates how crucial computation is to contemporary life, while also revealing its blind spots, structural dangers, and engineered weaknesses.
Code/Spaces
To take another example from aviation, consider the experience of being in an airport. An airport is a canonical example of what geographers call code/space. Code/space describe the interweaving of computation with the built environment and daily experience to a very specific extent: rather than merely overlaying and augmenting them, computation becomes a crucial component of them, such that the environment and the experience of it actually ceases to function in the absence of code.
In the case of the airport, code both facilitates and coproduces the environment. Prior to visiting an airport, passengers engage with an electronic booking system – such as SABRE – that registers their data, identifies them, and makes them visible to other systems, such as check-in desks and passport control. If, when they find themselves at the airport, the system becomes unavailable, it is not a mere inconvenience. Modern security procedures have removed the possibility of paper identification or processing: software is the only accepted arbiter of the process. Nothing can be done; nobody can move. As a result, a software crash revokes the building's status as an airport, transforming it into a huge shed filled with angry people. This is how largely invisible computation coproduces our environment – its critical necessity revealed only in moments of failure, like a kind of brain injury.
Code/spaces increasingly describe more than just smart buildings. Thanks to the pervasive availability of network access and the self-replicating nature of corporate and centralising code, more and more daily activities become dependent on their accompanying software. Daily, even private, travel is reliant on satellite routing, traffic information, and increasingly "autonomous" vehicles – which, of course, are not autonomous at all, requiring constant updates and input to proceed. Labour is increasingly coded, whether by end-to-end logistics systems or email servers, which in turn require constant attention and monitoring by workers who are dependent upon them. Our social lives are mediated through connectivity and algorithmic revision. As smartphones become powerful general-purpose computers and computation disappears into every device around us, from smart home appliances to vehicle navigation systems, the entire world becomes a code/space. Far from rendering the idea of a code/space obsolete, this ubiquity underscores our failure to understand the impact of computation on the very ways in we think.
When an e-book is purchased from an online service, it remains the property of the seller, its loan subject to revocation at any time – as happened when Amazon remotely deleted thousands of copies of 1984 and Animal Farm from customers' Kindles in 2009. Streaming music and video services filter the media available by legal jurisdiction and algorithmically determine "personal" preferences. Academic journals determine access to knowledge by institutional affiliation and financial contribution as physical, open-access libraries close down.
Computation Becomes Culture
The ongoing functionality of Wikipedia relies on an army of software agents – bots – to enforce and maintain correct formatting, build connections between articles, and moderate conflicts and incidences of vandalism. At the last survey, bots counted for seventeen of the top twenty most prolific editors and collectively make about 16 per cent of all edits to the encyclopaedia project: a concrete and measurable contribution to knowledge production by code itself. Reading a book, listening to music, researching and learning: these and many other activities are increasingly governed by algorithmic logics and policed by opaque and hidden computational processes. Culture is itself a code/space.
The danger of this emphasis on the coproduction of physical and cultural space by computation is that it in turn occludes the vast inequalities of power that it both relies upon and reproduces. Computation does not merely augment, frame, and shape culture; by operating beneath our everyday, casual awareness of it, it actually becomes culture.
That which computation sets out to map and model it eventually takes over. Google set out to index all human knowledge and became the source and arbiter of that knowledge: it became what people actually think. Facebook set out to map the connections between people – the social graph – and became the platform for those connections, irrevocably reshaping societal relationships. Like an air control system mistaking a flock of birds for a fleet of bombers, software is unable to distinguish between its model of the world and reality – and, once conditioned, neither are we.
Conditioned to Accept
This conditioning occurs for two reasons: because the combination of opacity and complexity renders much of the computational process illegible; and because computation itself is perceived to be politically and emotionally neutral. Computation is opaque: it takes place inside the machine, behind the screen, in remote buildings – within, as it were, a cloud. Even when this opacity is penetrated, by direct apprehension of code and data, it remains beyond the comprehension of most. The aggregation of complex systems in contemporary networked applications means that no single person ever sees the whole picture. Faith in the machine is a prerequisite for its employment, and this backs up other cognitive biases that see automated responses as inherently more trustworthy than nonautomated ones.
This phenomenon is known as automation bias, and it has been observed in every computational domain from spell-checking software to autopilots, and in every type of person. Automation bias ensures that we value automated information more highly than our own experiences, even when it conflicts with other observations – particularly when those observations are ambiguous. Automated information is clear and direct, and confounds the grey areas that muddle cognition. Another associated phenomenon, confirmation bias, reshapes our awareness of the world to bring it better into line with automated information, further affirming the validity of computational solutions, to the point where we may discard entirely observations inconsistent with the machine's viewpoint.
Automation Bias in Aviation
Studies of pilots in high-tech aircraft cockpits have produced multiple examples of automation bias. The pilots of the Korean Air Lines flight whose destruction led to the emancipation of GPS were victims of the most common kind. Shortly after takeoff from Anchorage, Alaska, on August 31, 1983, the flight crew programmed their autopilot with the heading given to them by air traffic control and handed over control of the plane. The autopilot was preprogrammed with a series of waymarks that would take it through the jetways over the Pacific to Seoul, but due either to a mistake in the settings, or an imperfect understanding of the mechanisms of the system, the autopilot did not continue to follow its preassigned route; rather, it stayed fixed on its initial heading, which took it further and further north of its intended route. By the time it left Alaskan airspace, fifty minutes into the flight, it was twelve miles north of its expected position; as it flew on, its divergence increased to fifty, then a hundred miles from its intended course. Over several hours, investigators related, there were several cues that might have alerted the crew to what was occurring. They noticed, but disregarded, the slowly increasing travel time between beacons. They complained about the poor radio reception as they drifted further from the normal air routes. But none of these effects caused the pilots to question the system, or to double-check their position. They continued to trust in the autopilot even as they entered Soviet military airspace over the Kamchatka Peninsula. As fighter jets were scrambled to intercept them, they flew on. Three hours later, still completely unaware of the situation, they were fired upon by a Sukhoi Su-15 armed with two air-to-air missiles, which detonated close enough to wreck their hydraulic systems. The cockpit transcript of the last few minutes of flight shows multiple failed attempts to re-engage the autopilot, as an automated announcement warns of an emergency descent.
Such events have been repeated, and their implications confirmed, in multiple simulator experiments. Worse, such biases are not limited to errors of omission, but include those of commission. When the Korean Air Lines pilots blindly followed the directions of an autopilot, they were taking the road of least resistance. But it has been shown that even experienced pilots will take drastic actions in the face of automated warnings, including against the evidence of their own observations. Oversensitive fire warnings in early Airbus A330 aircraft became notorious for causing numerous flights to divert, often at some risk, even when pilots visually checked for signs of fire multiple times. In a study in the NASA Ames Advanced Concepts Flight Simulator, crews were given contradictory fire warnings during preparation for takeoff. The study found that 75 per cent of the crews following the guidance of an automated system shut down the wrong engine, whereas when following a traditional paper checklist only 25 per cent did likewise, despite both having access to additional information that should have influenced their decision. The tapes of the simulations showed that those following the automated system made their decisions faster and with less discussion, suggesting that the availability of an immediate suggested action prevented them looking deeper into the problem.
Technological Lazines
Automation bias means that technology doesn't even have to malfunction for it to be a threat to our lives – and GPS is again a familiar culprit. In their attempt to reach an island in Australia, a group of Japanese tourists drove their car down onto a beach and directly into the sea because their satellite navigation system assured them there was a viable road. They had to be rescued as the tide rose around them, some fifty feet from the shoreline. Another group in Washington state drove their car into a lake when they were directed off the main road and down a boat ramp. When emergency services responded, they found the car floating in deep water, with only its roof rack visible. For rangers in Death Valley National Park, such occurrences have become so common that they have a term for it: Death by GPS, which describes what happens when travellers, unfamiliar with the area, follow the instructions and not their senses. In a region where many marked roads may be impassable to regular vehicles, and daytime temperatures can reach fifty degrees Celsius with no water available, getting lost will kill you. In these cases, the GPS signal wasn't spoofed, and it didn't drift. The computer was simply asked a question, and it answered – and humans followed that answer to their deaths.
At the foundation of automation bias is a deeper bias, firmly rooted not in technology, but in the brain itself. Confronted with complex problems, particularly under time pressure – and who among us is not under time pressure, all the time? – people try to engage in the least amount of cognitive work they can get away with, preferring strategies that are both easy to follow and easy to justify. Given the option of relinquishing decision making, the brain takes the road of least cognitive effort, the shortest cut, which is presented near-instantaneously by automated assistants. Computation, at every scale, is a cognitive hack, offloading both the decision process and the responsibility onto the machine. As life accelerates, the machine steps in to handle more and more cognitive tasks, reinforcing its authority – regardless of the consequences. We refashion our understanding of the world to better accommodate the constant alerts and cognitive shortcuts provided by automated systems. Computation replaces conscious thought. We think more and more like the machine, or we do not think at all.
In the lineage of the mainframe, the personal computer, the smartphone and the global cloud network, we see how we have come to live inside computation. But computation is no mere architecture: it has become the very foundation of our thought. Computation has evolved into something so pervasive and so seductive that we have come to prefer to use it even when simpler mechanical, physical, or social processes will do instead. Why speak when you can text? Why use a key when you can use your phone? As computation and its products increasingly surround us, are assigned power and the ability to generate truth, and step in to take over more and more cognitive tasks, so reality itself takes on the appearance of a computer; and our modes of thought follow suit.
Computers' Projecting into the Future
Just as global telecommunications have collapsed time and space, computation conflates past and future. That which is gathered as data is modelled as the way things are, and then projected forward – with the implicit assumption that things will not radically change or diverge from previous experiences. In this way, computation does not merely govern our actions in the present, but constructs a future that best fits its parameters. That which is possible becomes that which is computable. That which is hard to quantify and difficult to model, that which has not been seen before or which does not map onto established patterns, that which is uncertain or ambiguous, is excluded from the field of possible futures. Computation projects a future that is like the past – which makes it, in turn, incapable of dealing with the reality of the present, which is never stable.
Computational thinking underlies many of the most divisive issues of our times; indeed, division, being a computational operation, is its primary characteristic. Computational thinking insists on the easy answer, which requires the least amount of cognitive effort to arrive at. Moreover, it insists that there is an answer – one, inviolable answer that can be arrived at – at all.
Climate Change
The debate
on climate change, where it is not a simple conspiracy of petrocapitalism, is characterised by this computational inability to deal with uncertainty. Uncertainty, mathematically and scientifically understood, is not the same as unknowing. Uncertainty, in scientific, climatological terms, is a measure of precisely what we do know. And as our computational systems expand, they show us ever more clearly how much we do not know.
Computational thinking has triumphed because it has first seduced us with its power, then befuddled us with its complexity, and finally settled into our cortexes as self-evident. Its effects and outcomes, its very way of thinking, are now so much a part of everyday life that it appears as vast and futile to oppose as the weather itself. But admitting the myriad ways computational thinking is the product of oversimplification, bad data, and deliberate obfuscation also allows us to recognise the ways in which it fails, and reveals its own limitations. As we shall see, the chaos of the weather itself ultimately lies beyond its reach.
In the margins of his revision copy of Numerical Prediction, Lewis Fry Richardson wrote,
Einstein has somewhere remarked that he was guided towards his discoveries by the notion that the important laws of physics were really simple. R.H. Fowler has been heard to remark that, of two formulae, the more elegant is the more likely to be true. Dirac sought an explanation alternative to that of spin in the electron because he felt that Nature could not have arranged it in so complicated a way. These mathematicians have been brilliantly successful in dealing with mass-points and point-charges. If they would condescend to attend to meteorology the subject might be greatly enriched. But I suspect they would have to abandon the idea that the truth is really simple.
It took him forty years to formulate, but in the 1960s, Richardson finally found a model for this uncertainty; a paradox that neatly summarises the existential problem of computational thinking. While working on the "Statistics of Deadly Quarrels", an early attempt at the scientific analysis of conflict, he set out to find a correlation between the probability of two nations going to war and the length of their shared border. But he discovered that many of these lengths appeared as wildy different estimates in various sources. The reason, as he came to understand, was that the length of the border depended upon the tools used to measure it: as these became more accurate, the length actually increased, as smaller and smaller variations in the line were taken into account. Coastlines were even worse, leading to the realisation that it is in fact impossible to give a completely accurate account of the length of a nation's borders. This "coastline paradox" came to be known as the Richardson effect, and formed the basis for Benoît Mandelbrot's work on fractals. It demonstrates, with radical clarity, the counterintuitive premise of the new dark age: the more obsessively we attempt to compute the world, the more unknowably complex it appears.
Free Your Android
Android is a mostly free operating system developed mainly by Google. Unfortunately, the drivers for most devices and most applications from the "market" are not free (as in free speech, not free beer). They frequently work against the interest of the users, spy on them, and sometimes cannot even be removed.
This campaign can help you to regain control of your Android device and your data. We collect information about running an Android system as free as possible and try to coordinate the efforts in this area.
You want a mobile device that is really yours when you bought it? You want a mobile device that does not spy on you and hands over your data to big corporations? Then read on!
Why Your Mobile Device Needs To Be Free
Mobile devices are small computers that we carry around all the time. They know our current location and contain private pictures. We use them to communicate with our friends, our family and maybe our secret love. They also provide access to the internet and have built-in camera and microphone. Being such powerful tools, they can bring great risks to privacy, but at the same time we can achieve great things with them: it depends on who controls them. Freedom and Control
Most mobile devices are not controlled by the users, but by the manufacturer and the operator. The software that runs on them is not Free Software. Even Android phones ship with non-free software and proprietary add-ons that often work against in the full interest of the user. Software updates will only be made available as long as the manufacturer still has a commercial interest in your device. The applications (apps) available from the official market are commonly non-free. Nobody is allowed to study how they work and what they really do on your phone. Sometimes they just don't work exactly as you want, but often times they even contain malicious features.
Running exclusively Free Software on your device puts you in full control. Even though you may not have the skills to directly exercise all of your freedom, you will benefit from a vibrant community that can do it together.
Privacy
Our mobile devices contain more personal information than most private diaries. But proprietary systems, even most Android phones, are designed to hand over this data to companies like Google or Apple. Most users do not have full control over the personal data on their device. Convenient solutions for synchronisation and data backup trick more and more people into storing all their data on centralised servers run by some profit driven corporation. These are usually based in the US and are required to hand your data over to the US government on mere request. Whoever has personal information about us is able to manipulate us. Therefore, non-free devices are a threat to democracy and to our society.
Privacy is one of the most important reasons to support Free Software. Proprietary add-ons like Carrier IQ spy on smart-phone users without their knowledge. Many apps from the market contain malicious features. They read your private data, such as your address book and "phone home", or they use Google Analytics to send data to Google. These are just a few examples out of many that have been discovered so far. The lack of freedom impedes independent inspection and secret spy features only become known by accident.
Most smart-phones require you to connect and identify yourself to a centralised server before you can use them properly. Users have to trust the server without knowing what information is stored and how it is processed or related to other data. A phone running only Free Software does not require you to provide data to an untrustworthy company or pressure you to do so – at least this is very unlikely, as the software's actions would be obvious and the community would be able to develop an alternative version. The convenience of "value-added" services that are often coupled with such connections can be provided using Free Software as well. This keeps you in control of your personal data. Your diary remains in your possession. You can have your cake and eat it too.
Other Related Initiatives
Even though this campaign is about Android, there are other initiatives that make it possible to use small mobile computers with (mostly) Free Software. Due to the driver situation of mobile hardware, none of these initiatives really can run such machines in freedom. Non-free drivers and firmware are needed for many peripherals to work.
On our wiki, we collect other initiatives. Please help us to maintain this list.