Design And Performance Lab

7 ......Thinking Images: A Conversation with Paul Kaiser and Marc Downie



A series of manifestos and books -- including "Why the future doesn't need us" (Bill Joy), "The Second Coming" (David Gelernter), and The Age of Spiritual Machines: When Computers Exceed Human Intelligence (Ray Kurzweil) -- appeared at the turn of the century, first in print and then in a technology forum on the Web (, creating an ominous backdrop to the second international CROSS FAIR exhibition-conference held at the newly opened Choreographisches Zentrum in Essen, Germany.(1) The theme of the event, "The Intelligent Stage," seemed to address theatre and dance practitioners. Yet it turned out that most of the invited guests in fact don't practice on theatre stages but belong to a growing international network of digital artists, software designers, and robotics and multimedia engineers who show their work at electronic arts or new media museums, in specially designed virtual-reality caves, on screen or the Internet, or in public spaces. Within a period of a few years, the concept of an "intelligent stage" designed as an infra-networked laboratory-theatre for interactive experiments already seemed outdated.

The Intelligent Stage at Arizona State University, which I visited in 1999, was a black box studio set up with an array of hardware and software, video cameras, artificial perception systems, and synthesizers for MIDI interfaces allowing physical action to be translated into sound, music or video projection in real-time. At MIT's Media Lab, similar concepts had also been drawn up and tested. But CROSS FAIR's keynote speaker, Roy Ascott, proposed to look ahead to "moist media" -- constituted by the convergence of dry digital media and wet biological systems. For him, the creative use of telecommunications and the restructuring of consciousness are primary, and "intelligent stage" is a space in which one addresses questions such as:

- what is it to be human in the post-biological culture, and what is it to have mind and body distributed in cyberspace?

- how might artists deal with the responsibility of redefining nature and life itself?

- what might be the consequences for the arts of the convergence of computers, biology and nanotechnology?

Ascott emphasized two main distinguishing features of an intelligent stage: (1) its location and effect in cyberspace and in distribution across the Net; (2) its role as a space for modelling new structures and behaviors, both human and artificial, that follow the impact of new technologies and the globalization of culture The converted Zeche Zollverein, a former coal mine, was a perfect setting for these provocations which, in effect, deal with the transition of an industrial art form, based on technique, hierarchical structures of labor/production, and classical attributes of choreography, composition, and execution traditionally associated with the stage, to the digital era.

And it was in this setting that I saw Paul Kaiser's and Shelley Eshkar's Ghostcatching installation for the first time. A conventional physical and aesthetic understanding of performance is challenged by works such as Ghostcatching which present a new form of installation as "writing," one might argue, a performance writing constructed through motion capture and digital processing. The resulting digital performance bespeaks a rich artistic imagination and intelligence, but hardly in the sense of Bill Joy's and Ray Kurzweil's scenario of a posthuman condition in which computer software exceeds human thinking. Yet an open debate on what is meant by intelligence was lacking at CROSS FAIR, which was peculiar since the conceptual boundaries of humanly embodied intelligence, artificial intelligence and artifical life were addressed, if sometimes obliquely, by some of the artists. For a moment, however, let us entertain the idea that our most powerful 21st-century technologies -- robotics, computer science, genetic engineering, and nanotechnology -- operate according to the law of accelerating returns and intensify the progression of self-replicating cybernetic systems. How will the image of human nature or the human body be modified by computerized systems? How will images behave? How do humans and machines constantly modify one another? How do machines learn from t heir environment, and what does it matter for the "consciousness" or "intelligence" of performance?

In the following dialog, conducted via the Internet in early 2007, I am joined by Paul Kaiser and Marc Downie, two of the most prominent artists/scientists currently working in the field of digital creation. Together with Shelley Eshkar, they have formed a collaborative team in 2001 operating under the name of "OpenEnded Group." Though the three largely work together, they sometimes work in pairs, create solo artworks, and pursue collaborative projects with others, including key collaborators from a range of arts and science fields (architects, composers, electrical engineers, programmers). OpenEnded Group thus also reflects a particularly inspiring example of the organizational assemblage and working method in contemporary creative research.



Paul, you participated in the 2006 Monaco Dance Forum where you and your collaborators recreated how long does the subject linger on the edge of the volume...(2005) with Trisha Brown's dance company. At the festival, you also showed 22, a collaboration with Bill T. Jones which had emerged from the same artistic research begun several years earlier. I take it, from having seen some of your previous work, that you think of yourself as an image maker. Do you see how long does the subject linger on the edge of the volume... and 22 as live performance works which present a new kind of image art?

KAISER: Yes, we had the strong sense of opening a new door, especially with how long... In that particular piece, we were able to push the door open pretty wide.

So how is this a new kind of image? Well, to begin with, the artwork doesn't consist of a skein of pre-made pictures that are triggered "interactively" in the course of the performance. No, it works very differently from that. It has its own autonomy, thanks to the artificial intelligence that Marc Downie has endowed it with. Its imagery comes as it pictures things to itself, trying to make sense of what it sees onstage in real time as the dance unfolds.

Of course, it doesn't proceed from a completely blank slate any more than a newborn baby does. Instead it draws upon a series of structures and intentions that we've given to it, which in combination sometimes bring to mind simple living "creatures," to use Marc's term.

For example, the piece opens up with a triangle creature, whose intention is to move from stage right to stage left. It does so by hitching rides on points in the motion-captured dancers' bodies, guessing which ones are moving in the right direction. Thus it extends a line out to a likely point, and is then tugged that way if it's guessed correctly. Of course, sometimes its hunch is wrong, and it has to relinquish its grip on that point and await the next opportunity. In such a case, that line is left as a trace, and thus the whole image as it progresses is simultaneously a history of its attempts.

This touches on new element of this picturing-making: memory. For not only does our artwork work with a sense of present and of future -- by perceiving in real-time what is occurring on stage and guessing what might happen next -- but it also works with its memory of the past. Often its images come from its attempting to work out correspondences between past and present configurations on stage.



[Fig.1 Still frames from the dress rehearsal of how long does the subject linger at the edge of the volumeÉ (2005) at Arizona State University. Courtesy of Paul Kaiser and Marc Downie]


Of course, this is all very well and good on technical grounds, as computer science if you will. But our feeling is that the work should stand or fall on artistic grounds. It's up to the audience whether the quality of our line is exquisite or not, or whether their experience of time as it unfolds in the performance is somehow new and unexpected.

The images generated during the performance, in real time, have a memory as part of their behavior, as well as the power of anticipation? These are not properties we generally associate with the (photographic or filmic) notion of the image. How are we to understand the idea of such "creatures" in a theatre work? Are you suggesting that there is a new dimension of "artificial life" acting upon the performance?

KAISER: Yes, I think this work does push our notion of what imagery can be. Even "live" imagery of the kind normally seen on stage or in installations reacts only to the present moment (unless of course it's actually being driven by preset external cues). But Marc's creation of live "creatures" does give our imagery the ability both to remember and to form expectations. And it's crucial to bear in mind that each such creature's memories and expectations are its own -- that is, unlike each other's and, more to the point, unlike our own. But the question to ask is: why are artificial memory and anticipation so important? To answer this, let's see why our own abilities to remember to form expectations are so crucial to our experience of the world.

As we observe any event, our perception of it derives not simply from the present moment itself, but rather how it stands in contrast to what we think led up to it where we suspect it might be headed. In a complex event, like that of a great performance, such memories and expectations are never fixed, but are set in constant play as we continually readjust our perceptions and understanding. That's precisely what we want our imagery to do. Each creature works within its own limits to make sense of the world in just this way. And as I said before it pictures its understanding, and it's this process of picturing that becomes the live image.

Do you see the creatures as abstract figurations or as having a narrative dimension?

KAISER: The creatures are capable of being either abstract or figurative, and also of having an explicit narrative dimension. To my mind, we humans tend to interpret almost any sequence of events as cause-and-effect and therefore as narrative, so even a fairly abstract creature automatically suggests a story in much the way an inkblot does. The triangle creature I described above, for example, has a clear goal (to move across the stage) -- which we immediately interpret as its desire. And we soon make out its strategy (to link its points to those of the dancers moving in the same direction), which we interpret as some sort of narrative inter-connection.

One crucial aspect of this is that we see the virtual creature's effort - its mistakes and its misunderstandings, as well as its eventual success. Which again adds to its being curiously lifelike.

Are the creaturesÕ behaviors generated in real time from the actions on stage, or are they picturing something different, something of their own?

KAISER: The software always looks directly at what is happening on stage and then creates its imagery in response to its understanding of what it perceives. Of course, the picture it forms is idiosyncratic -- in part because a given creature attends to just one kind of pattern on stage, rather than to the aggregate of patterns that we in the audience are following. But because it attends so closely to that pattern, it often reveals relationships that we don't notice.

How do you and Marc understand the relationship of computing and artifical intelligence to the theatre, to choreography and movement, since here the "images" are neither film nor visualization or illustration. The creatures do have a geometric form, in my mind, animated forms, shapes, architectures. They are non-anthropomorphic, but as you suggest the audience might "read" a narrative into their behavior, humanize it. On the other hand, what intelligence do the creatures have? Do they observe motion, do they respond to the motorsensory (action observation, as the neurophysiologists might call it) or can they respond to choreographic content? Emotions, affect? What is the relationships between movement animation and consciousness/cognition?

DOWNIE: "Artificial Intelligence" informs our work in a number of ways, ranging from the mundane to the exquisite, from the general to the specific. To see this you can begin with an extremely broad definition of what "AI" is. Most simply put this is the field dedicated to getting computers to do "the right thing," by themselves, in situations where programmatic descriptions, of the kind that computers generally require, of what that "right thing" is are inaccessible to us. Stated this way, AI clearly ought to be central to almost any digital artists' practice -- as soon as, that is, they go beyond the commercially conventional ways of making images and music. What it is that we are wanting out of the computer -- the images, the relationships -- cannot be programmed in a forward or a forethought out way. Firstly, many of the "algorithms" are in fact just too complex -- too many values to tune, too many layered decisions, renders conventional engineering impossible. Secondly, and more importantly, our "specifications" for the computer programs -- what it is we want the images and relationships to do -- will only be discovered as we begin to glimpse them. Thus we seek solutions that allow us to work with computers rather than merely on them.

AI, as described this way, is still incredibly broad (and from both a programming and an artistic standpoint rather uselessly broad) but this should at least suggest that AI is not an arbitrary posture, that we introduce it not for its own sake, but rather because the field offers potential solutions to problems that many artists face.

That said, we are focused on a particular class of AI problems and solutions -- described variously as "agent based", "creature-like" or even "reactive" -- that emphasizes the computer as an embodied agent: that is one that is deeply coupled to its environment such that its actions on its environment -- mediated by the physical constraints of some "body" -- must be carefully produced and its perceptions of its environment -- mediated by its all too limited sensory apparatus -- must be carefully maintained. Such AI is also often where the most interesting integrative machine learning work occurs, and there learning takes the place of extensive human-level domain specific knowledge (think: birds rather than chess players). And it also makes perfect sense given our preference for images that move between abstraction and figuration to be near a field with a concern for virtual animated bodies. Finally, this "style" of AI is usually practiced in a frankly more interactive way by its community. These preferences dominate our choice of techniques --- not any need for our creatures to share a neurophysiological basis with the dancers (which they resolutely do not), nor do we have any goal for "human level" choreographic intelligence.

Thus, while this explanation should clarify much of the architecture of our imagery for how long... -- it is this thinking that our "thinking images" are about -- but it's important to realize that our creatures are not robots: their bodies and their physics are purely imaginary; their sensory apparatus are not sensing the "real world" but sensing the motion capture data and other agents in view; their motivation and affect systems are for their regulation not for our narrative; and the problems that they are forced to solve are not given by the natural world: they are also designed by ourselves as we see opportunities in Trisha's choreography.



[Fig.2 how long does the subject linger at the edge of the volume Monaco Dance Forum, 2006. Photos courtesy of MDF]




I am not sure how close theatre and dance are to games, let's say, and the "virtual animated bodies" in game environments if that is what you meant?

DOWNIE: I don't feel qualified to talk about how close theatre or dance are to games in general, either in the classical sense or in the sense of computer games. However there is a game-like aspect to our forms in two overlapping senses. Firstly, algorithms are rules --- nothing more. Our agents describe a set of rules that in application will produce the images seen as the artwork. While it's us that comes up with the rules initially, once set we tend to quite rigorously follow them to see where they might lead --- we play out the game without arguing over the rules. Coming up with a new set of rules is generally preferable to trying to break our previous set. Secondly, our agents often end up participating in games: the opening Triangle agent for how long... has to get from stage right to stage left by hitching a ride on the dancers at key moments. Clearly, neither of these two aspects are in any way unique to our work; there's precedent in every corner of art history, including, of course, the choreographers we have worked with.

There is also a relationship, but not a terribly interesting one, between our work at a technical level, and computer games. Like everybody else in the field we are parasitic to some extent on the consumer-level graphics hardware created for the game industry. Although recently it has become much easier to avoid the common aesthetics of computer games, this has come at the price of it being much harder to get anything on the screen at all. Finally, much of my code for making "abstract" bodies was tested much more rigorously than you'd expect while at the Media Lab -- and it still sees use in drawing and controlling much more computer-game-like characters and frighteningly "concrete" robots. I can testify that playing between the abstract and the figurative is technically much easier once you have the figurative worked out on a robot.

But you speak of more fully interactive practices in the AI community, while I understood Paul to imply that interactivity on stage is not necessarily your goal

DOWNIE: It is correct to say that "interactivity on stage" was not our goal for how long... Ultimately the choreography is fixed, and the dancers are accurate from evening to evening (to my eye, supernaturally so). There was no sense while we were making the work that the dancers would ultimately interact with the imagery during the performance. Rather we were seeking to interact with Trisha BrownÕs dance, and we took advantage of the three years that we worked on the project to build a set of tools and ideas to allow us to do just that. This difference is crucial and is present throughout our work -- our recent work Enlightenment is a cluster of 10 computers that interact with themselves; and at present Paul and I are making a series of prints: interactively -- but obviously they are not interactive prints. It's not that we are uninterested in the problem of interacting with "the public" we just haven't had any good ideas yet.

You said that you designed the creatures' "thinking" behavior in response to the choreography, is that correct to say? In terms of such thinking or learning, is sensing the kinematic data a mathematical/geometrical process of regulation (based on the computer's analysis of the tracked points in space), and an "improvisatory" transformation of a particular body model?

DOWNIE: The "thinking" in how long... certainly extends down to the lowest (and most tedious) technical levels of the piece. One of the lessons we quickly learned when we started to work with real-time motion capture was that we would have to reconstruct most of the motion capture pipeline ourselves, on our terms. The priorities of the hardware manufacturers are deeply misaligned with ours, emphasizing the quality of data over the quantity of it. Frustrating and time-consuming as this engineering was, it worked out happily: much of the piece evolved from "staging" these increasingly sophisticated algorithms of machine perception, and in particular staging their repeated failures and attempts to compensate. The piece would have been impossible to execute had we simply accepted the manufactures' "black boxes" as closed to us; and any AI would have found itself in a much impoverished environment. However, an unwillingness or an inability to understand, reconstruct or even reject the pre-made pieces of technology that populate the digital-art world remains the norm.

Paul, has the AI approach to physical performance privileged a certain kind of abstract movement or body model and a non-representational creation of geometry? How do you see such computational sensibilities, the digital graphics and animated geometries, relate to other forms of music or dance theatre, spoken word theatre or installations (for audience participation)? Do you foresee a growing use of such graphical environments in performance?

KAISER: Well, if you take a close look at the particular choreographers we're working with here -- Merce Cunningham and Trisha Brown -- you'll see that that their reliance on abstract diagrams predates ours by several decades! Both Merce and Trisha choreograph by means of such diagrams, creating amazing graphical depictions of such key dance elements as stage geometry, movement trajectories, temporal repetitions, body kinesphere, and so on. Look, for example, at even so early a work as Merce's Suite by Chance of 1953, and you'll see him charting the spatial and temporal axes of the work, with no recourse to stage pictures.




[Fig. Merce Cunningham. drawing (from The Dancer and the Dance. Merce Cunningham in conversation with Jacqueline Lesschaeve, London: Marion Boyars Publishers, 1992]


Trisha makes similar use of drawing, for example in creating her amazing 1975 solo Locus, in which she draws a simple cube and a string of letters, which she then uses to set her choreographic requirements. The cube she imagines as being roughly the size of her kinesphere (the space surrounding her body that she can potentially reach). By labelling each vertex of the cube (actually, cuboid) with a letter, she determines the order in which she must hit each point of space with any given part of her body. The resulting solo is of course a well-known tour-de-force, and Trisha's method led directly, I'd say, to Bill Forsythe's more elaborate invisible geometries.




[Fig: Trisha Brown drawing; Bill Forsythe drawing.]

Let me also point out that Trisha elevated this kind of drawing to a formal art practice, exhibiting the works in galleries and museums, so it's not as if she at least wanted to keep her diagrammatic thinking hidden from view -- behind the scenes, so to speak. To the contrary. Which is why she took so quickly and enthusiastically to the diagrammatic creatures of our how long... projections.

Parenthetically, I'd like to say something that I hope Marc will take up further, which is that in many ways Merce and Trisha were exploring ideas and processes in a manner very much parallel to what was going on simultaneously in computer science. I'm not suggesting that von Neumann and Turing were ever aware of Cunningham and Brown, or vice versa, but I will say this: what a shame! And it's not too late to cross-pollinate their ideas, even if they never had a chance to do so themselves.

However, I don't want to paint us into too tight a corner here, for your question implies that we committed ourselves exclusively to a style of diagrammatic abstraction and so on, which is far from the case. We've worked with equal interest and intensity with representation. Indeed my first encounter with the stage was through Robert Wilson's drawings, the vast majority of which are simple but incredibly effective series of stage scene thumbnails.


[Fig. Fig. Robert Wilson sketching scenes of Einstein on the Beach. Still frames from Visionary of Theater (1996). Courtesy Paul Kaiser / The Byrd Hoffman Watermill Foundation.]


More to the point, we were committed to an AI approach to representation in another collaboration done in parallel with how long... This was with Bill T. Jones, in a work entitled 22. This entailed an identical technical set-up to the Trisha piece, with 24 infrared cameras capturing Bill's movement in real-time, and our software responding simultaneously not through abstract creatures but through lifelike figures (a young boy and a man) that provided a thematic and visual counterpoint to Bill's solo stage presence.


[Fig. Production diagram for 22, from Choreographing the Extended Agent: performance graphics for dance theatre (doctoral dissertation; 2005). Courtesy Marc Downie]


I didn't use this example earlier, for the final piece fell well short of our aspirations for it, for complicated reasons really having nothing to do with us. From our standpoint, however, the virtual figures of 22 could work just as well as the abstract creatures of how long... in fulfilling this dream of "thinking pictures."

DOWNIE: I could mention that one of the "complicated reasons that had nothing to do with us" was that in 22 there was a need for the images to interact as much with the narrative that was being told in performance by Bill T. Jones as with his movement. If the getting a computer to "understand choreography" live is research, getting a computer to "understand narrative" live is simply foolhardy. So foolhardy in fact that we didn't attempt it in our contribution to this piece which, as it was, seemed to begin to sink under the weight of its rather traditional stage-managed cuing. Such short fallings, however, often indicate future opportunities and each of our more recent pieces have increasingly engaged language, and computational responses to it. This has led us, again, far away from the terrain of how long... or 22.

It's interesting that you mention the choreographers' and Robert Wilson's drawings. I tend to think that Wilson's scenography and directing have computational, almost mathematical-abstract quality, but of course he also works with the figure. Perhaps the terms figurative and non-represententational are no longer helpful in this context of complex systems, digital or AI. But given your interest in animation, graphic visual art, and the kind of complexities afforded by an agent-based aesthetics as well as your fascination with disequilibrium, for example, could you speak a little about your work not directly intended for the theatre but for galleries and public spaces? What are the aesthetic differences between the public projections, the gallery works and the collaborative performance works?

KAISER: I'm afraid I don't agree with you on the first point, Johannes. I'd say, actually, that Robert Wilson's scenography and directing are more mechanical than computational; it's certainly no secret that much of his staging is a revival of the sophisticated proscenium mechanics of the 19th century, repurposed for avant-garde spectacle. In any case, from what I've seen, he mostly works by drawing scenes from a stage perspective rather than diagramming relationships and movements more abstractly the way Merce, Trisha, and Bill Forsythe tend to do.

But to address the main part of your question, it's mainly an issue of context, which of course has aesthetic implications. The great thing about the stage is the way it frames not only the space of the artwork, but also the time of it. In our hectic day and age, it's quite a wonderful throwback to an earlier age where people more often gave things their undivided attention.

Certainly that's rarely the case for museum exhibits, much less for public art, so the pieces you make for those settings need to allow for random-access, as it were -- people come and go much more randomly in such settings, and the artwork has to withstand, address, or overcome that somehow.

For public artworks, you also have to imagine some of the viewers encountering the piece repeatedly over the days of its installation, which again is a very different kind of viewing experience. We try to make works that can disclose themselves in different ways depending on the depth and the frequency of their exposure.

DOWNIE: The parallels between Merce, Trisha, Bill Forsythe's and others' practices and digital art, at least as it could be practiced, are quite striking. If you pardon the language: these choreographers are producing algorithms that are executed on their dancers, observing the unforeseeable results of the computation, making adjustments, and iterating; culling the potential of their algorithmic ideas; carefully pruning the possibilities they create; constantly negotiating the constraints of human performance, perception and their interfaces with the abstract-computational. Of course, there are often wonderfully talented dancers that perform these computations, and we just have our keyboards and our stupid computers -- but those problems are separate, they are "just" technical. After enough work, enough engineering, you strive to find yourself in a position very similar to that a choreographer and their dancers. I've complained for a long time now that digital art, as a field that refuses to grow up and understand its history, or that there is a history, has generally ignored the possibility that there may be a useful prehistory in the performing arts.

Paul, do the projected animations you installed in public places reflect an aspect of image work that you have always been interested in? What has happened in your thinking and practice since your earlier collaborations with Bill T. Jones and Merce Cunningham?

KAISER: The simplest answer to your question, is that I started out as a filmmaker and I never gave up my desire for the kind of cinematic experience that large projections allow. It was obvious that I'd never sneak any of my work into the local multiplex cinema, so I found different spaces -- a huge scrim at the State Theater or a broad expanse of sidewalk at Rockefeller Center.

But there are important differences between the public works and the dance works. The first has to do with framing and audience. You realize when you step into the world of dance how distant it is from the everyday world, even despite various valiant efforts to break down that divide. But I came to realize, as I've said many times before, that the only people who go see modern dance are the people who go see modern dance. Very few outsiders stumble into it -- "stumbling in" becoming exactly what public art allows.


[Fig. Pedestrian at Rockefeller Center, 2002. Photograph © Peter Cunningham]


Most of the audience encountering Pedestrian on the street in Harlem or in a bus station in Seoul or on a market square in Bruges do so completely by accident, unexpectedly having to make sense of an artwork whose context does not automatically supply all the answers (as a gallery or museum or stage context invariably does).

But the other impetus behind some of the public artworks was a desire to get away from dance movement itself, which has never fascinated me so much as the movements I see through my window on the street below or from a bench in a playground when I take my daughters there. To me, children playing tag on a field or pedestrians manoeuvring through a busy intersection in midtown are supreme feats of choreography -- but choreography without a choreographer. Absolutely fascinating. Which is not to say that my perceptions of such everyday movements and patterns were not sharpened tremendously by our work with Merce and Bill T. and Trisha.

With your interest in more mundane, pedestrian and non-expert movement, are you going through your Judson Church phase?

KAISER: Well, it's an interesting parallel, and I sometimes wish we could have worked with Trisha when she herself was still in that phase. But the answer still is no, for two reasons.

First, when a self-conscious adult performer does ordinary movement on stage, well, the movement is no longer ordinary. Certainly it doesn't feel so to any audience member, which for us is the crucial part. So Judson Church movements are really not the same either physically or psychologically or spiritually as playground movements, and the latter interest us much more.

But second, when I was about 20 years old (back around 1979), I was already fascinated by pedestrian movement, and made an experimental film in Super-8 called Colourblind etc that took as its subject people crossing a single intersection on West End Avenue. In those days, of course, I knew nothing about postmodern dance. In any case I couldn't advance that interest any further for two decades, not only because the right tools didn't exist, but also because I didn't have the right collaborators (after all, Shelley was 9 years old at the time, and Marc was 2!)

Tell us briefly how OpenEnded Group evolved, are you three now a production & research company?

KAISER. The OpenEnded Group is more of a collective than a company -- there is no leader, for example, and we make all our decisions unanimously. Research is part of art-making, and art-making part of research.

The secret to great collaboration, I learned, is to work with people much smarter than I am. I met Shelley Eshkar in the mid-90s through his former teacher, Robert Breer, one of my filmmaking heroes, and it was immediately apparent that he was incredibly gifted. It was he who solved the problem of making a convincing "hand-drawn space" in 3D, and he who designed the hand-drawn bodies that made Ghostcatching and BIPED what they were. We've been working together now for almost 12 years and are essentially telepathic.

Marc Downie came in an arranged marriage, really, for a piece that the MIT Media Lab commissioned from us -- our abstract portrait of Merce Cunningham, Loops, which was based on our motion-capture of Merce's hand dances. When we first sat down with Marc over coffee, he nodded quietly at every preposterous goal we stated, and a couple of weeks later, had us shaping the piece in real-time in front of his screen at MIT. You have no idea how revolutionary that was! What had taken us overnight to render before, Marc was now drawing to the screen instantaneously. Independently, he had been pursuing some of the same hand-drawn, nonphotorealistic goals that we had, most notably in a series called Musical Creatures, so it was a natural fit. I also realized he was smartest person I'd ever met, so I helped steer him out of academia. The three of us having been working together ever since.

In a recent symposium at the 2007 Cyberarts Festival in Boston, I raised the question why there are so few substantive technological performance works or innovative dance works which use computer augmentation. The works that have had an impact we can count on one hand, why do you think this is so? If there are so few works, perhaps we cannot even speak of impact, in a cultural sense?

KAISER: The most obvious reason is expense. Practically speaking, dance budgets are small, and even set-up times tend to be very short. Talking to lighting designers, for example, I learned that they have far less time to design on stage for dance than they do for plays, for example -- and far less than for operas.

But a much more important reason has to do with method and approach, for the tools commonly used in interactive work (MaxMSP and Jitter, for example) largely lead to terrible results -- they seem to promote the shallowest and the easiest kinds of thinking, and it's no surprise that no masterpieces seem to have emerged therefrom! I'm no tools determinist, and it's certainly conceivable to create a great artwork with poor tools, but they certainly don't help. But Marc, who devoted part of his thesis to this question, can answer it better than I.

DOWNIE: Perhaps I am a "tools determinist" -- I just take a unusually broad definition of "tool," one that is expanded to include all of the ideas one uses to arrive at the artwork itself. And I believe that while pleasant surprises can happen, you generally end up in a place that bears the marks of how you got there. The separation of digital tool and formal idea is artificial and convenient only if you are in the unfortunate position of having to purchase your tools from strangers. It's worth studying the tools used for digital art as forms: not just from the perspective of what they can do (and how fast they can do it) but what ideas they embody and privilege or exclude and prevent. There are then two scandals: firstly, that there aren't a tremendous number of tools available to day; and secondly they are all the same! Max/MSP/Jitter and its brethren embody nothing more than a set of ideas firmly worked out in the mid 80s which themselves were constructed to be parallel to the ideas embodied by the analog synthesizers of the 50s, which in turn were simply inherited from the physics of electrical engineering. Outside the digital art world, nobody takes these ideas about interaction and programming seriously: not the art world, not computer science, not philosophy, not even physics. The third scandal is, of course, that nobody seems to care and digital art becomes increasingly un-interdisciplinary as it becomes un-serious to these fields. My response is two-fold: to make sure that I can take responsibility of any digital tool I use -- usually by making it or remaking it; and soon, to make our tools freely available. The goal of our upcoming "open source" release is not to skew the hegemony in our direction, but simply to disrupt it and make artists begin to question what many have simply "received" up until this point.

Do you think interactivity or interaction design in live performance on stage is limiting or artistically uninteresting? What forms of interactive art do you feel has had a significant artistic impact or corresponds to the increasing interactive design in the culture at large (communications, games, etc)?

DOWNIE: It's impossible to rule anything out, but I suspect that on stage "interaction design" is a bad place to start from and a bad place to end up. And as for the influence of interactive art I find that it has had almost no effect on even computers let alone culture are large. Digital artists continue to have no effect on: Apple's next operating system, Adobe's next image editing tool, Microsoft's next word processor, Google's next start up purchase, W3C's next web standard &c. (This makes the idea that artists are content to simply wait for better tools all the more appalling).

What is computer augmentation of performance, or what did the designers of the "intelligent stage" (at Arizona State University, for example) have in mind with the architectures of interactive, networked spaces? If universities are investing in such laboratories, why are arts organizations not following suit? BAM stopped supporting media arts, Dance Theatre Workshop discontinued its digital fellowship program. Or is the support for digital art mostly going to be found in visual arts organizations?

KAISER: If I were more politic, I would sidestep this question, but what the hell -- as you know, I tend to speak my mind. And in this case I do speak from experience, since I was one of the three artists chosen for the short-lived BAM digital installation program, and since we have worked at Arizona State and in several other university projects as well. (Indeed, you'll remember, Johannes, that you and I first met when you were teaching at Ohio State University, where the dance department was attempting to collaborate with the computer graphics department in a motion-capture studio.)

As is inevitably the case, the problem comes down to money. Universities and corporations have it; artists and art organizations do not. Let's look at universities first. Big universities are not only committed to advanced research, but many also have very large performance arts centers that are a crucial mainstay for contemporary American dance (I'm thinking of the Zellerbach stage in Berkeley, the Krannert Center in Illinois, the Hancher stage in Iowa, and quite a few others). These places are not only venues for dance, they also commission new works: a fantastic thing.

On paper, then, it makes sense that universities should go even further by supporting advances in digital art and performance, not only because they can tap into many sources of funding that are (unfairly!) out of the reach of artists, but also (and more appropriately) because they have the physical facilities (equipment, labs, stages) to host the development and the staging of complex projects. Universities even have, potentially, many scientific and artistic collaborators who could contribute to such work.

Why then has our experience been largely one of failure and exasperation? And, even worse, why do we harbor suspicions of fraudulence? The sad story is that universities tend to make lofty pronouncements (especially to funders), but fall terribly short in execution. Even worse, when they fail, they don't acknowledge their failure, but simply explain it away as the necessary consequence of experimentation and research, part of what they inevitably call the "process" (a word I've come to detest -- I think we need to get back to good old-fashioned "results" instead). The embarrassing fact is that such failures often come in areas of basic competence, not advanced research. University engineers, for example, simply won't know how to operate their expensive optical motion-capture equipment. When we were in (long, repeated) residence with Trisha Brown at Arizona State, they kept failing to capture any data whatsoever -- heartbreakingly, at times. We persuaded Trisha to don the motion-capture costume and to perform her Locus solo for posterity, only to learn much later on, and with no accompanying apology, that the data had never been acquired. With little skill in matters of simple competence, was it any surprise that the loftier research goals were rarely met? (Of course, you'd never know this from the press releases and status reports streaming forth; and meanwhile those hundreds of thousands of dollars had already dried up in the academic sands.)

Well, enough of this. If I sound bitter, that's because I am, but I'll spare you the excruciating details. Despite this terrible experience, I would like to believe that universities could support real research in the arts, but only if they really confront their previous failures in the field. And I would add that it was an art school, Cooper Union, that gave Shelley and me crucial support in working with Bill T. Jones on Ghostcatching very early on.

As for corporate support, that is more unabashedly about money or at least value: the value to companies being promotional (indeed, funds usually come directly from marketing departments, not from any charitable corporate offshoot). Sometimes such support is very simple Š in the mid to late 90s, for example, Compaq supported our work because we were using what were then their advanced NT workstations, and they could boast about that at the National Broadcasters convention and at SIGGRAPH.

The most pervasive, and the most pernicious, support for the performing arts has long come from Phillip Morris, the tobacco company Š blood money, really, with politically correct artists and art groups putting themselves through all kinds of odd contortions to justify that backing. My opposition to Phillip Morris got me in trouble at the Whitney, where a group show I was in found much-needed but tainted support from them. I protested publicly, and also paid the actual costs of installation myself.

I think itÕs remarkable that you have worked on some of the most crucial artistic performance collaborations over the past decade, especially if your initial background was in film -- now, you mentioned BAM's digital installation program -- why did BAM not encourage more of the kind of work you were producing?

KAISER: Again, I think it comes down to simple economics. The Brooklyn Academy initiative was with Lucent, which owns Bell Labs. Both BAM and Lucent wanted to associate themselves with the glory days of the 1960s, of the EAT collaborations (Rauschenberg, Breer, Trisha Brown, et al, working with engineers like Billy Kluver). But there was a key difference: originally, all that was done informally rather than institutionally, with the artists and engineers pairing up on their own. Bottom-up versus top-down.

In the later case, Lucent provided money and some equipment, and artists and engineers met in a kind of rapid dating game to see who could mate successfully. The actual results were pretty good Š for example, Ben Rubin and Mark Hansen worked together seamlessly to create Listening Post, a wonderful installation that did indeed draw on Bell Labs research meaningfully.

For my part, perhaps in reaction to the top-down institutional nature of the whole process Š and perhaps perversely Š I created an intensely personal work entitled Trace. Several research projects I saw at Bell Labs were about new technologies of surveillance, which took me back to my childhood as a diplomat's son in Eastern Europe, where we were constantly shadowed and bugged. Trace was about the sort of "self-surveillance" that that sort of self-consciousness led me to.

But the upshot was that Lucent's share prices tanked shortly thereafter, and when that support went away, so too did BAM's. In any case, it was always clear that BAM's heart was in performance, understandably enough. And they still had Phillip Morris to fund it.

DOWNIE: To tie these last two questions together I think it might be useful to ponder the overlap and differences between artists, engineers and scientists (speaking as an artist who has pretended to be a scientist and an engineer at times). The EAT collaborations are in some ways the canonical bottom-up meetings of artists with engineers. Despite having often irreconcilably different sets of tastes, engineers and artists share a dedication to actually making things, a pride in the things that they make and an understanding that they judge, and are judged on, the things that they end up making. This alignment of perspectives can be tremendously exciting when it happens, and it's possible for everybody to sure that they are getting back something from the collaboration. But collaborations inside a university are often formulated completely as science-art collaborations (certainly our most recent university experience was ultimately funded by the NSF). Academic science has as uneasy a relationship with engineering as it has with art. And since nobody can independently replicate, say, a motion recognition system built for Trisha Brown's movement, the junior science faculty and graduate students involved in such collaborations are left scrambling for the limited value that they can extract from the relationship in ways that are often independent of the resulting artwork. Artists then respond by defending the integrity of their process and the artifacts that they are going to be left with. In such pathological collaborations the only uncertainty is just how long it will be before everybody starts bolting for the exits.

Despite an abundance of interdisciplinary ventures by American universities, only a handful will accept artifacts (and even fewer, performances) as part of the tenure or advanced degrees processes for their "science" halves. No wonder then that dot-coms were a better bet than the Academy. Until this changes, interdisciplinary research is not being done by institutions except by accident or resistance.

Yes, I have similar experiences, but in the UK there is some effort made now to fuse the funding for sciences and humanities in so-called sci-art projects. More interesting is the bottom-up model and the many small research networks that I see growing across the world. Can we go back one step: what made you work on a project involving theatre director Bob Wilson? What kind of exhibition did you create of Wilson's works or drawings? Were you interested in his early long-durational or outdoor work, or his particular interest in working with autistic adolescents?

KAISER: A long story, but I'll shorten it. My start in multimedia was in working with severely learning disabled children, whom I taught for ten years in the 1980s. Well, "whom I taught" -- and who taught me. What I learned from them, primarily, was how to collaborate, for in creative classes I soon learned to give up instructing in favor of open-ended exploring. And when Apple's HyperCard program was released on the Macintosh in the mid-90s, we used its multimedia tools to carry on those collaborations on this new frontier. ThatÕs in fact when I came up with guiding principles like "drawing as performance" that carried through to many later artworks - and which proved to be one of the keys to my work with Wilson.

Fast forward to 1993 or so, when Wilson's foundation was thinking about a digital archive. A friend told them of my work, thinking of the parallel: for Bob Wilson had also started out by working with handicapped and "special needs" kids, two of whom directly inspired the first two (and to my mind most impressive) phases of his work. And then, too, there was the central practice of drawing, with Bob creating all of his work not verbally but visually, in the never-ceasing activity of his thumbnail sketching.

So just as with the children I'd taught, my goal was to capture the act of Bob's drawing, as it happened -- to put it back into the time of its performance. Which eventually I did. Working with Babette Mangolte, I filmed him drawing and narrating his recollections of such pivotal works as Deafman Glance, KA MOUNTAIN, and Einstein on the Beach, and used the resulting video as a guide to dissect the resulting drawings in Photoshop. I could then tie this drawn performance to the video, films, photographs, maquettes, etc., that I found in the archive, which allowed me to reconstruct works otherwise lost to time.

That was just one of several methods I used in making the piece, Visionary of Theater. In it, I was at pains to put Bob's best foot forward, to make the best possible case for his early work, and in particular to reveal the extraordinarily collaborative nature of it Š true collaboration with the likes of Christopher Knowles, Cindy Lubar, Andy deGroat, and so many others. Though the piece was exhibited widely, among other things in a show of Bob's drawings that I co-curated with Brooke Hodge of Harvard's Graduate School of Design, it was only this year that it finally reached its intended form of CD-ROM -- and has now -- belatedly, and on teetering, outdated technical underpinnings -- entered the collection of the Performing Arts Library at Lincoln Center.

So, yes, I put Wilson's work in the best possible light but all the while I was reacting against it and its kind. Too long a philosophical objection to go into here, but I can't omit the fact that IÕm deeply troubled by the severing of causality and of history that you find in Wilson and his forbears (Cage, for instance) and their many followers (who crowd the field of digital art). This objection relates to our critique of arbitrariness, the same critique Marc makes of the arbitrary "mapping" techniques embedded in the digital tools like Max MSP.

I think that dream logic, random juxtaposition, asynchronicity, and so on, have become easy evasions of hard thinking. The proof, perhaps, is the degree to which they've been gleefully adopted by mass media -- advertising, videogames, music videos, and so on. A welter of disconnects.

After Pedestrian, and later, Playground, Paul, you have now been commissioned to create public works in England (York Minster) -- are you becoming interested in very large scale work or are these projection works logical continuations of your image making for the theatre?

KAISER: Well, all the world's a stage, right? So, yes, we've been very interested in expanding the possibilities of public art, and in particular of making permanent works. In the United States at least, "public" is a slightly misleading term, for increasingly our public spaces are in fact "public/private." Bookstores replace libraries; malls replace public squares; and even civic spaces are often partly private, having been underwritten by companies or supported through advertising. Take Times Square, for example: how public is it, really?

So one question when trying to make vast and permanent public art projects is how to work in that odd seam between civic and corporate structures. WeÕve had two tantalizingly close prospects, though neither worked out in the end. One, which I canÕt discuss in any detail, was to put a single networked artwork in possibly hundreds of stores throughout the country. That would have been a fascinating encounter!

But the other one, entitled Horizon, was actually commissioned and announced, so I can describe it fully. Atlanta Airport was adding a new international terminal, for which the city's percent-for-the-arts set-aside ensured a pretty fair budget -- for our project, 3 million dollars. We were going to build a long LED display (280 by 30 feet) with intelligent imagery generated interactively and in real-time.

[Fig. "Airport as dollhouse" storyboard for Horizon (2005). Courtesy of the OpenEnded Group]


Shelley created an elaborate storyboard for the piece, one of which illustrates our basic idea best - "airport as dollhouse." Here the idea was to invert the sense of scale, by creating large virtual children who play with elements of the airport as if they were parts of a dollhouse. And so there were children's hands propelling airplanes down the runway, and their fingers becoming air bridges for disembarking passengers, passing them from one hand to the next. Here Shelley was inspired by the way in which children sometimes play by passing ladybugs back and forth -- carefully, delicately, but with an immense sense of power.

The idea was to reframe the airport experience, which in our post 9/11 world is a particularly unpleasant one. So for example as a passenger you're always aware of being under constant surveillance, with surveillance cameras constantly monitored to detect terrorist activities. Horizon would also have such cameras, but for completely benign means: to look at the weather, to study patterns of people's movement, and above all to play. So for example Shelley had another storyboard of children playing in Atlanta's Olympic fountains, which might be called up by the AI when it started to rain outside -- we'd celebrate the wet weather rather than bemoan it. Or the camera might allow the virtual children to spot people browsing among the displays of a newsstand, and then play hide-and-seek with them there. Or as dusk came on, it would bring down the night in our virtual world as well.

The key was to make a piece that never repeated, continually engaging not only the frequent business travellers who pass through that Atlanta hub regularly, but also of course the airport workers who would have to live with the piece. It's in this context that the idea of "thinking images" plays so crucial and so practical a role.

I hate to harp on money and the lack of it, but again this project was indefinitely suspended not because of our budget but rather because of enormous cost over-runs on the architectural and engineering design of the over-all terminal building. A comedy of errors, which has since devolved into a tangle of legal suits and counter-suits between the city and its various contactors. We've stayed well clear of that mess.

What a shame, your description is very vivid and I'd have loved to visit a playful airport, for a change. How do you see the current artistic culture in its gradual embrace of new media technologies or its embeddedness, what social or aesthetic aspects do you value, and how do you think about "digital culture"? Why is theatre in the US a relatively minor art form today which has barely developed new ideas or techniques, whereas so much current discourse on performance process, becoming, hybridity and interactive digital media tends to be progressive and also increasingly politically aware of the challenges we face in the biotechnological era?

KAISER. Hmm, I'm loathe to prognosticate on too broad a scale here. We work within our culture, and so while we definitely respond to the sorts of shifts and concerns you outline in your question, we certainly haven't risen to any elevated, Olympian position from which to assess them as seers.

But perhaps I can say a little bit about digital culture and what we make of it as artists. It's a commonplace that the Internet, especially as mediated by Google, has changed the context and the reach of information. Search is now a nearly instantaneous procedure, with tremendously wide scope, and it has become the quintessential mode of thought for our era -- the first recourse for nearly everyone, that first step we all take almost automatically. And it's not just Internet search that's so important -- DNA sequencing, for example, works essentially the same way.

So we've been interested in applying techniques of search in our own works. Which is not to say that we make so-called "web art" which uses -- and often tries to subvert -- the various protocols of the Internet for various postmodernist reasons. Instead, we bring search algorithms into our work itself. We're fascinated by this newfound ability to consider millions of possibilities and contingencies that are well beyond what the naked mind can address by itself.




[Fig. Pre-analysis visualization for Enlightenment (2006). Courtesy of the OpenEnded Group.]


And so, for example, for Enlightenment Marc wrote code that could analyze the 25 second coda to Mozart's last symphony and detect on it own not only the five themes of its invertible counterpoint, but also trace back related thematic material through the whole of the 4th movement. Similarly for a forthcoming public artwork, Breath, we perform other sorts of searches: for example, revealing essential semantic threads running through the entire Book of Psalms.

Now if search represents an advance in our culture, I must also point to a sore point of decline. We're often struck by how contemporary culture seems to be heading towards increasingly low resolution -- the voice quality on your cellphone is far worse than on your grandparents' old landline; the image quality on your iPod video or YouTube feed is mud compared to the 35 or 70mm films they enjoyed in cinemas; the language of email and chat is impoverished by contrast to that of their old-fashioned letter-writing. Attention itself is increasingly low-res, as I said earlier: the world seems designed for distraction.

So to this, as artists, we oppose ourselves, rather than just going with the cultural flow, so to speak. Visually, we insist on high resolution projections and displays: there's no reason why artists should have to accept the fuzzy interlacing of DVDs, for example. For a while now, we've said that we want digital resolution to feel like that of paper -- and as a logical conclusion, two of our current projects will be printed rather than projected. We're making lines so fine that we can't even see them on our screens, but only when they emerge from our printer.

But this is not some sort of retreat into tried-and-true old-fashioned art-making (the persistence of painting as an artform astonishes me): we're printing texts and imagery for projects that you could never have done by hand. If art advances, it does so unpredictably and not quite in lock-step with the culture at large.





(1) "CROSS FAIR 2000: The Intelligent Stage" took place 9th-12th November 2000 and was curated by Sške Dinkla and Stefan Hilterhaus for the Choreographic Center North-Rhine Westphalia in Essen. Roy Ascott's keynote address, "The Moist Scenario," is quoted with permission of the author. The exhibition and colloquium brought together numerous media artists, designers and choreographers to debate the implications of the "Intelligent Stage." Paul Kaiser (whose company was then called Riverbed) exhibited the Ghostcatching installation. Steina Vasulka, Michael Saup and Louis-Philippe Demers addressed technologies as independent, intelligent systems, and Jeffrey Shaw presented an overview of the innovative interactive installations created at the Zentrum fuer Kunst und Medientechnologie (ZKM) in Karlsruhe. Shaw spoke of immersive and interactive interface environments and referred to Nottingham University's Mixed Reality Lab where experiments with "MASSIVE," a multi-user distributed virtual reality system, helped the British ensemble Blast Theory to develop Desert Rain. The project was completed during their residency at the ZKM. Choreographers like William Forsythe went there to create CD-ROM projects (Improvisation Technologies) that required extensive digital video studio and computer processing facilities. With interface design by Volker Kuchelmeister and Christian Ziegler, Improvisation Technologies features a hypertextual content of over 60 video chapters showing lecture demonstrations in which Forsythe demonstrates the essential principles of his improvisation techniques. A solo by Forsythe, and other dance sequences performed by Frankfurt Ballet members, can be called up as further illustrations. As Ziegler pointed out at Cross Fair, the "intelligent stage" need not be understood as a physical location; it could as well refer to the specific nature of an interface design or platform on a CD-ROM or the Internet. For Ziegler, the CD-ROM is a "knowledge-reference system." For further information on Paul Kaiser, Marc Downie, and Shelley Eshkar's collective work, see: