Design And Performance Lab
|
SYNAESTHESIA AND MULTIMEDIA
Reflections on "Synaesthesia and Multimedia", a workshop at Erfurt University, December 2004
Johannes Birringer (Part 1 of this report is in German, Part II in english)
Second Day: Processing, Editing, and Interactive
Dramaturgy
Composing with digital media (defining various sequential and simultaneous layers of movement action, image projection, sound events within a choreographic structure) is here understood to be a choreographic process particularly concerned with kinesthetic impact but also with the conversion or correlation of modalities - the conversion of visual information into sound, of sound into motion, of motion into sound, etc. The correlation process among modalities is familiar in analog editing and production techniques (video, dance, theatre, music), but compared to the linear editing in analog video, digital media is numeric information and allows extreme manipulability of a database of images, graphics and sound. The data, no longer indexical (of analog physicality and an intrinsic relationship to the world and the body) but numeric code, are hyperplastic material which can be subjected to extreme modification, morphing, depression and distortion. This kind of hyper-editing is now often used in real-time performance synthesis and interactive software applications which foreground the ephemerality of digital video and audio (the metaphorical resemblance to the fleeting presence of dance and the vulnerability of bodies is obvious and may explain why dance and interactive technologies seem such a good match). My interest is less in hyper-editing, extreme processing and working with algorithms, randomization, and other generative programming options, and even less in interactive triggering of video/audio outputs (via the Eyecon motion tracking system or other motion sensing systems), but with transduction and "re-capture" -- a kind of constant conversion between physical and digital, a movement that is also movement of physical camera and sensory anticipation of the digital attenuation of the physical. Motion sensing of a physical movement calculates and computes not a signal but discrete numerical information. But how does the moving body sense the tracking and the conversions? How does it sense the excess over itself? Its loss? If bodies [in dance] are the main pivots and catalysts of the process (processing) -- how is the body, which in live art has been repeatedly inscribed, mapped, wrapped, cleansed, cut, beaten, shifted, changed, contorted and grafted, able to intensify its capture, its mobility and transferability (at the far end of potential?). How do performers travel with the digital? How does performance resonate with processual excess over the actual?
The "sequencing" of the temporal dramaturgy or structure of the performance-installation was found in discussions among the participants of the Media-Team, and this sequencing structure was also communicated to the Choreography Team which, as we knew, was working on a "loop" structure of the choreography, comprising small units or modules which would be performed in linear sequences and then repeated. 8 performers would be enacting the roles of passengers, sitting on a tram, leaving it and moving about the space, then re-entering the figurative tram. The choreographic movement thus attained a circular structure, which was in keeping with the leitmotif we had come to focus upon - the Gagarin orbital. The Choreography Team's decision to work on a loop form was in fact a very good decision, it allowed a dynamic and a flow not dependent on precise cues and interdependence of sound, image, and movement. The production method favored an open process, not the closed model of music drama (Wagner / Gesamtkunstwerk) which is hierarchically organized, nor the Cunningham/Cage/Rauschenberg paradigm of parallel but independent performance parameters (dance/sound/visual setting), but a convergence method which allowed stitching together (in real time) physical movement/linear sound/linear video with the kinesthetic of the digital loop form which carries its own textures and rhythms. Our converging method closely resembles interactive networked performance with live action and databases. (2) The simultaneities and convergences of the different kinesthetic media and physical actions were meant to emerge in a dynamic environment which was, to an extent, self-organizing, flexible, and sympathetic. There was no hierarchical order or confining structure, merely the time-line of a physical loop (choreography) joined with a linear filmic "narrative." Projected length: 20 minutes or more. The environment as a whole was not a programmed environment. It was not interactively designed, in the strict sense of computer-generated interaction designs where media outputs are controlled by physical and gestural action in real time. There was only one small detail (the "clarinet" at the Anger, a video shot of a vagrant musician sitting in a street corner and playing his instrument) which was moduled interactively into the overall flow. The sound of the clarinet was designed to be "played" by one of the dancers in real time (through her body movement), however the scene was actually never rehearsed Saturday night, and I suggested that it should not be included if the dancer didn't have an awareness of what it was she was doing. [The problem with computer-assisted interactive scenes is that they need to be effectively motivated and coherent in order to be sensed and "read" by an audience; if they don't work, they merely confuse and embarrass an audience that has been told to expect them. If they are trivial, they bore. If the mapping is very complex and non-transparent, the audience will not perceive meaningful relations between movement and sound. My observation of reactions to interactive scenes suggest that audiences in most cases want to perceive a causal relation between gesture and sound/image feedback.] The "story-board" / dramaturgy which I developed with a series of drawings about the spatial architecture and interaction environment between the elements was shown to team members Saturday afternoon and accepted as the provisional architectonic. It looked like this:
"R"
- Spatial diagram
The first time I consciously explored unknown urban territories and space was in 1988 after I had moved to Houston and became obsessed with its freeways and constant motor-vehicular motion, circulation, and abstraction, a city in constant motion without a past and a history but with tremendous power and arrogance of projection (the glass skyscrapers built from oil money, reaching into the vast and endless sky that resembled the empty desert of west Texas). The performance installation I made was called Invisible Cities. Ten years later, after I had returned to live in Houston for the second time, I became absorbed with the study of the MIR space station and its slow demise and abandonment. I went to NASA and worked with collaborators on a libretto about a fictional space journey, featuring a female cosmonaut abandoned on a spacecraft which drifts off into outer space; the opera Mirak premiered in December 1999, almost exactly five years ago. The physiological and sensorial interest of this work focused on the cosmonaut's body and her gradual loss of her muscular and motor-sensory functions as she lies strapped into her console. As her biofeedback system is affected by space sickness, melancholia, and madness, she begins to fantasize transformational processes in her body which are also induced by mnemonic reactions (memories that we visualized in film and through computer-generated visuals) to the disembodied voice of a Russian radio transmission operator at ground control. At the end, the cosmonaut imagines her transubstantiation into an insect. Most recently, during 2004, I have been working more intensely on investigations of sensory processing (vision, hearing, touch) and the relation of affect to "interactive" performance and inter-action with digital media. Ensayo sobre a cegueïra is an interactive opera project currently realized in progressive stages, each scene first produced as a separate installation (digital and sonic projection), before the work's production on a stage with singers and orchestra. The working idea behind Ensayo is the shifting or unbalancing of conventional coordinates; experience of loss of the optical sense (vision). The theme of (imaginary) blindness or agnosia is examined in regard to sensory processing and inter-activity relying on touch, hearing, voice, and altered proprioception. A major experienced sensation, during the exploratory wandering in Erfurt, was an experience of motion (the tram and its physical traces in the city): moving along the "Ring" (in Houston the ring road would be called "loop," in London the outer ring is called "orbital"). I felt I was moving in slow motion and in constant anticipation of the gliding trams that crossed my path. I became obsessed with the almost inaudible motion of the trams that cut their path through the body of the city. I tried to face (move into) the trams approaching me, and when they floated past me, I tried to catch their movement. I filmed in extreme proximity of their movement, as I felt I could sense and recognize their pattern, their speed. The central experience that I internalized, therefore, was not an image of stillness (the photographic, the punctum) but movement, a pattern recognition that I gradually began to associate also with certain sounds made by the trams and the electrical currents (the "network" of wires) I associated with the propulsion of the trams.
Does performance perception and cognition change, and how? Can we speak of a new interactive or digital perception?
1. Invisible Movement
Such sensory processing undermines the aesthetics of spectacle (optical visuality), and I believe it also has nothing really to do with technical "systems" such as Eyecon which set up a camera-tracking environment in which the performer/actor "activates" or "controls" data output (sound and video samples/files) and seems to inter-act with a response environment, if here environment is understood as including or constituting the sound and image-projection events that take place within a theatrical space. The technical "system" is a feedback system in which information travels and transduces bodily activity into computation (midi signals to software program) which controls the instant media outputs (sound, video). But this system is not an environment that affects proprioception, vision, hearing, smell and haptic sense unless we were to understand the sonic and the projected images, in their behaviors, as transformative, acting upon (producing) the sensing body and intensifying unaccustomed connections, orientations, deformations or interferences with the functioning of the body that now, and continually, must adapt to the unpredictable (?) conditions, the dissipative states. Interactivity in the account of my experience of the invisible dance in back to return can suggest a shared real-time environment in which "images" (perceptions) are extravisual, generative, synaesthetic (in a broad sense) -- contingent on sensorimotoric, haptic and auditory apprehensions and, possibly, confusions. Dorner's playing with the "blind" viewer and his acts of attention of course is an exception; in most cases of contemporary multimedia or digital dance the interactive relationship involves the dancer's or participant's intervention into the image-world of projected media (video, 3D virtual reality, animation). If back to return were a model of immersive performance, it indicates how dance can activate bodily modalities apart from sight.(3)
The process (proprioception, tactility, affectivity) through which human perception constructs "images" (and here we may need to ask what kind of images or mental constructs or cognitive movements we mean, and whether such mental constructs are conscious or preconscious) does not depend on the visual mode, but it would be interesting to examine contemporary interactive art installations in terms of their use of more auditory or haptic interface designs, and in terms of their abrogation or reliance on video and 3D visualization.(4) Other examples of interactive dance, for example telematic performances with partners in remote sites, point to a similar expansion and fragmentation of the visual image and our optical experience.(5) Dancing with virtual worlds or with telematic, distributed and non-Euclidian "space" opens up a perspectival flexibility which transforms the photographic and cinematic projection of the real. Furthermore, our response to the surface, the skin of the images, changes, and thus we could almost argue that we "wear" the filmic-projected textures differently, that we touch and sense the video images with our whole body.(6) This "wearing the digital" challenges choreographers to think of the relations between dance and projected image in different ways, especially regarding the familiar large-scale back-projections of video on stage, and the two-dimensional flat screens generally used in interactive dance (e.g. Palindrome) and in installations, including the wall-projections we used in the Hall in Erfurt. Issues of scale and proximity are crucial, in this context. If we want to posit a haptic visuality and "smooth space" for the participants' sensory perception of the digital, we need to examine how the body moves in this space, self-referencing and transducing the projected videospace, not only the video images themselves (naturalistic or digitally manipulated, low or high resolution, etc) but also the "skin" or membrane (screen, wall) which is the surface of projection.
Note on Transduction: Refering to "video haptics and erotics," Marks argues that a "visual medium that appeals to the sense of touch must be beheld by a whole body. I am not subjected to the presence of an other (such as a film image/film screen); rather, the body of the other confers being on me." (p.12) She understands video perception as bodily contact with the image, an exchange between two bodies in which beholder and image constitute each other. We need to note here that such mutual constitution presupposes a viewer (and would therefore relate to the experience of our Erfurt audience), but it does not precisely explain the transducing relationship of a performer (in multidirectional motion) to the audio-visual media conditions actualized and transformed in an interactive environment. Transduction, in the sense in which technical mediations link image and body in a space where they may no longer be separated ("real time"), implies that humans and things exchange properties, but in the case of a performance (unlike, say, a video game or an experiment in molecular biology), there are specific interlacings of the technical-corporeal, for example in regard to movement and the temporal ("real time"), and how time, through movement, touch, and the gestural, is informed by the synthesis or manipulation of images and sounds, or by delay, disruption or other contingent events that happen in the collective environment. What is the time between transmission and reception, between gesture and the visual perception of it in a performance? Real-time interactivity (cf. Eyecon) implies that there is no gap between gesture, touch, sound and image, but is there no gap between real time performer and viewer? In telematics the delay can be significant (4 seconds to 10 seconds and more). How is proprioception (as internalized quality of recognition, immediate viscerality, muscular memory, self-orienting abilities, habituated rhythm) transduced? How does vision perceive movement, visual form, and acoustic shapes? Does vision, as Massumi suggests, interrupt movement with (pre)formed images so that it must interrupt and delay itself in order to see movement as such? And how are the sensations and delayed resonances experienced collectively? How does the collective presence of an audience affect the performer movement and proprioception? Some of these questions are imprecise. They address the dynamics of movement (human performer) in perception, and hint at the activities of all sense modalities which are interlinked, while also posing the questions to the digital performance system. I have no scientific data on modulations and cross-modal perception, and thus cannot adequately address the temporal relations (which comes first? Is there sequence or simultaneity?) between the sensory production of a sensorimotor or haptic or acoustic space within the body, between stimulus and response. The processing speed of digital technologies has reached a point at which motion sensing systems can now transform data in real time and manipulate image-space in multiple and microcosmic or molecular ways. Interestingly, our metaphors for the dispersion or multiplication of the kinetic images are now often drawn from biology, chemistry or physics , and partly this is also an effect of the impact of cybernetics and neuoscience on contemporary digital art. Programmers and digital architects working with dancers have started to use "movement information" for the construction of synthetic image worlds and modelization which enable the dancer to act upon a "living" data environment that is as fluidly evolving, elastic, and changing as physical movement, but the projected images are no longer representational. Here our "Gagarin System" - in the concluding performance in Erfurt - turned out to be rather less compelling. We stayed to a large extent in the realist mode (movement images of the video; mundane and non-complex movement choreography based on a simple sequence of a few gestures and repetitions without particular variations and distortions, as we have learnt to try to perceive them in the work of William Forsythe or Akram Khan, Meg Stuart or Wayne McGregor). The departure from realist images, however, was suggested in the projected composited imagery created by Zelko Wiener and the hyperprocessed sonic elements created by Ursula Hentschläger.
The main problem I see in the conceptual development of "the Gagarin system" is its unfinished and unrealized dramaturgy; the performance also needed a spatial architecture that could have brought the various sensory media into relationship with each other - into a narrative coherence or a motivated relationship, which is of course an aesthetic supposition that I bring to such work.
Such linking could either be motivated (focused and inter-dependent), causal or non-causal, or structured in such a way that, for example, the inclusion of interaction-design becomes a meaningful dimension in the conception of the work. This relates to the "clarinet scene" as an isolated example of computer-assisted interactivity (using Eyecon), but it could also relate to the larger multi-medial shaping of the work as a whole. There also was no time to conceptually and physically (in rehearsal) explore with the whole ensemble the narrative strands or the patterns and leitmotifs of the assemblage, assuring that the dancers knew the filmic/photographic and digital (processed, real-time interactive, sonic) environment and could incorporate it or adapt to it. The dancer's relationship to their "outside" (the outside of their memorized movement vocabulary and repetitions and/or improvisations) was completely accidental, as they did not have the time to feel the sounds and images that the Media Team had constructed. Similarly, the Media Team did not have any opportunity to rehearse sufficiently with the dancers and the choreographer in order to make adjustments or changes in the visual, auditory and kinetic structuring of the materials for the "showing" of the work. What our ensemble presented in the public performance in Erfurt was not a "work" but a hastily constructed assemblage, an experimental arrangement that was not in fact based on current notions of real-time interaction or self-organizing systems, and thus, presumably, the opposite of what Palindrome generally does and what they were invited to do (Frieder Weiss, the company's engineer/progranmmer, was not present). We used a re-combinatory approach, for lack of time, and the overwhelming majority of the compositional elements were "recorded" (play-back), so to speak, and not emerging. This qualifies the aesthetic we used, as we did not find it possible to use "composition" in the sense of real-time improvisation or experience. I cannot speak here for the audience's experience of what they saw, heard, felt and perceived. I do not have the data for that. I can only speak from the point of view of the composition, the production.
I remember hearing one of Russian cosmonauts in the MIR say that the planet earth looked like a blue orange from his window in the space station. He smiled when he said it, gently floating (in zero gravity) towards the camera. I assume that Gagarin had no idea what he was seeing or where he was when he was speaking about his experience orbiting earth. He must have been hallucinating. I also believe that I do not know, now or then, what it was I experienced in the moving around Erfurt. I cannot remember the space, nor the trajectory of my walk across Erfurt, even though certain markers, marks of memory, are in my recollection, but these marks have nothing to do with the map of Erfurt (Gagarin Ring, Anger), nor do they really explain my experience and how the experience of the gliding trams and their incomprehensible sounds affected me and stayed with me.
The virtual world, which Hellen Sky and John McCormick (Company in Space) in their recent collaboration with Ruth Gibson and Bruno Martelli (Igloo company) called "Sentient Space," affords a different mental experience of the image-space.(7) Image-space here refers to the inevitable digital output (video, 3D VR projection) we are invited to look at when performers work interactively with a system: we do not look so much anymore at the physical performer, we look at the "visualizations." Sometimes we look at both, back and forth. Company in Space/Igloo's experiment with telepresence and motion capture took a complex conceptual turn towards real-time dynamic visualization systems in which the image-environment is generated by the crossing of several networked morphogenetic processes. Sky and McCormick explored telematic performance based on the idea of visual world-modeling. Movement data, captured in real time from Sky and Gibson (in exoskeletal motion sensing suits), were transmitted and processed to create "soft bodies," i.e. the data was not mapped onto animated figures but stretched, folded and manipulated to create topological models - semi-abstract and "architectural" images that act like the congealment of the space in-between the two dancers performing in remote locations. Folding terraces, tiles, mutating shapes, colors, objects floating and tilting and changing. Now why do they call these spaces sentient? And how do blue terraces taste? The "virtual" is here implicitly corporeal. But there is also a stretch of the imagination, as we may look between dancers and visualization, or try to look at what we think is the visualization of movement, in other terms. These terms, to some extent, are incomprehensible to our common sense perception. But we have walked on terraces, and our soles remember tiles.
Notes on Common Sense and external storage (F. Scott Taylor)
In international techno-art/science, F. Scott
Taylor suggests, there is currently a growing interest in the natural
phenomenon of "synaesthesia" particularly with relation to
the perception of multi-media performances and installations. The term,
also spelt "synesthesia," originates from the Greek etymological
roots syn, "together," and aesthesis, "perception,"
and refers to sensory cross-modal sensations like "coloured hearing"
and other sensory mixes. There are also more unusual features of synaesthesia
that can be found in the objective experiences of people who can "taste"
shapes, and in "audiomotor" synaesthesia where different body
postures can be precisely positioned according to the sounds of different
words. [cf. Richard E. Cytowic, "Synaesthesia: Phenomenology and
Neuropsychology - a Review of Current Knowledge", reprinted from
Psyche (1996), in Baron-Cohen, S. and Harrison, J. E., eds., Synaesthesia:
Classic and Contemporary Readings (Cambridge, Mass.: Blackwell Publishers
Inc., 1997), p.21.] Synaesthesia is a phenomenon common to all of us,
but few are conscious of the fact of its action. Those who are conscious
are termed "synesthetes." This renewed interest comes as the direct result of common inquiry into virtual disembodiment by telecommunications, which is ironic and paradoxical given the complex embodiment of synaesthesia in the mind-brain-body. And, furthermore, such interest is remarkable during a period devoted to the hyper-abstraction of human action and enterprise in terms of global real-time, when it is the effect and affect of the combined senses which gives us our perception of a time-space continuum in the first place. But the most extraordinary aspect of this inquiry is that while it is our synaesthetic condition which allows us to express ourselves verbally and mathematically -- that is, symbolically and abstractly -- we are rediscovering it at the exact juncture which threatens to transform techno-culture into the post-sensory and the post-symbolic, specifically through the utility of computational devices that surpass or may surpass human sense and sensibility (and what we need to remember as "common sense"). [Taylor here refers to Marshall McLuhan's comments on "common sense" but also takes into account several scientific studies from psychologists and neuroscientists.] Common comprehension of our selves and our world is composed of an infinite number of such basically analogical arrangements, that is, the concomitant sensations between and among the different senses; in fact the world wouldn't be consciously comprehensible without them. If new technological media are "new senses" then they alter and will alter further our common sense through a sort of aggressive occupation, colonization or usurpation. The degree to which they are extensions of ourselves, the degree to which they insinuate themselves into our sensibility, and the degree to which they render our senses vestigial must be met and measured in terms of the new experiences they provide. While our original psycho-physiological synaesthetic centre was devoted to the synthesis of external and internal environments for the sake of individual and cultural deportment, the new synaesthetic centre generated by our technologies is discovered in the overlap between our organic sensibility and our cyberspatial sensibility. What makes this centre untenable and impossible to fully comprehend and articulate, is the simple fact that while our old senses were co-ordinated and orchestrated internally through their organic, neurological analogues, the new media do so externally through digital code. In short, we now store and process our personal and cultural symbols externally. Consequently, when using digital code we become dependent upon our machines for the translation of sense into sensibility. With analogical code, however, there is a promise for a more natural and complete amalgamation, or, at the very least, an expanded upper limit for human comprehension. For example, while we may implant an individual with a computer chip, if it is analogue it may be an educational tool and teach behaviours which include intellectual synthesis so successfully that the chip could be removed when that was accomplished. As current Virtual Reality applications make abundantly clear, a digital approach makes it impossible for us to meld seamlessly with our technological creations and thereby create a third synaesthetic centre, neither entirely embodied nor disembodied, where, for better or worse, humanity would become its creations, and its creations would become human. At the moment such a tertiary centre simply inspires simulation sickness rather than sensory and extra-sensory unity.
In Biped (1999), choreographer Merce Cunningham combined large-scale projected moving images (developed by Shelley Eshkar/Paul Kaiser from the motion-captured movement data of Cunningham's dancers) and dance. The contrasting kinetic values between the dancer and the projected moving image in an immersive multimedia scenography provided the viewer with a type of synaesthetic experience which resulted from the interaction of kinesthesis proper to each medium. At Ars Electronica 2004 a work by Golan Levin, Zach Lieberman, Jaap Blonk and Joan La Barbara, Messa di Voce ("placing the voice"), was presented. Messa di Voce was an audiovisual performance during which speaking, shouting and singing were synaesthetically augmented in real-time by interactive visualization software. (8)
"R"
The reason I raise this issue is a conceptual and aesthetic consideration that I believe is of crucial significance if we want to examine a notion such as interactive perception or digital perception. In conclusion, we need to turn to the kinesthetic or synaesthetic experience of the physical/corporeal subjects in the environment, first the dancers, and secondly the audience members (I have no data for the latter, nor do I have any feedback data from the performers themselves, so my thinking here is speculative).
What I remember most, even after some time now, are the sounds of the tram, and when one of these vehicles passed me, I identified with the passengers sitting inside, I saw or felt myself gliding past myself, I felt I was pulled along the orbital of these trams, circulating the city, in an infinite rotation. When I think of "R", I imagine the loss of orientation one feels when one drifts, in midst of life or a sea of continuing losses, lost in a kind of trance, when one is no longer conscious of the path one walks or has stopped questioning the repetition, the unconscious orbit. The lost-ness, lost in bitter or resigned trance, spaced-out, the clarinet in the distance played by the vagrant sitting on the street corner raising his left hand begging for charity, that is what I remember feeling when watching the people in Erfurt on that cold December day, the Christmas shopping market, with the toothless Santa Claus and the slowly rotating ferris wheel a chimera, bright lights in the sky. Gagarin a black man, overlooked in a street corner. The sadness in the faces of silent, isolated white people sitting in the trams a darker reality that was hard to describe in words. It was ineluctable. Germany's Wintermärchen. Of course I may be projecting my own lost-ness, being at a loss, a kind of transference. At the end of line, when Katrin and I got off, an African student started to talk to her in French; I remember his joyous smile. He thought he knew her, but it was a misunderstanding. But the tram "mapped" something that I wanted to bring to our performance, re-collect and project for our audience, and that is why I asked Katrin Breitung to be my Gagarin. But it was not the map, nor the video, the recorded surfaces and the many lights, shimmering reflections in Katrin's face, yet something in the movement of the images that could be touched and felt, acted with. Smooth movement, close to the bones. The distant sound of the clarinet had a pivotal meaning for me as well, a catalyzing element, its melody mixed and then overturned by the metallic sound of the tram, the whirring sound of electricity, the silence, and then the computer voice of the announcer on the tram: Next Stop, Gagarin Ring. Digital world outside.
In his chapter "Strange Horizon: Buildings, Biograms, and ther Body Topologic," Massimi speaks of "biograms" when he describes the rhythms of our orientation and orienting abilities: our resources -- lived and relived experience, not diagrams (of synaesthetic forms and interfusions) but biograms, "event-perceptions, combining senses, tenses, and dimensions on a single surface." (p.187). Abstract movement on an abstract surface: intensive movement, occurring in place - or more accurately out-placed, in the event (p.187). What is an out-placed event? I would assume that Gagarin's orbiting the earth in his capsule is such an out-placed event, incomprehensible to human proprioception. What if the performance we try to make has to deal with this out-placedness, and how to compose it? Composition: not construction, Massumi would argue, and he makes an interesting distinction between the two processes that reflects back on our work. If artistic activity is "catalysis," he argues, then it is not a "construction." Construction takes already extracted variables and recombines them… Catalytic fusion is something else: it involves resituating variation. Klee calls this "composition" in contradistinction to "construction," composition as an experienced emergence… always moving to the edge, receding infinitely into the shadows. If construction recombines found elements or fragments, and composition involves the unfolding of an absolutely singular worlding relational whole, then it is important to specify that the whole actually never exists. It always moves. It is the openness of closed form, form continually running into and out of other dimensions of existence (p.174).
From a dance perspective, the"form"
closest to this mode of "composition" is improvisation. Marlon
Barrios Solano suggests that in his current work, which he describes
as "designing improvisational systems with dancers" under
the name Unstablelandscape, he has incorporated real-time interactive
digital technology into the process. They are not "choreographies"
with multimedia, nor videodances in the traditional sense, but "generative
hybrid systems that are set in motion in front of audiences/visitors
that may or may not digitally interact with them." His artistic
process, Barrios Solano says, intersects dance-theater improvisation,
embodied awareness training, algorithmic notions of movement and composition,
interactive video and sound design, basic electronics for interface
design and small robots. Improvisational performers, Barrios Solano suggests, train themselves to deploy their own psychophysical states in "real time," composing their experience while attending simultaneously to both the dynamics of the process and the compositional outcome. Improvisation and its generative compositional strategies imply a concept of mind as an organizational theory about patterns, causal relations, agency and control of the body actions. Practitioners developed strategies of phenomenological exploration in action and training methods to increase their agency or dynamically explore the embodied experience in terms of their own model of mind and its processes. The nature of improvisation itself led their models to be more dynamic, realizing a more plastic process of mind and embodiment through the exploration of improvisational movement. Due to the nature of dance improvisation--composing "on the spot" with physical actions and changes in time and space-- dance improvisers developed approaches and training methods which fuse kinesthetic knowledge and perceptual and behavioral psychology. These naturalizations of the creative-artistic process, in tandem with eastern philosophies like Zen Buddhism and Aikido and artistic influences such as those of John Cage, Merce Cunningham and Mabel Todd among others, facilitated a model for human embodiment, human actions in the world, and agency that allowed dance artists to deconstruct and dynamically re-conceive embodied practices involved in dance making practices as non-fixed, flexible and mutable.(9)
Massumi proposes that all emergent and mutable form brings its fringe of virtuality with it. Thus no particular medium has a monopoly on the virtual. Every medium produces its own virtuality, and "digital art" is in no way synonymous with "virtual reality." What matters is the "how" of expression, not the "what" of the medium, and especially not the simple abstractness of the elements that the medium allows to be combined (p.175). How, then, are we to orient ourselves, in the choreographic or improvisational relationship between corporeal presence/action and image/sound movement projection? Our presentation of "R" most likely failed to examine the relationship and the orientational system that our staging employed or suggested. What could have been a rich source of inquiry into the synaesthetic interfusions, improvisational becoming, the cross-modal perceptions of movement improvisation and the digital, did not take place on the level of production as our team had no time or occasion to interrogate the "construction" of the media landscapes and their affect on the dancers, and vice versa. The dancers in our performance were very similar to the people I observed in the tram, un-mindful of that which they hallucinated and could not see, the city outside, the reality of loss. What then, was the concrete experience? Massumi's philosophical argument suggests that as far as our vision is concerned, we often see something that is not there, or we have no visual memory of what we experience absent-mindedly, or our sense orientation indeed works differently. We often navigate using a form of non-visual memory, and Massumi refers to this as "bodily memory" (biogram) of movement, one of contorsion and rhythm rather than visible form. This argument gives us opportunities to rethink our process, pointing in the direction of interactive improvisation that uses in performance, and for the audience, what A.R. Luria (in his case study of S.) has described as proprioceptive twists (along a meandering walk), re-facing what we are normally not aware of --- facing movement-sensation, the unrecognizable, the peripheral, the between, the unpredictable, membranic connections to all the virtual seas of sensation. (p.204).[A.R.Luria, The Mind of the Mnemonist: A Little Book About a Vast Memory, Cambridge: Harvard University Press, 1968, p.35.] Gagarin's movement is adrift. Proprioception, Massumi suggests, is the sense most directly attuned to the movement of the body. It involves specialized sensors in the muscles and joints. Proprioception is a self-referential sense, in that what it most directly registers are displacements of the parts of the body relative to each other. Vision is an exoreferential sense, registering distances from the eye.(10)
What is most challenging here is the catalysis --- composing movement without remembering it, relating to the unrecognizable fleeting moments of light you see, while you are in a dark tunnel, and you re-surface without knowing where you are, there is sound and texture and light (projection), image movement of uncertain space. How does the dancer, and the audience, use cognitive maps to create visible forms for the experience? Or how does the dancer, and the audience, recall rhythms of movement (proprioceptive system of reference), half-conscious tendencies, psychoacoustic regions, disharmonies and strange patterns, half-formed symmetries, colors and vectors, polluted air, former countries and bygone political systems, black skin, empty spaces? How does spatial orientation deal with "immersive" moving image environments and distributed sound, how does movement through sound and images refer to its own variations, and can it do that with the digital projected world? Here, with the digital outside [coded storage], begins the composition, which is the movement of an extended orbital existence, hallucinatory yet habitual, always remembering yet unfolding in the dispersed moments of sensory presence, continuity of transformation. What we have learnt so far from the digital media is their infinite plasticity, and malleability. We are not in control. Collectively, we don't have a common identity even if we share the same technological infrastructure. So why would we belong together, with indeterminate gestures, like rubbing shoulders against the back seat, that are hardly intimate transactions? Why watch this? If we compose with media, perhaps it is best to grasp the digital as coming from our (analog) inside where it is vaguely felt, even when we don't know why we must face this possible loss.
Notes: (1) Laura U. Marks refers back to Deleuze and Guattari's adoption of the term "haptic" from the art historian Aloïs Riegl's study of haptic visuality in Egyptian and Roman art, commenting that the "haptic emerges in Deleuze and Guattari's description of 'smooth space,' a space that must be moved through by constant reference to the immediate environment, as when navigating an expanse of snow or sand. Close-range spaces are navigated not through reference to the abstractions of maps or compasses, but by haptic perception, which attends to their particularity." She then quotes from Deleuze/Guattari's A Thousand Plateaux: Capitalism and Schizophrenia: "It seems to us that the Smooth is both the object of a close vision par excellence and the element of a haptic space (which may be as much visual or auditory as tactile). The Striated, on the contrary, relates to a more distant vision, and a more optical space - although the eye in turn is not the only organ to have this capacity." Cf. Marks, Touch: Sensuous Theory and Multisensory Media (Minneapolis: University of Minnesota Press, 2002), p. xii. Regarding the study of perceptual experience, one would need to distinguish between the experience of the expert performers who train with interactive design for their work, and the general audience witnessing a multimedia work or interactive performance. I use the term "participant" here (also implying that the student performers in Erfurt were non-experts), preferring it over the term "user," although the latter has recently been applied to audiences visiting interactive installations and acting upon them. Cf. Johannes Birringer, "User Testing for Participatory Artworks," International Journal of Performance Arts and Digital Media (forthcoming). (2) The model that was approximated in our performance resembles the practice of "collaborative culture" which we often experience today in media arts workshops and networked performances that use real-time interactive systems (software platforms such as Keyworx, Max/Msp/Jitter, and Pure Data), live action and databases. Sher Doruff explains this dynamic of collaboration and emergence in the context of distributed networking and peer-to-peer perception, suggesting that the "advent of real-time, media-rich performative technologies, enabling synchronous multi-user creation" challenge our habits of cognition and control through such open, dynamic and shared media applications which are "hybrid models that combine learning through the iterative pattern-recognition of dense-multi-layered signifiers. Practical questions in determining degrees of collaborative agency in distributed, multi-user platforms center on perceptual learning curves.[…] In platforms where all media can be instantiated and commands sent by multiple users to change media properties, a learning curve looms, perhaps not dissimilar from proprioceptive skills acquired by dancers."(pp.81-82) Doruff then proposes three "levels of interaction" that require training and practice: 1. Media interaction: perception of intermedia
properties and parameters that 2. Control: perception of human -media interaction 3. Cooperation: human-human interaction, intervention and intention See Sher Doruff, "Collaborative Culture,"
in Brouwer, J., Mulder, A. , Charlton, S., eds. Making Art of Databases
(Rotterdam: V2_ Publishing/NAI Publishers, 2003), pp. 70-99. (3) For a fuller version of this writing, see
my "La Danse et la perception interactives," Nouvelles
de Danse 52, 99-115. For a critical study of immersive media, see
Oliver Grau, Virtual Art: From Illusion to Immersion (Cambridge:
MIT Press, 2003), pp. 343-44. For important insights into affective
perception, see Brian Massumi, Parables for the Virtual: Movement,
Affect, Sensation (Durham: Duke Univ. Press, 2002.). My reflections
on a new "digital" phenomenology are inspired by Mark B.N.
Hansen's theorization of the "digital image" (extending Henri
Bergson's theory of perception) and his approach to interactive information
environments which become a bodily process of filtering and composing
images. See his New Philosophy for New Media (Cambridge: MIT
Press, 2004), esp.pp.93-124, which also include a critical re-reading
of Massumi. (4) "Intimate Transactions," an interactive installation created in 2003-04 by the Australian Transmute Collective (Keith Armstrong, Lisa O"Neill, Guy Webster, et al) and recently shown at the New Territories festival in Glasgow, is an interesting case where the interface design is unusually thoughtful, sensual, and challenging from a synaesthetic perspective, but the ocular focus on making contact with another person through a visual world (video projection) is nevertheless maintained. In "Intimate Transactions" two participants situated in different physical or geographical locations will simultaneously interact with the work, reclining within in a new form of furniture (bodyshelf) that detects their bodily movements (feet and back of body). Each participant generates flowing combinations of digital imagery and sound, including ghostly, ethereal performing bodies (avatars), dynamic sculptural texts and immersive sound textures, by moving their feet on a floorboard or rubbing their back and shoulders against the shelf. Using the physical interface, gently moving their bodies on the "smart" or responsive surfaces, and by working both individually and collectively - observing their avatar-species float in space, connect and disconnect - participants are able to create unique "performances" of movement, image and sound which in fact influence an evolving "world" created from digital imagery and multi-channel sound. The implication of the single participant, performing in a closed space to connect with another person, into the visually projected world is obvious: when I began to "lean" into the interface, I had to first learn the interactive behavior (the movement of my feet, the rubbing with my shoulders and spine) that "controlled" or moved my avatar into contact with the other avatar, and thus I explored coordinating my feet and back movements in such as way that it (avatar) could in fact evolve, and I could form a certain sensory intimacy with the other person who was elsewhere, and this intimacy was created through my body's perceptional relationship to the logic (the movement patterns of the swirling, unpredictable creatures) of the digital scene. Most interestingly, I was also wearing a pouch in front of my stomach, which was wired to the network connecting me to the other participant, and when I was able to connect my avatar-creature with the other person's creature, the vibrations emitted from the pouch grew in intensity, and it felt as if my energy, connecting with the digital world and my fellow creatures, was radiating from the center of my body, which, as Lisa O'Neill suggests in her choreographic vision for the work, physiologically and emotionally streams out my centered relationship to space in the way it is taught in the Suzuki training method. The stomach vibrations created a sense perception of "streaming" which I had rarely ever experience before except in heightened moments of telematic performance, when the sonic energies of multi-channel sound, white noise, and flickering pixels pulsate through the networked studio and resonate deep inside the bowels of my body. Armstrong, whose work is influenced by the philosophy of ecology, thinks of the process as immersive; the transactions between participants make them sense their role in a complex web of relations and possibilities that connect living beings (not to speak of "living tissue" here, which would bring us to other contemporary experimentations, also conducted in Australia, linking performance and cellular biology/tissue cultures, for example in the work of SymbioticA, or in Sarah Jane Pell's Sub Culture project on liminal bio spheres, a collaboration she started with Oron Catts and Ionat Zurr involving living and semi living forms. Cf. www.symbiotica.uwa.edu.au). For "Intimate Transactions," see http://www.outlook.com.au/transmute/
(6) My reference to "wearing the digital" is based on current workshops I am conducting with fashion designers in Nottingham, where we explore the relations of fabric design and interactive design/performance. One aspect of this work includes exploring wearable computing and the inclusion of biofeedback sensors in the weaving of the textile design which can allow a direct influence on the projected environments around the body through tactile self-reference. The notion of "wearing" film is drawn from Jane M. Gaines " On Wearing the Film", in Stella Bruzzi and Pamela Church Gibson, eds., Fashion Cultures. London: Routledge, 2000, pp. 159-177. (7) Sentient Space was created as an experiment at the Digilounge Workshop in Chelmsford (UK) in February 2004. The workshop was convened by Scott deLahunta and produced by Essexdance and the British Arts Council. I am grateful to deLahunta for inviting me to the workshop, and to the artists for sharing their findings. Another experiment at the workshop was conducted by Carol Brown and her dancers. In their collaboration, Brown and digital architect Mette Ramsgard Thomsen devised an interactive environment (SPAWN ) to explore how kinetic information informs the behavior of virtual architectures. The dancers created duets or trios with the projected light of the architectures computed by the software Thomsen had written. Their bodily contours are tracked by the camera and processed by the computer program, and they can affect the morphological shapes of the image-light and image-color as their movement dynamics becomes embodied architecture. They literally enact the virtualization of their bodies. (8) This section is taken from a manuscript sent to me by an Canadian colleague after we spoke about the Erfurt workshop: F. Scott Taylor, "Synaesthesia and "The Name of Silence" in the Work of Steven Heimbecker," 2005, unpubl. manuscript [quoted by permission]. Heimbecker is a Canadian interactive digital artist who has created numerous multisensory installations. Taylor also refers to recent books by Laura U. Marks, The Skin of the Film: Intercultural Cinema, Embodiment, and the Senses (Durham: Duke University Press, 2000) and Touch: Sensuous Theory and Multisensory Media (Minneapolis: University of Minnesota Press, 2002) which offer fascinating insights into our corporeal relations of perception to filmic/video projection surfaces. Marks emphasizes the "haptic" as a term relating to touch that is referred to and further qualified by Deleuze and Guattari in their description of "smooth space," therefore the haptic refers to a space that is context dependent and, hence, felt (cf. Gilles Deleuze and Felix Guattari, "440: The Smooth and the Striated," A Thousand Plateaus: Capitalism and Schizophrenia, trans. Brian Massumi [Minneapolis: University of Minnesota Press, 987], pp. 474-500). For the phenomenological investigation of bodied space, see Maurice Merleau-Ponty, Phenomenology of Perception, trans. Colin Smith (London: The Humanities Press, Routledge & Kegan Paul Ltd., 1989). (9) Marlon Barrios Solano, "Designing Unstable
Landscapes: On bottom-up architectures, generative strategies and embodied
cognition in dance improvisation within dynamic multimedia environments,"
in Tanz im Kopf/Dance and Cognition, Yearbook 15 of the Gesellschaft
für Tanzforschung (Society for Dance Research), ed. Johannes Birringer
and Josephine Fenger (Münster: LIT Verlag; forthcoming). See also:
Birringer and Barrios Solano, Birringer and Barrios Solanor, "Materiality,
Embodiment, Interactive Technologies“:
http://www.dance.ohio-state.edu/~jbirringer/Dance_and_Technology/ips2M.html (10) This section engages Massumi, Parables for the Virtual, pp, 177-207.
Photo credits on these pages: Zelko Wiener; Ursula Hentschläger; Martha Steffen; Gesine K ästner; Johannes Birringer
Workshop organizers: Michael Giesecke, Christiane Heibach 3/4/2005
|
