Arts Track

Reinhard Gupfinger, University of Art and Design

Martin Kaltenbrunner, University of Art and Design

Luise Wolf, Independent author and journalist

Sound Shifting is an artistic research project that focuses on the physical representations of sound – this means the visualization and materialization of invisible phenomena that significantly shape our perception. We present a system that allows the transformation from sound to form in real-time by using a newly developed machine, the Audio Foam Cutter. This machine converts sound into polystyrene stripes that are being arranged to sculptural objects. The resulting sound sculptures of different sizes provide information about the represented sounds by their shape and aesthetic features and expand the range of our auditory perception to the tangible domain. The sound sculptures are snapshots of our soundscape and form a physical archive of sound representations. The Sound Shifting project aims to create an awareness of the materiality of sonic movements and affects.

Jennifer Weiler, Arizona State University

Piyum Fernando, Arizona State University

Todd Ingalls, Arizona State University

Stacey Kuznetsov, Arizona State University

The integration of new digital and physical fabrication tools with fine arts has the potential to provide new outlets for artistic expression, while at the same time raising questions about the role of material and process in artistic practice. In this work, we present Lithobox, a system that translates the traditional ceramic and lighting technique of lithophanes into a means of creating illuminated 3D models through a creative approach that utilizes both digital and tangible construction. Through work sessions with nine artists, we explored how the Lithobox fabrication impacted the way artists manifest design ideas and engage in creative exploration in crafting. At the TEI arts track, we plan to show our system and the physical lithophanes from our work with artists. The attendees will likely discuss the design, material, and artistic aspects of our exhibit. From these discussions, our goal is to gain insight into beneficial directions for integrating digital technology into traditional fine arts practices.

Xiaoyan Shen, Massachusetts Institution of Technology

Resonance Ver.M is an interactive video projection installation that allows the audience to have a peep into the artist's inside world. The project is the combination of a series of experimental executions including self-training, performance, EEG recording, interpretation of the bio-signal, subjective dream log, exhibition and interaction. The artwork shows an approach to the investigation of the dreams and a new form of "Human-Human interaction" through electrophysiological signals. It also introduces an innovative form of interaction between the reality and dream world, the conscious mind and the unconscious brain.

Malgorzata A. Zboinska, Chalmers University of Technology

Delia Dumitrescu, University of Borås

Hanna Landin, University of Borås

In this architectural research exploration, we challenge the notion of an interactive architectural surface as a single-layered, two-dimensional interaction interface. Instead, we propose the notion of an Interactive Voluminous Substance, which moves the interaction experience into four dimensions, shifting it from far-field, proximity-based interaction to a near-field, tactile one. We present four features of architectural expression that could potentially sustain the embodiment of this Substance: spatial positioning, geometry, expression, hybrid material composition and interaction design. If the future architectural interiors and exteriors are made from the Voluminous Architectural Substance, how it would feel to interact with it? We propose two physical prototypes and two interaction stories as speculative objects probing this question.

Jessica Broscheit, Hamburg University of Applied Sciences

Susanne Draheim, Hamburg University of Applied Sciences

Kai von Luck, Hamburg University of Applied Sciences

In this paper we explore human-environment interrelationships by utilizing both hybrid materially-oriented approaches and metaphorical representations. Inspired by the 'canary in a coalmine' metaphor we developed a tangible interface to sense the environment and provide a physical experience. The design utilizes life-like characteristics, like shape memory alloys and feathers to illustrate the metaphor. The aim of this approach is to propose a tangible interface as a mediator to provoke empathy for environmental issues. For that, the paper addresses an interdisciplinary field of design, society and technology through an embodied system.

Kimberly Lyle, Arizona State University

"After Words" is an interactive sculptural sound installation in which the viewer's participation causes basic units of speech to puncture the space, interrupting and overlapping yet remaining untied to any specific language. Inspired by early failed speech synthesizers that could only emit syllables, consonants, and vowels, these structures house small circuit boards which trigger audio files to play at random when the connected stand has air blown into it. The sounds emitted are made up of language's most basic building blocks and gesture towards a desire for language, but foreclose the possibility of meaning. Ultimately, "After Words" aims to create a space where sounds question logic, embrace nonsense, and untether the voice from language while poetically revealing connections between human and machine.

Karmen Franinović, Zurich University of the Arts

Luke Franzke, Zurich University of the Arts

Andrés Villa Torres, Labor 5020

Florian Wille, Zurich University of the Arts

In this paper, we discuss the opportunities and challenges of creating responsive environments with electroactive polymers (EAPs). Our previous research on tools and methods for EAPs enabled us to developed two public installations: SOLO and Electric Animal Plant. Going beyond the demonstration of EAPs, these projects explore aesthetics and interactivity of such shape-changing foils. We describe the creative process involved in the production of these works and present different exhibition setups. Further, the actuating and sensing capabilities of the material are discussed. Finally, we reflect on the response of the participants in two responsive environments and outline further research questions.

Tomas Laurenzo, City University of Hong Kong

Ekphrasis is a mixed-media installation consisting of close-up videos of a heavily scarred body projected onto an elastic screen. The screen has one string attached to its centre, which in turn is attached to a stepper motor. The motor pulls the screen in a controlled random pattern, stretching it and letting go, sometimes carefully, sometimes violently. The piece explores the relationship between the digital representation of the body and its corporeity, recreating the trauma that generated the scars on the medium itself, and proposing a new layer of abstraction that sustains the reflection on the significance of the human body.

Utsav Chadha, New York University

Mithru Vigneshwara, New York University

A display device, whilst seemingly static shows a lot of movement. Movement or change on such screens is perceived due to rapid, successive change in frames. Outside of screens, perceived movement on static objects may be achieved through a combination of optics and illusions. Kyne is a series of visual experiments, involving the perception of change in static distorted artifacts. It involves varied methods of obscuring or masking parts of static artifacts and taking advantage of persistence of vision to animate these artifacts. The artifacts explored use paper, refreshable static non-illuminated e-ink displays, and laser cut acrylic, and their masks include transparencies and digital projections. This paper illustrates the experiments conducted with different artifact-mask pairings and speculates on possible future pairings.

Kyung Yun Choi, MIT Media Lab

Hiroshi Ishii, MIT Media Lab

In this art project, the ephemeral and intangible aspects of human's communication are represented by soap-bubble. The shapeless, intangible, and insubstantial speech - once the speech is shouted out through speaker's mouth it disappears unless someone hears it immediately, or even it is heard, the message will be forgotten as time goes - is transferred to a semi-tangible yet still fleeting bubble. The bubble machine that we created provides person-to-person and person-to-space interaction. The machine has a iris mechanism that varies its outlet size reacting to the participant's speech pattern as if it tries to talk something. Once the participant pauses, the machine blows out various sizes of bubble. The floating bubble represents the subtle state of a message from interpersonal communications that lies in the middle of real and digital world. Also, it creates a certain delay until it pops, which is a metaphor of our behavior that we often delay to send out text-messages through chatting apps. We believe that anyone can be an artist. By open-sourcing the details of fabrication process and materials, we want to encourage people to build the machine, interact with it at any locations, and use and modify it as a art tool for realizing their own ideas whether it is for art or not.

Ryan Buyssens, New College of Florida

Digital information is typically only understood via lengthy explanations or data visualizations. It is my goal to use data to create physical objects that can not only represent the information that was used to create them but also to provide an interaction that can reinforce (or contradict) the core foundation of their creation. Manipulation of design with generative methods to create objects is one possible output. However, with the utilization of interactive technologies, digital information can be output as physical means through the control of objects interactively. This is particularly poignant in the format of an installation where multiples of objects can be controlled via data streams and have additional feedback through user interaction - which is the basis of my paper.

Tomas Laurenzo, City University of Hong Kong

Katia Vega, UC Davis

BOX is an artwork that exposes some of the social and political impact of artificial intelligence, computer vision, and automation. The project uses a commercially available computer vision system that predicts the interactor's ethnicity, and locks or unlocks itself depending on this prediction. The artwork showcases a possible use of computer vision, making explicit the fact that every technological implantation crystallises a political worldview.

Seth Dominicus Thorn, Arizona State University

Transference is a hybrid computational system for improvised violin performance. With hardware sensors and digital signal processing (DSP), the system shapes live acoustic input and computer-generated sound. An electromyographic (EMG) sensor unobtrusively monitors movements of the left hand, while a custom glove controller tracks bowing gestures of the right arm. Through continuous musical gesture the performer is able to actuate and perturb streams of computationally transmuted audio. No additional layers of windowing or semantically-inflected processes of machine learning mediate this process. Remaining at the level of signal processing, the lack of windowed and/or statistical mediation creates a sense of fine-grain tactility and physical transduction for the performer. The strategies employed are sufficiently generalizable to apply to situations beyond those imagined and implemented here within the scope of augmented violin performance.

Aurie Hsu, Oberlin Conservatory

Steven Kemper, Rutgers University

In "A Cyborg Manifesto," Donna Haraway describes how by the late twentieth century, humans have become hybridized with machines. While many criticize technology's encroachment on human lives, Haraway suggests accepting a kinship between organism and machine. The result is the cyborg, a hybrid body that fluidly transcends mechanical and organic boundaries. Why Should Our Bodies End at the Skin? for sensor-equipped dancer, robotic percussion, and live sound processing, explores these ideas of intersectionality and fluidity between organism and machine by connecting human action and mechanical tasks. This paper describes the creative framework and associated technologies involved in the development of the piece.

Raul Altosaar, OCAD University

Adam Tindale, OCAD University

Judith Doyle, OCAD University

A Very Real Looper (AVRL) is a non-visual virtual reality instrument inside of which a performer controls musical sounds and sequences through gesture and bodily movement. Contrary to how virtual reality (VR) is normally utilized, a performer playing AVRL is not disconnected from their surrounding environment through visual immersion, nor is their body restrained by a head-mounted display. Rather, AVRL uses VR sensors in conjunction with a game engine to map musical sounds and sequences onto physical objects and spaces. These are then by triggered by a performer simply wielding two controllers. AVRL thus combines the affordances of the physical world with the modularity of a game engine, consequently activating the expressive potential of the body inside of a large, highly reconfigurable, and musically augmented environment.

Thomas Ciufo, Mount Holyoke College

Line / Shape / Form is a solo improvisational performance project using the Eighth Nerve Guitar, a custom electric guitar that has a range of sensors built into the body of the instrument. This interactive system uses physical sensor data, a custom iPad interface, along with an array of real-time audio stream analysis data to control a range of computer-based digital signal processing transformations. This performance systems leverages a variety of unique hardware and software design strategies, all geared specifically towards improvisational performance. By combining a tactile, playable sound source with a dynamic interactive software system, the immediacy and richness of the guitar is extended by the power and flexibility of computation. This hybrid design, using physical sensors, generative strategies, audio stream analysis, and complex routing and mapping, creates an unpredictable, complex yet playable improvisational instrument.

Courtney Brown, Southern Methodist University

In Argentine tango, dancers typically respond to fixed musical recordings with improvised movements, each movement emerging in a wordless dialog between leader and follower. In the interactive work Machine Tango, this relation between dancers and music is inverted, enabling tango dancers to drive musical outcomes. Motion sensors are attached to dancer limbs, and their data is sent wirelessly to a computer, where algorithms turn the movement into sound. In doing so, the computer inserts itself in this on-going nonverbal conversation. Instead of traditional tango instruments such as the bandoneón, dancers generate and transform the sounds of typewriters and found sounds. System musical response to movement shifts during the dance, becoming more complex. The two dancers must navigate the resulting unstable musical structures as one body, responding with stylized tango movements. The difficulty of this task and the juxtaposition of the traditional with the experimental are integral to the performance aesthetic.

Jinsil Hwaryoung Seo,Texas A&M University

Michael Bruner, University of Texas at Austin

Nathan Ayres,Texas A&M University

Christine Bergeron,Texas A&M University

Alexandra Pooley,Texas A&M University

Austin Payne,Texas A&M University

Ashlyn Thompson,Texas A&M University

Kelsey Clark,Texas A&M University

Upwell is a mixed reality performance that allows audience members to explore virtual and physical worlds with two dancers. The environment provokes the feeling of being under water. A dander with a conventional VR head-mounted display and wearable controllers can navigate around a room scale virtual reality setup and interacts with dynamic visual and sound elements. Since the dancer wears custom-made wearable controllers on the palms, she can make intricate gestures to develop direct relationships with bioluminescent particles in the virtual water. The other dancer interacts with the visuals that created by the VR dancer without full understanding of the virtual world. Upwell can be utilized as a single person art installation as well as a performance projecting different views on a projection screen.