Skip to content
Thesis Info
- LABS ID
- 00590
- Thesis Title
- A Perceptual Approach to Audio-Visual Instrument Design, Composition and Performance
- Author
- Adriana Sa
- E-mail
- adrianasa AT clix.pt
- 2nd Author
- 3rd Author
- Degree
- PHD
- Year
- 2016
- Number of Pages
- 250
- University
- Goldsmiths, University of London
- Thesis Supervisor
- Atau Tanaka
- Supervisor e-mail
- a.tanaka AT gold.ac.uk
- Other Supervisor(s)
- Mick Grierson
- Language(s) of Thesis
- English
- Department / Discipline
- Dep. of Computing
- Copyright Ownership
- Adriana Sa / Creative Commons Licence
- Languages Familiar to Author
- Portuguese, English, Germain, Spanish, French, Italian
- URL where full thesis can be found
- research.gold.ac.uk/19431/
- Keywords
- NIME, audio-visual performance, music, 3D environments, perception, attention
- Abstract: 200-500 words
- The thesis presents a perceptual approach to audio-visual instrument design, composition and performance. The approach informs practical work as well as a parametric visualisation model, which can be used to analyse sensory dominance, sonic expression and spatial presence in any audio-visual performance language.
The practical work intends for the image to function as a stage scene, which reacts to the music and allows for attention to focus on the relation between the sounds themselves. This is challenging, because usually vision dominates over audition. To clarify the problem, the thesis extrapolates from audio-visual theory, psychology, neuroscience, interaction design and musicology. The investigations lead to three creative principles, which inform the design of an instrument that combines a custom zither and audio-visual 3D software. The instrument uses disparities between the acoustic and digital outputs so as to explore those creative principles: a) to threshold the performerʼs control over the instrument and the instrumentʼs unpredictability, in ways that convey musical expression; b) to facilitate perceptual simplification of visual dynamics; c) to create an audio-visual relationship that produces a sense of causation, and simultaneously confounds the cause and effect relationships. This latter principle is demonstrated with a study on audio-visual mapping and perception, whose conclusions are equally applicable to the audio-visual relationship in space. Yet importantly, my creative decisions are not driven by demonstrative aims. Regarding the visual dynamics, the initial creative work assures perceptual simplification, but the final work exposes a gray area that respects to how the audienceʼs attention might change over time.
In any case, the parametric visualisation model can reveal how any audio-visual performance work might converge or diverge from these three creative principles. It combines parameters for interaction, sonic & visual dynamics, audio-visual relationship, physical performance setup and semantics. The parameters for dynamics and semantics reflect how stimuli inform attention at a particular timescale. The thesis uses the model to analyse a set of audio-visual performance languages, to represent my solo performance work from a creative perspective, and to visualise the workʼs versatility in collaboration with other musicians.