Skip to main content
To KTH's start page

Spv: Sandra Pauletto

Sandra Pauletto
Sandra Pauletto associate professor

Project 1: Design for tomorrow’s sustainable behaviour

Background

Energy provides heating, lighting and more, but it affects the environment. In Europe, households account for 25% of the energy-related greenhouse gas emissions. The increasing popularity of smart meters allows to have detailed data about consumption in households. Residential energy feedback could contribute to 5%–10% energy consumption reduction, but there are several barriers. Visualisations of energy costs and savings are not enough. Your work will inform research done in the Sound for Energy project .

Task

Differently form other kinds of products, energy is not tangible. It is therefore difficult for user to be aware of their consumption. This project aims to investigate how digital sound can contribute to enhance smart meters and/or smart plugs to inform users of their energy consumption and nudge them towards a more energy efficient and sustainable behaviour.

Methods

Participatory methods will be used to design a sonic enhanced smart energy system prototype, the user experience and select appropriate sonic interaction design strategies. You will be able to exploit energy forecasting models already developed within the Sound for Energy project and real-time CO2 emissions calculation. Qualitative and quantitative evaluation methods will be used to assess how this prototype can facilitate understanding and engagement with energy efficiency and sustainability.

Initial references

  • Bertoldi, Paolo. "Overview of the European Union policies to promote more sustainable behaviours in energy end-users." Energy and Behaviour. Academic Press, 2020. 451-477.
  • European Environment Agency. 2013. Achieving energy efficiency through behaviour change: what does it take? doi.org/10.2800/49941 
  • Lockton, Dan, et al. "Powerchord: Towards ambient appliance-level electricity use feedback through real-time sonification." International Conference on Ubiquitous Computing and Ambient Intelligence. Springer, Cham, 2014.
  • Sonic Interaction Design book: mitpress.mit.edu/books/sonic-interaction-design

Supervisors: Sandra Pauletto, Rod Selfridge, Yann Selfrdige, Vincenzo Madaghiele

Project 2: The expression of emotions in sound design

Background

In this project you will study how contextual elements, this can be the visual aspects of the door and walls, the speed of opening of the door, the creaking of the door opening, a sentence accompanying the knocking sounds, etc. or acoustic characteristics such as reverb, might affect the perception of emotions in sound design. Sound effects such as footsteps, knocking sounds, etc. are quintessential dramatic tools in storytelling for many media (e.g. games, films). For example, the opening of a door often serves as a transition, a turning point to dramatic changes in the story. These sounds tells us not only about the emotion of the person producing them, but also create expectations and emotions on the audience listening to them. To understand how contextual elements affect our perception of these sounds could lead us to understand how we can maximise the emotional communication in audiovisual media such as film, games or VR.

Task

You will design an experiment focused on particular sound effects such as knocking sounds or footsteps that will measure the role of context in the perception of emotions in those sound effects. The experiment could be done in the context of in 2D or 3D media (film, game, 360 video or VR). In terms of sounds, we have a database of emotional labelled knocking sounds produced by Foley artist Ulf Olausson (link) and a data set of emotionally labelled digitally synthesized knocking sounds, however other sound effects such as footsteps could be considered too.
We also want to evaluate if the presence of a context can reduce any perceivable difference between the recorded and synthesized knocking sounds.

Methods

Creation of appropriate audiovisual scenes in 2D or 3D. Develop a perceptual experiment and gather quantitative and qualitative results and analyse them. Some knowledge of basic statistics will be useful.

Initial references

  • Pauletto, 2019, Invisible Seams: The Role of Foley and Voice Postproduction Recordings in the Design of Cinematic Performances, in Foundations in Sound Design for Linear Media: A Multidisciplinary Approach / [ed] Michael Filimowicz, Routledge, 2019.
  • M. Houel et al., "Perception of Emotions in Knocking Sounds: an Evaluation Study," in Sound and Music Computing Conference 2020, Torino, 24-26 June 2020, 2020.
  • A. Barahona-Rios och S. Pauletto, "Synthesising Knocking Sound Effects Using Conditional WaveGAN," i SMC Sound and Music Computing Conference 2020, 2020.
  • Iop, A., & Pauletto, S. (2021). Perception of Emotions in Multimodal Stimuli: the Case of Knocking on a Door. In Sound and Music Computing Conference.
  • Selfridge, Rod, et al. "Creating Historic Spaces in Virtual Reality Using Off-the-Shelf Audio Plugins." Audio Engineering Society Conference: 2019 AES International Conference on Immersive and Interactive Audio. Audio Engineering Society, 2019.
  • Mo, Ronald, Bin Wu, and Andrew Horner. "The effects of reverberation on the emotional characteristics of musical instruments." Journal of the Audio Engineering Society 63.12 (2016): 966-979.

Supervisor: Sandra Pauletto and Rod Selfridge

Project 3: The Sverige Radio Sound Studio: developing novel digital sound design tools through the modelling of historical sound effects

Background

Modern digital sound design has its roots in the art of Foley (creating sound effects in a studio with a variety of everyday objects). By digitally modelling some of these objects and their characteristics, we can expand the possibilities of modern sound design. Sverige radio has a large sound studio with a vast archive of sounding objects. Through our collaboration with studio manager Michael Johansson, we have access to this unique archive. This project will contribute to funded research which is a collaboration between KTH and KMH (Royal College of Music). See The Radio Sound Studio project .

Task

Through a collaboration with studio manager and sound maker Michael Johansson, we have selected a number sounding objects from the Sverige Radio Sound Studio. One example is the Squeaky Box a video of which can be found HERE. Other examples could be thunder sound effects have traditionally been performed by wobbling a thin metal sheet. We can provide you with initial digital models of these objects created using procedural audio and physical modelling in Pure Data. Your task will be to expand the sonic capabilities of these models by, for example, expanding their parameters' range and exploring a number of different interfaces (for example, the twisting action of a pepper mill could be used for both a thunder or a squeak).

Preliminary research questions

  1. What can we learn from successful historical sound design practices that we can transfer to digital sound design tools, and how do sound design models differ from models of musical instruments?
  2. How important is it to reproduce haptic and kinesthetic aspects of historical sound making practices for the final outcome?

Method

Taking inspiration from already developed prototypes, you will ideate, and develop new interface for these digital models and use them to perform the sounds in different contexts (music, art, game, film...). Through workshops with KTH sonic interaction design students and researchers, you will expand and test the capabilities of these models both for artistic and functional purposes. Finally, through system and user experience evaluations you will discuss how these models and new interafaces bring these sounding objects alive in new contexts.

References

  • Keenan, Fiona, and Sandra Pauletto. "‘Listening Back’: Exploring the Sonic Interactions at the Heart of Historical Sound Effects Performance." The New Soundtrack 7.1 (2017): 15-30.
  • Pauletto, Sandra. "Embodied Knowledge in Foley Artistry." The Routledge Companion to Screen Music and Sound (2017): 338.
  • Sonic Interaction Design book: mitpress.mit.edu/books/sonic-interaction-design

Supervisors: Sandra Pauletto and Rod Selfridge

Project 4: The Climate Change music box: understanding science to take action

Background

The goal of the project is to increase young people’s personal understanding of and engagement with climate change by providing them with a web-based app for making music that is based on environmental data employing novel musical sonification approaches.

Task

We will use Hans Lindetorp's web-based music framework  to prototype an application that can accept data from the most recent Intergovernmental Panel on Climate Change ( IPCC ) report or other data (including data about emissions, impact on natural and human systems, and so on), and will allow a non-expert music maker to produce a number of musical data sonification alternatives (different genres, keys, tempos, real-time interaction).

Methods

Participatory methods will be used to design the application, user experience and select appropriate sonification strategies. Qualitative and quantitative evaluation methods will be used to assess how digital tools for music making can facilitate understanding and engagement with climate change data.

Initial references

Supervisor: Sandra Pauletto

Project 5: Projects in Media Production

We are available to supervise a variety of projects in media production including projects about filmmaking, radio production, 360 video, VR/AR, and games.

If you have a project of this nature in mind, please come and talk to us and we will help define it.

Initial references

  • C. Manolas, S. Pauletto and J. Jang, "Soundtrack Loudness as a Depth Cue in Stereoscopic 3D Media," Convergence. The International Journal of Research into New Media Technologies, 2020.
  • R. Idrovo and S. Pauletto, "Immersive Point-of-Audition : Alfonso Cuarón’s Three-Dimensional Sound Design Approach," Music, Sound, and the Moving Image, 2019.
  • S. Pauletto, "The sound design of cinematic voices," The New Soundtrack, vol. 2, no. 2, pp. 127-142, 2012.
  • M. J. Lopez and S. Pauletto, "The Design of an Audio Film : Portraying Story, Action and Interaction through Sound.," Journal of Music & Meaning, vol. 8, no. 2, 2009.
  • S. Pauletto, "Invisible Seams : the Role of Foley and Voice Postproduction Recordings in the Design of Cinematic Performances," in Foundations in Sound Design for Linear Media : A Multidisciplinary Approach, Michael Filimowicz Ed., : Routledge, 2019.
  • Selfridge, Rod, et al. "Creating Historic Spaces in Virtual Reality Using Off-the-Shelf Audio Plugins." Audio Engineering Society Conference: 2019 AES International Conference on Immersive and Interactive Audio. Audio Engineering Society, 2019.

Supervisors: Sandra Pauletto, Rod Selfridge, Yann Seznec