Motion and Emotion with TensorFlow.js and Tone.js
Duration: 3 weeks
Project Contributions: This project was done in collaboration with Olof Ingelsten.
This project is a collection of explorations about kinaesthetic experiences in interaction, and gaining insights through multiple interactive sketches for iteration.
We used TensorFlow.js to capture the body’s position by detecting movement as input for further processing movement in spaces. On top of using TensorFlow.js to collect data through machine learning algorithm, we used Tone.js to work with sound as an expressive output.
Using our own body as a synthesiser, we experimented how sound reacts to the speed, distance, and scale in motion.
First round: testing
The purpose of this sketch was to learn the material in a fun way. We built a test sketch to learn how to set up camera, process data, and use TensorFlow.js & Tone.js.
For this sketch, we measured the distance between right shoulder, right elbow, and right hip. Then it makes a farting noise when the elbow is close to the hip. It was an effective way to reflect about the social implication of the actions that we perform because provocative action could not be ignored.
Second round: openness vs. closeness
Through this sketch we explored the feeling is openness or closeness through a motion, and how it demonstrates inward vs. outward expression. This sketch controls BPM by calculating distance between limbs and fist pumping increments.
Through this sketch, we discovered an element of discomfort, and how it influences your emotions and actions. The fist pumping action and squatting at the same time is performing contrary action to the pose. It expresses outward energy while keeping inward position, which causes tension between motion and emotions because it challenges your current state of being.
Generally speaking, people normally do not fist pump in the air while crouching down. On top of that, depending on your physical condition, you could have trouble balancing yourself while crouching down too. Like such, certain actions are unnatural depending on the pose and you feel the emotional resistance to certain actions. It is almost like a violence against your emotional state because if you feel reserved, perhaps you should be allowed to stay in that mood. The motions itself place you under a condition to want to stand up. You can either take that motion as a sense of encouragement to break out of your shell, or if taken negatively, you may feel forced against your will. Whether you feel the positive or negative motivations, there is an obvious observation of emotional transition from indifference to excitement.
Final round: Sightless Body Synthesiser
We continued with our explorations and iterated the concept of listening to smell. We kept the primary point of control to a nose, assign musical notes to each joints, then passed the sound around each joints using your own nose. Also, when the nose was close to a specific joint, the musical note increased in volume and sound traveled between ears.
It created a calm and meditative atmosphere. We saw a potential of making a design that allows dancers to manipulate music, so they become the creator of music instead of dancing to music. The dancer could express her inner feelings through movements and the performance will be available for audiences to watch. However, this sketch was demonstrated and perceived differently with earphone compared to speakers because it became a deeply individual experience with earphones.
Unfortunately, this sketch is lacking design intention because it is incomplete. The aesthetic experience is not necessarily connected to functionality. It would be beneficial to find richness and unpack the idea further.
Our success was finding nuance and quality of texture through the sketches. We avoided the use of hands and eyes, which are over used in current design space. We explored kinaesthetic movements, how the movement feels, how it is experienced, and how all theses aspects affect each other and come together as whole.