Artificial Intelligence (AI) is often used to recognize objects in photos or videos. Neural networks, a data-processing algorithm, are able to mimic the human brain’s process of thinking in order to distinguish objects in a frame. Neural networks are trained using a set of images that have pre-defined signal or background objects, with signal objects being the ones that the AI is supposed to identify later on in new sets of images. This way, the algorithm can be trained to identify certain patterns of pixels as signal objects.

A team of researchers at the University of Washington decided to create an AI program that could look at an audioless video of a person playing the piano and recreate the sound that the piano would have produced if the volume were unmuted. Audeo, the subsequent product, was tested for its ability to produce the corresponding sounds to multiple piano videos. These sounds were tested in a music-recognition app, which correctly identified 86% of Audeo’s music pieces. To compare, the app was able to correctly identify the 93% of the audio from the original piano videos, meaning that Audeo was able to recreate the piano sounds with relatively high accuracy.

Here is a quick demonstration of Audeo at work:

http://https://www.youtube.com/watch?v=8rS3VgjG7_c&feature=youtu.be

Audeo was able to produce the sounds using both logical precision and imaginative prediction. The program was trained with about 191,000 frames of YouTube videos of pianist Paul Barton playing different styles of music, from classical to jazz, from Baroque to late romantic. The AI programming allowed Audeo to identify the change in motion between each frame in each piano video in order to produce the correct notes. As it recognized the changes in the piano keys that were pressed, the program was able to generate a series of pitches to correspond with each timestamp. However, it had to be trained to fill the gaps in sound between each frame while attempting to replicate the dynamics and articulation of each note and phrase. This process of training the AI is modelled after how a piano teacher would teach a student to produce music that is filled with both accuracy and emotional creativity.

Future improvements on this program would include more settings to synthesize a wider range of sounds as well as the ability to transcribe audio into sheet music for particular instruments. That way, musicians can learn to play a song that they have recently heard on an instrument that they already know how to play. 

Works Cited

McQuate, S. (2021, February 4). ‘Audeo’ teaches artificial intelligence to play the piano. UW News. Retrieved February 6, 2021 from https://www.sciencedaily.com/releases/2021/02/210204192543.htm

Su, K., Liu, X., and Shlizerman, E. Audeo: Generating music just from a video of pianist movements. University of Washington. Retrieved February 6, 2021 from http://faculty.washington.edu/shlizee/audeo/

University of Washington. (2021, February 4). ‘Audeo’ teaches artificial intelligence to play the piano. ScienceDaily. Retrieved February 6, 2021 from https://www.sciencedaily.com/releases/2021/02/210204192543.htm