Google’s AI-powered Piano Genie is the New Way to Play the Instrument


Just like in other fields, machine learning is doing wonders in arts and music as well. The latest example of the AI-based instrument is the Piano Genie tool. The creative research team of Google known as Magenta developed the Piano Genie. The main feature of this AI tool is that you can improvise fluently on the piano by simply tapping eight buttons of this tool.

Piano Genie Creating Melodies for the New Users:

The Magenta team got the concept of Piano Genie by the games that simplify playing musical instruments like Guitar Hero and Rock Band. Their idea was to train the users in making up the pieces of melody on the fly instead of just tapping along the pre-written songs.

To develop this gadget, they trained an AI program on a huge dataset of classical piano music. They taught it how to predict the music notes that follow each other in a similar way as the smartphone texting feature predicts the word that you could write next.

According to one of the program developers, they wanted to design a tool that would allow someone to play piano who doesn’t know how to play the instrument. In this way, they would be able to create music on their own.

Since the Piano Genie is equipped with music prediction feature, the users will feel like it’s reading their minds. According to them, a lot of AI music based projects are able to generate entire melodies from a single starting note or chord. However, Piano Genie is a bit more innovative compared to other tools. It allows the users to have the command on the instrument by improvising note by note. At one side, it minimizes the latency in playing the note; while on the other side, it generates a unique feeling in the player.

AI powered piano Genie
Image: Youtube Video Screenshot

Donahue, a well-known pianist who’s been playing the piano for 20 years, used the Piano Genie tool. In his opinion, it was a really awesome experience on this tool. Occasionally, it seemed like that the AI device is capable of reading our mind and plays the exact note we intend to play next. And at some other times, it completely disobeys us but still suggests something reasonable on its own. That’s why the developers have named it as Piano Genie. Although you can wish for what you desire, you may not always get what you asked for.

The Magenta team used some common AI elements to develop the machine learning side of the tool. One of the main AI components of Piano Genie is the recurrent neural network. It is a type of program that’s particularly good at learning to mimic sequential data, such as music and writing.

They fed this neural network with a dataset of piano music that was taken from an international competition. This dataset proved to be quite useful in the development of the tool. The competition recorded all the performances in a file format that not only preserved the notes, but also the velocity of the notes. This velocity further translates the timbre and volume of the note.

They used this dataset as the main training data to build a predictive model of what piano notes would follow one another. This means that the Piano Genie stick to certain keys and scales while producing the notes. Although this variant can be tweaked.  According to Donahue, this competition dataset is quite useful as the main training data, since the users will be playing appropriately flashy things.

The team also designed a pair of encoders that would convert the output of the Genie tool into a format that suited their Guitar Hero-like controller. For this purpose, they shrink down the 88 notes, which is the standard number of piano keys, into just eight buttons. The main purpose of this last process was to hook up the whole program into a self-playing piano.


The programs like Piano Genie show that machine learning and AI can work to augment human creativity. It can turn humans into some sort of cyborgs by pairing our instinctive knowledge of when notes should be played with the computer’s ability to suggest which notes should come next. It proves to be a powerful combination as the new Piano Genie users feel delighted when they are able to create music just by pressing a couple of keys.


Please enter your comment!
Please enter your name here