Project

# Title Team Members TA Documents Sponsor
98 Real Time Piano Input Visualizer For Learning
 Jay Park
Nuwan Singhal
Sarayu Suresh
 Wenjing Song design_document1.pdf
proposal1.pdf
Team Members:
- Nuwan Singhal (nuwans2)
- Jay Park (jaypark3)
- Sarayu Suresh (sarayus2)

# Problem

Learning to play the piano, especially if self-taught, comes with many difficulties. Two of the main ones are learning how to read sheet music, as well as knowing if your timing is accurate. These hurdles can be difficult to overcome, and lead to people giving up as they have to put in a lot of preparation before they can start playing songs.

# Solution

Create a hardware solution that controls an RGB LED Matrix which responds to MIDI input from a piano. This can be used to learn songs by preloading data through an SD card and having it so visual cues tell the user when to press keys and which keys to press, waiting for users to press the correct key before moving on. Other features such as controlling the speed of the song and working with only one hand can be used to incrementally learn. It could also be used when teaching piano by instead outputting what key is currently being pressed, allowing students to have a better understanding of what their teacher is playing.

# Solution Components

## Subsystem 1 - Led Matrix board to display which keys to press and user interface

The LED board shows which keys should be pressed by the user who is trying to learn how to play the song. It lights up when the user needs to press the key for playing the song that is stored in the SD card. There can be multiple modes for playing the music as well as options related to speed and hand used. We plan on using the RGB LED Matrix 1528-2094-ND as it offers multiple colours and enough space to display a few octaves of the piano. We plan on designing this project to mainly work with 4 octaves as those are the most commonly used ones, but with this LED Matrix, extending it to more octaves remains as an option. For the MCU, we found that a STM32F446RET6 would be a possible option mainly due to the high speed which is needed to keep the LED Matrix persistent for vision without major flickering. Additionally, we already have a Nucleo-F446RE, which means it can be used as a dev board and part of it can later be used as an ST-Link.

## Subsystem 2 - SD card for storing and loading songs

Our PCB should have an SD card reader which will store MIDI files of various songs which can be read by the microcontroller and loaded onto the LED board so that users can play along visually to learn the track. We can use Micro SD Card Reader Module TS-891 for the SD card Reader, and we can use Sandisk ImageMate Sdxc Flash Memory for storing songs as MIDI Files.

## Subsystem 3 - USB input for reading and registering piano input in real time

The keyboard sends messages whenever a key is pressed or released. The microcontroller reads these messages and extracts the note number and timing information. This data is compared with the expected notes from the SD card to determine whether the user played the correct key. We can use a MIDI Jack (SDS-50J) to take in Midi input from a piano, this is a better option than using USB A since that will require our microcontroller to act as a USB host, while using MIDI input directly uses UART which is all we need for our use case.

## Subsystem 4 - User interface to control the system

Users need to be able to control certain parts of the system such as what song to play, which mode to operate in, the speed of the playback and actually starting and turning on and off the system. We will use external buttons and switches for these parts of the system as well as the LED matrix to display text related to the interface. We can use simple buttons since the main controls we need are ‘left’, ‘right’ and ‘select’. (Omrom B3F-4055)

## Subsystem 5 - Power Management

Outlet power supply can be used for our project to power up our project. We can convert AC current to DC current using an external 5V adapter barrel plug that we used in previous labs to convert mains to 5V DC. We can use a low dropout regulator (AMS1117-3.3) to convert it from 5V to 3.3V for the MCU. The LED Matrix we plan to use uses 5 Volts, so no converter is necessary for that, the LED Matrix uses bare wires to get powered, so we could use a 2 input screw terminal block (TB002-500-02BE). For the data of the LED board, we need a 16 pin header (900-0702461602-ND).

# Criterion For Success

The system successfully loads a song from the SD card and begins playback.
The LED board correctly displays which piano keys should be pressed for the selected song.
A user interface allows users to interact with the system and choose the song they wish to play as well as details such as the speed, which hand and switch from input to output mode.
USB MIDI input accurately detects pressed keys in real time and matches them to expected notes.
Input detects the length of the time that the key was pressed.
Also displays multiple notes when multiple notes need to be pressed because of the song.
The system measures timing differences between expected notes and user input.
The system should have the option to wait for the user to press the correct key before moving forward with the song
The system should have options to control the speed of playback of a song
At the end of a song, the system reports basic performance metrics such as number of correct notes and average timing error.

# Alternatives

Currently three main similar solutions exist. The first of which are software solutions like Synthesia, but these require an internet connection, a smart device which runs supported operating systems and the software itself costs money. Our solution is a separate device from phones or laptops making it more accessible to younger and older people, more affordable, and not requiring an internet connection to function. The other solution is a one dimensional LED strip that sits on piano keys, such as “The ONE Piano Hi-Lite”, these solutions also require a smart device to function, but more importantly only offer one dimensional lighting which means that users see which keys to press in advance which is an important feature for harder songs and for learning the timing. The last option would be products similar to the “PopuPiano” which is essentially a combination of a led strip and a piano, but we aim to offer a separate device that piano owners can use rather than a piano itself. Also this solution comes with many of the same drawbacks as the other two.

GYMplement

Srinija Kakumanu, Justin Naal, Danny Rymut

Featured Project

**Problem:** When working out at home, without a trainer, it’s hard to maintain good form. Working out without good form over time can lead to injury and strain.

**Solution:** A mat to use during at-home workouts that will give feedback on your form while you're performing a variety of bodyweight exercises (multiple pushup variations, squats, lunges,) by analyzing pressure distributions and placement.

**Solution Components:**

**Subsystem 1: Mat**

- This will be built using Velostat.

- The mat will receive pressure inputs from the user.

- Velostat is able to measure pressure because it is a piezoresistive material and the more it is compressed the lower the resistance becomes. By tracking pressure distribution it will be able to analyze certain aspects of the form and provide feedback.

- Additionally, it can assist in tracking reps for certain exercises.

- The mat would also use an ultrasonic range sensor. This would be used to track reps for exercises, such as pushups and squats, where the pressure placement on the mat may not change making it difficult for the pressure sensors to track.

- The mat will not be big enough to put both feet and hands on it. Instead when you are doing pushups you would just be putting your hands on it

**Subsystem 2: Power**

- Use a portable battery back to power the mat and data transmitter subsystems.

**Subsystem 3: Data transmitter**

- Information collected from the pressure sensors in the mat will be sent to the mobile app via Bluetooth. The data will be sent to the user’s phone so that we can help the user see if the exercise is being performed safely and correctly.

**Subsystem 4: Mobile App**

- When the user first gets the mat they will be asked to perform all the supported exercises and put it their height and weight in order to calibrate the mat.

- This is where the user would build their circuit of exercises and see feedback on their performance.

- How pressure will indicate good/bad form: in the case of squats, there would be two nonzero pressure readings and if the readings are not identical then we know the user is putting too much weight on one side. This indicates bad form. We will use similar comparisons for other moves

- The most important functions of this subsystem are to store the calibration data, give the user the ability to look at their performances, build out exercise circuits and set/get reminders to work out

**Criterion for Success**

- User Interface is clear and easy to use.

- Be able to accurately and consistently track the repetitions of each exercise.

- Sensors provide data that is detailed/accurate enough to create beneficial feedback for the user

**Challenges**

- Designing a circuit using velostat will be challenging because there are limited resources available that provide instruction on how to use it.

- We must also design a custom PCB that is able to store the sensor readings and transmit the data to the phone.