Project

# Title Team Members TA Documents Sponsor
98 Real Time Piano Input Visualizer For Learning
 Jay Park
Nuwan Singhal
Sarayu Suresh
 Wenjing Song proposal1.pdf
Team Members:
- Nuwan Singhal (nuwans2)
- Jay Park (jaypark3)
- Sarayu Suresh (sarayus2)

# Problem

Learning to play the piano, especially if self-taught, comes with many difficulties. Two of the main ones are learning how to read sheet music, as well as knowing if your timing is accurate. These hurdles can be difficult to overcome, and lead to people giving up as they have to put in a lot of preparation before they can start playing songs.

# Solution

Create a hardware solution that controls an RGB LED Matrix which responds to MIDI input from a piano. This can be used to learn songs by preloading data through an SD card and having it so visual cues tell the user when to press keys and which keys to press, waiting for users to press the correct key before moving on. Other features such as controlling the speed of the song and working with only one hand can be used to incrementally learn. It could also be used when teaching piano by instead outputting what key is currently being pressed, allowing students to have a better understanding of what their teacher is playing.

# Solution Components

## Subsystem 1 - Led Matrix board to display which keys to press and user interface

The LED board shows which keys should be pressed by the user who is trying to learn how to play the song. It lights up when the user needs to press the key for playing the song that is stored in the SD card. There can be multiple modes for playing the music as well as options related to speed and hand used. We plan on using the RGB LED Matrix 1528-2094-ND as it offers multiple colours and enough space to display a few octaves of the piano. We plan on designing this project to mainly work with 4 octaves as those are the most commonly used ones, but with this LED Matrix, extending it to more octaves remains as an option. For the MCU, we found that a STM32F446RET6 would be a possible option mainly due to the high speed which is needed to keep the LED Matrix persistent for vision without major flickering. Additionally, we already have a Nucleo-F446RE, which means it can be used as a dev board and part of it can later be used as an ST-Link.

## Subsystem 2 - SD card for storing and loading songs

Our PCB should have an SD card reader which will store MIDI files of various songs which can be read by the microcontroller and loaded onto the LED board so that users can play along visually to learn the track. We can use Micro SD Card Reader Module TS-891 for the SD card Reader, and we can use Sandisk ImageMate Sdxc Flash Memory for storing songs as MIDI Files.

## Subsystem 3 - USB input for reading and registering piano input in real time

The keyboard sends messages whenever a key is pressed or released. The microcontroller reads these messages and extracts the note number and timing information. This data is compared with the expected notes from the SD card to determine whether the user played the correct key. We can use a MIDI Jack (SDS-50J) to take in Midi input from a piano, this is a better option than using USB A since that will require our microcontroller to act as a USB host, while using MIDI input directly uses UART which is all we need for our use case.

## Subsystem 4 - User interface to control the system

Users need to be able to control certain parts of the system such as what song to play, which mode to operate in, the speed of the playback and actually starting and turning on and off the system. We will use external buttons and switches for these parts of the system as well as the LED matrix to display text related to the interface. We can use simple buttons since the main controls we need are ‘left’, ‘right’ and ‘select’. (Omrom B3F-4055)

## Subsystem 5 - Power Management

Outlet power supply can be used for our project to power up our project. We can convert AC current to DC current using an external 5V adapter barrel plug that we used in previous labs to convert mains to 5V DC. We can use a low dropout regulator (AMS1117-3.3) to convert it from 5V to 3.3V for the MCU. The LED Matrix we plan to use uses 5 Volts, so no converter is necessary for that, the LED Matrix uses bare wires to get powered, so we could use a 2 input screw terminal block (TB002-500-02BE). For the data of the LED board, we need a 16 pin header (900-0702461602-ND).

# Criterion For Success

The system successfully loads a song from the SD card and begins playback.
The LED board correctly displays which piano keys should be pressed for the selected song.
A user interface allows users to interact with the system and choose the song they wish to play as well as details such as the speed, which hand and switch from input to output mode.
USB MIDI input accurately detects pressed keys in real time and matches them to expected notes.
Input detects the length of the time that the key was pressed.
Also displays multiple notes when multiple notes need to be pressed because of the song.
The system measures timing differences between expected notes and user input.
The system should have the option to wait for the user to press the correct key before moving forward with the song
The system should have options to control the speed of playback of a song
At the end of a song, the system reports basic performance metrics such as number of correct notes and average timing error.

# Alternatives

Currently three main similar solutions exist. The first of which are software solutions like Synthesia, but these require an internet connection, a smart device which runs supported operating systems and the software itself costs money. Our solution is a separate device from phones or laptops making it more accessible to younger and older people, more affordable, and not requiring an internet connection to function. The other solution is a one dimensional LED strip that sits on piano keys, such as “The ONE Piano Hi-Lite”, these solutions also require a smart device to function, but more importantly only offer one dimensional lighting which means that users see which keys to press in advance which is an important feature for harder songs and for learning the timing. The last option would be products similar to the “PopuPiano” which is essentially a combination of a led strip and a piano, but we aim to offer a separate device that piano owners can use rather than a piano itself. Also this solution comes with many of the same drawbacks as the other two.

Iron Man Mouse

Jeff Chang, Yayati Pahuja, Zhiyuan Yang

Featured Project

# Problem:

Being an ECE student means that there is a high chance we are gonna sit in front of a computer for the majority of the day, especially during COVID times. This situation may lead to neck and lower back issues due to a long time of sedentary lifestyle. Therefore, it would be beneficial for us to get up and stretch for a while every now and then. However, exercising for a bit may distract us from working or studying and it might take some time to refocus. To control mice using our arm movements or hand gestures would be a way to enable us to get up and work at the same time. It is similar to the movie Iron Man when Tony Stark is working but without the hologram.

# Solution Overview:

The device would have a wrist band portion that acts as the tracker of the mouse pointer (implemented by accelerometer and perhaps optical sensors). A set of 3 finger cots with gyroscope or accelerometer are attached to the wrist band. These sensors as a whole would send data to a black box device (connected to the computer by USB) via bluetooth. The box would contain circuits to compute these translational/rotational data to imitate a mouse or trackpad movements with possible custom operation. Alternatively, we could have the wristband connected to a PC by bluetooth. In this case, a device driver on the OS is needed for the project to work.

# Solution Components:

Sensors (finger cots and wrist band):

1. 3-axis accelerometer attached to the wrist band portion of the device to collect translational movement (for mouse cursor tracking)

2. gyroscope attached to 3 finger cots portion to collect angular motion when user bend their fingers in different angles (for different clicking/zoom-in/etc operations)

3. (optional) optical sensors to help with accuracy if the accelerometer is not accurate enough. We could have infrared emitters set up around the screen and optical sensors on the wristband to help pinpoint cursor location.

4. (optional) flex sensors could also be used for finger cots to perform clicks in case the gyroscope proves to be inaccurate.

Power:

Lithium-ion battery with USB charging

Transmitter component:

1. A microcontroller to pre-process the data received from the 4 sensors. It can sort of integrate and synchronize the data before transmitting it.

2. A bluetooth chip that transmits the data to either the blackbox or the PC directly.

Receiver component:

1. Plan A: A box plugged into USB-A on PC. It has a bluetooth chip to receive data from the wristband, and a microcontroller to process the data into USB human interface device signals.

2. Plan B: the wristband is directly connected to the PC and we develop a device driver on the PC to process the data.

# Criterion for Success:

1. Basic Functionalities supported (left click, right click, scroll, cursor movement)

2. Advanced Functionalities supported(zoom in/out, custom operations eg. volume control)

3. Performance (accuracy & response time)

4. Physical qualities (easy to wear, durable, and battery life)