Project

# Title Team Members TA Documents Sponsor
31 Gamified Focus-Detection Tamagotchi
Khushi Murthy
Michael Yang
Noah Yi
Tim Jiang other1.pdf
proposal1.pdf
# Problem

We are currently living in a time where Gen Z’s attention span is being reduced due to short form media content and various social and technological distractions, leading to decreased productivity. As such, students who force themselves to sit through hours of studying will often find themselves struggling to stay motivated and on task. Methods such as the Pomodoro strategy have been shown to help increase productivity through structure, yet continuous accountability and immediate feedback when a user becomes distracted are still missing factors.

# Solution

We propose an interactive, desk-based smart companion inspired by a Tamagotchi. The device monitors several indicators of user focus and translates them into the emotional and physical behavior of a desktop pet. A host computer will collect simple focus-related signals, including active application information, keyboard and mouse activity, and whether the user is present at the desk. These signals will be combined into a focus score and transmitted to a microcontroller contained inside a 3D printed enclosure through USB serial communication. The microcontroller will maintain the pet's behavioral state and control its physical outputs. Sustained focus may cause the pet to appear happy or energetic, while extended inactivity or distraction may cause it to appear sad, tired, or perform a physical gesture such as waving its arms to attract the user’s attention. Therefore, our goal is for the companion to create an immediate, external feedback loop that gamifies focus and reinforces effective work habits.

The project is divided into three primary subsystems:

It will have 3 different subsystems:
Software Logic Subsystem: We will connect the board to the laptop to collect focus-related information from the host and compute a focus score.
Microcontroller Subsystem: The embedded controller itself that manages the pet’s behavioral state machine and controls the hardware outputs.
The Display: Provides the user-facing visual and mechanical feedback for the pet’s state

# Solution Components

## Subsystem 1 : Software Logic Subsystem

Description: Runs in the background on the host computer to monitor focus. It tracks active application titles, keyboard/mouse input activity, and uses the host webcam to detect desk presence via basic face tracking. To keep the image-processing component feasible, the webcam will not attempt complex activity, emotion, or identity recognition. It will perform a constrained presence-detection task using an existing computer-vision library such as OpenCV or MediaPipe. The system will assume a relatively fixed webcam position and desk environment. It aggregates these metrics into a scoring metric that represents focus state and sends periodic command packets over USB-serial to the hardware. The main software design challenge of this subsystem will be the fusion of multiple imperfect signals into a reliable and explainable focus metric, rather than relying on computer vision alone.

Parts:
Host PC
Software packages

## Subsystem 2 : Microcontroller and Custom Control PCB Subsystem

Description: A custom PCB will integrate the Raspberry Pi Pico with the system's power distribution, display interface, actuator-control circuitry, peripheral connectors, and protection components. The board will receive focus-state information from the host computer over USB and generate the electrical control signals required by the display and mechanical actuators.
Parts:
Microcontroller Board: Raspberry Pi Pico (Part #: SC0915)
USB interface
Voltage regulation and power distribution circuitry
Servo control interfaces
Display communication interface

## Subsystem 3 : Display and Actuation Subsystem

Description: Provides the physical feedback that makes the device function as a desktop companion. A small display will show animated facial expressions corresponding to the current behavioral state of the pet. One or more servo motors will provide additional mechanical feedback by moving physical components such as the pet's arms.

Parts:
Small OLED or TFT display
Micro servo motors
Mechanical arm linkage
3D-printed enclosure
Supporting power and signal circuitry

# Criterion For Success

Success would consist of meeting our objectives for both the hardware and software portion of the project. The software must successfully collect application activity, keyboard/mouse activity, and webcam-based desk-presence information and combine these inputs into a reproducible focus score or state. The host computer must successfully transmit low-latency focus-state updates to the hardware with outputs being shown within 1-2 seconds. The microcontroller must reliably maintain at least four distinct behavioral states, with each state transition determined by documented focus-score thresholds and timing conditions. Finally, Given the same sequence of focus inputs, the system should produce the same behavioral response. Every pet expression and mechanical action should therefore be traceable to a specific focus metric or state-machine transition.

Navigation Vest Suite For People With Eye Disability

Jiwoong Jung, Haoming Mei, Pump Vanichjakvong

 Navigation Vest Suite For People With Eye Disability

Featured Project

# Navigation Vest Suite For People With Eye Disability

Team Members & Experiences:

- Jiwoong Jung (jiwoong3): Experienced in Machine Learning, and some Embedded programming. Worked on many research and internships that requires expertise in Machine Learning, Software Engineering, Web Dev., and App Dev. Had some experience with Embedded programming for Telemetry.

- Haoming Mei (hmei7): Experienced in Embedded programming and PCB design. Worked with projects like lights, accelerometer, power converter, High-Fet Board, and motor control for a RSO that involve understanding of electronics, PCB design, and programming with STM32 MCUs.

- Pump Vanichjakvong (nv22): Experienced with Cloud, Machine Learning, and Embedded programming. Done various internships and classes that focuses on AI, ML, and Cloud. Experience with Telemetry and GPS system from RSO that requires expertises in SPI, UART, GPIOs, and etc with STM32 MCUs.

# Problem

People with Eye Disability often face significant challenges navigating around in their daily lives. Currently, most available solutions ranges like white canes and guide dogs to AI-powered smart glasses, many of which are difficult to use and can cost as much as $3,000. Additionally, problems arises for people with disability, especially in crowded/urban areas, and that includes injuries from collision with obstacles, person, or from terrains. According to the U.S department of Transportation's 2021 Crash Report , 75% of pedestrian fatalities occurred at locations that were not intersections. Thus we aim to design a navigation vest suite to help people with eye disability to deal with these issues.

https://crashstats.nhtsa.dot.gov/Api/Public/ViewPublication/813458.pdf

# Solution

We have devised a solution which helps ease visually impaired individuals in daily activities such as walking from two places, or navigating around a building with multiple obstacles. Our focus will for out-door navigation in urban areas, where obstacles, terrain, and pedestrians. But, if time permits we will also deal with traffics and crosswalks.

In order to achieve this, we will be utilizing 3 main components:

- Lidar sensors to help the wearer with depth perception tasks

- Vibration Motors to aid navigation (turning left/right)

- Magnetometer to enable more accurate GPS coordination

All the above components will contribute to the sensory fusion algorithm.

# Solution Components

## Subsystem 1

### Microcontroller System

We are planning to use a STM32 microcontroller as main processing unit for sensory data from lidar sensors (magnetometer and GPS if time permits) and object detection data from the **machine learning system**, and direction data from navigation app (our design on phone). We will use these information to generate vibration in the direction the wearer should navigate.

### Power Systems

The whole system will be battery-powered by a battery module, which contains 5V battery-cells. It will be connected to the **Microcontroller System**, which will also supply it to the **Machine Learning System**. We will also implement the necessary power protection, buck converter, regulator, and boost converters as necessary per sensors or components.

- Battery Module Pack

- Buck Converter (Step-Down)

- Boost Converter (Step-Up)

- Voltage Regulator

- Reverse Polarity Protection

- BMS

## Subsystem 2

### Navigation Locator Systems

Our navigation system will consist of an App which directly connects to the Google Maps API, paired with our existing sensors. We plan to utilize a magnetometer sensor, which will indicate the direction the user is facing (North, South, East, West, .etc). In order to pinpoint which direction the wearer needs to be heading, our built-in LiDAR sensors will enable us to create a SLAM (Simultaneous Localization and Mapping) to build a map of the environment. With these systems in place, we would be able to assist users in navigation. To deal with Terrain hazards, we will use the LiDAR to sensors to assist us in dealing with elevation changes the user needed to make.

- LiDAR

- Android App (Connected to Google API)

- Magnetometer

- Vibration Motors

Extra Features (if time permits):

- Audio Output (Text to Speech Generation from Raspberry PI 5 sends to microcontroller through AUX cable )

## Subsystem 3

### Machine Learning Systems

- We plan to employ a object detection model on a 16GB Raspberry PI 5 (already) along with a PI camera to detect objects, signs, and people on the road, which will be feed to the microcontroller

- Raspberry PI 5

- PI Camera

a) The image video model will be expected to be less than 5 billion parameters with convolutional layers, running on device in raspberry pi Obviously the processing power on the raspberry pi is expected to be limited, however we are planning to accept the challenge and find out ways to improve the model with limited hardware capabilities.

b) If the overall project for subtask a) becomes arduous or time consuming, we can utilize api calls or free open source models to process the image/video in real time if the user wants the feature enabled. The device is paired with the phone via the wifi chip on the raspberry pi to enable the API call. Some of the best candidates we can think of are the YOLO family models, MMDetection and MMTracking toolkit, or Detectron2 model that is developed by Facebook AI Research that supports real time camera feedbacks.

# Criterion For Success

### Navigational Motor/Haptic Feedback

1) The Haptic feedback (left/right vibration) should perfectly match with the navigation directions received from the app (turn left/right)

2) Being able to Detect Obstacles, Stairs, Curbs, and people.

3) Being able to detect intersections infront and the point of turn through the lidar sensory data.

4) Being able to obey the haptic feedback patterns that is designed. (tap front for walking forward, tap right to go right etc...)

### Object Detection

1) Using the Illinois Rules of the Road and the Federal Manual on Uniform Traffic Control Device Guidelines, we will be using total of 10-30 distinct pedestrian road signs to test the object detection capability. We will be using a formal ML testing methods like geometric transformations, photometric transformations, and background clutter. Accuracy will be measured by the general equation (Total Number of correctly classified Datasets)/(Total Number of Datasets)

2) The ML Model should be able to detect potential environmental hazards including but not limited to Obstacles, Stairs, Curbs, and people. We are planning onto gather multiple hazard senarios via online research, surveys, and in-person interviews. Based on the collected research, we will be building solid test cases to ensure that our device can reliably identify potential hazards. More importantly, we are planning design strict timing and accuracy measures metrics.

3) The ML model should be able to detect additional road structures such as curbs, crosswalks, and stairs to provide comprehensive environment awareness. We will be utilizing different crosswalks located on north quad and utilize the accuracy measurement techniques mentioned in 1)

### Power and Battery Life

1) The device should support at least 3 hours of battery life.

2) The device should obey the IEC 62368-1 safety standard. IEC 62368-1 safety standard lays out different classes such as ES1, ES2, and ES3 that considers electrical and flame

Project Videos