Project

# Title Team Members TA Documents Sponsor
30 Search and Identify
Ruidi Zhou
Shitian Yang
Yilai Liang
Yitao Cai
design_document2.pdf
design_document3.pdf
final_paper2.pdf
final_paper3.pdf
final_paper4.pdf
proposal1.pdf
proposal2.pdf
video1.mp4
Howard Yang
# Team members:
Yang, Shitian
sy39

Cai, Yitao
yitaoc3

Zhou, Ruidi
ruidi2

Liang, Yilai
yilail2

# Title:
Search and Identify

# Problem:
There is a noticeable gap in the availability of assistive applications tailored for homes, businesses, and individuals with mobility challenges. These groups lack efficient tools to swiftly locate everyday items, creating a significant inconvenience. This absence of specialized support not only hampers the day-to-day functionality within households and corporate environments but also poses a considerable barrier to independence for those with physical disabilities. Addressing this need with innovative solutions could dramatically improve the quality of life and operational efficiency by ensuring that vital items can be found quickly and easily, without unnecessary delay or reliance on others.

# Solution overview:
In solving the problem of accurately identifying specific items based on a user's immediate request, we are developing an innovative service-oriented robot capable of interactive processing. Our robot is equipped with a rotatable wireless camera mounted on a 360° steering engine which is controlled by a STM32F103c8t6 microcontroller under the drivetrain and power system, allowing it to visually scan its surroundings on receiving the user’s voice inquiry. When the software observes objects like cups, pencil cases, flowers, toys, and helmets, it processes the images to create an attention map. This map guides the robot to focus on and identify the specific object in question. Software component gives the object recognition feedbacks, the sensing system in the hardware component will output the 0/1 signal through signal control module and send it to the 360° steering engine. Such objects are easily achieved and can provide a suitable testing environment.

# Solution Component:
## Software Component:
1. Speech recognition: Transform audio instructions given by users into text tasks.
Prompt key item recognition: Simplify the text task prompt into a keyword or phrase.
2. Vision model: The vision model should take in the text prompt, and search for the object that matches the description best.
3. Algorithm: Our software parts will use a project on github called AbsVit as baseline, and we will remove the noise from the heatmap, and try to modify it to get the detail target object. The AbsVit is a algorithm and model for language-vision attention model.
## Hardware Component:
1. A drivetrain and power system: including a 360° steering engine, a wireless camera, a STM32F103c8t6 microcontroller, a 12V power source and a voltage converter. This system can rotate the camera to capture the pictures of its surroundings.
2. Control system: PC inputs the program into STM32F103c8t6 microcontroller, it will control the angles we want the camera to rotate each time and can control the time intervals between each rotation.
3. Storing system: including SD card which can store the pictures that the camera captures before, after software component finds that the object is found in the last picture, we can compare the pictures before and the picture which includes the object to verify the result.
4. Sensing system: including a signal control module. It can process the software component output into high-level or low-level signals and input 0/1 signals into drivetrain and power system to help it judge if it should operate or stop.

# Criterion for Success
1. Capable of identifying and navigating in indoor spaces, which have varying lighting situations including bright natural sunlight to dim artificial lights, and obstacles such as furniture and shelves.
2. The voice response system should also be easy to use, so it must respond timely and interact in natural language with the user (Users don’t need to learn the extra commands). The voice response system should also be easy to use, so it must respond timely and interact in natural language with the user.
3. When search request received from PC, our microcontroller of STM32F103c8t6 should send the correct impulse signal to control the 360° steering engine to automatically stop when the desired object is detected by the camera attached to it. The difference of the direction of the camera to the actual direction of the desired object should be within 3°.
4. Steering engine can rotate uniformly, smoothly, and continuously when no commands are given in a balanced and room temperature environment.

# Distribution of work
Yang, Shitian and Cai, Yitao:

Voice Recognition and Software Development:
Responsible for developing and testing the voice recognition system.

Yang, Shitian and Zhou, Ruidi:

Vision Module and Software Development:
Focus on developing and testing the vision module for object identification.

Zhou, Ruidi and Liang, Yilai:

Hardware and Microcontroller Development:
Responsible for developing and testing the hardware components, including the steering engine and microcontroller.

Cai, Yitao and Liang, Yilai:

Integration and Testing:
Oversee the integration of software and hardware components and conduct comprehensive testing of the entire system.

Seat U: Sensing System for Real-time Library Seat Occupation Detection

Jiayuan Huang, Hangzheng Lin, Jiaqi Lou, Hanyin Shao

Featured Project

# Problem

During the exam week, it is very difficult to find a seat in the library. Sometimes students cannot find a satisfying seat even if they walk through the library all around. Some students complain about unknown traffic in the library. For more convenient library seats seeking, students would like to know which other seats are empty ahead of time in order to decide whether they will go to the library and where to find available seats.

# Solution Overview

We will design a sensor-based device for each table to detect occupancy. The occupancy data will be uploaded through wifi to the cloud. There will be three states for each seat: occupied by people, occupied by items, or unoccupied. Then we will design an APP to visualize these data.

# Components

## The sensing subsystem:

• Data preprocessing and WiFi module to transfer data (ESP32)

• Multi-kinds of sensors to detect objects and collect data

• Wired power supply to support long-term real-time detection

## Human-computer interaction subsystem:

• Database server to store the collected data

• APP on the phone that allows clients to check the status of library seats

• It can indicate whether the seat is occupied with people (reserved by personal items), occupied without people, or available

# Criteria of Success

• Classify three different states of seats (occupied by people, occupied by items, or unoccupied)

• The accuracy of detecting whether a seat is reserved by items is above 90%

• The accuracy of detecting whether a seat is occupied by people is above 95%

• The sensor-based device APP is user-friendly and accurately visualizes the seat occupation

• The states of the seats get updated every 1 minute in the APP

• Adaptive to different kinds of table in the library (flexibility)

• Implement the database server bidirectionally: upload data from the device and download data to the APP