Project

# Title Team Members TA Documents Sponsor
34 A smart glove for HCI
 Hongwei Dong
Jinhao Zhang
Shanbin Sun
Zhan Shi
 design_document1.pdf
final_paper1.pdf
final_paper2.pdf
proposal2.pdf
 Pavel Loskot
# TEAM MEMBERS
Hongwei Dong (hd2), Shanbin Sun (shanbin3), Jinhao Zhang (jinhaoz2), Zhan Shi (zhans6)

# PROBLEM & SOLUTION OVERVIEW
In today's society, people are increasingly interacting with smart devices such as laptops and smartphones. This trend underscores the need for innovative methods to improve the efficiency of interaction with these devices. Among the emerging solutions, smart gloves hold great promise as a means to address this need. The smart glove is able to collect the positional information of the user's fingers. It then processes the information to recognize the user's gestures and maps the recognized gestures to predefined shortcuts, thereby facilitating efficient interaction between the user and the computer.

# PROJECT TITLE
A smart glove for HCI

# SOLUTION COMPONENTS
## Subsystem1: IMU based gesture sensing system
- MPU6050, a six DOF IMU, is placed on each fingertip to collect the position and angle information.
- I2C bus to communicate with the ESP32.

## Subsystem2: gesture recognition system
- Raw data pre-processing to obtain high accuracy gesture
- Pre-trained gesture recognition model using ESP-DL inference library
- User-defined gesture shortcut map to support any type of input

## Subsystem 3: Communication System
- Serialization and deserialization on both the device and host side to package the information to be transmitted in binary/JSON format.
- CP2102/CH340 USB module to support USB serial communication such as UART when the glove is charging or high-bandwidth transmission is required
- Bluetooth module to support Bluetooth (LE optional) communication for gesture and command transmission

## Subsystem 4: Power Management System
- High energy density lithium polymer battery (e.g. 1000-3000mAh) to power the IMUs, esp32, and peripheral components, ensuring a long wireless user experience.
- Step-up or step-down voltage regulator circuitry to meet the stable power requirements of each subsystem.

# Criterion For Success
- The MPU6050 IMU system should reliably collect raw gesture data, with stable I2C data transfer between the MPU6050 and the ESP32 without significant latency or data loss.
- The deep learning gesture recognition model used on the ESP32 should be able to map gestures to appropriate keystrokes, enabling mouse operations and various custom functions.
- The communication system should ensure seamless data transfer between the computer and mobile devices without significant latency.
- The battery management system should ensure that all components receive the correct voltage, with the charge management module and power monitoring functions operating correctly.

# DISTRIBUTION OF WORK
- Jinhao Zhang [EE]: Responsible for the design and implementation of the power subsystem and other circuit systems, including the design, test, and optimization of the battery charging module and power monitoring module.
- Hongwei Dong [ECE]: Responsible for raw gesture data pre-processing program. Training and deployment of the gesture recognition model. Serialization/deserialization of data structure for device and host.
- Shanbin Sun [ECE]: Responsible for collect the gesture data used for training the model. and I2C bus development. Develop the user interface to define the gesture shortcut map. Develop the IMU and device driver.
- Zhan Shi [EE]: Responsible for USB and Bluetooth communication between host and device. Develop the I2C bus for IMU to ESP32 communication. PCB design and verification. Unit testing of each subsystems.

Clickers for ZJUI Undergraduate

Bowen Li, Yue Qiu, Mu Xie, Qishen Zhou

Featured Project

# TEAM MEMBERS

Bowen Li (bowenli5)

Qishen Zhou (qishenz2)

Yue Qiu (yueq4)

Mu Xie (muxie2)

# PROBLEM

I-clicker is a useful teaching assistant tool used in undergraduate school to satisfy the requirement of course digitization and efficiency. Nowadays, most of the i-clickers used on campus have the following problems: inconsistency, high response delay, poor signal, manual matching. We are committed to making an i-clicker for our ZJUI Campus, which is economical, using 2.4G Wi-Fi signal connection, and on the computer to achieve matching. At the same time, it has to deal with the drawbacks as mentioned above.

# SOLUTION OVERVIEW

Compared with wired machines and mobile phone software, wireless i-clickers have the following advantages: they are easy to carry, they can accurately match and identify user tags, they are difficult to cheat and would not distract students. A wireless voting system consists of a wireless i-clicker, a wireless receiver on the administrator side, and a corresponding software program. In order to solve the problem of signal reception which is common in schools, we decided to use 2.4GHz Wi-Fi signal for data transmission. In addition, different from other wireless voting devices that carry out identity confirmation and bind identity information on the hardware side, we decided to make an identity binding system on the software side, and at the same time return it in the hardware unit for customer confirmation.

# SOLUTION COMPONENTS

A mature i-clicker should have a hardware part and a software part. The hardware part needs economical and effective hardware logic design. These include the storage and transportation of user key signals through a single chip computer program, a simple LCD1602 display to provide immediate feedback, a 2.4GHz Wi-Fi transmit-receive device for many-to-one wireless signal transmission, and a beautiful shell design. While the software component includes the conversion of hardware signals to software signals, a mature voting system, authentication of device owners, and signal return to hardware systems.

## SCM HARDWARE LOGIC SYSTEM:

Use SCM to compile the LCD module, return user input value. STC89C52RC can easily do this. Pass data to the NRF wireless transmission module.

## WIRELESS 2.4G SIGNAL TRANSMISSION SYSTEM:

A wireless signal detector should be a many-to-one signal transmission system. Bluetooth is one-to-one and Radio frequency is expensive. So, Wi-Fi signal transmission is the best choice. Each detector should load a transmitter and a receiver to transmit data to the administrator and get the data transmitted by the software.

## HARDWARE-TO-SOFTWARE SIGNAL TRANSFER SYSTEM:

A Hard-to-Soft system is necessary in any similar design. We should write a driver to process data.

## SOFTWARE DATA PROCESSING SYSTEM:

Software ought to process the data signal accurately and generate feedback to each i-clicker. Specifically, a software is needed in our design. The administrator can get user data and display it visually through statistical charts. This system should also have the function to associate user information to their answer. This is designed to score. A return signal should also be designed here. Users can receive feedback on their detector screen.

## USER IDENTIFICATION SYSTEM ON SOFTWARE:

Give an internal ID number to each i-clicker. Bind identity information (such as NetID, Student number) to i-clicker internal ID number on the software. Users can get their binding information on their screen by pushing a specific button. This data will be reset when a new packet is returned by the administrator.

## 3D PRINT SHELL:

A beautiful shell that fits the hardware system is needed. The shell should not be too large and the buttons must fit into the hardware.

# CRITERION FOR SUCCESS

Stability: Signal should be received easily. Signal loss inside a room shouldn’t occur, especially when there is a gap of two chairs.

Affordability: I-clickers should have a low cost. This facilitates mass production and popularization on campus.

Efficiency: The process from keystroke to signal collection and transmission shouldn’t have a high delay.

Beauty: Shell design should be accepted widely and be accessible to 3D printing.

Feedback: Users should get the feedback from the administrator easily. This is useful in arousing study enthusiasm of students.

Concurrency: The system should handle signals from a great deal of students in a short period correctly.

# DISTRIBUTION OF WORK

Qishen Zhou: Software data processing system and user information identification system.

Bowen Li: Hardware-to-software data transfer system and SCM hardware logic system.

Yue Qiu: Wireless signal transmission system and processing the data returned from the administrator.

Mu Xie: 3D print shell design and physical setup for the hardware part.