Individual Progress Report

Description

The Individual Progress Report (IPR) is a chance to put your contributions to the team's progress in writing. The report will discuss not only the components and subsystems you have personally been responsible for, but what components you have helped work on as well. It is important to talk about the relation between your work and your teammates' work as well.

Importantly, we want to see what you have worked on, what works and doesn't, and how you are planning on overcoming your challenges.

Requirements and Grading

This report should be 5-12 pages of your own work. This means that you cannot take full paragraphs or sections from your Design Document, since that was a collaborative effort. The IPR Grading Rubric describes what we look for in grading this assignment. The requirements are expanded on below:

  1. General: Concise writing is encouraged, but it is important that all pertinent information is conveyed. All figures should be labeled and formatted consistently.
  2. Formatting: Please refer to the Final Report Guidelines for general writing guidelines, since the format of this report should be very similar to that of the final report. Note that each component of the Final Report may be tailored to the parts of the project the individual has been active in.
  3. Introduction: First, discuss what portion of the system you have been active in designing connects to which portion of a different subsystem, and how these interact to complete an overall objective. Then discuss what you have accomplished, what you are currently working on, and what you still have left to do.
  4. Design: Discuss the design work you have done so far. It is expected that you have done calculations and/or found relevant equations, created circuits for your parts of the project, and simulated / drawn schematics for your parts. You may have already, at a high level, discussed how your part fits into the rest of the project, but you should expand on the technical details and interface between your module(s) and the other modules of the project.
  5. Verification: Testing and verification is also very important. Make sure you describe each test that was performed and its procedure in detail, and give quantitative, meaningful results. Also describe tests that have yet to be performed. We should be convinced that if all your tests will pass, your part of the project will work.
  6. Conclusion: Discuss a plan and timeline for completing your responsibilities and your project as a whole. Also explain the ethical considerations of your project by consulting the IEEE Code of Ethics, ACM Code of Ethics, or another relevant Code of Ethics.
  7. Citations: You need citations. Cite sources for equations, Application Notes you referenced in your design, and any literature you used to help design or verify your work. If you checked something from another course's lecture slides, Google'd for things related to your project, or anything similar, then you have something you need to cite. At the very least, since you have talked about the ethical considerations of your project as it relates to a published code of ethics (e.g., IEEE or ACM), you should cite those!

Submission and Deadlines

The IPR should be submitted on Blackboard in PDF format by the deadline listed on the Course Calendar.

An Intelligent Assistant Using Sign Language

Qianzhong Chen, Howie Liu, Haina Lou, Yike Zhou

Featured Project

# TEAM MEMBERS

Qianzhong Chen (qc19)

Hanwen Liu (hanwenl4)

Haina Lou (hainal2)

Yike Zhou (yikez3)

# TITLE OF THE PROJECT

An Intelligent Assistant Using Sign Language

# PROBLEM & SOLUTION OVERVIEW

Recently, smart home accessories are more and more common in people's home. A center, which is usually a speaker with voice user interface, is needed to control private smart home accessories. But a interactive speaker may not be the most ideal for people who are hard to speak or hear. Therefore, we aim to develop a intelligent assistant using sign language, which can understand sign languages, interact with people, and act as a real assistant.

# SOLUTION COMPONENTS

## Subsystem1: 12-Degree-of-Freedom Bionic Hand System

- Two moveable joints every finger driven by 5-V servo motors

- The main parts of the hand manufactured with 3D printing

- The bionic hand is fixed on a 2-DOF electrical platform

- All of the servo motors controlled by PWM signals transmitted by STM32 micro controller

## Subsystem2: The Control System

- The controlling system consists of embedded system modules including the microcontroller, high performance edge computing platform which will be used to run dynamic gesture recognition model and more than 20 motors which can control the delicate movement of our bionic hand. It also requires a high-precision camera to capture the hand gesture of users.

## Subsystem3: Dynamic Gesture Recognition System

- A external camera capturing the shape, appearance, and motion of objective hands

- A pre-trained model to help other subsystems to figure out the meaning behind the sign language. To be more specific, at the step of objects detection, we intended to adopt YOLO algorithm as well as Mediapipe, a machine learning framework developed by Google to recognize different sign language efficiently. Considering the characteristic of dynamic gesture, we also hope to adopt 3D-CNN and RNN to build our models to better fit in the spatio-temporal features.

# CRITERION OF SUCCESS

- The bionic hand can move free and fluently as designed, all of the 12 DOFs fulfilled. The movement of single joint of the finger does not interrupt or be interrupted by other movements. The durability and reliability of the bionic hand is achieved.

- The controlling system needs to be reliable and outputs stable PWM signals to motors. The edge computing platform we choose should have high performance when running the dynamic gesture recognition model.

- Our machine could recognize different sign language immediately and react with corresponding gestures without obvious delay.

# DISTRIBUTION OF WORK

- Qianzhong Chen(ME): Mechanical design and manufacture the bionic hand; tune the linking between motors and mechanical parts; work with Haina to program on STM32 to generate PWM signals and drive motors.

- Hanwen Liu(CompE): Record gesture clips to collect enough data; test camera modules; draft reports; make schedules.

- Haina Lou(EE): Implement the embedded controlling System; program the microcontroller, AI embedded edge computing module and implement serial communication.

- Yike Zhou(EE): Accomplish object detection subsystem; Build and train the machine learning models.