Project

# Title Team Members TA Documents Sponsor
15 Automated Pour-over Coffee Machine with Imitation Learning
Jie Wang
Jingyuan Huang
Rucheng Ke
William Qiu
design_document3.pdf
final_paper3.pdf
photo1.jpg
proposal2.pdf
Said Mikki
# RFA for Automated Pour-over Coffee Machine with Imitation Learning

# Problem

The art of pour-over coffee brewing, famous for its complex flavor and high quality, is heavily dependent on the skills and experience of a barista. This craftsmanship leads to variability in coffee quality due to human inconsistency. Additionally, it is challenging for common coffee enthusiasts to replicate professional barista techniques at home or in non-specialized settings.

# Solution Overview

We propose the development of **an intelligent Automated Pour-over Coffee Machine leveraging imitation learning algorithms**. This machine will mimic the techniques of professional baristas, ensuring consistency and high-quality in every cup. The project will involve designing a mechanical structure integrated with sensors and developing sophisticated software algorithms.

# Solution Components

## Component 1: Mechanical Design

- **Purpose:** To create a machine that can physically replicates the movements and precision of a barista.
- **Features:** An adjustable nozzle for water flow control, a mechanical arm for simulating hand movements, and a stable structure to house the coffee dripper.
- **Challenges:** Ensuring precise movement and durability of moving parts, and integrating the mechanical system with electronic controls for seamless operation.
- **Expectation:** A workable, fixed coffee machine first, then upgrade it.

## Component 2: Sensors and Data Collection

- **Purpose:** To gather precise data on barista techniques for the learning algorithm.
- **Features:** High-precision sensors capturing data on water flow, angle, speed, and trajectory during the pour-over process.
- **Challenges:** Accurately capturing the nuanced movements of a professional barista and ensuring sensor durability under varying conditions.

## Component 3: Imitation Learning Algorithm

- **Purpose:** To analyze and learn from the collected data, enabling the machine to replicate these actions.
- **Features:** Advanced algorithms processing visual and sensory data to mimic barista techniques, this requires to duplicate the state-of-the-art research result from Robotics field.
- **Challenges:** Developing an algorithm capable of adapting to different styles and ensuring it can be updated as it learns from new data.

## Optional Components:

- **Multimodal Origin Information Pre-Processing:** To adjust settings based on different coffee beans and grind sizes.
- **User Interface Design:** An intuitive interface for user customization and selection of coffee preferences.
- **ChatGPT Enhanced Custom Coffee Setting**: To make the machine more intelligent and like a human barista, SOTA artificial intelligence like LLMs should be involved to make it more a sort of an agent than a regular machine.

# Criterion for Success

- **Mechanical Precision:** The machine must accurately control water flow and replicate barista movements.
- **Algorithm Effectiveness:** The machine should consistently brew coffee that matches or surpasses the quality of a professional barista.
- **User Experience:** The interface should be user-friendly, allowing customization without overwhelming the user.
- **Reliability and Durability:** The machine should operate consistently over time with minimal maintenance.
- **Taste Test Approval:** The coffee produced must be favorably reviewed in taste tests against traditional pour-over coffee.

An Intelligent Assistant Using Sign Language

Qianzhong Chen, Howie Liu, Haina Lou, Yike Zhou

Featured Project

# TEAM MEMBERS

Qianzhong Chen (qc19)

Hanwen Liu (hanwenl4)

Haina Lou (hainal2)

Yike Zhou (yikez3)

# TITLE OF THE PROJECT

An Intelligent Assistant Using Sign Language

# PROBLEM & SOLUTION OVERVIEW

Recently, smart home accessories are more and more common in people's home. A center, which is usually a speaker with voice user interface, is needed to control private smart home accessories. But a interactive speaker may not be the most ideal for people who are hard to speak or hear. Therefore, we aim to develop a intelligent assistant using sign language, which can understand sign languages, interact with people, and act as a real assistant.

# SOLUTION COMPONENTS

## Subsystem1: 12-Degree-of-Freedom Bionic Hand System

- Two moveable joints every finger driven by 5-V servo motors

- The main parts of the hand manufactured with 3D printing

- The bionic hand is fixed on a 2-DOF electrical platform

- All of the servo motors controlled by PWM signals transmitted by STM32 micro controller

## Subsystem2: The Control System

- The controlling system consists of embedded system modules including the microcontroller, high performance edge computing platform which will be used to run dynamic gesture recognition model and more than 20 motors which can control the delicate movement of our bionic hand. It also requires a high-precision camera to capture the hand gesture of users.

## Subsystem3: Dynamic Gesture Recognition System

- A external camera capturing the shape, appearance, and motion of objective hands

- A pre-trained model to help other subsystems to figure out the meaning behind the sign language. To be more specific, at the step of objects detection, we intended to adopt YOLO algorithm as well as Mediapipe, a machine learning framework developed by Google to recognize different sign language efficiently. Considering the characteristic of dynamic gesture, we also hope to adopt 3D-CNN and RNN to build our models to better fit in the spatio-temporal features.

# CRITERION OF SUCCESS

- The bionic hand can move free and fluently as designed, all of the 12 DOFs fulfilled. The movement of single joint of the finger does not interrupt or be interrupted by other movements. The durability and reliability of the bionic hand is achieved.

- The controlling system needs to be reliable and outputs stable PWM signals to motors. The edge computing platform we choose should have high performance when running the dynamic gesture recognition model.

- Our machine could recognize different sign language immediately and react with corresponding gestures without obvious delay.

# DISTRIBUTION OF WORK

- Qianzhong Chen(ME): Mechanical design and manufacture the bionic hand; tune the linking between motors and mechanical parts; work with Haina to program on STM32 to generate PWM signals and drive motors.

- Hanwen Liu(CompE): Record gesture clips to collect enough data; test camera modules; draft reports; make schedules.

- Haina Lou(EE): Implement the embedded controlling System; program the microcontroller, AI embedded edge computing module and implement serial communication.

- Yike Zhou(EE): Accomplish object detection subsystem; Build and train the machine learning models.