ECE 110/120 Honors Lab Section : Computer Vision Robotic Arm

Statement of Purpose

Our project is a robotic arm that can sense the motion of a moving object (like a human arm) and mirror its motions in a plane. The goals are to have a robotic arm with 2 joints that can detect patterns of human arm motions, and learn to use its 2 joints to mimic the motion. Objectives include building the mechanical arm, attaching it with motors and controlling the arm with Raspberry Pi's, and having computer vision / segmentation software for the arm to follow.

This project has applications in robotic helper situations (an example being if a robot has to assist with some task that is dangerous for a human, like diffusing a bomb). The machine learning aspect of this project makes it unique.

Background Research

For the computer vision aspect, we have split the problem into two parts: segmenting a human's arm (identifying where a human's arm is in a picture) and then getting the arm to move accordingly. We will use Simple CV Computer Vision software (and run it on a Python console) on the Raspberry Pi. We will also use the Raspberry Pi Camera Module to directly feed in data to the Simple CV software. Simple CV has motion detection features that compares successive frames - we can essentially find the arm of a human and use that to detect how the arm is moving (in a general sense it is moving up or down). We can then send signals to the arm's motors, accordingly. The motor controllers will be linked to the Raspberry Pi- this is all essentially one integrated system.

What drives us to work on this project is its applications in areas of robotic helpers and automation. We also want to get more acquainted with machine learning by doing an applications-based project. We were inspired by a similar project in a previous semester which accomplished something similar (waving), yet we plan to extend our arm to have two joints (two motors) to more accurately mimic an arm.

Block Diagram / Flow Chart

Block Diagram

Flowchart

System Overview

For the block diagram, the camera module is the only input for the system, and the motors are the only output in the system. The central computer is a Raspberry Pi 4, which can handle all the openCV libraries and will take in the input from the camera and process the movement, and then it will send the signals to the motors, which will make the corresponding movements.

In the flowchart, the program will start by polling and detecting for a human arm. Once it does detect a human arm, the camera will detect if there is motion, and if there isn't any motion, the program will keep polling. Once it does detect motion, if the motion is a hand making the pause symbol, the program will end, but if it doesn't detect the pause motion, it will analyze the motion, create a vector, and send the necessary signals to the motors for them to act upon.

Parts

Possible Challenges

One challenge will be controlling the two joint system. We would need to identify the forearm of a human and the upper portion of their arm, and have the robotic arm mimic both parts individually. Also, Simple CV's motion detection might not provide information about directions of motion, which makes this project a bit harder. 

Another major problem is figuring out how to convert the motion of the human hand detected by the camera module to a vector that the motors can understand and act upon.

References

Monk, Simon. "Computer Vision With The Raspberry Pi". O’Reilly Media, 2022, https://www.oreilly.com/content/raspberry-pi-cookbook-computer-vision/.

"Adafruit DRV8833 DC/Stepper Motor Driver Breakout Board". Adafruit Learning System, 2022, https://learn.adafruit.com/adafruit-drv8833-dc-stepper-motor-driver-breakout-board?view=all.

List all references you used in your proposal. This is important, you do not want to be blamed for plagiarism. IEEE citation format is highly recommended. You can use citethisforme.com's IEEE citation generator to painlessly generate your references in this style.