Project

# Title Team Members TA Documents Sponsor
3 High Noon Sheriff Robot
Instructor's Choice
 Shuting Shao
Yilue Pan
Youcheng Zhang
Yuan Xu
 Yutao Zhuang design_document1.pdf
final_paper1.pdf
proposal1.pdf
 Timothy Lee
# MEMBERS:

- Yuan Xu [yuanxu4]
- Shuting Shao [shao27]
- Youcheng Zhang [yz64]
- Yilue Pan [Yilvep2]

# TITLE:

HIGH NOON SHERIFF ROBOT

PROBLEM:

Nowadays with the increasing number of armed attacks and shooting incidents. The update for public places needs to be put on the agenda. Obviously, we could not let police and security to do all the jobs since humans might neglect some small action of threat behind hundreds of people and could not respond quickly to the threat. A second of hesitation might cost an innocent life. Our team aims on making some changes to this situation since nothing is higher than saving lifes not only victims but also gunners. We find some ideas in the Old western movies when two cowboys are going to a high noon duel, the sheriff will pull out the revolver quicker than the other and try to warn him before everything is too late. If we can develop a robot that can detect potential threats and pull out weapons first in order to warn the criminal to abandon the crime or use non-lethal weapons to take him down if he continues to pull out his gun.

# SOLUTION OVERVIEW:

In order to achieve effective protection in a legal way, we have developed the idea of a security robot. The robot can quickly detect dangerous people and fire a gun equipped with non-lethal ammunition to stop dangerous events.
The robot should satisfy the following behavioral logic:

- When the dangerous person is acting normally and there is no indication of impending danger, the robot should remain in standby mode with its robot arm away from the gun.
- When the dangerous person is in a position ready to draw his gun or other indication of dangerous behavior, the robot is also in a drawn position and its arm is already clutching the gun.
- When the dangerous person touches his gun, The robot should immediately draw the gun, move the hammer and finish aiming and firing to control the dangerous person. This type of robot would need to include three subsystems: Detection system, Electrical Control system, and Mechanical system.
# SOLUTION COMPONENTS:

## [SUBSYSTEM #1: DETECTION SUBSYSTEM]

This subsystem consists of a camera and PC. We are going to use YOLO v5 to detect object, determine the position of human and the gun. Use DeepSORT to track the object, let the camera follow the opponent. Use SlowFast to detect opponent’s behavior.

## [SUBSYSTEM #2: ELECTRICAL CONTROL SYSTEM]

This subsystem consists of a STM32, two high speed motors, two gimbal motors, one motor for revolver action and position sensor. The STM32 serves as the controller for the motors. The high speed motor will be used to move the mechanical grab to grab the revolver and pull it out as fast as possible so that it will use the position sensor as the end stop point instead of PID control. The gimbal motors serve as Yaw and Pitch motion for the revolver to control the accuracy of the revolver so that it needs encoders to give the angle feedback.

## [SUBSYSTEM #3: MECHANICAL SYSTEM]

This subsystem consists of a three-degree-of-freedom robot arm and a clamping mechanism fixed to the end of the arm. The clamping mechanism is used to achieve the gripping of the gun, the moving of the hammer and the pulling of the trigger. The mechanical arm is used to lift and aim the gun.

# CRITERION FOR SUCCESS

- Move Fast. The robot must draw its gun and aim faster than the opponent；
- Warning First. If opponent’s hand moves close to the gun on his waist, the robot should draw the gun and aim it at the opponent without firing. If the opponent gives up drawing a gun and surrender, the robot should put its gun back in place. Otherwise, the robot will shoot at the opponent.
- Accurate shooting. Under the premise that the opponent may move, the robot must accurately shoot the opponent's torso.
# DISTRIBUTION OF WORK

- EE Student Shuting Shao: Responsible for object detection and object tracking.
- EE Student Yuan Xu: Responsible for behavior detection and video processing.
- EE Student Youcheng Zhang: Responsible for electrical control system.
- ME Student Yilue Pan: Responsible for the Mechanical system.

Autonomous Behavior Supervisor

Shengjian Chen, Xiaolu Liu, Zhuping Liu, Huili Tao

Featured Project

## Team members

- Xiaolu Liu (xiaolul2)

- Zhuping Liu(zhuping2)

- Shengjian Chen(sc54)

- Huili Tao(huilit2)

## Problem:

In many real-life scenarios, we need AI systems not only to detect people, but also to monitor their behavior. However, today's AI systems are only able to detect faces but are still lacking the analysis of movements, and the results obtained are not comprehensive enough. For example, in many high-risk laboratories, we need to ensure not only that the person entering the laboratory is identified, but also that he or she is acting in accordance with the regulations to avoid danger. In addition to this, the system can also help to better supervise students in their online study exams. We can combine the student's expressions and eyes, as well as his movements to better maintain the fairness of the test.

## Solution Overview:

Our solution for the problem mentioned above is an Autonomous Behavior Supervisor. This system mainly consists of a camera and an alarm device. Using real-time photos taken by the camera, the system can perform face verification on people. When the person is successfully verified, the camera starts to monitor the person's behavior and his interaction with the surroundings. Then the system determines whether there is a dangerous action or an unreasonable behavior. As soon as the system determines that there are something uncommon, the alarm will ring. Conversely, if the person fails verification (ie, does not have permission), the words "You do not have permission" will be displayed on the computer screen.

## Solution Components:

### Identification Subsystem:

- Locate the position of people's face

- Identify whether the face of people is recorded in our system

The camera will capture people's facial information as image input to the system. There exists several libraries in Python like OpenCV, which have lots of useful tools. The identification progress has 3 steps: firstly, we establish the documents of facial information and store the encoded faceprint. Secondly, we camera to capture the current face image, and generate the face pattern coding of the current face image file. Finally, we compare the current facial coding with the information in the storage. This is done by setting of a threshold. When the familiarity exceeds the threshold, we regard this person as recorded. Otherwise, this person will be banned from the system unless he records his facial information to our system.

### Supervising Subsystem

- Capture people's behavior

- Recognize the interaction between human and object

- Identify what people are doing

This part is the capture and analysis of people's behavior, which is the interaction between people and objects. For the algorithm, we decided initially to utilize that based on VSG-Net or other developed HOI models. To make it suitable for our system or make some improvement, we need analysis and adjustment of the models. For the algorithm, it is a multi-branch network: Visual Branch: extracting visual features from people, objects, and the surrounding environment. Spatial Attention Branch: Modeling the spatial relationship between human-object pairs. Graph Convolutional Branch: The scene was treated as a graph, with people and objects as nodes, and modeling the structural interactions. This is a computational work that needs the training on dataset and applies to the real system. It is true that the accuracy may not be 100% but we will try our best to improve the performance.

### Alarming Subsystem

- Staying normal when common behaviors are detected

- Alarming when dangerous or non-compliant behaviors are detected

It is an alarm apparatus connected to the final of our system, which is used to report dangerous actions or behaviors that are not permitted. If some actions are detected in supervising system like "harm people", "illegal experimental operation", and "cheating in exams", the alarming system will sound a warning to let people notice that. To achieve this, a "dangerous action library" should be prepared in advance which contains dangerous behaviors, when the analysis of actions in supervising system match some contents in the action library, the system will alarm to report.

## Criteria of Success:

- Must have a human face recognition system and determine whether the person is in the backend database

- The system will detect the human with the surrounding objects on the screen and analyze the possible interaction between these items.

- Based on the interaction, the system could detect the potentially dangerous action and give out warnings.

## DIVISION OF LABOR AND RESPONSIBILITIES

All members should contribute to the design and process of the project, we meet regularly to discuss and push forward the process of the design. Each member is responsible for a certain part but it doesn't mean that this is the only work for him/her.

- Shengjian Chen: Responsible for the facial recognition part of the project.

- Huili Tao: HOI algorithm modification and apply that to our project

- Zhuping Liu: Hardware design and the connectivity of the project

- XIaolu Liu: Detail optimizing and test of the function.

Project Videos