Project

# Title Team Members TA Documents Sponsor
9 Image acquisition, 3D reconstruction and a visual interactive digital heritage system
Chuanrui Chen
Denglin Cheng
Qianyan Shen
Ziying Li
 design_document1.pdf
design_document2.pdf
proposal1.pdf
 Shurun Tan
Spring 2024 ECE445 RFA

Image acquisition, 3D reconstruction and a visual interactive digital heritage system

# TEAM MEMBERS:

- Qianyan Shen (qianyan2)

- Ziying Li (ziyingl4)

- Chuanrui Chen (cc86)

- Denglin Cheng (denglin3)

# Problem

Cultural artifacts possess significant historical, cultural, and artistic value. However, due to the passage of time and the impact of natural deterioration, many artifacts face risks of damage, loss, or decay. Additionally, for history enthusiasts and researchers worldwide, detailed information about specific artifacts is not readily accessible.

Traditional photographs often fail to capture the intricate details of artifacts, hampering comprehensive research and preservation efforts. Furthermore, the absence of user-friendly interactive interfaces limits the interaction between enthusiasts and artifacts, impeding immersive experiences in virtual exploration of cultural heritage.

Therefore, our team aims to develop a system that can generate realistic 3D models of cultural artifacts and provide users with a user-friendly interactive interface for immersive exploration.

# Solution Overview

Our system will use advanced scanning and 3D reconstruction techniques to capture the detailed geometry of cultural artifacts. This will be achieved through a series of subsystems including a Stabilized Scanning Subsystem, 3D Reconstruction Subsystem, Database Subsystem, and Interactive Interface Subsystem. Please refer to the following subsystem descriptions for more detailed information.

# Solution Components

## Stabilized Scanning Subsystem
This subsystem aims to capture detailed 3D data of the workpiece with high precision and low noise by coordinating a self-stabilizing three-axis gimbal centered around the STM32 microcontroller.
We intend to use solidworks to build the three axis parts of the gimbal respectively, and print them out with a high-precision 3D printer, and then use the brushless motor to connect these parts, and control them with the STM32 code, so that it can achieve real-time angular correction, so that in the process of scanning can be done to achieve the lens anti-shake, reduce motion blur.

## 3D Reconstruction Subsystem
This subsystem aims to obtain a point cloud through RGBD images and perform 3D reconstruction using the point cloud.
We first use a depth camera to capture RGBD images of an object from different angles and preprocess the raw images by denoising and repairing. Then, we proceed with point cloud acquisition, registration, and reconstruction to obtain a 3D model.
To begin, we calibrate the camera to obtain the lens parameters. We then convert the 2D coordinate system of the depth image to a 3D point cloud and map the pixel colors from the RGB image to the 3D point cloud. Afterward, we process the obtained point cloud by applying denoising and sampling techniques, facilitating subsequent registration and reconstruction steps. By repeating these processes, we obtain point clouds from different angles, and we perform precise registration using the ICP （Iterative Closest Point） method to align them in a unified coordinate system. Finally, the 3D reconstruction is completed using the Poisson reconstruction algorithm or other techniques.

## Database Subsystem
This subsystem aims to store the basic information of the artifacts, including dynasties, historical backgrounds, stories, etc., and at the same time saving the generated complex 3D model data.
With database system, users can upload the information of artifacts from all over the world to the database, and can also retrive and view the artifacts from exotic countries. When a user wants to retrieve an artifact, the database will find the corresponding information from its own stored data according to the search item entered by the user and display it through the Interactive Interface Subsystem for users to view artifacts from around the globe.


## Interactive Interface Subsystem
This subsystem aims to provide a user-friendly interface that facilitates database interaction and basic visualization capabilities, delivering a visually pleasing experience to users and catering to their close-range viewing needs.

We aim to present brief introductions of multiple cultural artifacts on the interface, including physical photos, names, dynasties, and more. Upon selection, users can access the corresponding detailed information and the reconstructed 3D model by linking to the database. Specifically, we render the obtained 3D models and offer features such as rotation and scaling for users to observe the artifact's details. Additionally, the interface can include a filtering function to provide users with a certain degree of personalized service in selecting artifacts.

# Criterion for Success
Successfully captures information about the appearance of artifacts without requiring the user to manually adjust examples or angles to minimize the noise.
Accurate and detailed 3D scanning and reconstruction of artifacts.
A database subsystem for effective data management and data retrieval.
A user-friendly interactive interface provides an immersive experience in cultural heritage exploration.


# Divisions Of Labor And Responsibilities
Denglin Cheng is responsible for the modeling of the Stabilized Scanning Subsystem, 3D printing, and the design of the control circuits in the STM32, as well as the final assembly and debugging of the gimbal to ensure smooth scanning of the depth camera.

Qianyan Shen is responsible for RGBD image preprocessing, point cloud acquisition, alignment, and 3D reconstruction.

Ziying Li is responsible for enabling database system to store and retrive data and interact with front-end.

Chuanrui Chen is responsible for the specific design and implementation of the UI interface, requiring her to understand and utilize the database interface. She also assists in the acquisition of point clouds from RGBD images and the design of the control circuits in the STM32.




High Noon Sheriff Robot

Yilue Pan, Shuting Shao, Yuan Xu, Youcheng Zhang

Featured Project

# MEMBERS:

- Yuan Xu [yuanxu4]

- Shuting Shao [shao27]

- Youcheng Zhang [yz64]

- Yilue Pan [Yilvep2]

# TITLE:

HIGH NOON SHERIFF ROBOT

PROBLEM:

Nowadays with the increasing number of armed attacks and shooting incidents. The update for public places needs to be put on the agenda. Obviously, we could not let police and security to do all the jobs since humans might neglect some small action of threat behind hundreds of people and could not respond quickly to the threat. A second of hesitation might cost an innocent life. Our team aims on making some changes to this situation since nothing is higher than saving lifes not only victims but also gunners. We find some ideas in the Old western movies when two cowboys are going to a high noon duel, the sheriff will pull out the revolver quicker than the other and try to warn him before everything is too late. If we can develop a robot that can detect potential threats and pull out weapons first in order to warn the criminal to abandon the crime or use non-lethal weapons to take him down if he continues to pull out his gun.

# SOLUTION OVERVIEW:

In order to achieve effective protection in a legal way, we have developed the idea of a security robot. The robot can quickly detect dangerous people and fire a gun equipped with non-lethal ammunition to stop dangerous events.

The robot should satisfy the following behavioral logic:

- When the dangerous person is acting normally and there is no indication of impending danger, the robot should remain in standby mode with its robot arm away from the gun.

- When the dangerous person is in a position ready to draw his gun or other indication of dangerous behavior, the robot is also in a drawn position and its arm is already clutching the gun.

- When the dangerous person touches his gun, The robot should immediately draw the gun, move the hammer and finish aiming and firing to control the dangerous person. This type of robot would need to include three subsystems: Detection system, Electrical Control system, and Mechanical system.

# SOLUTION COMPONENTS:

## [SUBSYSTEM #1: DETECTION SUBSYSTEM]

This subsystem consists of a camera and PC. We are going to use YOLO v5 to detect object, determine the position of human and the gun. Use DeepSORT to track the object, let the camera follow the opponent. Use SlowFast to detect opponent’s behavior.

## [SUBSYSTEM #2: ELECTRICAL CONTROL SYSTEM]

This subsystem consists of a STM32, two high speed motors, two gimbal motors, one motor for revolver action and position sensor. The STM32 serves as the controller for the motors. The high speed motor will be used to move the mechanical grab to grab the revolver and pull it out as fast as possible so that it will use the position sensor as the end stop point instead of PID control. The gimbal motors serve as Yaw and Pitch motion for the revolver to control the accuracy of the revolver so that it needs encoders to give the angle feedback.

## [SUBSYSTEM #3: MECHANICAL SYSTEM]

This subsystem consists of a three-degree-of-freedom robot arm and a clamping mechanism fixed to the end of the arm. The clamping mechanism is used to achieve the gripping of the gun, the moving of the hammer and the pulling of the trigger. The mechanical arm is used to lift and aim the gun.

# CRITERION FOR SUCCESS

- Move Fast. The robot must draw its gun and aim faster than the opponent；

- Warning First. If opponent’s hand moves close to the gun on his waist, the robot should draw the gun and aim it at the opponent without firing. If the opponent gives up drawing a gun and surrender, the robot should put its gun back in place. Otherwise, the robot will shoot at the opponent.

- Accurate shooting. Under the premise that the opponent may move, the robot must accurately shoot the opponent's torso.

# DISTRIBUTION OF WORK

- EE Student Shuting Shao: Responsible for object detection and object tracking.

- EE Student Yuan Xu: Responsible for behavior detection and video processing.

- EE Student Youcheng Zhang: Responsible for electrical control system.

- ME Student Yilue Pan: Responsible for the Mechanical system.