Project

# Title Team Members TA Documents Sponsor
9 Image acquisition, 3D reconstruction and a visual interactive digital heritage system
Chuanrui Chen
Denglin Cheng
Qianyan Shen
Ziying Li
 design_document1.pdf
design_document2.pdf
final_paper1.pdf
final_paper2.pdf
proposal1.pdf
 Shurun Tan
Spring 2024 ECE445 RFA

Image acquisition, 3D reconstruction and a visual interactive digital heritage system

# TEAM MEMBERS:

- Qianyan Shen (qianyan2)

- Ziying Li (ziyingl4)

- Chuanrui Chen (cc86)

- Denglin Cheng (denglin3)

# Problem

Cultural artifacts possess significant historical, cultural, and artistic value. However, due to the passage of time and the impact of natural deterioration, many artifacts face risks of damage, loss, or decay. Additionally, for history enthusiasts and researchers worldwide, detailed information about specific artifacts is not readily accessible.

Traditional photographs often fail to capture the intricate details of artifacts, hampering comprehensive research and preservation efforts. Furthermore, the absence of user-friendly interactive interfaces limits the interaction between enthusiasts and artifacts, impeding immersive experiences in virtual exploration of cultural heritage.

Therefore, our team aims to develop a system that can generate realistic 3D models of cultural artifacts and provide users with a user-friendly interactive interface for immersive exploration.

# Solution Overview

Our system will use advanced scanning and 3D reconstruction techniques to capture the detailed geometry of cultural artifacts. This will be achieved through a series of subsystems including a Stabilized Scanning Subsystem, 3D Reconstruction Subsystem, Database Subsystem, and Interactive Interface Subsystem. Please refer to the following subsystem descriptions for more detailed information.

# Solution Components

## Stabilized Scanning Subsystem
This subsystem aims to capture detailed 3D data of the workpiece with high precision and low noise by coordinating a self-stabilizing three-axis gimbal centered around the STM32 microcontroller.
We intend to use solidworks to build the three axis parts of the gimbal respectively, and print them out with a high-precision 3D printer, and then use the brushless motor to connect these parts, and control them with the STM32 code, so that it can achieve real-time angular correction, so that in the process of scanning can be done to achieve the lens anti-shake, reduce motion blur.

## 3D Reconstruction Subsystem
This subsystem aims to obtain a point cloud through RGBD images and perform 3D reconstruction using the point cloud.
We first use a depth camera to capture RGBD images of an object from different angles and preprocess the raw images by denoising and repairing. Then, we proceed with point cloud acquisition, registration, and reconstruction to obtain a 3D model.
To begin, we calibrate the camera to obtain the lens parameters. We then convert the 2D coordinate system of the depth image to a 3D point cloud and map the pixel colors from the RGB image to the 3D point cloud. Afterward, we process the obtained point cloud by applying denoising and sampling techniques, facilitating subsequent registration and reconstruction steps. By repeating these processes, we obtain point clouds from different angles, and we perform precise registration using the ICP （Iterative Closest Point） method to align them in a unified coordinate system. Finally, the 3D reconstruction is completed using the Poisson reconstruction algorithm or other techniques.

## Database Subsystem
This subsystem aims to store the basic information of the artifacts, including dynasties, historical backgrounds, stories, etc., and at the same time saving the generated complex 3D model data.
With database system, users can upload the information of artifacts from all over the world to the database, and can also retrive and view the artifacts from exotic countries. When a user wants to retrieve an artifact, the database will find the corresponding information from its own stored data according to the search item entered by the user and display it through the Interactive Interface Subsystem for users to view artifacts from around the globe.


## Interactive Interface Subsystem
This subsystem aims to provide a user-friendly interface that facilitates database interaction and basic visualization capabilities, delivering a visually pleasing experience to users and catering to their close-range viewing needs.

We aim to present brief introductions of multiple cultural artifacts on the interface, including physical photos, names, dynasties, and more. Upon selection, users can access the corresponding detailed information and the reconstructed 3D model by linking to the database. Specifically, we render the obtained 3D models and offer features such as rotation and scaling for users to observe the artifact's details. Additionally, the interface can include a filtering function to provide users with a certain degree of personalized service in selecting artifacts.

# Criterion for Success
Successfully captures information about the appearance of artifacts without requiring the user to manually adjust examples or angles to minimize the noise.
Accurate and detailed 3D scanning and reconstruction of artifacts.
A database subsystem for effective data management and data retrieval.
A user-friendly interactive interface provides an immersive experience in cultural heritage exploration.


# Divisions Of Labor And Responsibilities
Denglin Cheng is responsible for the modeling of the Stabilized Scanning Subsystem, 3D printing, and the design of the control circuits in the STM32, as well as the final assembly and debugging of the gimbal to ensure smooth scanning of the depth camera.

Qianyan Shen is responsible for RGBD image preprocessing, point cloud acquisition, alignment, and 3D reconstruction.

Ziying Li is responsible for enabling database system to store and retrive data and interact with front-end.

Chuanrui Chen is responsible for the specific design and implementation of the UI interface, requiring her to understand and utilize the database interface. She also assists in the acquisition of point clouds from RGBD images and the design of the control circuits in the STM32.




Clickers for ZJUI Undergraduate

Bowen Li, Yue Qiu, Mu Xie, Qishen Zhou

Featured Project

# TEAM MEMBERS

Bowen Li (bowenli5)

Qishen Zhou (qishenz2)

Yue Qiu (yueq4)

Mu Xie (muxie2)

# PROBLEM

I-clicker is a useful teaching assistant tool used in undergraduate school to satisfy the requirement of course digitization and efficiency. Nowadays, most of the i-clickers used on campus have the following problems: inconsistency, high response delay, poor signal, manual matching. We are committed to making an i-clicker for our ZJUI Campus, which is economical, using 2.4G Wi-Fi signal connection, and on the computer to achieve matching. At the same time, it has to deal with the drawbacks as mentioned above.

# SOLUTION OVERVIEW

Compared with wired machines and mobile phone software, wireless i-clickers have the following advantages: they are easy to carry, they can accurately match and identify user tags, they are difficult to cheat and would not distract students. A wireless voting system consists of a wireless i-clicker, a wireless receiver on the administrator side, and a corresponding software program. In order to solve the problem of signal reception which is common in schools, we decided to use 2.4GHz Wi-Fi signal for data transmission. In addition, different from other wireless voting devices that carry out identity confirmation and bind identity information on the hardware side, we decided to make an identity binding system on the software side, and at the same time return it in the hardware unit for customer confirmation.

# SOLUTION COMPONENTS

A mature i-clicker should have a hardware part and a software part. The hardware part needs economical and effective hardware logic design. These include the storage and transportation of user key signals through a single chip computer program, a simple LCD1602 display to provide immediate feedback, a 2.4GHz Wi-Fi transmit-receive device for many-to-one wireless signal transmission, and a beautiful shell design. While the software component includes the conversion of hardware signals to software signals, a mature voting system, authentication of device owners, and signal return to hardware systems.

## SCM HARDWARE LOGIC SYSTEM:

Use SCM to compile the LCD module, return user input value. STC89C52RC can easily do this. Pass data to the NRF wireless transmission module.

## WIRELESS 2.4G SIGNAL TRANSMISSION SYSTEM:

A wireless signal detector should be a many-to-one signal transmission system. Bluetooth is one-to-one and Radio frequency is expensive. So, Wi-Fi signal transmission is the best choice. Each detector should load a transmitter and a receiver to transmit data to the administrator and get the data transmitted by the software.

## HARDWARE-TO-SOFTWARE SIGNAL TRANSFER SYSTEM:

A Hard-to-Soft system is necessary in any similar design. We should write a driver to process data.

## SOFTWARE DATA PROCESSING SYSTEM:

Software ought to process the data signal accurately and generate feedback to each i-clicker. Specifically, a software is needed in our design. The administrator can get user data and display it visually through statistical charts. This system should also have the function to associate user information to their answer. This is designed to score. A return signal should also be designed here. Users can receive feedback on their detector screen.

## USER IDENTIFICATION SYSTEM ON SOFTWARE:

Give an internal ID number to each i-clicker. Bind identity information (such as NetID, Student number) to i-clicker internal ID number on the software. Users can get their binding information on their screen by pushing a specific button. This data will be reset when a new packet is returned by the administrator.

## 3D PRINT SHELL:

A beautiful shell that fits the hardware system is needed. The shell should not be too large and the buttons must fit into the hardware.

# CRITERION FOR SUCCESS

Stability: Signal should be received easily. Signal loss inside a room shouldn’t occur, especially when there is a gap of two chairs.

Affordability: I-clickers should have a low cost. This facilitates mass production and popularization on campus.

Efficiency: The process from keystroke to signal collection and transmission shouldn’t have a high delay.

Beauty: Shell design should be accepted widely and be accessible to 3D printing.

Feedback: Users should get the feedback from the administrator easily. This is useful in arousing study enthusiasm of students.

Concurrency: The system should handle signals from a great deal of students in a short period correctly.

# DISTRIBUTION OF WORK

Qishen Zhou: Software data processing system and user information identification system.

Bowen Li: Hardware-to-software data transfer system and SCM hardware logic system.

Yue Qiu: Wireless signal transmission system and processing the data returned from the administrator.

Mu Xie: 3D print shell design and physical setup for the hardware part.