Project

# Title Team Members TA Documents Sponsor
42 Human-Robot Interaction for Object Grasping with Visual Reality and Robotic Arms
 Jiayu Zhou
Jingxing Hu
Yuchen Yang
Ziming Yan
 design_document1.pdf
final_paper1.pdf
final_paper2.pdf
final_paper3.pdf
final_paper4.pdf
proposal3.pdf
 Gaoang Wang
Human-Robot Interaction for Object Grasping with Mixed Reality and Robotic Arms

#Team Members:

Student 1 jiayu9

Student 2 zimingy3

Student 3 yucheny8

Student 4 hu80

#Problem

Current robotic systems lack intuitive and seamless human-robot interaction for object manipulation. Traditional teleoperation methods often require complex controllers, making it difficult for users to interact naturally. With advancements in Mixed Reality (MR) and robotic systems, it is possible to develop an intuitive interface where users can manipulate objects in a virtual space, and a robotic arm replicates these actions in real-time. This project aims to bridge the gap between human intention and robotic execution by integrating MR with robotic grasping, enabling precise and efficient remote object manipulation.

#Solution

Our solution involves creating a Mixed Reality-based control system using Microsoft HoloLens, allowing users to interact with virtual objects via hand gestures. These interactions are then translated into real-world robotic grasping motions using a robotic arm. The system consists of three key subsystems: (1) Digital Twin Creation, (2) MR-based Interaction & Control, and (3) Robotic Arm Execution. This approach ensures seamless synchronization between virtual and real-world interactions, improving accessibility and usability for robotic object manipulation.

#Solution Components

Subsystem 1: Digital Twin Creation

This subsystem focuses on generating accurate 3D models of real-world objects for use in Mixed Reality.

Components:

RealityCapture Software – for photogrammetry-based 3D model generation.

Gaussian Splatting – for efficient and high-fidelity neural rendering of objects.

Camera (e.g., DSLR or Smartphone with high resolution) – to capture ~100 images per object.

Blender/Meshlab – for 3D model optimization and format conversion.

Unity with MRTK (Mixed Reality Toolkit) – to integrate digital twins into MR.

Subsystem 2: Mixed Reality Interaction & Control

This subsystem enables users to interact with digital twins via Microsoft HoloLens.

Components:

Microsoft HoloLens 2 – to provide an immersive MR experience.

MRTK (Mixed Reality Toolkit) in Unity – for hand tracking and object interaction.

Azure Kinect (optional) – for improved depth sensing and object recognition.

Custom Hand Gesture Recognition Algorithm – to detect and map user actions to grasping commands.

Subsystem 3: Robotic Arm Execution

This subsystem translates user interactions into real-world robotic grasping.

Components:

Robotic Arm (e.g., UR5, Kinova Gen3, or equivalent) – for object grasping.

ROS (Robot Operating System) with MoveIt! – for motion planning and control.

Unity-to-ROS Bridge (WebSocket or ROSBridge) – for communication between HoloLens and ROS.

Custom Grasping Algorithm – to ensure stable and efficient object manipulation.

External Camera for Robot Arm Reference – to assist with object localization and depth perception, improving grasping accuracy. This subsystem translates user interactions into real-world robotic grasping.

#Criterion for Success

--Successfully generate and import at least 10 digital twin objects into Mixed Reality.

--Users should be able to interact with objects using hand gestures tracked by HoloLens.

--The system should accurately map hand gestures to robotic arm movements in real-time.

--The robotic arm should replicate the grasping motion within 2 minutes of user interaction.

--Ensure seamless integration between MR and robotic control, with minimal latency.

--Conduct a successful live demonstration showing MR-based grasping and real-world execution.

Master Bus Processor

Featured Project

General Description

We will design a Master Bus Processor (MBP) for music production in home studios. The MBP will use a hybrid analog/digital approach to provide both the desirable non-linearities of analog processing and the flexibility of digital control. Our design will be less costly than other audio bus processors so that it is more accessible to our target market of home studio owners. The MBP will be unique in its low cost as well as in its incorporation of a digital hardware control system. This allows for more flexibility and more intuitive controls when compared to other products on the market.

Design Proposal

Our design would contain a core functionality with scalability in added functionality. It would be designed to fit in a 2U rack mount enclosure with distinct boards for digital and analog circuits to allow for easier unit testings and account for digital/analog interference.

The audio processing signal chain would be composed of analog processing 'blocks’--like steps in the signal chain.

The basic analog blocks we would integrate are:

Compressor/limiter modes

EQ with shelf/bell modes

Saturation with symmetrical/asymmetrical modes

Each block’s multiple modes would be controlled by a digital circuit to allow for intuitive mode selection.

The digital circuit will be responsible for:

Mode selection

Analog block sequence

DSP feedback and monitoring of each analog block (REACH GOAL)

The digital circuit will entail a series of buttons to allow the user to easily select which analog block to control and another button to allow the user to scroll between different modes and presets. Another button will allow the user to control sequence of the analog blocks. An LCD display will be used to give the user feedback of the current state of the system when scrolling and selecting particular modes.

Reach Goals

added DSP functionality such as monitoring of the analog functions

Replace Arduino boards for DSP with custom digital control boards using ATmega328 microcontrollers (same as arduino board)

Rack mounted enclosure/marketable design

System Verification

We will qualify the success of the project by how closely its processing performance matches the design intent. Since audio 'quality’ can be highly subjective, we will rely on objective metrics such as Gain Reduction (GR [dB]), Total Harmonic Distortion (THD [%]), and Noise [V] to qualify the analog processing blocks. The digital controls will be qualified by their ability to actuate the correct analog blocks consistently without causing disruptions to the signal chain or interference. Additionally, the hardware user interface will be qualified by ease of use and intuitiveness.