# Title Team Members TA Documents Sponsor
3 AR Sandbox
Haowen Zheng
Haoze Gao
Qiran Pan
Yiheng Zhang
Timothy Lee
## Team members

Haoze Gao, haozeg2

Haowen Zheng, haowenz5

Qiran Pan, qiranp2

Yiheng Zhang, yihengz5

## Title Of Project

AR Sandbox Redesign

## Problem

Introducing a smart sandbox with augmented reality (AR) capabilities that projects contour maps in real-time onto the sand surface, making geography education for children not only informative but also significantly more enjoyable. However, currently available educational sandboxes are mostly cumbersome and limited to public spaces like activity centers rather than serving as personalized learning tools.

Furthermore, the existing AR projectors designed for sandboxes exhibit primitive features, characterized by a notably low refresh rate and harsh direct light. We are committed to addressing these drawbacks and are working towards the development of a new and improved AR sandbox. This innovative solution aims to overcome the limitations of bulkiness, offering a more accessible and personal learning experience. Additionally, we are focused on enhancing the AR functionality to deliver a smoother experience with higher refresh rates and reduced glare, ensuring a more comfortable and engaging educational tool for children.

## Solution Overview

We would develop a next-generation sandbox with augmented reality (AR) projection and interaction capabilities. In comparison to the popular versions available in the market, our AR projector is set to achieve a higher refresh rate, easier control without external touch screen, and the overall structure will be designed to be foldable while ensuring both high load-bearing capacity and stability.

## Solution Components

### Sensor Subsystem

- **RGBD** camera (ToF or structured light) and associated software for acquiring RGB image and processing depth information

### Processing Subsystem

With the use of GPU acceleration

- **Human body detection** to overcome the interference from human hands and head. With this to enable multi-user collaboration
- **User Interface** with gesture control. Use hand gestures to interact with the screen projected on sand.
- **Real-time topography rendering**: Constructing topography map from depth information with GPU acceleration

### Display Subsystem

- Displaying on sand requires high luminance projector and associated **calibration software**. The software needs to track for image alignment

### Structure Subsystem

- The sand table should be made of materials and designs with sufficient strength to carry sand and prevent people from damaging the wall of the sand table when in use.
- The sand table will be foldable, which will reduce the volume and facilitate carrying and storage.
- The sand table can be separated from the sand while folding, which will make the sand table more conducive to cleaning, increasing durability, and conducive to rapid deployment in different use scenarios.
- We will add an additional vibration device so that the sand surface can be quickly restored to level when necessary.

## Criterion For Success

For our criteria for success, we outline the following requirements:

1. Physical Structure: The sandbox must have a robust physical structure capable of fully supporting the weight of the sand without any leakage. It should also withstand lateral forces of around 40kg exerted by children pulling on the sides of the sandbox.
2. AR Projector: The projector should be capable of accurately projecting contour maps onto the sandbox with more than 1 people playing with sand at a refresh rate of higher than 30fps. To verify the correctness of the contour maps, we will artificially create distinctive landforms such as ridges, valleys, and saddles, and compare them with the projected contour maps to ensure accurate alignment.

VoxBox Robo-Drummer

Featured Project

Our group proposes to create robot drummer which would respond to human voice "beatboxing" input, via conventional dynamic microphone, and translate the input into the corresponding drum hit performance. For example, if the human user issues a bass-kick voice sound, the robot will recognize it and strike the bass drum; and likewise for the hi-hat/snare and clap. Our design will minimally cover 3 different drum hit types (bass hit, snare hit, clap hit), and respond with minimal latency.

This would involve amplifying the analog signal (as dynamic mics drive fairly low gain signals), which would be sampled by a dsPIC33F DSP/MCU (or comparable chipset), and processed for trigger event recognition. This entails applying Short-Time Fourier Transform analysis to provide spectral content data to our event detection algorithm (i.e. recognizing the "control" signal from the human user). The MCU functionality of the dsPIC33F would be used for relaying the trigger commands to the actuator circuits controlling the robot.

The robot in question would be small; about the size of ventriloquist dummy. The "drum set" would be scaled accordingly (think pots and pans, like a child would play with). Actuators would likely be based on solenoids, as opposed to motors.

Beyond these minimal capabilities, we would add analog prefiltering of the input audio signal, and amplification of the drum hits, as bonus features if the development and implementation process goes better than expected.