Project
| # | Title | Team Members | TA | Documents | Sponsor |
|---|---|---|---|---|---|
| 34 | Purrceptron |
Meg Naidu Neel Rajesh Vinny G Smith |
Denghan Xiong | ||
| # Purrceptron Team Members: - Megna Naidu (megnan2) - Neel Rajesh (neelrr2) # Problem Companionship and consistent, positive interaction have emotional and psychological benefits, which is why emotional support animals are widely used to help with stress, anxiety, and loneliness. However, owning a living emotional support animal isn't practical for many people for reasons like housing restrictions, allergies, financial limitations, travel schedules, and lack of time preventing people from being able to care for a real pet. Existing "virtual pet" apps and simple robotic toys don't fill this gap well either. Apps lack any physical presence, and most robotic pets on the market either follow scripted, non-adaptive behavior or are expensive assistive-care robots. There is a need for a low-cost, physically interactive companion that can build a responsive, evolving relationship with its user without the practical burdens of a living animal. # Solution The proposed solution is a robotic companion pet that senses and responds to physical interaction from its user and its environment, and that maintains an internal emotional/behavioral state which evolves over time based on how it is treated. Rather than a static toy or a purely digital pet, this device combines physical sensing and multimodal feedback (visual, audio, and physical/motion) so that interactions feel tangible and personalized. Positive, frequent interaction should make the pet display happier, more energetic behavior, while neglect over time should shift it toward more subdued or "lonely" behavior", creating a feedback loop that encourages the user to engage with it regularly, similar to caring for a real pet. At a high level, the system is composed of four subsystems: a Sensing Subsystem that detects user interaction and environmental input, a Control Subsystem (microcontroller + PCB) that processes sensor data and runs the pet's internal state/personality logic, a Feedback Subsystem that expresses the pet's current state through lights, sound, and movement, and a Power Subsystem that allows the device to operate untethered. # Solution Components ## Sensing Subsystem This subsystem is responsible for detecting how the user physically interacts with the pet (petting, tapping, holding) as well as ambient conditions (light, sound, motion/handling), which feed into the pet's internal state calculations. - Capacitive touch sensor — Adafruit MPR121 (12-channel capacitive touch breakout) for detecting petting/touch location and duration - IMU (accelerometer + gyroscope) — InvenSense MPU-6050 for detecting motion, shaking, being picked up, or being set down - Electret microphone with amplifier — Adafruit MAX9814 for detecting ambient sound/voice presence (amplitude/duration based, not speech recognition) ## Vision Subsystem Captures video of the user's face and posture and streams it over WiFi to the host device, where it is used to infer emotional state and body language. - Camera module with onboard WiFi — AI-Thinker ESP32-CAM (OV2640 2MP camera + ESP32-S), used purely for image capture and WiFi video streaming (MJPEG/HTTP or RTSP stream to host) ## Edge Controller Subsystem Reads sensor data, drives actuators/feedback, and handles WiFi communication with the host, but does not run the state machine or any perception itself, keeping the onboard compute footprint minimal. - Microcontroller — Espressif ESP32-WROOM-32 (sufficient GPIO/I2C/I2S for sensors and actuators, built-in WiFi for MQTT communication with host) - Real-time clock module — DS3231 RTC breakout, to timestamp interaction events accurately even across power loss - MQTT client (software, runs on ESP32) — publishes sensor events to host, subscribes to feedback commands from host Custom PCB integrating the microcontroller, sensor interconnects, and power regulation ## Remote Processing Subsystem Runs on a separate host device with significantly more compute than the onboard MCU, and is responsible for the two most processing-intensive tasks: reading the camera feed for emotion/body-language cues, and maintaining the pet's internal state (mood, energy, attachment) over time. Sends resulting feedback commands back to the Edge Controller over WiFi. - Host compute device — Raspberry Pi 4 Model B (4GB), running the perception and state-machine software (assumption: a small local host on the same network as the pet; a laptop could substitute during development) - MQTT broker — Mosquitto, running on the host, mediating all communication between Edge Controller, Vision Subsystem, and Remote Processing software - Facial expression / body-language inference — OpenCV + a lightweight pretrained facial emotion classification model, combined with MediaPipe Face/Pose landmarks for basic posture cues (leaning in, turning away, etc.); chosen specifically because it runs in real time on Pi-class hardware without needing a GPU or cloud service - State machine logic (software) — combines sensing events (touch/IMU/audio) and vision-derived emotion/posture signals into the pet's mood/energy/attachment state, and issues feedback commands to the Edge Controller ## Feedback Subsystem Expresses the pet's internal state back to the user through light, sound, and motion, giving it a readable "personality" in the moment. Driven directly by the Edge Controller based on commands received from the host. - RGB LED matrix — Adafruit 16x8 or similar (e.g., HT16K33-driven matrix) for simple animated facial expressions/eyes - Small speaker + audio amplifier — MAX98357A I2S Class-D amplifier driving a 3W 4Ω speaker, for chirps/tones representing emotional states - Micro servo motors — SG90 (x2–3) for simple physical movement (ear twitch, head tilt, tail wag) to add a physical expressiveness layer beyond lights/sound ## Power Subsystem Allows the pet to be handled and interacted with freely, untethered from a wall outlet. - 3.7V LiPo battery, 1200–2000 mAh (capacity to be finalized based on current draw testing, including WiFi/camera streaming load) - LiPo charging/protection circuit — TP4056 module with battery protection IC - 3.3V/5V voltage regulation — AMS1117 (or equivalent) to supply regulated power to both onboard microcontrollers and peripherals # Criterion For Success - The system must correctly register a touch input from the capacitive sensor and the Vision Subsystem must stream video to the host - The system must correctly classify at least 3 distinct physical interaction types (e.g., pet, tap, pick-up/shake) with at least 90% accuracy over 30 trials per interaction type. - The Remote Processing Subsystem must correctly classify at least 3 distinct facial emotion categories (e.g., smiling/positive, neutral, frowning/negative) with at least 80% accuracy against a labeled test set of at least 50 images/frames. - Round-trip latency from a sensor event on the Edge Controller to a corresponding feedback command being received back from the host must be under 300 ms, measured over at least 20 trials. - The pet's internal state (mood/energy) must persist correctly across a power cycle, verified by powering off, waiting a set interval, and confirming the state on power-up matches the expected decayed value within a defined tolerance. - The Feedback Subsystem must produce a visibly/audibly distinguishable output (LED pattern + sound) for at least 3 distinct internal states (e.g., happy, neutral, lonely), confirmed by a blind test where observers can correctly identify the displayed state at least 90% of the time. - The device must run untethered on battery power for a minimum of 2 continuous hours of intermittent interaction and active video streaming before requiring a recharge. - The custom PCB must successfully interface with all sensors and output peripherals with no more than a 5% sensor read failure/dropout rate over a 10-minute continuous test. |
|||||