Project
| # | Title | Team Members | TA | Documents | Sponsor |
|---|---|---|---|---|---|
| 69 | A Comprehensive Approach to Tumor Detection using RGB, NIR, and Immersive 3D Visualization |
Amy He TJ Shapiro Zach Mizrachi |
Jason Zhang | design_document1.pdf final_paper1.pdf other1.pdf |
|
| ECE 445 Senior Design RFA A Comprehensive Approach to Tumor Detection using RGB, NIR, and Immersive 3D Visualization Team Members: - Zach Mizrachi (zdm3) - TJ Shapiro (tylers5) - Yue (Amy) He (yuehe4) # Problem The most widely used approach for tumor removal today is traditional surgery, which introduces a host of problems. This traditional method relies solely on the surgeon's visual and tactile feedback, which is subject to human error. The surgeon is also operating on his or her own view of the tumor, which is often limited when the tumor is not easily visible. All of the above can lead to excess damage being done to the patient in order to increase tumor visibility, or accidental damage caused by human error. # Solution We propose a camera system meant to assist a surgeon in their removal of a tumor. The system is intended to perform two main tasks: detect the tumor by segmenting it from the surrounding biological material, and reconstruct the detected tumor in 3D. The camera system is small and highly mobile, such as to allow the surgeon to view all areas of the tumor. The presented solution will improve the visual capabilities of the surgeon, allowing for continuous visualization and informed decision making. The setup: putting some fluorescent drug over the area of interest, and the tissue would reflect NIR light while the tumor wouldn’t. We use that to distinguish between the tumor area and the healthy area via the tumor-detecting pen system. This method has been validated in the pilot study. Specifically, we intend to visualize the operating surface in real time in the Apple Vision Pro, highlighting the tumor in augmented reality from the NIR. This will allow the surgeon to record the area of interest guided by the highlighting from NIR, contributing to more accurate photos for the tumor reconstruction. Then, in post processing, we will generate a 3D model of the tumor that will allow the surgeon to have a more detailed view of the region of surgical interest. ## Subsystem ### Casing Module - **Part Name:** 3D Print - **Part #:** N/A - **Protocol:** N/A - **Purpose of Part:** Hold all components rigidly together ### Imaging Module - **Part Name:** Beam Splitter **Part #:** Edmund Optics, Family ID #2185, Visible and NIR Plate Beamsplitters **Protocol:** N/A **Purpose of Part:** Take in Visible Light, split the beam into two equal beams - **Part Name:** NIR Filter **Part #:** 49950 - RT – Raman 785nm Laser Longpass Set **Protocol:** N/A **Purpose of Part:** Filter beam for NIR light - **Part Name:** NIR Sensor **Part #:** LI-OV5640-MIPI-AF-NIR **Protocol:** MIPI **Purpose of Part:** Record NIR signal - **Part Name:** RGB Filter **Part #:** Chroma 27040 - Lum **Protocol:** N/A **Purpose of Part:** Filter beam for RGB light - **Part Name:** RGB Sensor **Part #:** Digikey, 2289-LI-IMX185-MIPI-M12-ND **Protocol:** MIPI **Purpose of Part:** Record RGB signal - **Part Name:** Lens **Part #:** Edmund Optics, Family ID #1748, Uncoated Double-Convex (DCX) Lenses **Protocol:** N/A **Purpose of Part:** Focus the light on the camera sensors ### Processing Module - **Part Name:** NVIDIA Jetson - **Part #:** Digikey, 1597-102110417-ND - **Protocol:** MIPI - **Purpose of Part:** Image analysis and sensor fusion - **Additional Notes:** - Uses SPI to connect the device and the display - GPU is responsible for generating a 3D representation based on the input data - Storing the frames from real-time for later processing ### PCB Components - **Part Name:** IMU - **Part #:** Digikey LSM6DSO iNEMO™ - **Protocol:** SPI/I2C, and MIPI I3CSM serial interface - **Purpose of Part:** Record pose information for the camera via 'Structure from Motion' Algorithm. See Software Overview. ### Modeling Module - **Part Name:** Apple Vision pro - **Part #:**N/A - **Protocol:** N/A - **Purpose of Part:** Communicates with Mac, which is communicating with Jetson. Project 3D reconstruction of tumor detection/biological information via head-mounted display through augmented reality. This will be done using Apple’s proprietary VisionPro platform as well as SwiftUI and ARKit frameworks. ## Hardware Components Explain what the subsystem does. Explicitly list what sensors/components you will use in this subsystem. Include part numbers. We look to replicate a similar hardware setup to the parent study of this project. In this, we will work closely with Professor Gruev to ensure a feasible approach to the hardware system. Software Overview: For 3D models to be useful in a surgical scenario, we need the reconstruction to have high levels of detail. For this, we prioritize detail over real-time analysis, and look to implement an open source Structure from Motion algorithm. To further improve upon existing algorithms, we intend to fuse IMU data to eliminate the need to estimate camera pose. We believe this will improve the accuracy of our 3D models. Existing work has shown that implementing IMU within SFM is not only feasible but improves the robustness of 3D models for small objects. In this work, we look to follow a similar approach to existing literature. # Criterion For Success Describe high-level goals that your project needs to achieve to be effective. These goals need to be clearly testable and not subjective. This project can be separated into goals for 3 stages. Hardware: 3D Print a casing, allowing for adjustment of beam splitter distance to image sensors assemble all electrical components correctly Successfully integrate IMU with PCB Software Receive and validate all data on NVIDIA Jetson RGB Data NIR Data IMU Data Filter RGB images by using NIR region of interest Set up and run open SFM software on NVIDIA Improve SFM model with IMU Perform optimal frame selection using IMU Augmented/Virtual Reality Establish communication between Jetson and Vision Pro Set up pass through mode on Vision Pro, with NIR tumor highlighting View 3D SFM Point Cloud on Vision Pro Interact with Point Cloud on Vision Pro Each bullet here is a goal that we would like to achieve over the course of the semester. Given the difficulty of the task, we plan on utilizing the IMU to improve SFM as a final step, after the pipeline to the Vision Pro has been completed. |
|||||