Project
| # | Title | Team Members | TA | Documents | Sponsor |
|---|---|---|---|---|---|
| 6 | SMART GLASSES FOR HANDS-FREE DATASHEET RETRIEVAL |
Hridik Hingorani Preity Varanasi Shiv Bahl |
Yushi Zhou | ||
| Team Members: Hridik Hingorani Shiv Bahl Preity Varanasi Problem When working with electronic hardware, engineers frequently need to stop what they are doing to identify a component, search for its datasheet, and locate specific information such as pinouts, voltage limits, timing specifications, or recommended operating conditions. This interrupts the workflow and is particularly inconvenient when the user is soldering, probing a circuit, or otherwise using both hands. Our goal is to reduce this interruption by creating a low-cost wearable system that can identify an electronic component being viewed by the user and automatically retrieve the corresponding datasheet. Solution We propose building a pair of smart glasses containing a camera and a custom embedded PCB. When the user looks at a component and activates the system, the camera will capture an image and send it wirelessly to a laptop. The laptop will use computer vision and OCR to extract identifying information such as the component's part number or package markings. Once the component has been identified, the laptop will locate the appropriate datasheet and process it so that important information can be retrieved quickly. The result will initially be displayed on the laptop. The system is intended primarily for clearly labeled electronic components such as integrated circuits, modules, sensors, and other devices whose identifying markings can be captured by the camera. High-Level System Smart Glasses Camera → Embedded PCB → Wi-Fi Communication ↓ Laptop Image Processing/OCR → Component Identification → Datasheet Retrieval → Datasheet Processing → User Output Hardware We will design a custom PCB mounted on or integrated into the glasses. The PCB will include: ESP32-based microcontroller Camera interface Wireless communication User input for triggering image capture Power regulation Battery/power management Necessary supporting circuitry The embedded system will be responsible for capturing images, managing the camera, handling user input, and wirelessly transmitting data to the laptop. The glasses themselves will be inexpensive commercially available frames modified to hold our electronics. Software The laptop-side software will: Receive the image from the glasses. Process the image to improve readability of component markings. Use OCR/computer vision to determine the component identifier. Search for and retrieve the correct datasheet. Parse the datasheet and make relevant specifications accessible to the user. Display the component identity, datasheet, and requested information on the laptop. The LLM will only answer questions using information retrieved from the identified component's datasheet rather than relying solely on its existing knowledge. Requirements For a successful final demonstration, the system should: Capture a usable image from the wearable camera. Wirelessly transfer the image from the glasses to the laptop. Correctly identify a predefined set of clearly marked electronic components. Retrieve the correct datasheet associated with the identified component. Extract and display relevant information from that datasheet. Operate using our custom PCB rather than a standalone commercial development board. Be wearable and operate without a wired connection between the glasses and laptop. We will create a test set of electronic components and quantitatively evaluate component-identification accuracy and end-to-end response time. Stretch Goal Our stretch goal is to make the system fully hands-free after component identification by adding voice interaction. The user would be able to ask questions such as: "What is the maximum supply voltage?" "What does pin 4 do?" "What value capacitor does the manufacturer recommend here?" The system would transcribe the question, search the retrieved datasheet, and provide an answer grounded specifically in that datasheet. Future Work A future version could integrate a small near-eye or AR display into the glasses so that information could be presented directly in the user's field of view. An AR display is not part of the scope of this semester's project. Complexity The project combines several independently testable hardware and software subsystems: Custom wearable PCB design Camera interfacing Battery and power-management circuitry Wireless embedded communication Image processing OCR/component identification Automated datasheet retrieval Datasheet parsing and information extraction Integration between the embedded hardware and laptop software A major technical challenge will be reliably extracting part markings from small electronic components under different viewing angles, distances, orientations, and lighting conditions. Uniqueness Existing smart glasses and visual assistants are generally designed for broad image recognition or general-purpose AI assistance. Our project is specifically designed around electronics work. Instead of simply describing what the camera sees, the system will identify a specific electronic component, locate its technical documentation, and provide information grounded in the manufacturer's datasheet. The project therefore combines a purpose-built wearable embedded platform with a specialized datasheet retrieval and processing pipeline. Scope The core project will focus on labeled electronic components and laptop-based output. We will not attempt to recognize every possible component or construct an AR display during this semester. Restricting the identification problem to a controlled but varied set of components allows us to quantitatively evaluate the system while still addressing the major technical challenges of wearable image acquisition, wireless communication, component identification, datasheet retrieval, and system integration. |
|||||