CS598

CS598: 3D Vision

Time: Tue/Thurs 2:00-3:15pm

Location: Siebel 0216

teaser

Description

Modeling, understanding, and generating the 3D world are among the primary goals of computer vision. This course is intended for graduate students with a research interest in 3D vision. Various topics will be explored through a combination of lectures on fundamental techniques and concepts, as well as individual paper reading group discussions. Unlike conventional one-to-many seminar-style reading groups, this class will adopt a new approach called a Role-Playing Reading Group. In each lecture, a group of students will play various roles to provide a holistic perspective on a specific research topic or paper. Topics include 3D representations, monocular geometry, multi-view geometry, 3D and 4D scene understanding, and 3D content creation. The course project consists of a proposal, experiments, a report, and a final poster presentation.

Prerequisites: You should have a graduate-level understanding of computer vision (equivalent of CS543 or CS445+CS444 combined) , including camera models, image filters, two-view geometry, feature detection and matching, and recognition. You should also have a graduate-level understanding of machine learning (equivalent of CS446). You should be engaged in or interested in research in 3D vision

Course Objective: Through this course, you will gain a broad and deep knowledge of the state-of-the-art in 3D vision, demonstrated and reinforced by group presentations and write-ups. You will also gain experience in identifying and validating research directions, as well as hands-on skills in implementing various 3D vision algorithms.

Requirements: This advanced-level course aims to help you master advanced techniques and gain insights into cutting-edge research topics. It assumes that students already have prerequisite knowledge; hence, it is not intended to teach fundamental knowledge or basic skills in computer vision. The discussions and lectures are heavily research-oriented and are better suited for graduate students in research programs related to areas such as vision, AR/VR, and robotics. Weekly readings, group presentations, coding assignments, write-ups, active class participation, and deliverables related to the final project are required.

Participation: We expect you to attend all lectures and discussions in-person unless pressing and unforeseen conflicts arise. Conflicts that are persistent (e.g. registering for two classes with half an hour overlap) are not excused.

Syllabus: Please see this this link.

Slack Sign-Up: Please sign up using this link.

Grading:
  • 35%: Role-play Discussion
    • 25%: Non-Hacker Role-playing Presentations (5% each, tentative)
    • 10%: Hacker’s Deliverables (code, demo, presentation)
  • 40%: Final Project
    • 10% Proposal (3-4 pages)
    • 15% Poster & Demo Presentation
    • 15% Final Report (6-8 pages)
  • 10%: Research Topic Survey (4 pages)
  • 10%: Quiz (2.5% each)
  • 5%: Participation in class and on slack

Staff

Shenlong Wang
Instructor
Zhi-Hao Lin
TA

Office Hours: Thurs 3:15-4:30pm (Siebel 4124)

Schedule

Date Topic Lecture Team Paper for Discussion (Tentative) Notes Reference List (Tentative)
Intro & Fundamentals
Aug 27 (Tuesday) Intro, Logitics, and Overview; 3D Basics [slides] Shenlong
Aug 29 (Thursday) Image Formation & Camera Basics [slides] Shenlong Quiz-1: Rotation representations and Conversion
Sept 3 (Tuesday) Correspondence & Flow [slides] Shenlong
3D Modeling
Sept 5 (Thursday) Two-view geometry [slides] Shenlong Quiz-1 due
Sept 10 (Tuesday) 3D Modeling: Structure-from-Motion [slides] Shenlong Quiz-2: Linear system solvers
Sept 12 (Thursday) MVS & Depth Fusion TBD KinectFusion [slides]
Sept 17 (Tuesday) SLAM TBD DROID-SLAM [slides] Quiz-2 due (extended to Sept 18th)
Sept 19 (Thursday) Learning-based SFM TBD Camera as Rays [slides]
3D Representation
Sept 24 (Tuesday) 3D Representation Overview [slides] Shenlong
Sept 26 (Thursday) Mesh & Tedrahedron TBD DMTet [slides] Quiz-3: 3D Representation Conversion
Oct 1 (Tuesday) No class - Shenlong at ECCV
Oct 3 (Thursday) No class - Shenlong at ECCV Survey Due at Oct. 6
Oct 8 (Tuesday) Neural Volumes TBD DeepSDF
Oct 10 (Thursday) Deep Learning on 3D TBD fVDB
Rendering & Inverse Rendering
Oct 15 (Tuesday) Rendering & Lighting Basics Zhi-Hao
  • Proposal due at Oct 15
  • Quiz-4: Reflectance Model Summary
Oct 17 (Thursday) Differentiable Rendering TBD NVDiffRast
Oct 22 (Tuesday) Neural Radiance TBD Gaussian Splats Quiz-4 due
Oct 24 (Thursday) Neural Surface TBD NeuS
Oct 29 (Tuesday) Neural Inverse rendering TBD Rendering synthetic objects into legacy photographs
Ill-posed 3D Reasoning
Oct 31 (Thursday) Monocular Geometry TBD Recovering surface layout from an image
Nov 5 (Tuesday) Few-View Geometry TBD Mast3R
Content Creation
Nov 7 (Thursday) 3D Editing TBD Enhancing Photorealism Enhancement
Nov 12 (Tuesday) No class (CVPR deadline)
Nov 14 (Thursday) No class (CVPR deadline)
Nov 19 (Tuesday) 3D Simulation TBD PhysGaussian
Nov 21 (Thursday) 3D Generation TBD Get3D
Dynamic 3D
Dec 3 (Tuesday) Parametric Articulated Shapes TBD SMPL
Dec 5 (Thursday) Dynamic 3D Understanding TBD Dynamic Fusion
Dec 10 (Tuesday) Final Presentation (Poster)
Dec 12 (Thursday) Final Project Due