CS598

CS598: 3D Vision

Time: Tue/Thurs 2:00-3:15pm

Location: Siebel 0216

teaser

Description

Modeling, understanding, and generating the 3D world are among the primary goals of computer vision. This course is intended for graduate students with a research interest in 3D vision. Various topics will be explored through a combination of lectures on fundamental techniques and concepts, as well as individual paper reading group discussions. Unlike conventional one-to-many seminar-style reading groups, this class will adopt a new approach called a Role-Playing Reading Group. In each lecture, a group of students will play various roles to provide a holistic perspective on a specific research topic or paper. Topics include 3D representations, monocular geometry, multi-view geometry, 3D and 4D scene understanding, and 3D content creation. The course project consists of a proposal, experiments, a report, and a final poster presentation.

Prerequisites: You should have a graduate-level understanding of computer vision (equivalent of CS543 or CS445+CS444 combined) , including camera models, image filters, two-view geometry, feature detection and matching, and recognition. You should also have a graduate-level understanding of machine learning (equivalent of CS446). You should be engaged in or interested in research in 3D vision

Course Objective: Through this course, you will gain a broad and deep knowledge of the state-of-the-art in 3D vision, demonstrated and reinforced by group presentations and write-ups. You will also gain experience in identifying and validating research directions, as well as hands-on skills in implementing various 3D vision algorithms.

Requirements: This advanced-level course aims to help you master advanced techniques and gain insights into cutting-edge research topics. It assumes that students already have prerequisite knowledge; hence, it is not intended to teach fundamental knowledge or basic skills in computer vision. The discussions and lectures are heavily research-oriented and are better suited for graduate students in research programs related to areas such as vision, AR/VR, and robotics. Weekly readings, group presentations, coding assignments, write-ups, active class participation, and deliverables related to the final project are required.

Participation: We expect you to attend all lectures and discussions in-person unless pressing and unforeseen conflicts arise. Conflicts that are persistent (e.g. registering for two classes with half an hour overlap) are not excused.

Syllabus: Please see this this link.

Slack Sign-Up: Please sign up using this link.

Grading:
  • 35%: Role-play Discussion
    • 25%: Non-Hacker Role-playing Presentations (5% each, tentative)
    • 10%: Hacker’s Deliverables (code, demo, presentation)
  • 40%: Final Project
    • 10% Proposal (3-4 pages)
    • 15% Poster & Demo Presentation
    • 15% Final Report (6-8 pages)
  • 10%: Research Topic Survey (4 pages)
  • 10%: Quiz (2.5% each)
  • 5%: Participation in class and on slack

Staff

Shenlong Wang
Instructor
Zhi-Hao Lin
TA

Office Hours: Thurs 3:15-4:30pm (Siebel 4124)

Schedule

Date Topic Lecture Team Paper for Discussion (Tentative) Notes Reference List (Tentative)
Intro & Fundamentals
Aug 27 (Tuesday) Intro, Logitics, and Overview; 3D Basics [slides] Shenlong
Aug 29 (Thursday) Image Formation & Camera Basics [slides] Shenlong Quiz-1: Rotation representations and Conversion
Sept 3 (Tuesday) Correspondence & Flow [slides] Shenlong
3D Modeling
Sept 5 (Thursday) Two-view geometry [slides] Shenlong Quiz-1 due
Sept 10 (Tuesday) 3D Modeling: Structure-from-Motion [slides] Shenlong Quiz-2: Linear system solvers
Sept 12 (Thursday) MVS & Depth Fusion TBD KinectFusion [slides]
Sept 17 (Tuesday) SLAM TBD DROID-SLAM [slides] Quiz-2 due (extended to Sept 18th)
Sept 19 (Thursday) Learning-based SFM TBD Camera as Rays [slides]
3D Representation
Sept 24 (Tuesday) 3D Representation Overview [slides] Shenlong
Sept 26 (Thursday) Mesh & Tedrahedron TBD DMTet [slides] Quiz-3: 3D Representation Conversion
Oct 1 (Tuesday) No class - Shenlong at ECCV
Oct 3 (Thursday) No class - Shenlong at ECCV Survey Due at Oct. 6
Oct 8 (Tuesday) Neural Volumes TBD DeepSDF [slides]
Oct 10 (Thursday) Deep Learning on 3D TBD PointTransformerV3 [slides]
Rendering & Inverse Rendering
Oct 15 (Tuesday) Rendering & Lighting Basics [slides] Zhi-Hao
  • Proposal due at Oct 15
Oct 17 (Thursday) Differentiable Rendering TBD NVDiffRast [slides]
Oct 22 (Tuesday) Neural Radiance TBD Gaussian Splats [slides]
Oct 24 (Thursday) Neural Surface TBD NeuS [slides]
Oct 29 (Tuesday) Neural Inverse rendering TBD Rendering synthetic objects into legacy photographs [slides]
Ill-posed 3D Reasoning
Oct 31 (Thursday) Monocular Geometry TBD Recovering surface layout from an image [slides]
Nov 5 (Tuesday) Few-View Geometry TBD Mast3R [slides]
Content Creation
Nov 7 (Thursday) 3D Editing TBD Enhancing Photorealism Enhancement [slides]
Nov 12 (Tuesday) No class (CVPR deadline)
Nov 14 (Thursday) No class (CVPR deadline)
Nov 19 (Tuesday) 3D Simulation TBD PhysGaussian [slides]
Nov 21 (Thursday) 3D Generation TBD Get3D [slides]
Dynamic 3D
Dec 3 (Tuesday) Parametric Articulated Shapes TBD SMPL [slides]
Dec 5 (Thursday) Dynamic 3D Understanding TBD Dynamic Fusion [slides]
Dec 10 (Tuesday) Final Presentation (Poster)
Dec 12 (Thursday) Final Project Due