In this class, we will discuss both classical and current research on natural language understanding and generation for embodied AI (both virtual agents and robots) What are the main challenges in getting systems to follow (or give) instructions or engage in a dialogue with human users in real or simulated 3D spaces, including games such as Minecraft? What tasks, scenarios, and platforms are suitable for research in this domain? How far can LLMs get us? Where do we have go to beyond them?
35% In-Class Paper Presentation and Report
50% Research Project
10% Paper Reviews
5% Class Participation
(subject to further revisions)
| Date | Topic | ||
| 01/24 | Introduction | ||
| 01/26 | Research in Julia's group | ||
| 01/31 | Research in Julia's group | ||
| 02/02 | Relevant Linguistic Concepts | ||
| 02/07 | Relevant Linguistic Concepts | ||
| 02/09 | Project Discussions | ||
| 02/14 | /td> | ||
| 02/16 | Project Resources | ||
| 02/21 | |||
| 02/23 | |||
| 02/28 | |||
| 03/01 | MC-Saar-Instruct, IGLU, Large Language Models as Minecraft Agents | ||
| 03/06 | MineDojo, Craft an Iron Sword CerealBar | ||
| 03/08 | |||
| 03/20 | |||
| 03/22 | |||
| 03/27 | |||
| 03/29 | |||
| 04/03 | |||
| 04/05 | |||
| 04/10 | |||
| 04/12 | |||
| 04/17 | |||
| 04/19 | |||
| 04/24 | |||
| 04/26 | |||
| 05/01 |