In this class, we will discuss both classical and current research on natural language understanding and generation for embodied AI (both virtual agents and robots) What are the main challenges in getting systems to follow (or give) instructions or engage in a dialogue with human users in real or simulated 3D spaces, including games such as Minecraft? What tasks, scenarios, and platforms are suitable for research in this domain? How far can LLMs get us? Where do we have go to beyond them?


35% In-Class Paper Presentation and Report
50% Research Project
10% Paper Reviews
5% Class Participation

Schedule and Syllabus

(subject to further revisions)

Date Topic
01/24 Introduction pdf
01/26 Research in Julia's group pdf
01/31 Research in Julia's group pdf
02/02 Relevant Linguistic Concepts pdf
02/07 Relevant Linguistic Concepts pdf
02/09 Project Discussions
02/14 /td>
02/16 Project Resources pdf
02/21 pdf
02/28 pdf
03/01 MC-Saar-Instruct, IGLU, Large Language Models as Minecraft Agents pdf
03/06 MineDojo, Craft an Iron Sword CerealBar pdf
03/08 pdf
03/20 pdf
03/22 pdf
03/27 pdf
03/29 pdf
04/03 pdf
04/05 pdf
04/10 pdf
04/12 pdf
04/17 pdf
04/19 pdf
04/24 pdf
04/26 pdf
05/01 pdf

Online resources

NLP papers: ACL anthology