UniTN: Grounded Language Processing 22-23

Grounded Language Processing 22-23

The Grounded Language Processing course is taught by Raffaella Bernardi (UniTN), the TA is Alberto Testoni. Classes are on Tuesdays (13:00-15:00), Wednesdays (13:00-15:00) and Thursdays (15:00-17:00) in Rovereto, Palazzo Fedrigotti, Corso Bottini 31, 3rd floor (aula seminari).

The course is part of the degree in Artificial Intelligent Systems, but any student at UniTN interested in the topic can attend it as a Free Choice Course -- following the rules of the Program in which he/she is enrolled.

UniTn students, who are interested in attending the course but cannot attend it in presence, are welcome to email me -- we plan to teach the course using a digital board so to facilitate virtual participation.

If you are planning to attend the course, please add info about you in this form, it will help us planning the course better.

What this course is about This course focuses on the new emerging field of Grounded Language Processing (GLP), a subarea of AI that studies the connection between natural language, perception and action in the world. It gives students an overview of recent advances by revisiting also the long standing challenges set by the AI community at its start. It makes connections between natural language processing (NLP) and computer vision and robotics. It covers both grounded Natural Language Understanding and grounded Natural Language Generation and unified architecture for these two crucial components of AI agents. If time allows, the course ends by providing students with hints towards connection between GLP and Robotics and by comparing humans’ neural representations and attention mechanisms behind grounded NL and State-of-the-Art multimodal models.

Each main section consists both of frontal and hands-on experience.

Prerequisites: The course presupposes knowledge in Machine Learning, Natural Language Processing and possibly Computer Vision.
Grading Criteria: paper review 15%, presentation of a research question and its SOTA 35%, Project 50%. Details can be found here

Main Surveys

Papers for Reading Groups

  1. Michael J. Mayo (2003) Symbol Grounding and its Implications for Artificial Intelligence
  2. Lazaridou, Bruni and Baroni Is this a wampimuk? Cross-modal mapping between distributional semantics and the visual world ACL 2014
  3. Douwe Kiela, Alexis Conneau, Allan Jabri, Maximilian Nickel (2018) Learning Visually Grounded Sentence Representations
  4. Jize Cao, Zhe Gan, Yu Cheng, Licheng Yu, Yen-Chun Chen, Jingjing Liu Behind the Scene: Revealing the Secrets of Pre-trained Vision-and-Language Models
  5. Oriol Vinyals, Alexander Toshev, Samy Bengio, Dumitru Erhan Show and Tell: A Neural Image Caption Generator
  6. Will Monroe, Robert X.D. Hawkins, Noah D. Goodman and Christopher Potts Colors in Context: A Pragmatic Neural Model for Grounded Language Understanding
  7. Anderson, P., He, X., Buehler, C., Teney, D., Johnson, M., Gould, S., & Zhang, L. (2018). Bottom-up and top-down attention for image captioning and visual question answering. In Proceedings of the IEEE conference on computer vision and pattern recognition (pp. 6077–6086).
  8. Jesse Thomason, Aishwarya Padmakumar, Jivko Sinapov, Nick Walker, Yuqian Jiang, Harel Yedidsion, Justin Hart, Peter Stone, Raymond J. Mooney (2019) Improving Grounded Natural Language Understanding through Human-Robot Dialog [pre-print]

Open Access Codes

  • Blog on ACL 2022 by Mubashara Akhtar

    • Other interesting papers

    Last modified: Tue Dec 6 14:21:05 CET 2022