The 4th Credit Hour
If you are taking this class for four hours credit, you will have to complete a literature review.
We won't accept group work.
You will need to read and describe several (5-7) NLP papers
on a particular task or topic, and produce a written report that compares and critiques these approaches.
Please read our Academic Integrity Policy. Your literature survey must also be your own work.
The report must be written in your own words. You must properly cite all sources that have been
consulted, including papers, text books, websites and existing software.
You are not allowed to use tools like ChatGPT for any assignments in this class, except for the purpose of generating output that an assignment requires you to analyze.
Details
The purpose of the 4th credit hour literature review is to get you to read, understand, and critique original research papers in Natural Language Processing.
Your review should focus on 4–5 NLP papers that are connected in some way, for example because they address the same task or problem, or because they use similar techniques for different tasks or problems. We will give you a LaTeX Template that you will need to use to write your review in. We expect your final review to be 6-8 pages.
The best reviews are guided by some sort of question that you wish to answer (e.g. "which linguistic features help for semantic role labeling?", "how can NLP be used to identify fake news in social media?", "how well do large language models generate English?", "what datasets exists for NLP in the biological domain, and how do they differ?", "what linguistic structures do neural sequence models learn?", "how well does information extraction work in Hindi", "does commonsense knowledge help in question answering", etc.). The focus of your question needs to be on NLP. Topics that are not well suited for this class include questions such as "how do companies apply sentiment analysis in their products" (that is not really a topic of research), "can we use RNNs to model the Dow Jones" (the Dow Jones is not natural language), etc. We encourage you to consider questions that you personally find interesting (e.g. because you look at NLP for your mother tongue, or because you just find the topic fascinating).
Note that you can review the same paper in very different ways, depending on the question that you wish to address in your review (one review might focus on an algorithmic contribution, another on a particular evaluation technique, or a dataset, etc.) We also encourage you to look beyond papers that have been published in the last few years. You may find a lot of interesting ideas and insights in older works, even if the modeling techniques are perhaps obsolete.
Your review should 1.) clearly describe the question you wish to answer, 2.) summarize the aspects of each paper that are relevant to your question (in enough detail that your reader doesn't need to have read the original paper), 3.) discuss how these papers address your question (and how they differ in addressing this question, if appropriate), and 4) give your own assessment (e.g. a critique of the work described in the papers, and/or suggestions for future work to address your question.
We will provide you with a list of a few broad topics with some recent papers. There are many other good topics that you could address (we will try to add more to this list, but this list will never be exhaustive), and you should see the listed papers more as starting points than as a prescribed set that you should include in your review.
If you choose your own set of papers, the majority of the papers you discuss need to be published in the ACL anthology, although one or two of your papers can also come from other venues (AAAI, IJCAI, NeurIPS/NIPS, ICML, KDD and related conferences often have papers that are on or related to NLP, and if you focus on multimodal NLP, you might find papers in vision conferences like CVPR, ICCV, ECCV etc.). You will need to submit the pdf of your report here.
Timeline
Briefly, here is what we are planning to do:
In week 4, we will release a list of potential topics and lists of papers associated with each topic, and you will need to let us know which topic you choose. You can also propose your own topic, but Prof. Hockenmaier will need to first approve your suggestion and list of papers.
By week 12, you will then have to submit a draft review on which you will get feedback from your peers.
For your final report (at the end of week 14), you will have to submit a report that describes each of the papers in detail, and also critiques them. This report will also be peer graded (so you will have to grade three reports of your peers as well).
Grading Rubric
Here is the rubric for the final literature review that will be used. You can get up to 100 points. 4 points are for the file format (please use a PDF that was formatted according to our template), 40 points are to check that the length of your review is appropriate (7-8 pages [or more, if you want to], reviewing 4-5 NLP papers, or papers that have a high relevance to NLP). 10 points are for style and presentation, and the rest is for content.
Note that we will check your papers for plagiarism (to the web, the original papers, and to your peers). Plagiarism is a violation of the student code. We understand that a literature review might use expressions that are taken from the original papers, and that is okay for technical definitions, for example, but please do not copy entire sentences or paragraphs from the original papers unless you clearly indicate that you are quoting them, and do not copy from your peers or other places on the web.
Basics: File format
How many papers that are of relevance to an NLP class did this review discuss in detail? (Note that some of these papers might appear inconferences in related fields such as machine learning, AI, computer vision, etc. That is completely fine, as long as the paper still has relevance to NLP).
Any good paper contains an introduction that briefly states what this paper is about. The introduction may also give an overview of the structure of the rest of the paper (although this is a matter of personal preference how detailed this overview is). This question is more about the structure of the document than about the content of the introduction. The next question will address the content of the introduction.
A great literature review should have a clear topic or research questions, and the topic or research question should be well explained and motivated (even if the reader may disagree with the motivation). That is, it should be clear to the reader why they should (or perhaps should not!) read this review, and what they will learn by reading it. Note that motivation does not mean that you are trying to "sell" your topic. What matters is whether you make it clear what the reader will get out of your review. This is typically explained in the introduction or in a separate section that follows the introduction.
A great literature review provides enough technical background, so that even a reader who is unfamiliar with the topic that is being discussed can follow it. This may mean that important technical terms that are highly relevant are (briefly) explained, or that papers are placed in a broader context.
A great literature review should make it very clear why each of the discussed papers was included. Why are these papers worthy of a review? What can the reader learn from them? How has a particular paper been important? (We understand that you may have just chosen a random set of papers for this exercise, so our standards here are less stringent than they would be if we were deciding whether to publish your review, but you should still explain how each papers fits into your narrative. And please don't just say that an author or a topic are "famous"; that is actually irrelevant in scholarly discourse)
A great literature review should of course do a great job at actually describing the papers that it discusses (without copying the text from the original papers verbatim). Note that even a great review might choose to focus on only part of a paper that it discusses, because not the entire paper is of relevance to the topic of the review. That's completely okay (as long as the review is cohesive).
A great literature review should go beyond just describing what was done in the papers that were discussed, and add something extra: an assessment,interpretation, analysis or discussion. Obviously, you are just beginning to learn about the topics that are discussed in these papers, so this is challenging, but you should at least attempt to go beyond just listing what is in each paper.
In a great literature review, the contribution of each paper to the topic or research question is clearly described after the paper has been summarized. That is, what have we learned by discussing this paper about our overall question? Or how does this differ from what we saw in an earlier paper?
A great literature review shouldn't just describe each paper in isolation. Is there any overarching discussion, analysis or conclusion that ties the individual papers together (either as they are being introduced or at the end of the review)
A great literature review contains a complete bibliography of all the works it cites, and the references are complete enough so that the reader can find the papers (authors and titles are not sufficient; there needs to be a year, and a venue where this was published; page numbers are less important/irrelevant for some online venues, so we won't hold it against you if you don't include them for other papers as well, although in a real publication you should).
A well-written paper has explicit structure (in the form of sections/subsections with clear titles) and tries to provide cohesiveness to the reader by guiding them through the paper.
A great literature review is well presented; tables and figures have captions, are labeled and referred to in the body of the paper,
- 0 pts The student did not submit a PDF file.
- 2 pts The student submitted a PDF file that was not formatted in LaTeX or used a different template than the one provided.
- 4 pts The student submitted a PDF file that was formatted in LaTeX and that uses the provided template.
- 5 pts The review consists of only one or two pages (not counting references)
- 10 pts The review consists of only three or four pages (not counting references)
- 15 pts The review consist of only five or six pages (not counting references)
- 20 pts The review consists of seven to eight papers (not counting references) or more
How many papers that are of relevance to an NLP class did this review discuss in detail? (Note that some of these papers might appear inconferences in related fields such as machine learning, AI, computer vision, etc. That is completely fine, as long as the paper still has relevance to NLP).
- 0 pts None
- 5 pts Only a single paper (that is relevant to NLP) was discussed in detail.
- 10 pts Only two papers (that are relevant to NLP) were discussed in detail.
- 15 pts Only three papers(that are relevant to NLP) were discussed in detail.
- 20 pts (At least) four or five papers(that are relevant to NLP) were discussed in detail.
Any good paper contains an introduction that briefly states what this paper is about. The introduction may also give an overview of the structure of the rest of the paper (although this is a matter of personal preference how detailed this overview is). This question is more about the structure of the document than about the content of the introduction. The next question will address the content of the introduction.
- 0 pts This paper has no introduction, but goes straight into one of the papers.
- 1 pt The introduction in this paper is very short (e.g. it just mentions the topic in one sentence, and then the next section starts discussing the first paper).
- 2 pts The introduction in this paper is very clear on what this paper is about.
A great literature review should have a clear topic or research questions, and the topic or research question should be well explained and motivated (even if the reader may disagree with the motivation). That is, it should be clear to the reader why they should (or perhaps should not!) read this review, and what they will learn by reading it. Note that motivation does not mean that you are trying to "sell" your topic. What matters is whether you make it clear what the reader will get out of your review. This is typically explained in the introduction or in a separate section that follows the introduction.
- 0 pts It was unclear what this review was about or why this topic matters
- 2 pts It was only somewhat clear what this review was about or why this topic matters
- 4 pts It was very clear what this review was about and why this topic matters.
A great literature review provides enough technical background, so that even a reader who is unfamiliar with the topic that is being discussed can follow it. This may mean that important technical terms that are highly relevant are (briefly) explained, or that papers are placed in a broader context.
- 0 pts The paper did not attempt to provide any technical background that would help the reader understand the papers that are discussed
- 2 pts The paper only provided some, but not enough, technical background that would help the reader understand the papers that are discussed
- 4 pts The paper provided enough technical background to help the reader understand the papers that are discussed.
A great literature review should make it very clear why each of the discussed papers was included. Why are these papers worthy of a review? What can the reader learn from them? How has a particular paper been important? (We understand that you may have just chosen a random set of papers for this exercise, so our standards here are less stringent than they would be if we were deciding whether to publish your review, but you should still explain how each papers fits into your narrative. And please don't just say that an author or a topic are "famous"; that is actually irrelevant in scholarly discourse)
- 0 pts It was unclear why any of these papers were included or why the reader should care about them.
- 2 pts It was only somewhat clear why these papers were included or why the reader should care about them (or it was clear, but only for a subset of the papers)
- 4 pts It was very clear for each of the papers that were discussed in depth why it was included and what the reader should care about them.
A great literature review should of course do a great job at actually describing the papers that it discusses (without copying the text from the original papers verbatim). Note that even a great review might choose to focus on only part of a paper that it discusses, because not the entire paper is of relevance to the topic of the review. That's completely okay (as long as the review is cohesive).
- 0 pts The discussed papers were poorly described, or the description was copied (more or less) verbatim from the papers.
- 4 pts The author made an attempt to describe the papers in their own words, but the description was not very clear. The reader gets confused, or is missing important details.
- 8 pts The author made an attempt to describe the papers in their own words, and the description was fairly clear. The reader can mostly follow, and is not missing too many important details.
- 12 pts The author made an attempt to describe the papers in their own words, and the description was very clear. The reader can follow completely, and is getting all the important details.
A great literature review should go beyond just describing what was done in the papers that were discussed, and add something extra: an assessment,interpretation, analysis or discussion. Obviously, you are just beginning to learn about the topics that are discussed in these papers, so this is challenging, but you should at least attempt to go beyond just listing what is in each paper.
- 0 pts The author did not attempt to provide any further interpretation, analysis or discussion of any of the papers
- 2 pts The author attempted to provide a little further interpretation, analysis or discussion of the papers, but should have done more.
- 4 pts The author attempted to provide a sufficient amount of further interpretation, analysis or discussion of the papers.
In a great literature review, the contribution of each paper to the topic or research question is clearly described after the paper has been summarized. That is, what have we learned by discussing this paper about our overall question? Or how does this differ from what we saw in an earlier paper?
- 0 pts There is no attempt to tie the papers to the overall research question or topic
- 2 pts There is some attempt to tie the papers to the overall research question or topic, but it is unclear what each paper contributes to the discussion.
- 4 pts This paper clearly states for each paper how it contributes to the research question.
A great literature review shouldn't just describe each paper in isolation. Is there any overarching discussion, analysis or conclusion that ties the individual papers together (either as they are being introduced or at the end of the review)
- 0 pts No, the review just describes each paper in isolation.
- 4 pts Yes, the review contains an overall discussion or conclusion, but it is very brief or perfunctory.
- 8 pts Yes, the review contains an overall discussion or conclusion that really ties everything together
A great literature review contains a complete bibliography of all the works it cites, and the references are complete enough so that the reader can find the papers (authors and titles are not sufficient; there needs to be a year, and a venue where this was published; page numbers are less important/irrelevant for some online venues, so we won't hold it against you if you don't include them for other papers as well, although in a real publication you should).
- 0 pts This paper does not have any bibliography
- 1 pt The bibliography only includes a few of the works discussed in the paper, or the references are so incomplete that a reader would not be able to identify the original papers.
- 3 pts The bibliography includes all (or most) of the works discussed in the paper (it may omit a reference to a dataset or software package; it's fine if websites appear only in footnotes), and the reader is able to identify the original papers
- 4 pts The bibliography includes all (or most) of the works discussed in the paper (it may omit a reference to a dataset or software package; it's fine if websites appear only in footnotes), as well as additional background references where appropriate, as well as references to important background papers. The reader is able to identify the original papers.
- 0 pts It seemed like this paper was not proofread at all. There were so many grammatical errors (including run-on sentences, incomplete sentences, etc.) and typos that the writing was difficult to follow in many places.
- 2 pts It seemed like this paper needed more proofreading. There were a few grammatical errors (including run-on sentences, incomplete sentences, etc.)or typos, but the writing was still understandable throughout the paper.
- 4 pts This paper seemed to be carefully proofread. There were (almost) no grammatical errors (including run-on sentences, incomplete sentences, etc.) or typos, and the writing felt polished.
A well-written paper has explicit structure (in the form of sections/subsections with clear titles) and tries to provide cohesiveness to the reader by guiding them through the paper.
- 0 pts This review has no overall structure or cohesiveness. There are no sections or subsections, and/or there is no attempt to guide the reader through the document, so it is really unclear how the different parts of this paper are connected
- 2 pts This review does not have enough overall structure or cohesiveness. There may be clearly labeled sections, but it is only somewhat clear how the different parts of this paper are connected (perhaps the author tried to draw connections between the different parts, but this either did not come across, or it did not make much sense).
- 4 pts This review has a clear overall structure and is very cohesive. It is very clear how the different parts of this paper are connected., and the reader is clearly guided through the document.
A great literature review is well presented; tables and figures have captions, are labeled and referred to in the body of the paper,
- 0 pts The figures and tables in this review (if present)do not have captions and are not referred to in the body of the paper
- 1 pt The figures and tables in this review (if present)do not have captions or are not referred to in the body of the paper
- 2 pts The figures and tables in this review (if present)have captions and are referred toand described in the body of the paper