LING83600: Language Technology

Fall 2021

CUNY Graduate Center

Lecture: Wednesday 4:15-6:15, GC 9205

Office hours: Tuesday 2-4, GC 7400.01 (and online by request)

Synopsis

This course explores how computers process human language. Key technologies emphasized include neural networks, computational phonology, morphology, and syntax, machine translation, speech recognition, and speech synthesis. Students are expected to be familiar with basic linguistic notions like phoneme, morpheme, (syntactic) head, constituent, etc., and to be comfortable developing in the Python program language.

Objectives

Students will become familiar with the basic models guiding the aforementioned technologies. They will also learn to critically read the academic literature in speech and language technology.

Materials

We will use readings from several texts, including the draft 3rd edition of Jurafsky & Martin's Speech and Language Processing, Eisenstein's An Introduction to Natural Language Processing, and various papers from the ACL Anthology. Students are encouraged to use the Computational Linguistics Laboratory (7400.13) for practice and assignments.

Assignments

Assignments in this class are "labs", small software development projects accompanied by a write-up describing the general approach taken and any challenges encountered. Students be graded on their correctness and readability of their code and write-up. Students will use GitHub Classroom for assignment turn-in.

The final assignment will be an open-ended project. Students are encouraged to conceive of projects relevant to their research interests. Students should discuss project plans with the instructor during office hours to confirm that it is both feasible and of appropriate scope.

Grading

80% of students' grades will be derived from the assignments; the remaining 20% will be reserved for participation and attendance. Assignments must be submitted on time or will receive a 0 grade (barring a documented emergency).

Accommodations

The instructor will attempt to provide all reasonable accommodations to students upon request. If you believe you are covered under the Americans With Disabilities Act, you are encouraged to direct accommodations requests to Matthew G. Schoengood, Vice President for Student Affairs.

Attendance

Students are expected to attend all lectures and practica. The instructor is not responsible for reviewing materials missed to absence.

Integrity

In line with the Student Handbook policies on plagiarism, students are expected to complete their own work. The instructor reserves the right to refer violations to the Academic Integrity Officer.

Respect

Students are expected to be considerate of their peers and to treat them with respect during class discussions.

Schedule

(Approximate and subject to change)

8/25		Words; frequency	Slides Lecture	J&M 20.5.1	(Yang)
9/1		WordNet; embedding	Slides 1, 2 Lecture	J&M ch. 6, ch. 18, Eisenstein ch. 14	(Faruqui & Dyer, Garg et al.)
9/8	No class
9/15	No class
9/22		Neural networks I	Slides 1 2 Notebook Lecture Reading guide	Eisenstein ch. 2	(Goldberg ch. 2)
9/29	HW1 due [solution]	Neural networks II	Slides Lecture Reading guide	Eisenstein ch. 3, Goldberg ch. 3, ch. 4, ch. 5
10/6		Text classification	Lecture Demos
10/13		Sequences; phonology	Slides 1 2 3 4 Lecture	Goldberg ch. 14, ch. 15	(Lee et al., Luong et al.)
10/20		Morphology	Slides Lecture	Gorman & Sproat 2021 ch. 6	(Gorman et al. 2019)
10/27	HW2 due [solution]	CFGs; CFG parsing	Handout Slides Lecture	Eistenstein ch. 10	(J&M ch. 12, ch. 13)
11/3		Dependency parsing	Handout Lecture	Eisenstein ch. 11	(J&M ch. 14, ch. 15, Kübler et al. ch. 1-3)
11/10		Machine translation I	Slides Handout Lecture	Eisenstein ch. 18
11/17	HW3 due [solution]	Machine translation II	Handout Blog Lecture	Rush	(Vaswani et al.)
11/24	(Asynchronous option?)	Text normalization	Slides 1 2 Lecture	Zhang et al.	(Gorman & Sproat 2016, Gorman et al. 2021)
12/1		Speech recognition	Slides Lectures	Jelinek p. 4-5, J&M ch. 26
12/8		Speech synthesis	Slides Lecture		(van den Oord et al. 2016)
12/14	HW4 due [solution]

Final project ideas

Bibliography

Bird, S., Klein, E., and Loper, E. n.d. Natural Language Processing with Python. URL: https://www.nltk.org/book/.
Eisenstein, J. 2019. Introduction to Natural Language Processing. MIT Press.
Faruqui, M. and Dyer, C. 2015. Non-distributional word vector representations. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 2: Short Papers), pages 464-469.
Garg, N., Shiebinger, L., Jurafsky, D.. and Zou, J. 2018. Word embeddings quantify 100 years of gender and ethnic stereotypes. Proceedings of the National Academy of the Sciences 115(16): E3635-3644.
Goldberg, Y. 2017. Neural Network Methods for Natural Language Processing. Morgan & Claypool.
Gorman, K., McCarthy, A. D., Cotterell, R., Vylomova, E., Silfverberg, M., and Markowska, M. 2019. Weird inflects but OK: making sense of morphological generation errors. In Proceedings of the 23rd Conference on Computational Natural Language Learning, pages 140-151.
Gorman, K. and Sproat. R.. 2016. Minimally supervised number normalization. Transactions of the Association for Computational Linguistics 4: 507-519.
Gorman, K. and Sproat, R. 2021. Finite-State Text Processing. Morgan & Claypool.
Gorman, K., Kirov, C., Roark, B., and Sproat, R. 2021. Structured abbreviation expansion in context. In Findings of the Association for Computational Linguistics: EMNLP 2021, pages 995-1005.
Jelinek, F. 1997. Statistical Methods for Speech Recognition. MIT Press.
Jurafsky, D., and Martin, J. H. 2009. Speech and Language Processing. 2nd edition. Pearson.
Kübler, S., McDonald, R. and Nivre, J. 2009. Dependency Parsing. Morgan & Claypool.
Lee, J. L., Ashby, L. F.E., Garza, M. E., Lee-Sikka, Y., Miller, S., Wong, A., McCarthy, A. D., and Gorman, K. 2020. Massively multilingual pronunciation mining with WikiPron. In Proceedings of the 12th Language Resources and Evaluation Conference, pages 4223-4228.
Luong, T., Pham, H., and Manning, C. D. 2015. Effective approaches to attention-based neural machine translation. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 1412-1421.
van den Oord, A., Dielman, S., Zen, H., Simoyan, K., Vinyals, O., Graves, A. Kalchbrenner, K., Senior, A., and Kavukcuoglu, K. 2016. WaveNet: a generative model for raw audio. arXiv 1609.03499.
Rush, A. 2018. The annotated transformer. Blog post accessed 2021/11/17.
Yang, C.. 2013. Who's afraid of George Kingsley Zipf? Or: Do children and chimps have language? Significance 10:29-34.
Vaswani, A., Shazeer, N., Parmar, N. Uszkoreit, J., Jones, L., Gomez, A. N, Kaiser, Ł. and Polosukhin, I. 2017. Attention is all you need. In Advances in Neural Information Processing Systems 30, pages 5998-6008.
Zhang, H., Sproat, R., Ng, A. H., Stahlberg, F. Peng, X., Gorman, K., and Roark, B. 2019. 2019. Neural models of text normalization for speech applications. Computational Linguistics 45(2): 293-337.