Fall, 2023
|
|
|
|
|
|
|
||
Lecture |
Date |
Lecture Topics |
Readings and Resources |
Homeworks |
| 1 | T 9/5 | Administrative matters |
Syllabus: HTML Here is a useful list of posts on Medium.com concerning NLP: PDF. Medium is generally very good, and worth the $5 fee for a month of access; subscription is optional -- I will post PDFs of readings from Medium. |
Review tutorials as listed in Useful Links above, as needed. You can ignore the tutorials on Latex and Pandas for the present. |
| 2 | Th 9/7 | What is NLP? Slides: PDF |
Overview of NLP: This page is a decent overview of current NLP; you can stop when you see "IBM Watson." The two linked blog posts are worth reading as well. Chapter 1 from J&M second edition is here, and worth a look, especially for the history, and a fairly complete bibliography. (They have not yet revised this chapter for the third edition we will be using.) For more details, the Wikipedia page on NLP is also useful. Good overview of Machine Learning models: HTML Good overview of the normal NLP pipeline: HTML. |
HW 01: : IPYNB, ZIP, PDF (in case you want to refer to the sample outputs) Solution:IPYNB |
| 3 | T 9/12 | Basic Notions: Words, documents, corpora, and language models. Low-level processing of text data: regular expressions; stemming, lemmatization, normalization. Slides: PDF |
J&M Ch. 2 Here is a good summary of publicly-available corpora for a variety of NLP tasks: HTML (I like this blog very much, generally high quality posts.) The Unreasonable Effectiveness of Data When do Language Models Need a Billion Words in their Dataset? Here is a Python version of the Porter Stemmer, a well-known algorithm for stemming: TXT. The original paper is here: PDF. It is instructive to look through the code, which makes a number of passes over a text file, and applies rules to remove suffixes. |
|
| 4 | Th 9/14 | Language modeling: Bag of Words, Probabilistic LMs, Conditional Probabilities, N-grams
Slides: PDF |
J&M Ch. 3 |
HW 02: Wrangling text (a screenplay) into a useful data set; IPYNB, ZIP Due Th 9/21 |
| 5 | T 9/19 | Generative language models in detail: Smoothing; Perplexity;
Slides: PDF |
J&M Ch. 3 Two short, clear posts about perplexity. |
|
| 6 | Th 9/21 | Vector Models: BOW as Term Frequency; TF-IDF; Cosine Similarity;
Slides: PDF |
J&M Ch. 4 | HW 03: ZIP Due M 10/9 |
| 7 | T 9/26 | Vector Models Continued: Principal Component Analysis; Distributional semantics; Word embeddings and word2vec; Sentence and text embeddings
Slides: PDF |
J&M Ch. 6
Beautiful tutorial on Word2Vec and word embeddings: The Illustrated Word2Vec Excellent introduction to word embeddings: PDF Excellent series of articles on word2Vec: PDF |
|
| 8 | Th 9/28 | Introduction to Machine Learning; Unsupervised ML; Clustering with K-Means; (Hierarchical Clustering)
Slides: PDF |
Tutorial on Clustering and K-Means: HTML |
|
| 9 | T 10/3 | Prelude to supervised ML: Linear regression, Logistic regression; Classification using logistic regression; Implementing logistic regression: Loss functions, gradient descent, optimization
Slides: PDF |
J&M Ch. 5
Short summary of loss functions in ML: HTML Well-written and concise blog post on the math behind classification using LR: HTML Good summary of gradient descent algorithms: HTML Good summary of Loss Functions: HTML Excellent blog post about Optimizers: PDF |
|
| 10 | Th 10/5 | Supervised machine learning; Deep learning with Artificial Neural Networks; Applications of feedforward NNs in NLP; Classification with FFNNs
Slides: PDF Echo 360 Lecture recordings (full and slide view) are on YT. |
J&M Ch. 5 Stanford Tutorial on Deep Learning: HTML Outstanding tutorial and animation of how NN's work: Ch 1, Ch 2, Ch 3, Ch 4 Well-done article (with diagrams and an example) on backpropagation: HTML Good blog post about the Cross-Entropy Cost Function: HTML |
|
| T 10/10 | This day is a Monday schedule but I will hold a Zoom lecture from 6-8pm on T 10/10 on the typical workflow for developing deep learning projects in Pytorch. I intend this for those with less experience with machine learning, but unless you have previous experience with Pytorch, you should plan to attend or view the video.
Lecture Recording from Zoom: HTML Here are the two notebooks I discussed during the Zoom lecture: ZIP |
Here are some useful tutorials on Pytorch:
YouTube 30 minute introduction And here are the tutorial notebooks from Pytorch Pocket Reference: HTML |
||
| 11 | Th 10/12 | (Zoom Lecture) Classification Workflow in detail: Evaluation, Parameter Tuning, Generalization.
Slides: PDF |
J&M Ch. 7 | HW 04: Visualizing sentence and text embeddings with PCA; Classification with FFNNs: ZIP Be sure check out another great blog post by Jason Brownlee (also linked in the hw) which contains essential information for Problem One: HTMLHW 04 Problem 1 Walkthrougbh: HTML HW 04 Problem 2 Walkthrougbh: HTML Due F 10/27 |
| 12 | T 10/17 | Neural Networks for Sequence Data: RNNs, GRUs, LSTMs; Deep RNNs; Bidirectional RNNs
Slides: PDF Echo 360 Lecture recordings (full and slide view) are on YT. |
A blog post on RNNs well worth reading: The Unreasonable Effectiveness of Recurrent Neural Networks A very clear blog post introducing LSTMs: Understanding LSTM Networks |
|
| 13 | Th 10/19 | Applications of RNNs; POS tagging, Named Entity Recognition; Generative Models
Slides: PDF Echo 360 Lecture recordings (full and slide view) are on YT. |
||
| 14 | T 10/24 | RNN Generative Models: Character-level modeling; Word-level modeling, Beam Search; Word and Text Embeddings
Slides: PDF Echo 360 Lecture recordings (full and slide view) are on YT. |
CharacterLevelLSTM.ipynb
Readable research paper comparing various generation strategies with actual human generation: Neural Text Degeneration |
|
| 15 | Th 10/26 | Machine Translation; The Attention Mechanism
Slides: PDF Echo 360 Lecture recordings (full and slide view) are on YT. |
Good introduction to Attention: HTML Excellent talk (long, 48 mins) on Attention: HTML Useful article on Attention in RNNs with detailed example: PDF Some very useful data sets: Europarl translation dataset |
|
| 16 | T 10/31 | Advanced Features of Neural Networks
Slides: PDF Echo 360 Lecture recordings (full and slide view) are on YT. |
||
| 17 | Th 11/2 | Discussion of semester project; Project requirements: HTML Advanced features continued: 1D convolutional layers, Embedding layers, Network geometries for NLP, Pretrained networks. Slides: PDF Echo 360 Lecture recordings (full and slide view) are on YT. |
Good tutorial on Embedding Layers in Pytorch: HTML The list of 33 topics: PDF Here is a great summary of current topics of interest in NLP: PDF. If you are looking for a project idea, pick one of these and Google around to find appropriate blog posts. |
HW 05: RNNs: Generative models at character-level and word-level with beam search; POS tagging with Hidden Markov Models and RNNs: IPYNB, ZIP Viterbi Algorithm with option for Log Space: IPYNB, ZIP Milton Paradise Lost: TXT Due F 11/27 |
| 18 | T 11/7 | Attention and BRNNs; Transformers: Encoder
Slides: PDF Echo 360 Lecture recordings (full and slide view) are on YT. |
Clear introduction to positional encodings: PDF Short introduction to residual connections: HTML Great introduction to Transformers (with video version): The Illustrated Transformer Longer introduction with more algorithmic details: HTML Walkthrough of Pytorch code for transformers: HTML Hugging Face Transformers library (very complete): HTML |
|
| Th 11/9 | Lecture cancelled | |||
| 19 | T 11/14 | Discussion of generative language model refinements; Transformers: Decoder; Transfer Learning;
Slides: PDF Here is the sample_choice(...) function discussed in lecture: TXT |
Beautiful Tutorial on Bert and Transfer Learning: The Illustrated BERT Here is the web site of Jay Alammar, author of the excellent "Illustrated XXX" series of blogs, lots of good stuff here: HTML Excellent new book: Natural Language Processing with Transformers: Amazon, also available online through BU Mugar Library. |
|
| 20 | Th 11/16 | Transfer Learning concluded; The Transformer Family; Summary of Most Important NLP Tasks (if time) Slides: PDF Echo 360 Lecture recordings (full and slide view) are on YT. |
|
|
| T 11/21 | No Lecture | |||
| Th 11/23 | Thanksgiving Break | |||
| 21 | T 11/28 | GPT and related algorithms
Slides: PDF Echo 360 Lecture recordings (full and slide view) are on YT. |
Beautiful blog post on GPT models (with additional details on attention): The Illustrated GPT Excellent summary of current LLM research: HTML |
HW 06: Transformers Problem One: IPYNB Problem Two: IPYNB Problem Three: IPYNB Datasets for HW 06: ZIP Due F 12/22 |
| Th 11/30 | Guest Lecture
Slides: PDF Echo 360 Lecture recordings (full and slide view) are on YT. |
|||
| 22 | T 12/5 | Machine Learning for Audio: FFT, Mel Spectrograms,
Slides: PDF Echo 360 Lecture recordings (full and slide view) are on YT. |
||
| 23 | Th 12/7 | Automatic Speech Recognition (Speech-to-Text)
Slides: PDF Echo 360 Lecture recordings (full and slide view) are on YT. |
Excellent blog post on ASR: PDF | |
| 24 | T 12/12 | Conclusions: Future of AI; Can machines think? Will they replace us?
Slides: PDF Echo 360 Lecture recordings (full and slide view) are on YT. |