Skip to main content Link Search Menu Expand Document (external link)
DSGA 1011

Calendar

  • Previous version: Spring 2023
  • The schedule below is tentative and subject to change.
  • All course materials can be found on Github.

Supervised learning

Sep 6
Text classification [recording]
HW 1 out [zip] [pdf]
  1. Course overview
  2. Supervised learning basics
  3. Feature-based text classification
  4. Additional readings: JM Ch4.1, JM Ch5
Sep 11
Section Text classification [slides] [notebook] [recording]
Sep 13
Word embedding [recording]
  1. Distributed representation of words
  2. Learning word vectors
  3. Additional readings: JM Ch6
Sep 18
Section Word vector algebra [slides] [notebook] [recording]
Sep 20
Sequence modeling [recording]
HW 1 due
HW 2 out [zip] [pdf]
  1. Neural network basics
  2. RNN and its variants
  3. Attention and Transformers
  4. Additional readings: D2L Ch9.4-9.7, D2L 10.1, D2L Ch11.1-11.7
Sep 25
Section HPC and PyTorch tutorial [Check HPC Access] [HPC Basics] [PyTorch Basics] [Recording]
Sep 27
Sequence generation [recording]
  1. Encoder-decoder models
  2. Decoding algorithms
Oct 2
Section Machine translation and encoder-decoder models [notes] [recording]
Oct 4
Tasks and applications [recording]
  1. The landscape of NLP tasks
  2. Evaluation
  3. Final project tips
Oct 10
Section Beam search; NLP datasets [slides] [notebook] [recording]

Representation learning

Oct 11
Pretraining and finetuning (basics) [recording]
HW 2 due
  1. Data
  2. Self-supervised learning
  3. Adaptation / finetuning
Oct 16
Section Transformer [slides] [notebook] [recording] HW 3 out [zip] [pdf]
Oct 18
Pretraining and finetuning (advanced) [recording]
  1. Efficient pretraining
  2. Efficient finetuning
  3. Additional reading: Efficient transformers: a survey, also see papers linked in slides
Oct 23
Section Efficient Inference [slides] [recording]
Oct 25
Holistic evaluation [recording]
Proposal due on 27th See submission instruction, team signup and latex template.
  1. Fairness, robustness, privacy
  2. Additional reading: Holistic evaluation of language models
Oct 30
Section Model Evaluation [slides] [notebook]

NLP via language modeling

Nov 1
Scaling language models [recording]
  1. N-gram language models
  2. Scaling and emergent abilities
Nov 3
HW 3 due
Nov 6
Section Training LLMs [slides] [notebook]
Nov 8
Prompt engineering [recording]
HW 4 out [zip] [pdf] [tex]
Nov 13
Section Prompt Engineering and Huffman Code [slides] [recording] [annotated slides]
Nov 15
Aligning language models (basics) [recording]
  1. Why do we need alignment?
  2. Basic alignment techniques
Nov 20
Section Reinforcement learning basics [slides] [recording]
Nov 22
Fall break (no lecture)
HW4 due
Nov 27
Section Project presentation and report writing [slides] [recording]
Nov 29
Aligning language models (advanced) [recording]
  1. RL for text generation
  2. RL from human feedback
Dec 4
Section RLHF and its extensions [slides] [annotated slides] [recording]
Dec 6
Guest lecture: Aligning Semi-Parametric Language Models by Victoria Lin
Dec 11
Final project presentation [recording]
Dec 13
Final project presentation
Report due