Appendix B — Course Lecture Progression

This section describes an example progression through the course material based on my “Machine Learning for Mechanical Engineering” course at ETHZ. It assumes two lecture+exercise sessions twice a week, for a duration of 1 hour and 45 minutes each with a 5-10 minutes break in the middle of each session. This provides a sample of how you might work through the content in this book, and also acts as a reference to the students. My course is structured such that lectures and exercises occur in the same session, often alternating between lecture and in-class demonstrations or exercises, and this is frequently reflected in each of the book chapters. In class, for time reasons, I may skip some of the longer derivations in the book, since these can be effectively studied on their own.

B.1 Part 0: Review and Foundations

These lectures introduce the course and also cover some background information that will be foundational and critical later in the course. For illustrative and notational purposes, we will use Linear Regression as a simple model to understand these foundations first, and this will allow us to build up to more complex models as we go through the course.

B.1.1 Lecture 1: Course Introduction and Review of ML Basics

B.1.2 Lecture 2: Review of (Stochastic) Gradient Descent

  • Read Stochastic Gradient Descent
  • Read Why Momentum Really Works
  • Review of Regularization for Linear Regresson models, including weight decay (L2) and L1/sparsity control, and effects on loss functions.
  • Review of Linear Unsupervised Learning (e.g., PCA, Sparse PCA, etc.)

B.1.3 Lecture 3: Automatic Differentiation and Taking Derivatives

  • Read Taking Derivatives, except for Advanced topics like Implicit Differentiation and JVP/HVP (this is for a later lecture)
  • In-Class example of Forward and Reverse AD on a simple function
  • In-Class example of AD through a Verlet Integrator

B.1.4 Lecture 4: Advanced Differentiation

  • Read the remaining parts of Taking Derivatives
  • In-Class example of Implicit Differentiation
  • In-Class example with JVPs for computing some example properties of Linear Models, for example, the Fisher Information Matrix.
  • (Time Permitting) Examples of Projected Gradient or Proximal Gradient methods.

B.2 Part 1: Advanced Neural Network Models

B.2.1 Lecture 5: Review of Neural Network Models

  • Read Review of Neural Networks
  • In-Class example of Visualizing NN Layers using ConvNetJS
  • In-Class examples of Auto-Encoders on Simple Data, Auto-Encoders on Airfoil Data, and Least Volume Auto-Encoders (including Spectral Normalization)

B.2.2 Lecture 6: Push-Forward Generative Models

B.2.3 Lecture 7: Interlude – Techniques for Building and Debugging ML Models

B.2.4 Lecture 8: Optimal Transport

B.2.5 Lecture 9: Stochastic Generative Models

  • Read Stochastic Generative Models
  • In-Class examples of Diffusion Models on Synthetic Data and the Airfoil Problem
  • In-Class examples of Sequence Data generation (e.g., the Tangrams Example)

B.2.6 Lecture 10: Latent Generative Neural Models

  • In-Class example of Latent Diffusion Models via Least Volume AEs.
  • In-Class example of continuous+discrete models
  • Practical Demonstrations of complex models on EngiBench

B.2.7 Lecture 11:

  • Read Introduction to Transformers
  • In-Class experiment with the Attention Mechanism
  • In-Class demonstrations of Latent Transformer models (e.g., VQGAN)

B.2.8 Lecture 12: Review of Reinforcement Learning

B.2.9 Lecture 13: Actor-Critic and Policy Gradient Methods

  • Policy Gradients

B.2.10 Lecture 14: Advanced Policy Methods

  • Diffusion Policies
  • Transformer Policies
  • Latent Generative Policies (e.g., Dreamer and variants)

B.2.11 Lecture 15: Message Passing, Graph Neural Networks, and other Structured Data

B.3 Part 2: Probabilistic Models and Kernels

B.3.1 Lecture 16: Review of Probabilistic Models

  • Read ?sec-reviewprobability of Probabilistic Models, since this reviews the information from the prior course. We will revisit the later sections in a later lecture.
  • In-Class exercise deriving the MLE for various common distributions

B.3.2 Lecture 17: Introduction to Probablistic Programming

  • Bayesian Linear Regression in a Probabilistic Programming Language
  • Introduction to Approximate Inference Methods, such as MCMC and Variational Inference

B.3.3 Lecture 18: Large-Scale Bayesian Models and Debugging Probabilistic Inference

  • Stochastic Variational Inference and Stein Variational Gradient Descent
  • Pitfalls of Probabilistic Models and Debugging Inference

B.3.4 Lecture 19: Kernel Basics

  • Representer Theorem
  • Kernelized Ridge Regression
  • Regularization of Kernels and effects of Fourier Spectra

B.3.5 Lecture 20: Gaussian Processes and Bayesian Optimization

  • Basics of Gaussian Processes
  • Bayesian Optimization
  • Effect of Dimension on GPs and related variants (AdditiveGPs, etc.)

B.4 Part 3: Engineering-relevant Machine Learning Topics

B.4.1 Lecture 21: Active Learning and Semi-Supervised Learning

  • Notebook on Active Learning and Semi-Supervised Learning

B.4.2 Lecture 22: Transfer Learning and Foundation Models

  • Fine-Tuning a Pre-trained model
  • Jointly embedded models

B.4.3 Lecture 23: Integrating Everything Together

  • In-Class exercise integrating all prior topics using EngiBench/EngiOpt