Deep Learning

The "Black" Art & "Elusive" Science of Training Deep Neural Networks

School of Computing, University of Nebraska-Lincoln Spring 2021 (3-week session): CSCE 496/896

Synopsis: Deep Learning, a cornerstone of modern Artificial Intelligence, tackles complex challenges in computer vision, natural language processing, and speech recognition. Deep Neural Networks (DNNs)—vast networks of artificial neurons arranged in hierarchical layers—uncover intricate patterns by creating increasingly meaningful data representations. Unlocking their potential requires a deep understanding of effective architectures, learning algorithms, and optimization strategies. Often dubbed a "black art" due to its complexity, training DNNs can feel like summoning a genie. This course aims to demystify this process, exploring state-of-the-art DNN architectures and delving into the "black" art and "elusive" science of training them.

Philosophy: Inspired by my mentor Richard Feynman's philosophy on learning—“What I cannot create, I do not understand”—this course emphasizes hands-on implementation. True mastery of DNN training comes from building models from scratch, without high-level libraries like Scikit-Learn, Keras, or PyTorch. Begin with a Multi-Layer Perceptron (MLP) using NumPy (pseudocode), then tackle a Convolutional Neural Network (CNN) using NumPy (pseudocode).

Instructor: Dr. M. R. Hasan
Office Hours: See the course Canvas page
Lecture Time: Monday through Friday: 10:00 AM - 11:30 AM via Zoom

Explore my GitHub tutorials on Machine Learning and Deep Learning for additional resources. Key repositories include:

Schedule

Week	Date	Topic & PDF Slides	Video Links
1	Jan 4	Introduction Linear Neural Networks - Linear Regression	[Jan 4] Course Introduction (from 2:40) [Jan 4] Introduction to Deep Learning (from 9:00) [Jan 4] Linear Neural Network - Linear Regression
1	Jan 5	Linear Neural Networks - Logistic Regression	[Jan 5] Review of Previous Lecture [Jan 5] Linear Neural Network - Regression - Gradient Descent [Jan 5] Linear Neural Network - Binary Classification - Introduction
1	Jan 6	Linear Neural Networks - Softmax Regression Jupyter Notebook: Linear Neural Networks using TensorFlow-Keras	[Jan 6] Review of Previous Lecture [Jan 6] Linear Neural Network - Logistic Regression - Binary Cross Entropy Loss [Jan 6] Linear Neural Network - Logistic Regression - TensorFlow Demo [Jan 6] Linear Neural Network - Softmax Regression
1	Jan 7	Multi-Layer Perceptron (MLP): Introduction Multi-Layer Perceptron (MLP): Training - Backpropagation Algorithm I	[Jan 7] Review: On the Horizon of LNNs [Jan 7] Nonlinear Neural Network - Multi-Layer Perceptron [Jan 7] MLP Training: The Backpropagation Algorithm
1	Jan 8	Multi-Layer Perceptron (MLP): Training - Backpropagation Algorithm II Art & Science of Training Deep MLPs-1 Jupyter Notebooks: Handcrafting MLP using NumPy Using TensorFlow-Keras for Building Effective MLP Classifiers MLP - Curious Dabbling MLP - Investigation of the Spatial Invariance Property	[Jan 8] Review: Backpropagation Algorithm [Jan 8] Backpropagation Algorithm, Loss Function, Final Layer Activation [Jan 8] Art & Science of Training Deep MLPs
1	Jan 9	Art & Science of Training Deep MLPs-2	[Jan 9] Vanishing & Exploding Gradient Problem-I
2	Jan 11	Hidden Layer Activation Functions Art & Science of Training Deep MLPs-3 Art & Science of Training Deep MLPs-4	[Jan 11] Review: Hidden Layer Activations (sigmoid, tanh, ReLU) & Weight Initializers (LeCun, Glorot) [Jan 11] Vanishing & Exploding Gradient Problem-II: ReLU, He Initializer, Variants of ReLU [Jan 11] Vanishing & Exploding Gradient Problem-III: Gradient Clipping & Batch Normalization [Jan 11] Achilles' Heel of SGD: Learning Rate Schedule and Optimal Learning Rate by Hessian
2	Jan 12	Art & Science of Training Deep MLPs-5 CNN-1-Introduction	[Jan 12] Review: Achilles' Heel of SGD [Jan 12] Achilles' Heel of SGD: Topology of Cost Function [Jan 12] Fast Optimizers [Jan 12] Convolutional Neural Networks (CNNs): Limitations of Dense MLPs
2	Jan 13	CNN-2-Convolution	[Jan 13] CNN: Introduction & Architecture [Jan 13] CNN: Convolution
2	Jan 14	CNN-3-Padding CNN-4-Pooling	[Jan 14] CNN: Convolution (continued) [Jan 14] CNN: Padding [Jan 14] CNN: Pooling
2	Jan 15	CNN-5-Training-I CNN-6-Training-II CNN-7-Cheat Sheet CNN-8-Notable Architectures-I	[Jan 15] CNN: Global Average Pooling; Depthwise Pooling [Jan 15] CNN: Cheat Sheet, Study of LeNet-5 [Jan 15] CNN: ResNet
2	Jan 17		[Jan 17] CNN: Notable Architectures-I
3	Jan 18	Martin Luther King Holiday
3	Jan 19	CNN-9-Notable Architectures-II RNN-1-Introduction CNN Visualization: CNN Interactive Visualization Read Chollet: ch 5.4 Zeiler & Fergus (2013) Visualizing and Understanding Convolutional Networks How Convolutional Neural Networks See the World Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localization Jupyter Notebooks: CNN: Image Classification - Tricks of the Trade Data Augmentation for Deep Learning CNN: A Tour	[Jan 19] CNN: Notable Architectures-II [Jan 19] CNN: Training & Visualization [Jan 19] Recurrent Neural Networks (RNN): Introduction
3	Jan 20	RNN-2-Architecture RNN-3-Training	[Jan 20] RNN: Review [Jan 20] RNN: Architecture [Jan 20] RNN: Training
3	Jan 21	RNN-4-Training Issues RNN-5-Preventing Unstable Gradients RNN-6-Gated RNNs RNN-7-Gated RNNs-Prevent Vanishing Gradient RNN-8-Effective RNN Design Jupyter Notebooks: RNN: A Tour Efficient Methodology for Deep Learning	[Jan 21] RNN: Interpretation of Loss Gradient [Jan 21] RNN: Training Issues [Jan 21] RNN: Preventing Unstable Gradients [Jan 21] RNN: Gated RNNs

Text Resources

Lecture slides and Jupyter notebooks provide a thorough and extensive account of the topics.
Core Deep Learning Texts:
- Dive into Deep Learning by Aston Zhang, Zachary C. Lipton, Mu Li, and Alexander J. Smola
- Deep Learning by Ian Goodfellow, Yoshua Bengio, and Aaron Courville
- Deep Learning with Python by Francois Chollet
- Hands-On Machine Learning with Scikit-Learn, Keras and TensorFlow (2nd Edition, 2019) by Aurélien Géron (O'Reilly)
Foundational Machine Learning Texts:
- Machine Learning: A Probabilistic Perspective by Kevin P. Murphy
- Pattern Recognition and Machine Learning by Christopher M. Bishop
- Introduction to Machine Learning (3rd ed.) by Ethem Alpaydin
Python Refresher/Introduction:
- Data Science from Scratch by Joel Grus (O’Reilly)
- Python for Data Analysis (2nd Edition) by Wes McKinney (O'Reilly)
- Python Machine Learning by Sebastian Raschka (Packt Publishing)
Statistics, Linear Algebra & Calculus:
- Advanced Engineering Mathematics (10th Ed.) by Erwin Kreyszig
- All of Statistics: A Concise Course in Statistical Inference by Larry Wasserman
Interesting & Enlightening Texts:
- The Master Algorithm by Pedro Domingos
- Artificial Intelligence: A Guide for Thinking Humans by Melanie Mitchell
- The Deep Learning Revolution by Terrence J. Sejnowski
- Rebooting AI: Building Artificial Intelligence We Can Trust by Gary Marcus and Ernest Davis
- Thinking, Fast and Slow by Daniel Kahneman
- Interpretable Machine Learning by Christoph Molnar

The "Black" Art & "Elusive" Science of Training Deep Neural Networks

Schedule

Text Resources

Deep Learning Courses Elsewhere

Collaboration Tool

Google Colab Tutorials

Python

Open Data Repositories

ML Podcasts

Journals

Conference Proceedings