ICS 471 Deep Learning

(Term 221)

Course Information
Prerequisites
Textbook & Resources
Schedule
Grading
Late Policy

Current Catalog Description

The course will review linear models and stochastic optimization. It will develop an in-depth understanding of Feedforward networks, Loss functions, Back-propagation training, Regularization, Convolutional neural networks, Recurrent and recursive networks, Vanishing gradient problem, Long-short term memory (LSTM) model, Gated recurrent units (GRUs), Processing sequences, images, and hierarchical structures, Auto-encoders, Transfer learning, and Generative adversarial networks. The course will develop models for several domain problems such as automatic speech recognition, image recognition, drug discovery, and recommendation systems, etc.

Instructor

Hamzah Luqman

Class meetings: Sunday, Tuesday 11:00 am - 12:15 pm ; Building 23 room 011; Sunday, Tuesday 02:00 pm - 03:15 pm ; Building 24 room 135
Office hours: Sunday & Tuesday: 10:00 am -10:45 am!
Blackboard: https://blackboard.kfupm.edu.sa/
Instructor Email: hluqman@kfupm.edu.sa

Course Objectives

Provide the student with in-depth knowledge of Deep Learning.
Teach student how to build neural networks, and how to lead successful deep learning projects.
Provide students with the required knowledge on how to develop and employ convolutional networks, RNNs, LSTM, Adam, Dropout, BatchNorm, Xavier/He initialization, and more.

Course Learning Outcomes

After completion of this course, the student shall be able to:

Be able to explain different neural network architectures (shallow versus deep) and their usage in current standard intelligent software applications.
Be able to analyse the various optimization techniques used to train shallow and deep neural network models.
Be able to implement, train, and evaluate shallow and deep neural networks using existing software libraries.
Be able to plan and carry out a full pipeline project model using shallow and deep neural networks.

Prerequisites

COE 292, MATH 208, STAT 319

Textbook

Deep Learning, Ian Goodfellow, Aaron Courville, and Yoshua Bengio, MIT Press

Tentative Schedule

Item	Topics	Additional readings, if any	Quizzes/Assignments
1	Course logistics
2	Introduction to deep learning Foundation of neural network and deep learning		Assignment 0
3	Images Clasification and KNN	CS231: Image classification CS231: Linear classification
4	Foundation of neural network and deep learning
5	Neural networks training and optimization Perceptron and LMS	DL book: Regularization for DL DL book: Optimization for Training Deep Models	Assignment 01
6	Feedforward networks and Loss functions	DL book: Deep Feedforward Nets	Quiz 1
7	Back propagation Calculus of back propagation	CS231: Optimization and Backpropagation
8	Convergence in neural networks Rates of convergence Loss surfaces Learning rates, and optimization methods RMSProp, Adagrad, Momentum	CS231: Optimization and Backpropagation DL book: Optimization for Training Deep Models
9	Stochastic gradient descent Optimization Acceleration Overfitting and regularization Tricks of the trade: Choosing a divergence (loss) function Batch normalization Dropout	CS230: Neural Networks training
10	Convolutional Neural Networks (CNNs)	DL book: Convolutional Networks CS231: CNN	Assignment 2 Quiz 2
11	Convolutional Neural Networks Models of vision
12	Backpropagation through CNNs Increasing output map size Transform invariance Alexnet, Inception, VGG
13	Transfer Learning	TCS231: ransfer Learning and Fine-tuning CNNs
14	Recurrent Neural Networks (RNNs) Modeling series Back propagation through time (BPTT) Bidirectional RNNs	DL book RNN chapter	Assignment 3
	Midterm [October 25, 2022]
15	Stability Exploding/vanishing gradients Long Short-Term Memory Units (LSTMs) and GRU	Understanding LSTM Networks
16	Sequence To Sequence Modeling		Quiz 03
17	Auto-encoder models
18	Attention Models	Attention? Attention!
19	Transformers	Attention Is All You Need The Illustrated Transformer	Assignment 4 Quiz 4
20	Generative Adversarial Networks (GANs)
	Final Exam

Grading

10% Quizzes
25% Homework (5 homeworks)
15% Project
20% Midterm
30% Final exam

Late policy for deliverables

Every student has 4 free late days (one day per homework) for this course.
After all free late days are used up, penalty is 25% for each additional late day.

Important notes

Students are expected to be courteous toward the instructor and their classmates throughout the duration of this course.
All cell phones and e-devices must be “on silent” mode during the classes and “turned off” during the exam times.
Attendance is taken at the beginning of the class.
Unexcused Lecture Absences Policies: Two late attendances are considered as one absence. Six unexcused absences result in DN grade.
An unexcused absence can become an excused absence ONLY by an official letter and must be presented as soon as possible.
Assignments must be submitted on the due date.
No make up for exams or any other class work will be made.
Any issues related to grading should be raised within one week of the release of the grades
ZERO-TOLERANCE for CHEATING, whether in exams, quizzes, written/programming assignments, and course project. Plagiarism, copying and other anti-intellectual behavior are prohibited by the university regulations.