DDA4220 Deep Learning and Applications

MDS6224 Deep Learning and Their Applications

MBI6011 Deep Learning

Spring 2023


Deep Learning and Application is the course designed to learn some basic components of modern deep neural networks, as well as the relevant applications from a practical perspective. The course covers topics including convolution neural network, recurrent neural network, graph neural network, neural radiance fields, attention mechanism, and Transformer. Besides, some carefully selected computer vision and NLP applications are also introduced in this course. Some projects are arranged to practice the concepts learned during the course. By taking this course, students can master the use of python libraries to build deep neural networks and address some basic artificial intelligence tasks.

Teaching Team

Ruimao Zhang



Jie Yang

Ph.D., since 2021, TA


Ruiying Liu

Ph.D., since 2022, TA


Shunlin Lu

Ph.D., since 2023, TA



  • Lecture: are on Monday/Wednesday 3:30--5:00 PM. in Room 111, Zhixin Building.
          Note: lectures will be remote for the first two weeks, and hybrid afterwards. The Zoom link is posted on BB.

  • Office Hours:

          Ruimao Zhang: Tuesday 3:00--4:00 PM. Room 519, Daoyuan Building

          Jie Yang: Tuesday 10:00-11:00 AM. Tencent meeting link: https://meeting.tencent.com/dm/1pvUtzh7amoz (No. 963-5662-5312)

          Ruiying Liu: Monday 10:00-11:00 AM. The bigger room of SDS Research Lab (4th Floor, Zhi Xin Building) Seat No. 88.

          Shunlin Lu: Friday 20:00-21:00 PM. Tencent meeting link: https://meeting.tencent.com/dm/Nt429IFgBhnU (No. 304-4741-4210)

  • Contact: If you have any question, please reach out to us via email or post it to BB.

  • Course Information

    This course is designed as the first course for students who are interested in deep learning technology. It mainly focus on some basic components of modern deep neural networks, as well as the relevant applications from a practical perspective. While the advanced topics cover the cutting-edge technology in recent years, giving students the opportunity to know the trend of technology development. In particular, the topics include:

  • Convolutional Neural Network

  • Computer Vision Basic

  • Recurrent Neural Network

  • Natural Language Processing Basic

  • Attention Mechanism and Transformer

  • Pretrained Language Model

  • Self-supervised Learning

  • Unified Language and Vision Model

  • Prerequisites

  • Basic machine learning knowledge: It requires the students have taken at least one course on machine learning.

  • Proficiency in LaTex: All the reports need to be written by using LaTex. A template will be provided. If you are not familiar with LaTex, please learn from the tutorial in advance.

  • Proficiency in Python: All the assignments will be in Python, especially the PyTorch framework.

  • Proficiency in Openmmlab: All the assignments will be realized on the Openmmlab platform

  • Proficiency in GitHub: All the source codes need to be submitted in GitHub.

  • Textbooks

    Recommended Books

  • Deep Learning, by Ian Goodfellow and Yoshua Bengio and Aaron Courville

  • Dive into Deep Learning, by Aston Zhang, Zachary C. Lipton, Mu Li, and Alexander J. Smola

  • Grading Policy

    Assignments (30%)
  • Assignment 1 (15%) Image classification by using deep models

  • Assignment 2 (15%) Language processing and classification by using deep models

  • Midterm Exam (25%)

    We will have 20~30 questions about the basic concept. The scope of the mid-term exam is from lecture 1 to lecture 9 (including Transformer).

    Final Project (40%)

    The final project is teamwork with no more than 3 teammates. You need to write a project report (max 5 pages not including references) for the final project. Here is the report template. You also need to prepare a “10 minutes” code explanation (using the Zoom) to explain the core content of your code, its reasonableness, and the steps to run it, which is highly relevant to the quality and readability of your code. After the final project deadline, feel free to make your project open source.
    【Note】If your team submits a project report after the submission deadline, the maximum project credit for all team members is 50%.

  • Code reproducibility (15%): Determine code quality and readability through visual inspection and code explanation. The code needs to ensure that it can be executed effectively and is consistent with the experimental results in the report (10%). The structure of the code and the functions implemented need to be clear and easy to understand (5%). Teams that choose the same topic are not allowed to plagiarize, otherwise, all members of the corresponding two teams will fail the course.

  • Project reprort (25%): Please follow the report template to organize your content and ensure the following items.

    (1) Reference to prior work (3%): Cite existing related work and describe the lineage of technology development, as well as their relations to your algorithms.

    (2) Technical correctness (4%): Present your algorithms or systems for your project. Provide key information for judging whether it is technically correct.

    (3) Experimental validation (5%): Present your experimental setups, and experiments conducted. Discuss the motivation of the designed experimental setups and the corresponding evaluation metrics.

    (4) Findings and Analysis (8%): Report the final results of your algorithms (performances and visualization results). Analyze and understand your system by inspecting key outputs and intermediate results (ablation study). Discuss how it works, when it succeeds and when it fails, and try to interpret why it works and why not.

    (5) Clear in writing (2%): The report is written in a precise and concise manner so the report can be easily understood.

    (6) Overall quality of report (3%): An overall evaluation of the final project report, including but not limited in writing, creativity, convincing experiments and analysis.

  • Originality and creativity (5%): Original and creative ideas are encouraged. If you try to leverage some techniques outside the scope of this course (2%) and correctly use them to improve your algorithms (3%), you will get extra scores.

  • Participation (5%)

    Here are some ways to earn the participation credit. Note that the relevant scores are used to make up for grades.

  • Completing feedback surveys: We will send out two feedback surveys during the semester to improve the course. All students get 1.5% per survey (in total 3%) for completing the surveys.

  • Course and Teaching Evaluation (CTE): The school will send requests for CTE to all students. The CTE is worth 2% credit.

  • Schedule

    Date Lecture Description Reading Material Lecture Note Events/Deadlines
    Jan. 4 & Jan. 9 Lecture 1: Introduction Textbook: Deep Learning
    Textbook: Dive into Deep Learning
    [ Slides]
    [ Video-1]
    [ Video-2]
    Jan. 11 & Jan. 16 & Feb. 8 Lecture 2: Convolution Neural Network 1 Learn LaTeX in 30 minutes
    Official tutorials of GitHub
    [ Slides]
    [ Video-1]
    [ Video-2]
    [ Video-3]
    Jan. 11 Tutorial 1: Introduction to Pytorch Colab official tutorial
    Pytorch Quickstarts
    PyTorch Installation
    [ Material]
    [ Video]
    Feb. 13 & Feb. 15 & Feb. 20 & Feb. 22 Lecture 3: Convolution Neural Network 2 Article Management Tool
    Recommended papers:
    GoogleNet, ResNet
    [ Slides]
    [ Video-1]
    [ Video-2]
    [ Video-3]
    [ Video-4]
    Assignment 1 out
    [ PDF] [ Template] [ Data]
    Feb. 15 Tutorial 2: Network Training Pipeline Homepage of OpenMMlab [ Material]
    [ Video]
    Feb. 22 & Feb. 27 Lecture 4: Computer Vision Basic 1 Recommended papers:
    Training Tricks, Adagrad, Adam
    [ Slides]
    [ Video-1]
    [ Video-2]
    Mar. 1 & Mar. 6 & Mar. 8 Lecture 5: Computer Vision Basic 2 [ Slides-1]
    [ Slides-2]
    [ Video-1]
    [ Video-2]
    [ Video-3]
    Topics of the final project out
    Mar. 1 Tutorial 3: Deep Learning Applications I [ Material]
    [ Video]
    Mar. 13 & Mar. 15 & Mar. 20 Lecture 6: Recurrent Neural Network Recommended papers:
    LSTM, GRU, Sequence-to-sequence Modeling
    [ Slides]
    [ Video-1]
    [ Video-2]
    [ Video-3]
    Assignment 1 due
    Mar. 19, 11:59 PM.
    Mar. 15 Tutorial 4: Deep Learning Applications II [ Video]
    Mar. 22 & Mar. 27 Lecture 7: Natural Language Processing Basic Recommended papers:
    Wrod2vec, Glove,
    [ Slides]
    [ Video-1]
    [ Video-2]
    Assignment 2 out
    [ PDF] [ Template]
    Mar. 27 Tutorial 5: Introduction to NLP framework [ Material]
    [ Video]
    Mar. 29 & Apr. 3 Lecture 8: Attention and Transformer Recommended papers:
    non-local, Transformer, ViT
    [ Slides]
    [ Video]
    Apr. 4 Tutorial 6: Deep Learning Applications III [ Video]
    Apr. 10 Midterm Exam
    Apr. 12 & Apr. 17 Lecture 9: Pretrained Language Models Recommended papers:
    BERT, GTP-1, GTP-2, GTP-3, InstructGPT
    [ Slides]
    [ Video-1]
    [ Video-2]
    Assignment 2 due
    Apr. 23 & Apr. 24 Lecture 10: Self-supervised Learning in Vision [ Slides]
    [ Video-1]
    [ Video-2]
    Apr. 26 & May 6 Lecture 11: Unified Language and Vision Models [ Slides]
    [ Video-1]
    [ Video-2]
    Final Project Submission Final Project Report due (11:59 PM.)