Neural Networks and Deep Learning
Code: 106573 ECTS Credits: 6| Degree | Type | Year | Semester |
|---|---|---|---|
| 2504392 Artificial Intelligence | OB | 2 | 2 |
Contact
- Name:
- Dimosthenis Karatzas
- Email:
- dimosthenis.karatzas@uab.cat
Use of Languages
- Principal working language:
- english (eng)
- Some groups entirely in English:
- Yes
- Some groups entirely in Catalan:
- No
- Some groups entirely in Spanish:
- No
Prerequisites
To have completed the first-year subjects of Fundamentals of Machine Learning, Fundamentals of Programming, Fundamentals of Mathematics, Probability and Statistics, and Data Engineering.
Objectives and Contextualisation
This subject aims to give a practical introduction to neural network models and deep learning.
The students will consolidate and extend their theoretical background, by building on top of previous subjects on machine learning and complementing previous knowledge with new concepts on neural network design, deep learning frameworks, and the training process for such models.
The students should finish this subject, having a broad knowledge of different neural network architectures and their typical use scenarios, and a demonstrated capacity to critically choose the right architecture and training mechanisms for each task.
Finally, the students will receive hands-on training and acquire practical experience on using current deep learning frameworks to solve specific tasks.
Competences
- Develop critical thinking to analyse alternatives and proposals, both one's own and those of others, in a well-founded and argued manner.
- Develop strategies to formulate and solve different learning problems in a scientific, creative, critical and systematic way, knowing the capabilities and limitations of the different existing methods and tools.
- Identify, understand and apply the fundamental concepts and techniques of knowledge representation, reasoning and computational learning for the solution of artificial intelligence problems.
- Introduce changes to methods and processes in the field of knowledge in order to provide innovative responses to society's needs and demands.
- Students must be capable of collecting and interpreting relevant data (usually within their area of study) in order to make statements that reflect social, scientific or ethical relevant issues.
- Work cooperatively to achieve common objectives, assuming own responsibility and respecting the role of the different members of the team.
Learning Outcomes
- Develop critical thinking to analyse alternatives and proposals, both one's own and those of others, in a well-founded and argued manner.
- Formulate the most appropriate cost function for a specific problem according to the characteristics of the experimental data and to the requirements/restrictions of the problem.
- Plan, execute and interpret the training process of a deep neural network model by applying a suitable optimisation method, employing the appropriate pre-processing and data initialisation techniques, as well as the necessary regularisation techniques.
- Students must be capable of collecting and interpreting relevant data (usually within their area of study) in order to make statements that reflect social, scientific or ethical relevant issues.
- Understand deep learning mechanisms based on neural networks in order to be able to design and apply the most appropriate architecture for a given problem.
- Understand the concepts of bias and variance, and be able to use data preparation methods and regularisation techniques to obtain generalisable solutions from the available data.
- Understand the gradient descent algorithm and its variants and how to apply it to solve regression and classification problems.
- Weigh up the risks and opportunities of both your own and others' proposals for improvement.
- Work cooperatively to achieve common objectives, assuming own responsibility and respecting the role of the different members of the team.
Content
- Neural networks
- McCulloch-Pitts Neurons and Perceptron
- Multi-layer perceptron
- Backpropagation
- Training process
- Initialization
- Optimization algorithms
- Regularization techniques
- Tasks and cost functions
- Deep neural network architectures
- Convolutional networks
- Recurrent networks
- Unsupervised learning (autoencoders)
- Metric learning (Siamese and triplet networks)
- Generative Adversarial Networks
Methodology
Neural network design is guided by the types of problems that it aims to solve. Throughout this subject it will be that typology of problems that will provide the motivation of each section and will direct the organization of the contents.
There will be two types of sessions:
Theory classes: The objective of these sessions is for the teacher to explain the theoretical background of the subject. For each one of the topics studied, the theory and mathematical formulation is explained, as well as the corresponding algorithmic solutions.
Laboratory sessions: Laboratory sessions aim to facilitate interaction, collaborative work and to reinforce the comprehension of the topics seen in the theory classes. During laboratory sessions the students will work through practical cases that require the design of solutions using the methods studied in theory classes. Problem solving will be initiated in the class and will be complemented by a weekly set of problems to work through at home.
The above activities will be complemented by a system of tutoring and consultations outside class hours.
All the information of the subject and the related documents that the students need will be available at the virtual campus (cv.uab.cat).
Annotation: Within the schedule set by the centre or degree programme, 15 minutes of one class will be reserved for students to evaluate their lecturers and their courses or modules through questionnaires.
Activities
| Title | Hours | ECTS | Learning Outcomes |
|---|---|---|---|
| Type: Directed | |||
| Laboratory Sessions | 21 | 0.84 | 5, 7, 1, 6, 2, 3, 8, 4, 9 |
| Theory Classes | 24 | 0.96 | 5, 7, 6, 2, 3 |
| Type: Supervised | |||
| Tutoring | 2 | 0.08 | 5, 7, 1, 6, 2, 3, 8 |
| Work on problems / projects | 40 | 1.6 | 5, 1, 2, 3, 8, 4, 9 |
| Type: Autonomous | |||
| Individual studying | 20 | 0.8 | 5, 7, 6, 2, 3 |
| Work on problems / projects | 30 | 1.2 | 5, 1, 2, 3, 8, 4, 9 |
Assessment
To assess the level of student learning, a formula is established that combines theoretical and practical knowledge acquisition, and the ability to solve problems.
Final Grade
The final grade is calculated weighted in the following way and according to the different activities that are carried out:
Final grade = 0.5 * Theory Grade + 0.1 * Problems Portfolio Grade + 0.4 * Practical Evaluation
This formula will be applied as long as the theory and the practical evaluation grades are higher than 5. There is no restriction on the problems portfolio grade. If doing the calculation of the formula yields >= 5 but the theory grade or practical evaluation grade does not reach the minimum required, then a final grade of 4.5 will be given.
Theory Grade
The theory grade aims to assess the individual abilities of the student in terms of the theoretical content of the subject, this is done continuously during the course through two partial exams. The overall theory grade is the average of the grades of the two partial exams.
The mid-term exam (Exam 1) is done in the middle of the semester and serves to eliminate part of the subject if it is passed. The final exam (Exam 2) is done at the end of the semester and serves to eliminate the rest of subject if it is passed.
These exams aim to assess the abilities of each student in an individualized manner, both in terms of solving problems using the techniques explained in class, as well as evaluating the level of conceptualization that the student has made of the techniques seen. In order to obtain a final pass theory grade, it will be required for the both partial exam grades to be higher than 4. If doing the calculation of the formula yields >= 5 but the grades of any of the two partial exams do not reach the minimum required, then the grade for theory that will be used in the final calculation will be 4.5.
Recovery exam. In case the theory grade does not reach the adequate levelto pass, the students can take a recovery exam, destined to recover the failed part (1, 2 or both) of the continuous evaluation process.
Problems Portfolio Grade
The aim of the problems (exercises) is for the student to train with the contents of the subject continuously and become familiar with the application of the theoretical concepts. As evidence of this work, the presentation of a portfolio is requested in which the exercises worked out will be collated. To obtain a problems grade it is required that the student submits a minimum of 70% of the problem sets, in the opposite case, the problems grade will be 0.
Practical Evaluation Grade
The practical abilities of the students will be evaluated twice during the semester. Each student will be asked to defend, either orally or in writing, the solutions that he has provided to the weekly problems. This evaluation will be individual and may focus on a subset of problems. Eventually, larger problems can be posed to be done autonomously to which a computational solution will have to be presented.
The overall practical evaluation grade is the average of the two practical evaluations.
In case of not passing any of the practical evaluations, the student will be able to recover the failed part, restricted to a maximum grade of 7/10.
Important notes
Notwithstanding other disciplinary measures deemed appropriate, and in accordance with the academic regulations in force, evaluation activities will be suspended with zero (0) whenever a student commits any academic irregularities that may alter such evaluation (for example, plagiarizing, copying, letting copy, ...). The evaluation activities qualified in this way and by this procedure will not be recoverable. If you need to pass any of these assessment activities to pass the subject, this subject will be failed directly, without opportunity to recover it in the same course.
In case there the student does not deliver any exercise solutions, does not participate in any practical evaluation, and does not take any exam, the corresponding grade will be a "non-evaluable". In any other case, the “no shows” count as a 0 for the calculation of the weighted average.
In order to pass the course with honours, the final grade obtained must be equal or higher than 9 points. Because the number of students with this distinction cannot exceed 5% of the total number of students enrolled in the course, it is given to whoever has the highest final marks. In case of a tie, the results of the partial exams will be taken into account.
More details about the evaluation process will be given at class during the first weeks of the semester. In case of any discrepancy between this guide and the information given at class, the information provided at class takes precedence.
Assessment Activities
| Title | Weighting | Hours | ECTS | Learning Outcomes |
|---|---|---|---|---|
| Practical examination | 40% | 4 | 0.16 | 1, 8, 4 |
| Problem deliverables | 10% | 5 | 0.2 | 1, 8, 4, 9 |
| Written exams | 50% | 4 | 0.16 | 5, 7, 6, 2, 3 |
Bibliography
Books:
- Deep Learning, Ian Goodfellow, Yoshua Bengio, and Aaron Courville, MIT Press, 1st Ed. 2016
- Deep learning with Python, François Chollet, Manning Publications, 1st Ed., 2017
- Pattern Recognition and Machine Learning, Christopher Bishop, Springer, 2011
Online Books:
- Michael Nielsen’s Neural Networks and Deep Learning
http://neuralnetworksanddeeplearning.com/ - Zhang, Z.C. Lipton, M. Li, A.J. Smola, “Dive into Deep Learning”, 2021
https://d2l.ai/
Software
For the practical activities of the course we will use Python (NumPy, MatPlotLib, SciKit Learn) and PyTorch