Optimization for Learning
Optimering för maskininlärning

FRTN50, 7.5 credits, A (Second Cycle)

Valid for: 2025/26
Faculty: Faculty of Engineering LTH
Decided by: PLED F/Pi
Date of Decision: 2025-04-10
Effective: 2025-05-05

General Information

Depth of study relative to the degree requirements: Second cycle, in-depth level of the course cannot be classified
Elective for: D5-mai, E4-mi, F5, F5-r, F5-mai, F5-mtm, I4-fir, M4, MMSR2, Pi5-ssr, Pi5-mtm
Language of instruction: The course will be given in English

Aim

Learning from data is becoming increasingly important in many different engineering fields. Models for learning often rely heavily on optimization; training a machine is often equivalent solving a specific optimization problem. These problems are typically of large-scale. In this course, we will learn how to solve such problems efficiently. The large-scale nature of the problems renders traditional methods inapplicable. We will provide a unified view of algorithms for large-scale convex optimization and treat algorithms for the nonconvex problem of training deep neural networks.

Learning outcomes

Knowledge and understanding
For a passing grade the student must

• know basic convex analysis
• understand the connection between machine learning and optimization
• have an understanding on the role of regularization in learning from an optimization point of view
• understand unifying framework for large-scale convex optimization
• understand concepts such as nonexpansiveness, and averagedness and their relation to monotone operators and their role for convergence of algorithms
• understand how to derive specific algorithms from the few general ones
• understand methods for avoiding numerical issues in deep neural network training.

Competences and skills
For a passing grade the student must

• be able to describe optimality conditions that are useful for large-scale methods
• be able to describe the building blocks that are the foundations of large-scale optimization algorithms and why they are used
• be able to analyze performance of optimization algorithms
• be able to solve optimization problems numerically using software and own implementations
• be able to present results in writing.

Judgement and approach
For a passing grade the student must

• understand what algorithm that should be used for different machine learning training problems
• be able to participate in the team-work needed to solve the hand-in assignments.

Contents

The course has lectures, exercises, and two hand-in assigments.

The lectures will cover:

convexity, models for learning, unified convex optimization algorithm view, fixed-point iterations, monotone operators, nonexpansive mappings, stochastic methods, reduced variance methods, block-coordinate methods, nonconvex stochastic gradient descent and variations for for deep learning training.

Examination details

Grading scale: TH - (U, 3, 4, 5) - (Fail, Three, Four, Five)
Assessment:

Written exam (5 hours), 2 hand-in exercises. In case of less than 5 registered students, the exam may be given in oral form.

The examiner, in consultation with Disability Support Services, may deviate from the regular form of examination in order to provide a permanently disabled student with a form of examination equivalent to that of a student without a disability.

Modules
Code: 0125. Name: Exam.
Credits: 7.5. Grading scale: TH - (U, 3, 4, 5). Assessment: Passed exam
Code: 0225. Name: Hand-in 1.
Credits: 0.0. Grading scale: UG - (U, G). Assessment: Passed hand-in
Code: 0325. Name: Hand-in 2.
Credits: 0.0. Grading scale: UG - (U, G). Assessment: Passed hand-in

Admission

Assumed prior knowledge: FMAN61 Optimization
The number of participants is limited to: No

Reading list

• Lecture slides and notes.

Contact

Course coordinator: Pontus Giselsson, pontusg@control.lth.se
Director of studies: Björn Olofsson, bjorn.olofsson@control.lth.se
Course homepage: https://canvas.education.lu.se/courses/37223

Further information