Statistical learning theory 2024/25 — различия между версиями
Материал из Wiki - Факультет компьютерных наук
Bauwens (обсуждение | вклад) |
Bauwens (обсуждение | вклад) |
||
| Строка 24: | Строка 24: | ||
|| [https://www.dropbox.com/s/oncvg4mxulbt56d/00book_intro.pdf?dl=0 ch00] [https://www.dropbox.com/s/i9pc4kf0zsdeksb/01book_onlineMistakeBound.pdf?dl=0 ch01] | || [https://www.dropbox.com/s/oncvg4mxulbt56d/00book_intro.pdf?dl=0 ch00] [https://www.dropbox.com/s/i9pc4kf0zsdeksb/01book_onlineMistakeBound.pdf?dl=0 ch01] | ||
|| [https://www.dropbox.com/scl/fi/qs5wqr97qoyh3l2gfju48/01sem.pdf?rlkey=6lvzcbfkw6lj9y77ep64nq7lk&dl=0 prob01] | || [https://www.dropbox.com/scl/fi/qs5wqr97qoyh3l2gfju48/01sem.pdf?rlkey=6lvzcbfkw6lj9y77ep64nq7lk&dl=0 prob01] | ||
| − | || <-- | + | || <!-- [https://www.dropbox.com/scl/fi/kksvt6ttgf06u8uce6g9z/01sol.pdf?rlkey=ldcqaewvg7cqdlfqkt7ltckej&dl=0 sol01] --> |
|- | |- | ||
| [https://www.youtube.com/watch?v=gQm1G3Ep-5s ?? Sept] | | [https://www.youtube.com/watch?v=gQm1G3Ep-5s ?? Sept] | ||
| Строка 31: | Строка 31: | ||
|| [https://www.dropbox.com/s/p3auugqwc89132b/02book_sequentialOptimalAlgorithm.pdf?dl=0 ch02] [https://www.dropbox.com/s/b00dcqk1rob7rdz/03book_perceptron.pdf?dl=0 ch03] | || [https://www.dropbox.com/s/p3auugqwc89132b/02book_sequentialOptimalAlgorithm.pdf?dl=0 ch02] [https://www.dropbox.com/s/b00dcqk1rob7rdz/03book_perceptron.pdf?dl=0 ch03] | ||
|| [https://www.dropbox.com/scl/fi/di1k87aq44ss07mq4s6pi/02sem.pdf?rlkey=yu476v8z77bal6ma029frnilm&dl=0 prob02] | || [https://www.dropbox.com/scl/fi/di1k87aq44ss07mq4s6pi/02sem.pdf?rlkey=yu476v8z77bal6ma029frnilm&dl=0 prob02] | ||
| − | || <-- | + | || <!-- [https://www.dropbox.com/scl/fi/d2wuka77bu18j9plivwl5/02sol.pdf?rlkey=yp2eprgxpc7r2antyidjd8qiw&dl=0 sol02] --> |
|- | |- | ||
| [https://www.youtube.com/watch?v=H7kvz2rxX4o ?? Sept] | | [https://www.youtube.com/watch?v=H7kvz2rxX4o ?? Sept] | ||
| Строка 38: | Строка 38: | ||
|| [https://www.dropbox.com/scl/fi/oetz6dwz77jdlta1k03li/04book_predictionWithExperts.pdf?rlkey=f0u947kiq9bjjaa46iv68lr7j&dl=0 ch04] [https://www.dropbox.com/scl/fi/cx7hsxzwg2f8ep4qcuefc/05book_introProbability.pdf?rlkey=rfq0y9cgzqvl1dlxkccc3qebv&dl=0 ch05] | || [https://www.dropbox.com/scl/fi/oetz6dwz77jdlta1k03li/04book_predictionWithExperts.pdf?rlkey=f0u947kiq9bjjaa46iv68lr7j&dl=0 ch04] [https://www.dropbox.com/scl/fi/cx7hsxzwg2f8ep4qcuefc/05book_introProbability.pdf?rlkey=rfq0y9cgzqvl1dlxkccc3qebv&dl=0 ch05] | ||
|| [https://www.dropbox.com/scl/fi/3w9yk9rb7l0pb2l6q94fc/03sem.pdf?rlkey=jt40nnw35t7e8je9nj1ef22f1&dl=0 prob03] | || [https://www.dropbox.com/scl/fi/3w9yk9rb7l0pb2l6q94fc/03sem.pdf?rlkey=jt40nnw35t7e8je9nj1ef22f1&dl=0 prob03] | ||
| − | || [https://www.dropbox.com/scl/fi/w26siug25arwoilgb5t7i/03sol.pdf?rlkey=7waf3ddt4fvz0xeicicayoilt&dl=0 sol03] | + | || <!-- [https://www.dropbox.com/scl/fi/w26siug25arwoilgb5t7i/03sol.pdf?rlkey=7waf3ddt4fvz0xeicicayoilt&dl=0 sol03] --> |
|- | |- | ||
| | | | ||
| Строка 48: | Строка 48: | ||
|| [https://www.dropbox.com/s/nh4puyv7nst4ems/06book_sampleComplexity.pdf?dl=0 ch06] | || [https://www.dropbox.com/s/nh4puyv7nst4ems/06book_sampleComplexity.pdf?dl=0 ch06] | ||
|| [https://www.dropbox.com/scl/fi/n4004z5mmrn3nbr9ggtt9/05sem.pdf?rlkey=sntvs95trliffbc2vh5vge06b&dl=0 prob05] | || [https://www.dropbox.com/scl/fi/n4004z5mmrn3nbr9ggtt9/05sem.pdf?rlkey=sntvs95trliffbc2vh5vge06b&dl=0 prob05] | ||
| − | || [https://www.dropbox.com/scl/fi/55530savq0vn6apra7oo4/05sol.pdf?rlkey=ql9q3a7s7k5dkggymul4p4s2o&dl=0 sol05] | + | || <!-- [https://www.dropbox.com/scl/fi/55530savq0vn6apra7oo4/05sol.pdf?rlkey=ql9q3a7s7k5dkggymul4p4s2o&dl=0 sol05] --> |
|- | |- | ||
| [https://www.youtube.com/watch?v=8J5B9CCy-ws ?? Oct] | | [https://www.youtube.com/watch?v=8J5B9CCy-ws ?? Oct] | ||
| Строка 55: | Строка 55: | ||
|| [https://www.dropbox.com/s/eurz2vkvt1wa5zm/07book_growthFunctions.pdf?dl=0 ch07] [https://www.dropbox.com/scl/fi/50oxlmjkx59hjrq82yqvx/08book_VCdimension.pdf?rlkey=5dtlcis378kqu24ttko6s7zpf&dl=0 ch08] | || [https://www.dropbox.com/s/eurz2vkvt1wa5zm/07book_growthFunctions.pdf?dl=0 ch07] [https://www.dropbox.com/scl/fi/50oxlmjkx59hjrq82yqvx/08book_VCdimension.pdf?rlkey=5dtlcis378kqu24ttko6s7zpf&dl=0 ch08] | ||
|| [https://www.dropbox.com/scl/fi/wvj022mv9w82mlynp4t28/06sem.pdf?rlkey=k8bieoxn7zlkfkzyhi311n26s&dl=0 prob06] | || [https://www.dropbox.com/scl/fi/wvj022mv9w82mlynp4t28/06sem.pdf?rlkey=k8bieoxn7zlkfkzyhi311n26s&dl=0 prob06] | ||
| − | || [https://www.dropbox.com/scl/fi/gcr4n00ef62ezrta7atll/06sol.pdf?rlkey=b9rgqxgmnlxouvsl5eevpwg3d&dl=0 sol06] | + | || <!-- [https://www.dropbox.com/scl/fi/gcr4n00ef62ezrta7atll/06sol.pdf?rlkey=b9rgqxgmnlxouvsl5eevpwg3d&dl=0 sol06] --> |
|- | |- | ||
| [https://www.youtube.com/watch?v=zHau8Br_UFQ ?? Oct] | | [https://www.youtube.com/watch?v=zHau8Br_UFQ ?? Oct] | ||
| Строка 62: | Строка 62: | ||
|| [https://www.dropbox.com/scl/fi/15zjsv1w9coq2py9djlai/09book_riskBounds.pdf?rlkey=4lnyo8kcd226qlybrdgyt36i8&dl=0 ch09] | || [https://www.dropbox.com/scl/fi/15zjsv1w9coq2py9djlai/09book_riskBounds.pdf?rlkey=4lnyo8kcd226qlybrdgyt36i8&dl=0 ch09] | ||
|| [https://www.dropbox.com/scl/fi/neso7q9vq8ouix208u841/07sem.pdf?rlkey=k8dxkxwqdxf3kjsclzt9vwiw5&dl=0 prob07] | || [https://www.dropbox.com/scl/fi/neso7q9vq8ouix208u841/07sem.pdf?rlkey=k8dxkxwqdxf3kjsclzt9vwiw5&dl=0 prob07] | ||
| − | || [https://www.dropbox.com/scl/fi/dw3u10rhy33pv37z5zf5m/07sol.pdf?rlkey=wssi52zoiveccmpy2197ry5pt&dl=0 sol07] | + | || <!-- [https://www.dropbox.com/scl/fi/dw3u10rhy33pv37z5zf5m/07sol.pdf?rlkey=wssi52zoiveccmpy2197ry5pt&dl=0 sol07] --> |
|- | |- | ||
| [https://www.youtube.com/watch?v=yMsUH1brAs8 ?? Oct] | | [https://www.youtube.com/watch?v=yMsUH1brAs8 ?? Oct] | ||
| Строка 69: | Строка 69: | ||
|| [https://www.dropbox.com/scl/fi/ohtmf1fwsu9c6vkrj6e5a/10book_measureConcentration.pdf?rlkey=dqsgskp8slui6xoq9c7tx680b&dl=0 ch10] [https://www.dropbox.com/s/hfrvhebbsskbk6g/11book_RademacherComplexity.pdf?dl=0 ch11] | || [https://www.dropbox.com/scl/fi/ohtmf1fwsu9c6vkrj6e5a/10book_measureConcentration.pdf?rlkey=dqsgskp8slui6xoq9c7tx680b&dl=0 ch10] [https://www.dropbox.com/s/hfrvhebbsskbk6g/11book_RademacherComplexity.pdf?dl=0 ch11] | ||
|| [https://www.dropbox.com/scl/fi/g278mmezenlyxd1my0ta9/08sem.pdf?rlkey=hvqmbumpd0xb6pumdgv5bqx6u&dl=0 prob08] | || [https://www.dropbox.com/scl/fi/g278mmezenlyxd1my0ta9/08sem.pdf?rlkey=hvqmbumpd0xb6pumdgv5bqx6u&dl=0 prob08] | ||
| − | || [https://www.dropbox.com/scl/fi/06yobqe58fiecsobp4yrb/08sol.pdf?rlkey=9c7t1y4nxxtg14vpndsyyko2u&dl=0 sol08] | + | || <!-- [https://www.dropbox.com/scl/fi/06yobqe58fiecsobp4yrb/08sol.pdf?rlkey=9c7t1y4nxxtg14vpndsyyko2u&dl=0 sol08] --> |
|- | |- | ||
| | | | ||
| Строка 79: | Строка 79: | ||
|| [https://www.dropbox.com/s/573a2vtjfx8qqo8/12book_regression.pdf?dl=0 ch12] [https://www.dropbox.com/scl/fi/hxeh5btc0bb2f52fnqh5f/13book_SVM.pdf?rlkey=dw3u2rtfstpsb8mi9hnuc8poy&dl=0 ch13] | || [https://www.dropbox.com/s/573a2vtjfx8qqo8/12book_regression.pdf?dl=0 ch12] [https://www.dropbox.com/scl/fi/hxeh5btc0bb2f52fnqh5f/13book_SVM.pdf?rlkey=dw3u2rtfstpsb8mi9hnuc8poy&dl=0 ch13] | ||
|| [https://www.dropbox.com/scl/fi/rp2m0dvovdjbvzdl7t1bl/09sem.pdf?rlkey=v1jsm5dagh7tymci5pkqn5gox&dl=0 prob09] | || [https://www.dropbox.com/scl/fi/rp2m0dvovdjbvzdl7t1bl/09sem.pdf?rlkey=v1jsm5dagh7tymci5pkqn5gox&dl=0 prob09] | ||
| − | || [https://www.dropbox.com/scl/fi/e598w1t8tzqxfvn1d4ww1/09sol.pdf?rlkey=yr1gzu8kg2rdkubaelicljj46&dl=0 sol09] | + | || <!-- [https://www.dropbox.com/scl/fi/e598w1t8tzqxfvn1d4ww1/09sol.pdf?rlkey=yr1gzu8kg2rdkubaelicljj46&dl=0 sol09] --> |
|- | |- | ||
| [https://youtu.be/9FhFxLHR4eE ?? Nov] | | [https://youtu.be/9FhFxLHR4eE ?? Nov] | ||
| Строка 86: | Строка 86: | ||
|| [https://www.dropbox.com/scl/fi/lozpqk5nnm8us77qfhn7x/14book_kernels.pdf?rlkey=s8e7a46rm3znkw13ubj3fzzz0&dl=0 ch14] | || [https://www.dropbox.com/scl/fi/lozpqk5nnm8us77qfhn7x/14book_kernels.pdf?rlkey=s8e7a46rm3znkw13ubj3fzzz0&dl=0 ch14] | ||
|| [https://www.dropbox.com/scl/fi/9mjmb6deu08ipf38s57bh/10sem.pdf?rlkey=z1khm4i8r39eeqmhargte24s4&dl=0 prob10] | || [https://www.dropbox.com/scl/fi/9mjmb6deu08ipf38s57bh/10sem.pdf?rlkey=z1khm4i8r39eeqmhargte24s4&dl=0 prob10] | ||
| − | || [https://www.dropbox.com/scl/fi/a5c0buap9b1h1ojdbhp3u/10sol.pdf?rlkey=8ft5tjyy1sl5dkj4p4hh8phbc&dl=0 sol10] | + | || <!-- [https://www.dropbox.com/scl/fi/a5c0buap9b1h1ojdbhp3u/10sol.pdf?rlkey=8ft5tjyy1sl5dkj4p4hh8phbc&dl=0 sol10] --> |
|- | |- | ||
| [https://www.youtube.com/watch?v=OgiaWrWh_WA ?? Nov] | | [https://www.youtube.com/watch?v=OgiaWrWh_WA ?? Nov] | ||
| Строка 93: | Строка 93: | ||
|| [https://www.dropbox.com/s/e7m1cs7e8ulibsf/15book_AdaBoost.pdf?dl=0 ch15] | || [https://www.dropbox.com/s/e7m1cs7e8ulibsf/15book_AdaBoost.pdf?dl=0 ch15] | ||
|| [https://www.dropbox.com/scl/fi/ykbzx314pdn3mn3jiehli/11sem.pdf?rlkey=hpmtks20a3k5zsvr8jm1iqc35&dl=0 prob11] | || [https://www.dropbox.com/scl/fi/ykbzx314pdn3mn3jiehli/11sem.pdf?rlkey=hpmtks20a3k5zsvr8jm1iqc35&dl=0 prob11] | ||
| − | || [https://www.dropbox.com/scl/fi/c805j4f54ioiozphvh9j0/11sol.pdf?rlkey=6rrxlweaiko1lm0z2ua4k7mqk&dl=0 sol11] | + | || <!-- [https://www.dropbox.com/scl/fi/c805j4f54ioiozphvh9j0/11sol.pdf?rlkey=6rrxlweaiko1lm0z2ua4k7mqk&dl=0 sol11] --> |
|- | |- | ||
| [https://youtu.be/GL574ljefJ8 ?? Nov] | | [https://youtu.be/GL574ljefJ8 ?? Nov] | ||
Версия 15:28, 13 сентября 2024
Содержание
General Information
Lectures: on TBA in room TBA and in zoom by Bruno Bauwens
Seminars: on TBA in room TBA and in TBA by Nikita Lukianenko.
To discuss the materials and practical issues, join the telegram group The course is similar to last year.
Course materials
| Video | Summary | Slides | Lecture notes | Problem list | Solutions |
|---|---|---|---|---|---|
| Part 1. Online learning | |||||
| ?? Sept | Philosophy. The online mistake bound model. The halving and weighted majority algorithms. | sl01 | ch00 ch01 | prob01 | |
| ?? Sept | The perceptron algorithm. Kernels. The standard optimal algorithm. | sl02 | ch02 ch03 | prob02 | |
| ?? Sept | Prediction with expert advice. Recap probability theory (seminar). | sl03 | ch04 ch05 | prob03 | |
| Part 2. Distribution independent risk bounds | |||||
| ?? Oct | Necessity of a hypothesis class. Sample complexity in the realizable setting, examples: threshold functions and finite classes. | sl04 | ch06 | prob05 | |
| ?? Oct | Growth functions, VC-dimension and the characterization of sample comlexity with VC-dimensions | sl05 | ch07 ch08 | prob06 | |
| ?? Oct | Risk decomposition and the fundamental theorem of statistical learning theory | sl06 | ch09 | prob07 | |
| ?? Oct | Bounded differences inequality, Rademacher complexity, symmetrization, contraction lemma. | sl07 | ch10 ch11 | prob08 | |
| Part 3. Margin risk bounds with applications | |||||
| ?? Nov | Simple regression, support vector machines, margin risk bounds, and neural nets with dropout regularization | sl08 | ch12 ch13 | prob09 | |
| ?? Nov | Kernels: RKHS, representer theorem, risk bounds | sl09 | ch14 | prob10 | |
| ?? Nov | AdaBoost and the margin hypothesis | sl10 | ch15 | prob11 | |
| ?? Nov | Implicit regularization of stochastic gradient descent in overparameterized neural nets (recording with many details about the Hessian) | ch16 ch17 | |||
| ?? Dec | Part 2 of previous lecture: Hessian control and stability of the NTK. |
Background on multi-armed bandits: A. Slivkins, [Introduction to multi-armed bandits https://arxiv.org/pdf/1904.07272.pdf], 2022.
The lectures in October and November are based on the book: Foundations of machine learning 2nd ed, Mehryar Mohri, Afshin Rostamizadeh, and Ameet Talwalker, 2018.
Grading formula
Final grade = 0.35 * [score of homeworks] + 0.35 * [score of colloquium] + 0.3 * [score on the exam] + bonus from quizzes.
All homework questions have the same weight. Each solved extra homework task increases the score of the final exam by 1 point.
There is no rounding except on the final grade. Arithmetic rounding is used.
Autogrades: if you only need 6/10 on the exam to pass with maximal final score, it will be given automatically. This may happen because of extra questions and bonuses from quizzes.
Homeworks
Deadline every 2 weeks, before the seminar at 16h00. Homework problems from
seminars 1 and 2 on September 25, seminars 3 and 4 on October 9, seminars 5 and 6 on November 6, seminars 7 and 8 on November 13, seminars 9 and 10 on November 27 December 4, seminar 11 before the start of the exam.
Email to brbauwens-at-gmail.com. Start the subject line with SLT-HW. Results will be here.
Late policy: 1 homework can be submitted at most 24 late without explanations.
Colloquium
Rules and questions from last year.
Date: TBA
Problems exam
TBA
-- You may use handwritten notes, lecture materials from this wiki (either printed or through your PC), Mohri's book
-- You may not search on the internet or interact with other humans (e.g. by phone, forums, etc)
Office hours
Bruno Bauwens: TBA
Nikita Lukianenko: Write in Telegram, the time is flexible
