EE Seminar: Efficient Machine Learning: Fewer Parameters, Less Data, Better Initialization
https://tau-ac-il.zoom.us/j/88213860721?pwd=4dEhqsyvBuVusyLgIGbZZpYB5qc2pX.1
Meeting ID: 882 1386 0721
Passcode: 236297
Electrical Engineering Systems ZOOM Seminar
Speaker: Noga Bar
Ph.D. student under the supervision of Prof. Raja Giryes
Monday, 27th April 2025, at 16:00
Efficient Machine Learning: Fewer Parameters, Less Data, Better Initialization
Abstract
Modern machine learning systems achieve impressive performance, but often at a substantial cost in terms of computation, model size, and data requirements. This raises a fundamental question: how can we make learning both efficient, and how can we design methods that make better use of available resources
In this talk, I will present a perspective on efficient machine learning through three complementary directions. First, I will discuss how pruning at initialization can identify effective subnetworks before training, reducing the number of parameters without sacrificing performance. I will further show how sketching-based techniques can be used to improve both the performance and the theoretical understanding of such subnetworks.
Next, I will demonstrate how learning can be performed with only a small number of examples, selected in an unsupervised manner by identifying informative and diverse training points without relying on labels.
Finally, I will present theoretical results on recurrent neural network initialization in regimes where both the width and the sequence length grow to infinity. These results show that Glorot initialization, which is often taken as a standard baseline, breaks down when the sequence length is allowed to grow, revealing a mismatch between standard practice and theoretical analyses in more realistic long-sequence settings. This highlights and helps address a gap in our understanding of initialization in recurrent architectures.
-סמינר זה ייחשב כסמינר שמיעה לתלמידי תואר שני ושלישי-
This Seminar Is Considered A Hearing Seminar For Msc/Phd Students-
כדי לקבל קרדיט שמיעה יש לחתום בצ'ט שם מלא ומספר ת.ז.
Speaker: Noga Bar
Ph.D. student under the supervision of Prof. Raja Giryes
Monday, April 27th at 16:00
Efficient Machine Learning: Fewer Parameters, Less Data, Better Initialization
Abstract:
Modern machine learning systems achieve impressive performance, but often at a substantial cost in terms of computation, model size, and data requirements. This raises a fundamental question: how can we make learning both efficient, and how can we design methods that make better use of available resources
In this talk, I will present a perspective on efficient machine learning through three complementary directions. First, I will discuss how pruning at initialization can identify effective subnetworks before training, reducing the number of parameters without sacrificing performance. I will further show how sketching-based techniques can be used to improve both the performance and the theoretical understanding of such subnetworks.
Next, I will demonstrate how learning can be performed with only a small number of examples, selected in an unsupervised manner by identifying informative and diverse training points without relying on labels.
Finally, I will present theoretical results on recurrent neural network initialization in regimes where both the width and the sequence length grow to infinity. These results show that Glorot initialization, which is often taken as a standard baseline, breaks down when the sequence length is allowed to grow, revealing a mismatch between standard practice and theoretical analyses in more realistic long-sequence settings. This highlights and helps address a gap in our understanding of initialization in recurrent architectures.
