EE Seminar: Expressive Efficiency and Inductive Bias of Convolutional Networks: Analysis and Design through Hierarchical Tensor Decompositions
(The talk will be given in English)
Speakers: Dr. Nadav Cohen
School of Computer Science and Engineering, Hebrew University
Monday, May 8th, 2017
15:00 - 16:00
Room 011, Kitot Bldg., Faculty of Engineering
Expressive Efficiency and Inductive Bias of Convolutional Networks: Analysis and Design through Hierarchical Tensor Decompositions
Abstract
The driving force behind convolutional networks - the most successful deep learning architecture to date, is their expressive power. Despite its wide acceptance and vast empirical evidence, formal analyses supporting this belief are scarce. The primary notions for formally reasoning about expressiveness are efficiency and inductive bias. Efficiency refers to the ability of a network architecture to realize functions that require an alternative architecture to be much larger. Inductive bias refers to the prioritization of some functions over others given prior knowledge regarding a task at hand. Through an equivalence to hierarchical tensor decompositions, we study the expressive efficiency and inductive bias of various architectural features in convolutional networks (depth, width, pooling geometry, inter-connectivity, overlapping operations etc). Our results shed light on the demonstrated effectiveness of convolutional networks, and in addition, provide new tools for network design.
The talk is based on a series of works published in venues such as COLT, ICML, CVPR and ICLR, with collaborators Or Sharir, Ronen Tamari, David Yakira, Yoav Levine and Amnon Shashua.
Bio
Nadav Cohen is concluding his PhD at the School of Computer Science and Engineering in the Hebrew University of Jerusalem, under the supervision of Prof. Amnon Shashua. His research focuses on the theoretical and algorithmic foundations of deep learning. In particular, he is interested in the application of tensor analysis for the study of convolutional network architectures.
Nadav holds a BSc in electrical engineering and a BSc in mathematics (both summa cum laude), conducted under the Technion Excellence Program for distinguished undergraduates. For his research in graduate studies, he was awarded the Google Doctoral Fellowship in Machine Learning and the Rothschild Postdoctoral Fellowship.
Starting September 2017, Nadav will be a postdoctoral member at the Institute for Advanced Study (IAS) in Princeton, hosted by Prof. Sanjeev Arora, Elad Hazan and Yoram Singer.
