Long Feng Science Forum Seminar Series | Seminar #10

Dear All,

You are cordially invited to the tenth seminar of Long Feng Science Forum Seminar Series. It will be delivered by Prof. Zhi-Qin John XU (Shanghai Jiao Tong University) on October 10 (Monday). This seminar will discuss the "Simple Implicit Regularizations in Deep Learning".

Seminar Information

Time & Date: 9-10 am on October 10 (Monday), Beijing time

Host: Prof. Dong WANG, The Chinese University of Hong Kong, Shenzhen

Speaker: Prof. Zhi-Qin John XU,  Shanghai Jiao Tong University

Abstract: 

Why do neural network models that look so complex usually generalize well? To understand this problem, we study deep learning training and find some simple implicit regularization effects. The first is the frequency principle that neural networks learn from low frequency to high frequency. In this presentation, I will show some understanding and application based on the frequency principle. The second is the parameter condensation effect, which makes the number of effective neurons in large networks far less than the actual number of neurons. We attempt to understand the mechanism of condensation from the embedding principle of loss landscape and gradient flow. We further derive the implicit regularization of Dropout and find that it can promote condensation formation and limit the complexity of model fitting. These implicit regularizations all reflect the characteristic that neural networks tend to use simple functions to fit data in the training process, so as to achieve better generalization. Finally, I will discuss some important theoretical issues in neural networks.

Biography

Prof. Zhi-Qin John XU

Associate Professor

Institute of Natural Sciences,

School of Mathematical Sciences,

Shanghai Jiao Tong University

Zhi-Qin John Xu is an associate professor at Shanghai Jiao Tong University (SJTU). Zhi-Qin obtains B.S. in Physics (2012) and a Ph.D. degree in Mathematics (2016) from SJTU. Before joining SJTU, Zhi-Qin worked as a postdoc at NYUAD and Courant Institute from 2016 to 2019. He published papers in the Journal of Machine Learning Research, AAAI, NeurIPS, Communications in Computational Physics, SIMODS, European Journal of Neuroscience and Communications in Mathematical Sciences etc. He is a managing editor of the Journal of Machine Learning.

Participation

A. Zoom Online Meeting

https://us06web.zoom.us/j/87145493057?pwd=elVVT1V2Wk1hNzFmcmJmcytLSU5HZz09

Meeting ID: 871 4549 3057

Passcode: 1010

B. CUHK(SZ) WeChat Channel

Please feel free to scan the QR code below to access  CUHK(SZ) WeChat Channel to participate in the live broadcasting:

C. Weizan Live Broadcasting Platform

Or visit the Weizan live broadcasting platform (Welcome to click "Read more" and access the platform):

https://wx.vzan.com/plug-ins/?v=637922847453656237#/FixupIndex/980009879?shareuid=415330463

点击以下链接，进入理工时刻：