BEGIN:VCALENDAR
VERSION:2.0
PRODID:Linklings LLC
BEGIN:VTIMEZONE
TZID:America/Chicago
X-LIC-LOCATION:America/Chicago
BEGIN:DAYLIGHT
TZOFFSETFROM:-0600
TZOFFSETTO:-0500
TZNAME:CDT
DTSTART:19700308T020000
RRULE:FREQ=YEARLY;BYMONTH=3;BYDAY=2SU
END:DAYLIGHT
BEGIN:STANDARD
TZOFFSETFROM:-0500
TZOFFSETTO:-0600
TZNAME:CST
DTSTART:19701101T020000
RRULE:FREQ=YEARLY;BYMONTH=11;BYDAY=1SU
END:STANDARD
END:VTIMEZONE
BEGIN:VEVENT
DTSTAMP:20260522T150123Z
LOCATION:D167/174
DTSTART;TZID=America/Chicago:20181112T140000
DTEND;TZID=America/Chicago:20181112T143000
UID:submissions.supercomputing.org_SC18_sess151_ws_mlhpce117@linklings.com
SUMMARY:On Adam-Trained Models and a Parallel Method to Improve the Genera
 lization Performance
DESCRIPTION:Guojing Cong (IBM) and Luca Buratti (IBM, University of Bologn
 a)\n\nAdam is a popular stochastic optimizer that uses adaptive estimates 
 of lower-order moments to update weights and requires little hyper-paramet
 er tuning. Some recent studies have called the generalization and out-of-s
 ample behavior of such adaptive gradient methods into question, and argued
  that such methods are of only marginal value. Notably for many of the wel
 l-known image classification tasks such as CIFAR-10 and ImageNet-1K, curre
 nt models with best validation performance are still trained with SGD with
  a manual schedule of learning rate reduction.  \n\nWe analyze Adam and SG
 D trained models for 7 popular neural network architectures for image clas
 sification tasks using the CIFAR-10 dataset.  Visualization shows that for
  classification Adam trained models frequently "focus''  on areas of the i
 mages not occupied by the objects to be classified.  Weight statistics rev
 eal that Adam trained models have larger weights and L2 norms than SGD tra
 ined ones. Our experiments show that weight decay and reducing the initial
  learning rates improves generalization performance of Adam, but there sti
 ll remains a gap between Adam and SGD trained models. \n\nTo bridge the ge
 neralization gap, we adopt a K-step model averaging parallel algorithm wit
 h the Adam optimizer. With very sparse communication, the algorithm achiev
 es high parallel efficiency. For the 7 models on average the improvement i
 n validation accuracy over SGD is 0.72%, and the average parallel speedup 
 is 2.5 times with 6 GPUs.\n\nTag: Deep Learning, Machine Learning\n\nRegis
 tration Category: Workshop Reg Pass\n\n
END:VEVENT
END:VCALENDAR