BEGIN:VCALENDAR
VERSION:2.0
PRODID:Linklings LLC
BEGIN:VTIMEZONE
TZID:America/Chicago
X-LIC-LOCATION:America/Chicago
BEGIN:DAYLIGHT
TZOFFSETFROM:-0600
TZOFFSETTO:-0500
TZNAME:CDT
DTSTART:19700308T020000
RRULE:FREQ=YEARLY;BYMONTH=3;BYDAY=2SU
END:DAYLIGHT
BEGIN:STANDARD
TZOFFSETFROM:-0500
TZOFFSETTO:-0600
TZNAME:CST
DTSTART:19701101T020000
RRULE:FREQ=YEARLY;BYMONTH=11;BYDAY=1SU
END:STANDARD
END:VTIMEZONE
BEGIN:VEVENT
DTSTAMP:20260522T150115Z
LOCATION:D167/174
DTSTART;TZID=America/Chicago:20181112T143000
DTEND;TZID=America/Chicago:20181112T150000
UID:submissions.supercomputing.org_SC18_sess151_ws_mlhpce129@linklings.com
SUMMARY:Aluminum: An Asynchronous, GPU-Aware Communication Library Optimiz
 ed for Large-Scale Training of Deep Neural Networks on HPC Systems
DESCRIPTION:Nikoli Dryden (University of Illinois, Lawrence Livermore Nati
 onal Laboratory); Naoya Maruyama, Tim Moon, Tom Benson, and Andy Yoo (Lawr
 ence Livermore National Laboratory); Marc Snir (University of Illinois); a
 nd Brian Van Essen (Lawrence Livermore National Laboratory)\n\nWe identify
  communication as a major bottleneck for training deep neural networks on 
 large-scale GPU clusters, taking over 10x as long as computation. To reduc
 e this overhead, we discuss techniques to overlap communication and comput
 ation as much as possible. This leads to much of the communication being l
 atency-bound instead of bandwidth-bound, and we find that using a combinat
 ion of latency- and bandwidth-optimized allreduce algorithms significantly
  reduces communication costs. We also discuss a semantic mismatch between 
 MPI and CUDA that increases overheads and limits asynchrony, and propose a
  solution that enables communication to be aware of CUDA streams. We imple
 ment these optimizations in the open-source Aluminum communication library
 , enabling optimized, asynchronous, GPU-aware communication. Aluminum demo
 nstrates improved performance in benchmarks and end-to-end training of dee
 p networks, for both strong and weak scaling.\n\nTag: Deep Learning, Machi
 ne Learning\n\nRegistration Category: Workshop Reg Pass\n\n
END:VEVENT
END:VCALENDAR