BEGIN:VCALENDAR
VERSION:2.0
PRODID:Linklings LLC
BEGIN:VTIMEZONE
TZID:America/Chicago
X-LIC-LOCATION:America/Chicago
BEGIN:DAYLIGHT
TZOFFSETFROM:-0600
TZOFFSETTO:-0500
TZNAME:CDT
DTSTART:19700308T020000
RRULE:FREQ=YEARLY;BYMONTH=3;BYDAY=2SU
END:DAYLIGHT
BEGIN:STANDARD
TZOFFSETFROM:-0500
TZOFFSETTO:-0600
TZNAME:CST
DTSTART:19701101T020000
RRULE:FREQ=YEARLY;BYMONTH=11;BYDAY=1SU
END:STANDARD
END:VTIMEZONE
BEGIN:VEVENT
DTSTAMP:20260522T150117Z
LOCATION:D220
DTSTART;TZID=America/Chicago:20181111T141200
DTEND;TZID=America/Chicago:20181111T141500
UID:submissions.supercomputing.org_SC18_sess160_ws_whpc103@linklings.com
SUMMARY:Optimizing Python Data Processing for the DESI Experiment on the N
 ERSC Cori Supercomputer
DESCRIPTION:Laurie A. Stephey and Rollin C. Thomas (Lawrence Berkeley Nati
 onal Laboratory, National Energy Research Scientific Computing Center (NER
 SC)) and Stephen J. Bailey (Lawrence Berkeley National Laboratory)\n\nThe 
 goal of the Dark Energy Spectroscopic Instrument (DESI) experiment is to b
 etter understand dark energy by making the most detailed 3D map of the uni
 verse to date. The images obtained each night over a period of 5 years sta
 rting in 2019 will be sent to the NERSC Cori supercomputer for processing 
 and scientific analysis. \n\nThe DESI spectroscopic pipeline for processin
 g these data is written exclusively in Python. Writing in Python allows th
 e DESI scientists to write very readable scientific code in a relatively s
 hort amount of time. However, the drawback is that Python can be substanti
 ally slower than more traditional HPC languages like C, C++, and Fortran. 
 \n\nThe goal of this work is to increase the efficiency of the DESI spectr
 oscopic data processing at NERSC while satisfying their requirement that t
 he software remain in Python. As of this writing we have obtained speedups
  of over 6x and 7x on the Cori Haswell and KNL partitions, respectively. S
 everal profiling techniques were used to determine potential areas for imp
 rovement including Python's cProfile, line_profiler, Intel Vtune, and Tau.
  Once we identified expensive kernels, we used the following techniques: 1
 ) JIT-compiling hotspots using Numba (the most successful strategy so far)
 , 2) reducing MPI data transfer where possible (i.e. replacing broadcast o
 perations with scatter), and 3) re-structuring the code to compute and sto
 re important data rather than repeatedly calling expensive functions. We w
 ill continue using these strategies and also explore the requirements for 
 future architectures (for example, transitioning the DESI workload to GPUs
 ).\n\nTag: Diversity, Education, Hot Topics\n\nRegistration Category: Work
 shop Reg Pass\n\nSession Chairs: Toni Collis (Women in High Performance Co
 mputing); Weronika Filinger (Edinburgh Parallel Computing Centre (EPCC); U
 niversity of Edinburgh, Scotland); and Misbah Mubarak (Amazon Web Services
 )\n\n
END:VEVENT
END:VCALENDAR