BEGIN:VCALENDAR
VERSION:2.0
PRODID:Linklings LLC
BEGIN:VTIMEZONE
TZID:America/Chicago
X-LIC-LOCATION:America/Chicago
BEGIN:DAYLIGHT
TZOFFSETFROM:-0600
TZOFFSETTO:-0500
TZNAME:CDT
DTSTART:19700308T020000
RRULE:FREQ=YEARLY;BYMONTH=3;BYDAY=2SU
END:DAYLIGHT
BEGIN:STANDARD
TZOFFSETFROM:-0500
TZOFFSETTO:-0600
TZNAME:CST
DTSTART:19701101T020000
RRULE:FREQ=YEARLY;BYMONTH=11;BYDAY=1SU
END:STANDARD
END:VTIMEZONE
BEGIN:VEVENT
DTSTAMP:20260522T150116Z
LOCATION:D173
DTSTART;TZID=America/Chicago:20181111T122000
DTEND;TZID=America/Chicago:20181111T123000
UID:submissions.supercomputing.org_SC18_sess163_ws_works114@linklings.com
SUMMARY:End-to-End Online Performance Data Capture and Analysis for Scient
 ific Workflows
DESCRIPTION:George Papadimitriou (University of Southern California), Cong
  Wang (Renaissance Computing Institute (RenCI)), Karan Vahi and Rafael Fer
 reira da Silva (University of Southern California), Anirban Mandal (Renais
 sance Computing Institute (RenCI)), Zhengchun Liu (Argonne National Labora
 tory), Rajiv Mayani and Mats Rynge (University of Southern California), Ma
 riam Kiran (Energy Sciences Network (ESnet)), Vickie E. Lynch (Oak Ridge N
 ational Laboratory), Rajkumar Kettimuthu (Argonne National Laboratory), Ew
 a Deelman (University of Southern California), Jeffrey S. Vetter (Oak Ridg
 e National Laboratory), and Ian T. Foster (Argonne National Laboratory)\n\
 nWith the increased prevalence of employing workflows for scientific compu
 ting and a push toward exascale computing, it has become paramount that we
  are able to analyze characteristics of scientific applications to better 
 understand the impact on the underlying infrastructure and vice-versa. Suc
 h analysis can help drive the design, development, and optimization of the
 se next generation systems and solutions. In this paper, we present the ar
 chitecture, integration with existing well-established and newly developed
  tools, to collect online performance statistics of workflow executions fr
 om various, heterogeneous sources and publish them in a distributed databa
 se (Elasticsearch). Using this architecture, we are able to correlate onli
 ne workflow performance data with data from the underlying infrastructure,
  and present them in a useful and intuitive way via an online dashboard. W
 e have validated our approach by executing two classes of real-world workf
 lows, both under normal and anomalous conditions. The first is an I/O-inte
 nsive genome analysis workflow; the second, a CPU- and memory-intensive ma
 terial science workflow. Based on the data collected in Elasticsearch, we 
 are able to demonstrate that we can correctly identify anomalies that we i
 njected. We identify this end-to-end data collection of workflow performan
 ce data as an important resource of training data for automated machine le
 arning analysis.\n\nTag: Reproducibility, Scientific Computing, Scientific
  Workflows, Workflows, HPC, Data Intensive\n\nRegistration Category: Works
 hop Reg Pass\n\n
END:VEVENT
END:VCALENDAR