Difference between revisions of "Atlas:Analysis Challenge"

Un article de lcgwiki.
Jump to: navigation, search
Ligne 14: Ligne 14:
 
* GRIF and CC-IN2P3 TOP BDII : should be available as they are used remotely by some sites
 
* GRIF and CC-IN2P3 TOP BDII : should be available as they are used remotely by some sites
 
==== FR Cloud ST (2009 plans) ====
 
==== FR Cloud ST (2009 plans) ====
* Weekly ST : 50 jobs per site  
+
* Weekly ST : 50 jobs per site - all sites involved
 +
  LPSC : ready to be tested but only 16 DS available
 +
  BEIJING : Validation on-going - Available DS on MCDISK 68
 
* Optimization and Site Stress test: On demand and dedicated to sites which are interrested.  
 
* Optimization and Site Stress test: On demand and dedicated to sites which are interrested.  
 
  GRIF : GRIFOPN test (as soon as it is available) Total of 1000 jobs will
 
  GRIF : GRIFOPN test (as soon as it is available) Total of 1000 jobs will
Ligne 26: Ligne 28:
 
       DS available on IN2P3-LPC_MCDISK: 326
 
       DS available on IN2P3-LPC_MCDISK: 326
 
  LAPP-CPPM : interrested by a stress test ~130 DS available there
 
  LAPP-CPPM : interrested by a stress test ~130 DS available there
LPSC : ready to be tested but only 16 DS available
 
 
  TOKYO : Input DS available on TOKYO-LCG2_MCDISK: 1000
 
  TOKYO : Input DS available on TOKYO-LCG2_MCDISK: 1000
BEIJING : Validation on-going - Available DS on MCDISK 68
 
 
==== First exercise on the FR Cloud (December 2008) ====
 
==== First exercise on the FR Cloud (December 2008) ====
 
=====  Phase 1 : Site stress test oraganized by ATLAS and run centrally in a controlled manner (2 days) =====
 
=====  Phase 1 : Site stress test oraganized by ATLAS and run centrally in a controlled manner (2 days) =====

Version du 16:32, 23 janvier 2009

02/12/08 : E.Lançon, F.Chollet (Thanks to Cédric Serfon)

Information & Contact

Mailing list ATLAS-LCG-OP-L@in2p3.fr

Goals

  • measure "real" analysis job efficiency and turn around on several sites of a given cloud
  • measure data access performance
  • check load balancing between different users and different analysis tools (Ganga vs pAthena)
  • check load balancing between analysis and MC production

Required services @ T1 and GRIF

  • LFC catalog : lfc-prod.in2p3.fr
  • ATLAS Disk space : ATLASUSERDISK on T1 SE (fail-over for outputs in case of problems with T2 disk storage)
  • GRIF and CC-IN2P3 TOP BDII : should be available as they are used remotely by some sites

FR Cloud ST (2009 plans)

  • Weekly ST : 50 jobs per site - all sites involved
 LPSC : ready to be tested but only 16 DS available
 BEIJING : Validation on-going - Available DS on MCDISK 68 
  • Optimization and Site Stress test: On demand and dedicated to sites which are interrested.
GRIF : GRIFOPN test (as soon as it is available) Total of 1000 jobs will
be distributed to ATLAS CEs with following DS available being accessed
from LAL, SACLAY and LPNHE
       - GRIF-SACLAY_MCDISK: 332
       - GRIF-LPNHE_MCDISK: 334
       - GRIF-LAL_MCDISK: 457
LPC : to be planned when new storage arrays installation and network  
      upgrade done
      DS available on IN2P3-LPC_MCDISK: 326
LAPP-CPPM : interrested by a stress test ~130 DS available there
TOKYO : Input DS available on TOKYO-LCG2_MCDISK: 1000

First exercise on the FR Cloud (December 2008)

Phase 1 : Site stress test oraganized by ATLAS and run centrally in a controlled manner (2 days)

DA challenges have been performed on IT and DE clouds in october 08. Proposition has been made to extend this cloud-by cloud challenge to the FR Cloud. See ATLAS coordination DA challenge meeting (Nov. 20)

First exercise will help to identify breaking points and bottlenecks. It is limited in time (a few days) and requires careful attention of site administrators during that period,in particular network (internal & external), disk, cpu monitoring. This first try (Stress tests) can be run centrally in a controlled manner. The testing framework is ganga-based.

Sites: IN2P3-LPC, GRIF-LPNHE, TOKYO-LCG2, IN2P3-CPPM 
Max Jobs Per Site: 300
Sites: TOKYO
Max Jobs Per Site: 300
Input DS Patterns: mc08.*Wmunu*.recon.AOD.e*_s*_r5*tid* 
                   mc08.*Zprime_mumu*.recon.AOD.e*_s*_r5*tid* 
                   mc08.*Zmumu*.recon.AOD.e*_s*_r5*tid* 
                   mc08.*T1_McAtNlo*.recon.AOD.e*_s*_r5*tid* 
                   mc08.*H*zz4l*.recon.AOD.e*_s*_r5*tid* 
                   mc08.*.recon.AOD.e*_s*_r5*tid*
ST summary
Phase 2 : Pathena Analysis Challenge
  • Data Analysis exercice open to physicists with their favorite application
  • Physicists involved : Julien Donini, Arnaud Lucotte, Bertrand Brelier, Eric Lançon, LAL ?, LPNHE ?

Planning

  • Dec 8 : stop of MC production
  • Dec. 8-9: 1rst round with Tokyo, CPPM, LPC (LAN limited to 1Gbps), GRIF-LPNHE
  • Dec 17 : restart of MC production
  • Dec 14 : stop of MC production
  • Dec. 15-16 : 2nd round with LAPP, CC-IN2P3-T2 (to be confirmed), Tokyo, CPPM, LPC, possibly GRIF (SACLAY, IRFU, LPNHE), RO-07 and RO-02
  • Dec 17 : restart of MC production
  • Dec 17 : Beginning of Analysis Challenge (Phase 2)

Target and metrics

  • Nb of events : Few hundred up to 1000 jobs/site
  • Rate (evt/s) : up to 15 Hz
  • Efficiency (success/failure rate) : 80 %
  • CPU utilization : CPUtime / Walltime > 50 %