Difference between revisions of "Atlas:Functionnal-test-lyon-oct06"

Un article de lcgwiki.
Jump to: navigation, search
 
(→‎News and actions)
 
(12 intermediate revisions by 8 users not shown)
Ligne 2: Ligne 2:
 
a performance measurement yet.
 
a performance measurement yet.
  
−
* 15 October : Start of the test
+
== News and actions ==
  
−
[[Atlas | Page principale du Twiki ATLAS]]
+
* 27 October:
 +
** Problem of BNL-LYON resolved (pending since 18 October)
 +
** All FTS submissions are blocked in 'Submit' mode. Requested intervention by LYON team
 +
* 26 October :
 +
** D. Cameron corrected DDM monitoring because files already transfered during T0->T1 are not monitored when doing T2->T1. But induced a bug in the speed transfer (80 MB/s LYON->T2 is wrong). Bug is corrected
 +
but not yet validated.
 +
* 25 October :
 +
** It looks that T2->T1 transfers went well. But no measurement because of limitation in DDM monitoring
 +
** Transfer of 3 GB files from CERN to T1s. OK for LYON but problems in dcache for files > 2 GB appeared
 +
(ls -l says 1 B) and dccp does not work in BNL.
 +
* 24 October :
 +
** FTS server in Lyon seems to be overloaded (appear 'File exists' in T2)
 +
** One of the dcache servers for LYONDISK (but were are the data of this test) cannot sustain the load ('gridftp errors')
 +
** Results for the FR cloud:
 +
LYON - 396 files transferred
 +
        Beijing, LAL done
 +
        CPPM  50% done (50% ASSIGNED)
 +
        LAPP  70% done (30% in progress)
 +
        LPC  30%      (70%)
 +
        LPNHE 20%      (80%)
 +
        SACLAY60%      (40%)
 +
        TOKYO 70%      (30%)
 +
* 23 October :
 +
** BNL accessible again (routing problem in Lyon)
 +
** All ddmf1 files reached LYONDISK ( ~40 files per dataset)
 +
* 22 October : Start of the test
 +
* 21 October :
 +
** No possible transfer between T2s and Lyon T1 (all FTS jobs in Pending mode)
 +
** No contact with BNL from LYON (FTS, srm-get-metadata, afs). Not the case  for TRIUMF or if test is done from CERN
 +
* 19 October : Available disk space:
 +
** sedpm.mrs.grid.cnrs.fr : 188 GB
 +
** lpnse1.in2p3.fr : 2790 GB
 +
** lapp-se01.in2p3.fr 804 GB
 +
** clrlcgse03.in2p3.fr: 2270 GB
 +
** grid05.lal.in2p3.fr: 1159 GB
 +
** node12.datagrid.cea.fr 3760 GB
 +
** dgse0.icepp.jp: 6760 GB
 +
** atlasse01.ihep.ac.cn: 1018 GB
 +
 
 +
== [http://lcg2.in2p3.fr/wiki/index.php/Atlas#Resources_monitoring Ressource monitoring] ==
 +
 
 +
== Summary ==
 +
'''''mail from Alexei Klimentov (7 October 2006) '''''
 +
 
 +
* The next DDM functional test is scheduled for October 2006 after SC4
 +
* We will start to generate data in a week from now (Oct 15th)
 +
* Please update table 'Sites' (DDM ops WiKi Functional test) for your site. Data will be generated only for the sites listed in the table. (DDM ops WiKi : https://twiki.cern.ch/twiki/bin/view/Atlas/DDMOperations)
 +
*Test duration 7 - 10 days
 +
 
 +
*The test scope (preliminary) :
 +
:repeat Step1 and Step2 of the previous test
 +
:Step 1: CERN->Tier1->Tier2s data transfer
 +
:Step 2: Tier2s->Tier1 data transfer
 +
 
 +
Bulk data transfer
 +
:CERN -> Tier1s
 +
:Tier1->Tier2s (2-3 Tier-2s per cloud)
 +
 
 +
*Data transfer of large 2+GB files (if available)
 +
*Evaluation of data transfer robustness vs dataset size
 +
*FTS open access  (data transfer between different clouds)
 +
*Two more functional tests are already scheduled Year 2006 (week 47-48, TBC), and Jan 2007. Probably in January we will try to evaluate data transfer performance.
 +
 
 +
 
 +
* several suggestions, comments, questions
 +
** will we use DDM ops mailing list to discuss test scope or HyperNews ?
 +
** Tier-1s will need 200-300GB disk space for the October test
 +
** Tier-2s (if they will participate in bulk data transfer) will need 100-200 GB disk space for the October test
 +
** There are no special requirements for the SW installation. Run what you are using for SC4
 +
 
 +
==[[Atlas | Page principale du Twiki ATLAS]]==

Latest revision as of 18:17, 29 octobre 2006

The goal of this test is to check that data produced can be moved between sites (functionnalities are working). It is not a performance measurement yet.

News and actions

  • 27 October:
    • Problem of BNL-LYON resolved (pending since 18 October)
    • All FTS submissions are blocked in 'Submit' mode. Requested intervention by LYON team
  • 26 October :
    • D. Cameron corrected DDM monitoring because files already transfered during T0->T1 are not monitored when doing T2->T1. But induced a bug in the speed transfer (80 MB/s LYON->T2 is wrong). Bug is corrected

but not yet validated.

  • 25 October :
    • It looks that T2->T1 transfers went well. But no measurement because of limitation in DDM monitoring
    • Transfer of 3 GB files from CERN to T1s. OK for LYON but problems in dcache for files > 2 GB appeared

(ls -l says 1 B) and dccp does not work in BNL.

  • 24 October :
    • FTS server in Lyon seems to be overloaded (appear 'File exists' in T2)
    • One of the dcache servers for LYONDISK (but were are the data of this test) cannot sustain the load ('gridftp errors')
    • Results for the FR cloud:

LYON - 396 files transferred 

       Beijing, LAL done
       CPPM  50% done (50% ASSIGNED)
       LAPP  70% done (30% in progress)
       LPC   30%      (70%)
       LPNHE 20%      (80%)
       SACLAY60%      (40%)
       TOKYO 70%      (30%)
  • 23 October :
    • BNL accessible again (routing problem in Lyon)
    • All ddmf1 files reached LYONDISK ( ~40 files per dataset)
  • 22 October : Start of the test
  • 21 October :
    • No possible transfer between T2s and Lyon T1 (all FTS jobs in Pending mode)
    • No contact with BNL from LYON (FTS, srm-get-metadata, afs). Not the case for TRIUMF or if test is done from CERN
  • 19 October : Available disk space:
    • sedpm.mrs.grid.cnrs.fr : 188 GB
    • lpnse1.in2p3.fr : 2790 GB
    • lapp-se01.in2p3.fr 804 GB
    • clrlcgse03.in2p3.fr: 2270 GB
    • grid05.lal.in2p3.fr: 1159 GB
    • node12.datagrid.cea.fr 3760 GB
    • dgse0.icepp.jp: 6760 GB
    • atlasse01.ihep.ac.cn: 1018 GB

Ressource monitoring

Summary

mail from Alexei Klimentov (7 October 2006)

  • The next DDM functional test is scheduled for October 2006 after SC4
  • We will start to generate data in a week from now (Oct 15th)
  • Please update table 'Sites' (DDM ops WiKi Functional test) for your site. Data will be generated only for the sites listed in the table. (DDM ops WiKi : https://twiki.cern.ch/twiki/bin/view/Atlas/DDMOperations)
  • Test duration 7 - 10 days
  • The test scope (preliminary) :
repeat Step1 and Step2 of the previous test
Step 1: CERN->Tier1->Tier2s data transfer
Step 2: Tier2s->Tier1 data transfer

Bulk data transfer

CERN -> Tier1s
Tier1->Tier2s (2-3 Tier-2s per cloud)
  • Data transfer of large 2+GB files (if available)
  • Evaluation of data transfer robustness vs dataset size
  • FTS open access (data transfer between different clouds)
  • Two more functional tests are already scheduled Year 2006 (week 47-48, TBC), and Jan 2007. Probably in January we will try to evaluate data transfer performance.


  • several suggestions, comments, questions
    • will we use DDM ops mailing list to discuss test scope or HyperNews ?
    • Tier-1s will need 200-300GB disk space for the October test
    • Tier-2s (if they will participate in bulk data transfer) will need 100-200 GB disk space for the October test
    • There are no special requirements for the SW installation. Run what you are using for SC4

Page principale du Twiki ATLAS

Récupérée de « http://lcg.in2p3.fr/wiki/index.php?title=Atlas:Functionnal-test-lyon-oct06&oldid=2432 »