Atlas:Functionnal-test-lyon-oct06

The goal of this test is to check that data produced can be moved between sites (functionnalities are working). It is not a performance measurement yet.

News and actions

• 27 October:
  • Problem of BNL-LYON resolved (pending since 18 October)
  • All FTS submissions are blocked in 'Submit' mode. Requested intervention by LYON team
• 26 October :
  • D. Cameron corrected DDM monitoring because files already transfered during T0->T1 are not monitored when doing T2->T1. But induced a bug in the speed transfer (80 MB/s LYON->T2 is wrong). Bug is corrected

but not yet validated.

• 25 October :
  • It looks that T2->T1 transfers went well. But no measurement because of limitation in DDM monitoring
  • Transfer of 3 GB files from CERN to T1s. OK for LYON but problems in dcache for files > 2 GB appeared

(ls -l says 1 B) and dccp does not work in BNL.

• 24 October :
  • FTS server in Lyon seems to be overloaded (appear 'File exists' in T2)
  • One of the dcache servers for LYONDISK (but were are the data of this test) cannot sustain the load ('gridftp errors')
  • Results for the FR cloud:

LYON - 396 files transferred

       Beijing, LAL done
       CPPM  50% done (50% ASSIGNED)
       LAPP  70% done (30% in progress)
       LPC   30%      (70%)
       LPNHE 20%      (80%)
       SACLAY60%      (40%)
       TOKYO 70%      (30%)

• 23 October :
  • BNL accessible again (routing problem in Lyon)
  • All ddmf1 files reached LYONDISK ( ~40 files per dataset)
• 22 October : Start of the test
• 21 October :
  • No possible transfer between T2s and Lyon T1 (all FTS jobs in Pending mode)
  • No contact with BNL from LYON (FTS, srm-get-metadata, afs). Not the case for TRIUMF or if test is done from CERN
• 19 October : Available disk space:
  • sedpm.mrs.grid.cnrs.fr : 188 GB
  • lpnse1.in2p3.fr : 2790 GB
  • lapp-se01.in2p3.fr 804 GB
  • clrlcgse03.in2p3.fr: 2270 GB
  • grid05.lal.in2p3.fr: 1159 GB
  • node12.datagrid.cea.fr 3760 GB
  • dgse0.icepp.jp: 6760 GB
  • atlasse01.ihep.ac.cn: 1018 GB

Summary

mail from Alexei Klimentov (7 October 2006)

• The next DDM functional test is scheduled for October 2006 after SC4
• We will start to generate data in a week from now (Oct 15th)
• Please update table 'Sites' (DDM ops WiKi Functional test) for your site. Data will be generated only for the sites listed in the table. (DDM ops WiKi : https://twiki.cern.ch/twiki/bin/view/Atlas/DDMOperations)
• Test duration 7 - 10 days

• The test scope (preliminary) :

repeat Step1 and Step2 of the previous test

Step 1: CERN->Tier1->Tier2s data transfer

Step 2: Tier2s->Tier1 data transfer

Bulk data transfer

CERN -> Tier1s

Tier1->Tier2s (2-3 Tier-2s per cloud)

• Data transfer of large 2+GB files (if available)
• Evaluation of data transfer robustness vs dataset size
• FTS open access (data transfer between different clouds)
• Two more functional tests are already scheduled Year 2006 (week 47-48, TBC), and Jan 2007. Probably in January we will try to evaluate data transfer performance.

• several suggestions, comments, questions
  • will we use DDM ops mailing list to discuss test scope or HyperNews ?
  • Tier-1s will need 200-300GB disk space for the October test
  • Tier-2s (if they will participate in bulk data transfer) will need 100-200 GB disk space for the October test
  • There are no special requirements for the SW installation. Run what you are using for SC4