lcgwiki - Contributions [en]

Atlas:FDR2

2008-06-18T21:42:49Z

Erming: /* 17th June */

= Monitoring pages/graphs =
=== FTS ===
http://cctoolsafs.in2p3.fr/fts/monitoring/prod/ftsmonitor.php?vo=atlas

=== T0-T1 (ALL) ===
; FDR monitoring page - T1s
http://atldq2pro.cern.ch:8000/ft/mon/fdrmon_tier1s.html

http://dashb-atlas-data-tier0.cern.ch/dashboard/request.py/site
*Throughput 
:http://dashb-atlas-data-tier0.cern.ch/dashboard/templates/plots/OVERVIEW.throughput.14400.png
*Errors 
:http://dashb-atlas-data-tier0.cern.ch/dashboard/templates/plots/OVERVIEW.num_file_xs_error.14400.png

=== T0-T1 (Lyon) ===
http://dashb-atlas-data-tier0.cern.ch/dashboard/request.py/site?statsInterval=4&name=LYON
*Throughput T0-LYON 
:http://dashb-atlas-data-tier0.cern.ch/dashboard/templates/plots/LYON.throughput.14400.png
*Errors T0-LYON 
:http://dashb-atlas-data-tier0.cern.ch/dashboard/templates/plots/LYON.num_file_xs_error.14400.png
*Throughput Others-LYON 
:http://dashb-atlas-data.cern.ch/dashboard/templates/plots/LYON.T1.throughput.14400.png
*Errors Others-LYON :http://dashb-atlas-data.cern.ch/dashboard/templates/plots/LYON.T1.num_file_xs_error.14400.png

=== T1-T1 ===

=== T1-T2 (Lyon) ===
; FDR monitoring page
http://atldq2pro.cern.ch:8000/ft/mon/fdrmon_TiersInfo.html

http://atldq2pro.cern.ch:8000/ft/mon/fdrmon_T1-T2_matrix_day.html

; Dashboard
http://dashb-atlas-data.cern.ch/dashboard/request.py/site?statsInterval=4&name=LYON
*Throughput 
:http://dashb-atlas-data.cern.ch/dashboard/templates/plots/LYON.T2.throughput.14400.png
*Errors
:http://dashb-atlas-data.cern.ch/dashboard/templates/plots/LYON.T2.num_file_xs_error.14400.png

=== Network Graphs ===
; IN2P3 Weathermap
* http://netstat.in2p3.fr/weathermap/
* http://netstat.in2p3.fr/weathermap/graphiques/

; cc-in2p3 : [http://netstat.in2p3.fr/weathermap/graphiques/lyo-cern.html lyo-cern] (lhcopn-in2p3.cern.ch) http://netstat.in2p3.fr/weathermap/graphiques/lyo-cern-daily.gif
; lyon-nord : [http://netstat.in2p3.fr/weathermap/graphiques/lyo-nrd.html lyo-nrd] http://netstat.in2p3.fr/weathermap/graphiques/lyo-nrd-daily.gif
; orsay : [http://netstat.in2p3.fr/weathermap/graphiques/orsay.html orsay] (in2p3-orsay.cssi.renater.fr http://netstat.in2p3.fr/weathermap/graphiques/orsay-daily.gif
; lal-cc : [http://netstat.in2p3.fr/weathermap/graphiques/lal-cc.html Liaison LAL - CC] http://netstat.in2p3.fr/weathermap/graphiques/lal-cc-daily.gif
; lal : [http://netstat.in2p3.fr/weathermap/graphiques/lal.html lal] http://netstat.in2p3.fr/weathermap/graphiques/lal-daily.gif
; lpnhe: [http://netstat.in2p3.fr/weathermap/graphiques/lpnhe-nrd.html lpnhe-nrd] (in2p3-jussieu.cssi.renater.fr) http://netstat.in2p3.fr/weathermap/graphiques/lpnhe-nrd-daily.gif [http://netstat.in2p3.fr/weathermap/graphiques/lpnhe.html lpnhe] (Paris-LPNHE.in2p3.fr) http://netstat.in2p3.fr/weathermap/graphiques/lpnhe-daily.gif
; lapp : [http://netstat.in2p3.fr/weathermap/graphiques/ann-nrd.html ann-nrd] http://netstat.in2p3.fr/weathermap/graphiques/ann-nrd-daily.gif
; lpc: [http://netstat.in2p3.fr/weathermap/graphiques/lpc-cf.html lpc-cf]
http://netstat.in2p3.fr/weathermap/graphiques/lpc-cf-daily.gif
; lpsc : [http://netstat.in2p3.fr/weathermap/graphiques/lpsc.html lpsc] http://netstat.in2p3.fr/weathermap/graphiques/lpsc-daily.gif
; cppm : [http://netstat.in2p3.fr/weathermap/graphiques/cppm.html cppm] 
http://netstat.in2p3.fr/weathermap/graphiques/cppm-daily.gif
; international (tokyo etc.) : [http://netstat.in2p3.fr/weathermap/graphiques/parisnrd.html parisnrd] 
http://netstat.in2p3.fr/weathermap/graphiques/parisnrd-daily.gif
; GEANT - NYC : [http://dc-snmp.wcc.grnoc.iu.edu/manlan/show-graph.cgi?title=sw.newy32aoa.manlan.internet2.edu--te10/1&rrdname=sw.newy32aoa.manlan.internet2.edu--te10_1.rrd MANLAN] 
http://dc-snmp.wcc.grnoc.iu.edu/manlan/img/sw.newy32aoa.manlan.internet2.edu--te10_1-std5.gif
; NYC - TOKYO : [http://dc-snmp.wcc.grnoc.iu.edu/manlan/show-graph.cgi?title=sw.newy32aoa.manlan.internet2.edu--te11/1&rrdname=sw.newy32aoa.manlan.internet2.edu--te11_1.rrd MANLAN] 
http://dc-snmp.wcc.grnoc.iu.edu/manlan/img/sw.newy32aoa.manlan.internet2.edu--te11_1-std5.gif
; tokyo : [http://www.icepp.jp/monitor/gridftp/monitor.html gridftp monitor] http://www.icepp.jp/monitor/gridftp/img/all.png

= T0-T1 =

= T1-T2 =
==== General informations ====
;FDR page:
https://twiki.cern.ch/twiki/bin/view/Atlas/FullDressRehearsal

;DDM_in_FRD2: https://twiki.cern.ch/twiki/bin/view/Atlas/DDMOperationsGroup#DDM_in_FRD2

;Replication status(checked ~ every 2h?)
http://pandamon.usatlas.bnl.gov:25880/server/pandamon/query?mode=listFDR

;Lyon FTS monitor: http://cctoolsafs.in2p3.fr/fts/monitoring/prod/ftsmonitor.php?vo=atlas

;Contacts:

* IN2P3-CPPM_DATADISK :
* IN2P3-LPSC_DATADISK :
* IN2P3-LPC_DATADISK :
* GRIF-LAL_DATADISK : (grid.admin a lal.in2p3.fr)
* GRIF-LPNHE_DATADISK :
* GRIF-SACLAY_DATADISK :
* BEIJING-LCG2_DATADISK : (yanxf a ihep.ac.cn, Erming.Pei a cern.ch)
* RO-07-NIPNE_DATADISK : (ciubancan a nipne.ro)
* RO-02-NIPNE_DATADISK : (tpreda a nipne.ro)
* TOKYO-LCG2_DATADISK : (lcg-admin a icepp.s.u-tokyo.ac.jp)

===== Shares =====
Only srmv2+ space tokens sites will collect data. The list of streams is : Egamma, Bphysics, MinBias, Muon and Jet. If no precise stream name is associated to the share, the site will get any stream. The site in parenthesis will collect the complementary datasets. The T2 ressources at T1 means that users without production role can access CPUs at the T1 without the Production role.
* US :
o AGLT2_DATADISK : 100 %
o MWT2_DATADISK: 100 %
o NET2_DATADISK: 100 %
o SLACXRD_DATADISK: 100 %
o SWT2_CPB_DATADISK: 100 %
o T2 CPU ressources in BNL to access AOD
* CA :
o SFU-LCG2_DATADISK : 40%
o VICTORIA-LCG2_DATADISK : 30%
o ALBERTA-LCG2_DATADISK : 30%
o TORONTO-LCG2_DATADISK : 50 %
* IT :
o INFN-MILANO_DATADISK : 25%
o INFN-FRASCATI_DATADISK : 25%
o INFN-ROMA1_DATADISK : 100%
o INFN-NAPOLI-ATLAS_DATADISK : 100 %
* TW :
o TW-FTT_DATADISK : 50 %
* NL
o JINR-LCG2_DATADISK : 50% Jet (RU-PROTVINO-IHEP_DATADISK)
o JINR-LCG2_DATADISK : 50% MinBias (RU-PROTVINO-IHEP_DATADISK)
o JINR-LCG2_DATADISK : 50% Muon (RU-PROTVINO-IHEP_DATADISK)
o JINR-LCG2_DATADISK : 50% Bphysics (RU-PNPI_DATADISK)
o RU-PROTVINO-IHEP_DATADISK : 50% Jet (JINR-LCG2_DATADISK)
o RU-PROTVINO-IHEP_DATADISK : 50% Muon (JINR-LCG2_DATADISK)
o RU-PROTVINO-IHEP_DATADISK : 50% MinBias (JINR-LCG2_DATADISK)
o RU-PROTVINO-IHEP_DATADISK : 50% Egamma (RU-PNPI_DATADISK)
o RU-PNPI_DATADISK : 50% Egamma (RU-PROTVINO-IHEP_DATADISK)
o RU-PNPI_DATADISK : 50% Bphysics (JINR-LCG2_DATADISK)
o RU-PNPI_DATADISK : 50% Jet (RRC-KI_DATADISK)
o RU-PNPI_DATADISK : 50% MinBias (RRC-KI_DATADISK)
o RRC-KI_DATADISK : 50% Jet (RU-PNPI_DATADISK)
o RRC-KI_DATADISK : 50% MinBias (RU-PNPI_DATADISK)
o RRC-KI_DATADISK : 50% Egamma
o RRC-KI_DATADISK : 50% Bphysics
o CSTCDIE_DATADISK : 20%
* FR :
o IN2P3-LAPP_DATADISK : 100 % Egamma
o IN2P3-CPPM_DATADISK : 100 % Egamma
o IN2P3-LPSC_DATADISK : 5 %
o IN2P3-LPC_DATADISK : 100% Muon, 50% Egamma
o BEIJING-LCG2_DATADISK : 100 % Egamma, 50% Muon, 35% MinBias
o RO-07-NIPNE_DATADISK : 10%
o RO-02-NIPNE_DATADISK : 10%
o GRIF-LAL_DATADISK : 45%
o GRIF-LPNHE_DATADISK : 25%
o GRIF-SACLAY_DATADISK : 30 %
o TOKYO-LCG2_DATADISK : 100%
o T2 CPU ressources in LYON to access AOD
* DE :
o DESY-HH_DATADISK : 100%
o CSCS-LCG2_DATADISK: 100%
o DESY-ZN_DATADISK : 50%
o GOEGRID_DATADISK : 50%
o WUPPERTALPROD_DATADISK: 25%
o CYFRONET-LCG2_DATADISK : 15%
o PRAGUELCG2_DATADISK : 25%
o LRZ-LMU_DATADISK : 35%
o UNI-FREIBURG_DATADISK : 100% Jet, 100% Muon
* UK :
o UKI-SOUTHGRID-CAM-HEP_DATADISK : 50 % Muon (UKI-NORTHGRID-LANCS-HEP_DATADISK
o UKI-NORTHGRID-LANCS-HEP_DATADISK : 50 % Muon (UKI-SOUTHGRID-CAM-HEP_DATADISK)
o UKI-NORTHGRID-SHEF-HEP_DATADISK : 50 % MinBias (UKI-SCOTGRID-GLASGOW_DATADISK)
o UKI-SCOTGRID-GLASGOW_DATADISK : 50 % MinBias (UKI-NORTHGRID-SHEF-HEP_DATADISK)
o UKI-LT2-RHUL_DATADISK : 50 % Egamma (UKI-NORTHGRID-LIV-HEP_DATADISK)
o UKI-NORTHGRID-LIV-HEP_DATADISK : 50 % Egamma (UKI-LT2-RHUL_DATADISK)
o UKI-SOUTHGRID-RALPP_DATADISK : 50 Egamma (UKI-LT2-RHUL_DATADISK)
o UKI-SOUTHGRID-BHAM-HEP_DATADISK : 50 Jet (UKI-SOUTHGRID-OX-HEP_DATADISK)
* ES :
o IFIC-LCG2_DATADISK: 50%
o UAM-LCG2_DATADISK: 25%
o IFAE_DATADISK: 25%
* NDGF :
o SIGNET_DATADISK : 50 %

== Logbook ==

=== 2nd June ===

* Since Lyon is in donwtime, so there's no transfers from LYON to T2s today.
http://cc.in2p3.fr/cc_accueil.php3?lang=fr

*Free Space

Token Total Guaranteed Unused
------------------------------------------
BEIJING-LCG2_DATADISK 6.366 6.366 6.343
GRIF-LAL_DATADISK 6.0 6.0 5.732
GRIF-LPNHE_DATADISK 3.906 3.906 3.89
GRIF-SACLAY_DATADISK 6.0 6.0 5.994
IN2P3-CPPM_DATADISK 1.0 1.0 0.977
IN2P3-LAPP_DATADISK 4.0 4.0 4.0
IN2P3-LPC_DATADISK 3.0 3.0 3.0
IN2P3-LPSC_DATADISK 0.488 0.488 0.482
RO-02-NIPNE_DATADISK 3.0 3.0 3.0
RO-07-NIPNE_DATADISK 2.0 2.0 2.0
TOKYO-LCG2_DATADISK 15.0 15.0 13.643

=== 3rd June ===

* Lyon is still in donwtime, no transfers from LYON to T2s in this morning.

; Throughput(snapshot) (MB/s) : [[Image:LYON.throughput.14400_20080603.png]]
; Number of transfer errors(snapshot) : [[Image:LYON.num_file_xs_error.14400_20080603.png]]

=== 4th June ===
;22h02
* Still no transfers, desipte that all the services in LYON shows OK.

;Throughput(snapshot)
[[Image:LYON.throughput.3600_20080604.png]]

;Services in LYON
[[Image:services_cc-in2p3_2008060422h15.PNG]]

;Free space

BEIJING-LCG2_DATADISK 6.366 6.366 6.343
BEIJING-LCG2_MCDISK 5.457 5.457 5.457
GRIF-LAL_DATADISK 6.0 6.0 5.732
GRIF-LAL_MCDISK 7.0 7.0 3.724
GRIF-LPNHE_DATADISK 3.906 3.906 3.89
GRIF-LPNHE_MCDISK 1.0 1.0 0.194
GRIF-SACLAY_DATADISK 6.0 6.0 5.994
GRIF-SACLAY_MCDISK 2.0 2.0 1.713
IN2P3-CPPM_DATADISK 1.0 1.0 0.977
IN2P3-CPPM_MCDISK 0.781 0.781 0.765
IN2P3-LAPP_DATADISK 4.0 4.0 4.0
IN2P3-LAPP_MCDISK 2.0 2.0 0.745
IN2P3-LPC_DATADISK 3.0 3.0 3.0
IN2P3-LPC_MCDISK 2.0 2.0 1.936
IN2P3-LPSC_DATADISK 0.488 0.488 0.482
IN2P3-LPSC_MCDISK 2.0 2.0 1.518
RO-02-NIPNE_DATADISK 3.0 3.0 3.0
RO-02-NIPNE_MCDISK 1.0 1.0 0.995
RO-07-NIPNE_DATADISK 2.0 2.0 2.0
RO-07-NIPNE_MCDISK 1.0 1.0 0.654
TOKYO-LCG2_DATADISK 15.0 15.0 13.643
TOKYO-LCG2_MCDISK 10.0 10.0 10.0

=== 5th June ===

* Data produced on Tuesday are supposed to be shipped in this night.

;14h50
* Now the RAW data has begun to shipped to LYON.

;Throughput(snapshot)(MB/s)
[[Image:LYON.throughput.14400_RAW_20080605_1816.png]]

;Number of transfer errors (snapshot)
[[Image:LYON.num_file_xs_error.14400_20080605_1823.png]]

;21h30
* IN2P3-CC_MCDISK has some problem of staging file. But it should have nothing to do with T0 export.

* 47 datasets have been exported to T1s. The total is 90.

;Throughput(snapshot)(MB/s)
[[Image:LYON.throughput.14400 20080605 2140.png]]

;Number of transfer errors (snapshot) -- It's the problem of staging file in T1 and nothing to do with FDR transfers.
[[Image:LYON.num_file_xs_error.14400_20080605_2140.png]]

=== 6th June ===
;09h50

;Throughput(snapshot) (MB/s)
[[Image:LYON.throughput.14400 200806060 0958.png]]

;Number of transfer errors (snapshot)
[[Image:LYON.num file xs error.14400 20080606 0959.png]]

;10h03
* Now all the RAW datasets have been exported to T1. The total number is 90.
http://atlas.web.cern.ch/Atlas/tzero/prod2/monitoring/tables/ddm_RAW_status.html

=== 7th June ===
* at CERN, will re-start the bulk processing with using the upcoming AtlasPoint1-14.1.0.12 release cache.

;11h30
* SRM in LYON is still in bad shape. Experts are working on it.

=== 8th June ===
;14h20
* FDR2 AODs/DPDs/TAGs/ESDs are being exported to LYON (IN2P3-CC_DATADISK), but haven't been shipped to T2s.

;Number of datasets transfered by far
$ dq2-list-dataset-site IN2P3-CC_DATADISK|grep fdr08_run2.*|wc -l
68

;Throughput(T0-T1,snapshot)(MB/s)
[[Image:LYON.throughput.14400 200806081420.png]]

;Number of transfer errors
[[Image:LYON.num file xs error.14400 200806081420.png]]

;Throughput(T1-T2,snapshot)(MB/s)
[[Image:LYON.T2.throughput.14400 T1-T2 200806081420.png]]

=== 9th June ===
;0h20
* FDR datasets has begun to be shipped to T2s.
* Some T2s (e.g. BEIJING) subscribed dedicated datasets, which will be done later in today's morning.

;Shipped datasets

IN2P3-CC_DATADISK 315
BEIJING-LCG2_DATADISK 0
GRIF-LAL_DATADISK 79
GRIF-LPNHE_DATADISK 39
GRIF-SACLAY_DATADISK 38
IN2P3-CPPM_DATADISK 0
IN2P3-LAPP_DATADISK 12
IN2P3-LPC_DATADISK 12
IN2P3-LPSC_DATADISK 32
RO-02-NIPNE_DATADISK 29
RO-07-NIPNE_DATADISK 46
TOKYO-LCG2_DATADISK 134

;Free space

BEIJING-LCG2_DATADISK 6.366 6.366 6.343
GRIF-LAL_DATADISK 6.0 6.0 5.579
GRIF-LPNHE_DATADISK 3.906 3.906 3.817
GRIF-SACLAY_DATADISK 6.0 6.0 5.927
IN2P3-CPPM_DATADISK 1.0 1.0 0.977
IN2P3-LAPP_DATADISK 4.0 4.0 4.0
IN2P3-LPC_DATADISK 0 0 0
IN2P3-LPSC_DATADISK 0.488 0.488 0.439
RO-02-NIPNE_DATADISK 3.0 3.0 2.986
RO-07-NIPNE_DATADISK 2.0 2.0 1.896
TOKYO-LCG2_DATADISK 15.0 15.0 13.313

;Troughputs(snapshot)(MB/s)
[[Image:LYON.throughput.14400 200806090046.png]]

;Number of transfer erros(snapshot)
* Note: No errors found in FDR2 transfers. Errors showed in this snapshot are since other production data transfers.
[[Image:LYON.num file xs error.14400 200806090047.png]]

;09h24

;Troughputs(snapshot)(MB/s)
[[Image:LYON.throughput.14400 200806090922.png]]

;Number of transfer erros(snapshot)
* Note: No errors found in FDR2 transfers. Errors showed in this snapshot are since other production data transfers.
[[Image:LYON.num file xs error.14400 20806090922.png]]

;Shipped datasets

IN2P3-CC_DATADISK 319
BEIJING-LCG2_DATADISK 0
GRIF-LAL_DATADISK 79
GRIF-LPNHE_DATADISK 39
GRIF-SACLAY_DATADISK 38
IN2P3-CPPM_DATADISK 0
IN2P3-LAPP_DATADISK 12
IN2P3-LPC_DATADISK 12
IN2P3-LPSC_DATADISK 32
RO-02-NIPNE_DATADISK 59
RO-07-NIPNE_DATADISK 49
TOKYO-LCG2_DATADISK 134

;Free space
BEIJING-LCG2_DATADISK 6.366 6.366 6.343
GRIF-LAL_DATADISK 6.0 6.0 5.579
GRIF-LPNHE_DATADISK 3.906 3.906 3.817
GRIF-SACLAY_DATADISK 6.0 6.0 5.927
IN2P3-CPPM_DATADISK 1.0 1.0 0.977
IN2P3-LAPP_DATADISK 4.0 4.0 4.0
IN2P3-LPC_DATADISK 3.0 3.0 3.0
IN2P3-LPSC_DATADISK 0.488 0.488 0.439
RO-02-NIPNE_DATADISK 3.0 3.0 2.945
RO-07-NIPNE_DATADISK 2.0 2.0 1.885
TOKYO-LCG2_DATADISK 15.0 15.0 13.313

;14h00
Many failed with RO-02-NIPNE_DATADISK.

[FTS] FTS State [Failed] FTS Retries [1] Reason [TRANSFER error during TRANSFER phase: [PERMISSION] the server sent an error response: 550 550 rfio write failure: Permission denied.] Source Host [ccsrm.in2p3.fr] 16
[FTS] FTS State [Failed] FTS Retries [1] Reason [TRANSFER error during TRANSFER phase: [TRANSFER_TIMEOUT] gridftp_copy_wait: Connection timed out] Source Host [ccsrm.in2p3.fr] 9
[FTS] FTS State [Failed] FTS Retries [1] Reason [TRANSFER error during TRANSFER phase: [GRIDFTP] the server sent an error response: 426 426 Transfer aborted (Unexpected Exception : java.io.IOException: Broken pipe)] Source Host [ccsrm.in2p3.fr] 2
[FTS] FTS State [Failed] FTS Retries [1] Reason [The process serving the transfer (status = PREPARING) is no longer active (could not open file /proc/3775/cmdline)] Source Host [ccsrm.in2p3.fr] 1
[FTS] FTS State [Failed] FTS Retries [1] Reason [The process serving the transfer (status = PREPARING) is no longer active (could not open file /proc/30076/cmdline)] Source Host [ccsrm.in2p3.fr] 1

;20h10
* FDR datasets have finished transferred to all T2s except BEIJING-LCG2_DATADISK and RO-02-NIPNE_DATADISK.
* Many "Connection timed out" errors found at BEIJING-LCG2_DATADISK and RO-02-NIPNE_DATADISK.

[FTS] FTS State [Failed] FTS Retries [1] Reason [SOURCE error during PREPARATION phase: [CONNECTION] failed to contact on remote SRM [httpg://ccsrm.in2p3.fr:8443/srm/managerv2]. Givin' up after 3 tries] Source Host [ccsrm.in2p3.fr] 111
[FTS] FTS State [Failed] FTS Retries [1] Reason [TRANSFER error during TRANSFER phase: [TRANSFER_TIMEOUT] gridftp_copy_wait: Connection timed out] Source Host [ccsrm.in2p3.fr] 95
[FTS] FTS State [Failed] FTS Retries [1] Reason [SOURCE error during PREPARATION phase: [REQUEST_TIMEOUT] failed to prepare source file in 180 seconds] Source Host [ccsrm.in2p3.fr] 3

; Shipped datasets

IN2P3-CC_DATADISK 134
BEIJING-LCG2_DATADISK 79
GRIF-LAL_DATADISK 67
GRIF-LPNHE_DATADISK 32
GRIF-SACLAY_DATADISK 26
IN2P3-CPPM_DATADISK 52
IN2P3-LAPP_DATADISK 52
IN2P3-LPC_DATADISK 52
IN2P3-LPSC_DATADISK 32
RO-02-NIPNE_DATADISK 49
RO-07-NIPNE_DATADISK 49
TOKYO-LCG2_DATADISK 126

=== 10th June ===

;Summary

All T2s except BEIJING-LCG2_DATADISK and RO-02-NIPNE_DATADISK were in good status and received all the exported datasets when FDR transferring.
The problem of "Connection timed out" with transfers to BEIJING will be hopefully solved by tuned the transfer duration >1800s by Lionel.
RO-02-NIPNE is said being upgraded from SL3 to SL4.

=== 11st June ===

; Anouncement
All FDR2 datasets will be deleted from all sites since part of the detector was not included in the reconstruction. It was not detected by the Data Quality because it is run on the express stream which is correct.

Old datasets will be deleted centrally by Stephane and new datasets that would be named begin with "ccrc08-run.0024*" will be re-generated and re-exported soon in today.

T2s will receive the same share rate as requested for FDR (except that DPD are not produced).

=== 12nd June ===
;01h20
* Function Test datasets(named beginning with "ccrc08_run2.*") to T2s.
For this moment, all T2 sites seems in good shape to accept datasets except RO-07-NIPNE and RO-02-NIPNE that are in downtime.

* Datasets shipped (begins with "ccrc08_run2.*")

IN2P3-CC_DATADISK 387
BEIJING-LCG2_DATADISK 46
GRIF-LAL_DATADISK 141
GRIF-LPNHE_DATADISK 79
GRIF-SACLAY_DATADISK 90
IN2P3-CPPM_DATADISK 22
IN2P3-LAPP_DATADISK 22
IN2P3-LPC_DATADISK 36
IN2P3-LPSC_DATADISK 17
RO-02-NIPNE_DATADISK 20
RO-07-NIPNE_DATADISK 33
TOKYO-LCG2_DATADISK 310

;Troughput
[[Image:LYON.T2.throughput.14400 200806120120.png]]

;Transfer errors
* Note that the errors showed in this snapshot involves all transfers mixed with other transfers(e.g MC/Panda). So it may not reflect the real situation FT transfers.
[[Image:LYON.T2.num file xs error.14400 200806120120.png]]

=== 13rd June ===
=== 14th June ===
* The re-produced FDR2 datasets have been re-exported to T1s.
* Now 3 RAW datasets have been exported to IN2P3-CC_DATATAPE.

=== 15th June ===
* Now 5 RAW datasets have been exported to IN2P3-CC_DATATAPE.
* There's something wrong with the configurations resulted in low transfer rate.

=== 16th June ===
* Configuration problems found in dCahce at LYON resulted in failure of transfers to T2s.

=== 17th June ===

* FDR2 datasets have been exported to T2s. Except

IN2P3-CC_DATADISK Total:218 Completed:218 AOD:30 TAG:30 DPD:150 ESD:8
BEIJING-LCG2_DATADISK Total:89 Completed:89 AOD:15 TAG:15 DPD:59 ESD:0
GRIF-LAL_DATADISK Total:94 Completed:94 AOD:15 TAG:15 DPD:64 ESD:0
GRIF-LPNHE_DATADISK Total:57 Completed:57 AOD:9 TAG:9 DPD:39 ESD:0
GRIF-SACLAY_DATADISK Total:59 Completed:59 AOD:6 TAG:6 DPD:47 ESD:0
IN2P3-CPPM_DATADISK Total:42 Completed:42 AOD:6 TAG:6 DPD:30 ESD:0
IN2P3-LAPP_DATADISK Total:42 Completed:42 AOD:6 TAG:6 DPD:30 ESD:0
IN2P3-LPC_DATADISK Total:69 Completed:69 AOD:11 TAG:11 DPD:47 ESD:0
IN2P3-LPSC_DATADISK Total:15 Completed:15 AOD:3 TAG:3 DPD:9 ESD:0
RO-02-NIPNE_DATADISK Total:27 Completed:10 AOD:2 TAG:2 DPD:6 ESD:0
RO-07-NIPNE_DATADISK Total:27 Completed:27 AOD:5 TAG:5 DPD:17 ESD:0
TOKYO-LCG2_DATADISK Total:210 Completed:210 AOD:30 TAG:30 DPD:150 ESD:0

=== 18th June ===

* FDR2 datasets have been completely transferred to all T2s.

IN2P3-CC_DATADISK 218
BEIJING-LCG2_DATADISK 89
GRIF-LAL_DATADISK 94
GRIF-LPNHE_DATADISK 57
GRIF-SACLAY_DATADISK 59
IN2P3-CPPM_DATADISK 42
IN2P3-LAPP_DATADISK 42
IN2P3-LPC_DATADISK 69
IN2P3-LPSC_DATADISK 15
RO-02-NIPNE_DATADISK 27
RO-07-NIPNE_DATADISK 27
TOKYO-LCG2_DATADISK 210

IN2P3-CC_DATADISK Completed:218 AOD:30 TAG:30 DPD:150 ESD:8
BEIJING-LCG2_DATADISK Completed:89 AOD:15 TAG:15 DPD:59 ESD:0
GRIF-LAL_DATADISK Completed:94 AOD:15 TAG:15 DPD:64 ESD:0
GRIF-LPNHE_DATADISK Completed:57 AOD:9 TAG:9 DPD:39 ESD:0
GRIF-SACLAY_DATADISK Completed:59 AOD:6 TAG:6 DPD:47 ESD:0
IN2P3-CPPM_DATADISK Completed:42 AOD:6 TAG:6 DPD:30 ESD:0
IN2P3-LAPP_DATADISK Completed:42 AOD:6 TAG:6 DPD:30 ESD:0
IN2P3-LPC_DATADISK Completed:69 AOD:11 TAG:11 DPD:47 ESD:0
IN2P3-LPSC_DATADISK Completed:15 AOD:3 TAG:3 DPD:9 ESD:0
RO-02-NIPNE_DATADISK Completed:27 AOD:5 TAG:5 DPD:17 ESD:0
RO-07-NIPNE_DATADISK Completed:27 AOD:5 TAG:5 DPD:17 ESD:0
TOKYO-LCG2_DATADISK Completed:210 AOD:30 TAG:30 DPD:150 ESD:0

IN2P3-CC_DATADISK Completed:218 Jet:43 Egamma:46 Bphys:43 Minbias:43 Muon:43
BEIJING-LCG2_DATADISK Completed:89 Jet:0 Egamma:42 Bphys:0 Minbias:20 Muon:27
GRIF-LAL_DATADISK Completed:94 Jet:18 Egamma:16 Bphys:18 Minbias:21 Muon:21
GRIF-LPNHE_DATADISK Completed:57 Jet:12 Egamma:14 Bphys:12 Minbias:12 Muon:7
GRIF-SACLAY_DATADISK Completed:59 Jet:12 Egamma:12 Bphys:12 Minbias:9 Muon:14
IN2P3-CPPM_DATADISK Completed:42 Jet:0 Egamma:42 Bphys:0 Minbias:0 Muon:0
IN2P3-LAPP_DATADISK Completed:42 Jet:0 Egamma:42 Bphys:0 Minbias:0 Muon:0
IN2P3-LPC_DATADISK Completed:69 Jet:0 Egamma:27 Bphys:0 Minbias:0 Muon:42
IN2P3-LPSC_DATADISK Completed:15 Jet:2 Egamma:6 Bphys:7 Minbias:0 Muon:0
RO-02-NIPNE_DATADISK Completed:27 Jet:3 Egamma:4 Bphys:7 Minbias:7 Muon:6
RO-07-NIPNE_DATADISK Completed:27 Jet:6 Egamma:9 Bphys:7 Minbias:2 Muon:3
TOKYO-LCG2_DATADISK Completed:210 Jet:42 Egamma:42 Bphys:42 Minbias:42 Muon:42

Atlas:FDR2

2008-06-18T19:41:24Z

Erming: /* 17th June */

= Monitoring pages/graphs =
=== FTS ===
http://cctoolsafs.in2p3.fr/fts/monitoring/prod/ftsmonitor.php?vo=atlas

=== T0-T1 (ALL) ===
; FDR monitoring page - T1s
http://atldq2pro.cern.ch:8000/ft/mon/fdrmon_tier1s.html

http://dashb-atlas-data-tier0.cern.ch/dashboard/request.py/site
*Throughput 
:http://dashb-atlas-data-tier0.cern.ch/dashboard/templates/plots/OVERVIEW.throughput.14400.png
*Errors 
:http://dashb-atlas-data-tier0.cern.ch/dashboard/templates/plots/OVERVIEW.num_file_xs_error.14400.png

=== T0-T1 (Lyon) ===
http://dashb-atlas-data-tier0.cern.ch/dashboard/request.py/site?statsInterval=4&name=LYON
*Throughput T0-LYON 
:http://dashb-atlas-data-tier0.cern.ch/dashboard/templates/plots/LYON.throughput.14400.png
*Errors T0-LYON 
:http://dashb-atlas-data-tier0.cern.ch/dashboard/templates/plots/LYON.num_file_xs_error.14400.png
*Throughput Others-LYON 
:http://dashb-atlas-data.cern.ch/dashboard/templates/plots/LYON.T1.throughput.14400.png
*Errors Others-LYON :http://dashb-atlas-data.cern.ch/dashboard/templates/plots/LYON.T1.num_file_xs_error.14400.png

=== T1-T1 ===

=== T1-T2 (Lyon) ===
; FDR monitoring page
http://atldq2pro.cern.ch:8000/ft/mon/fdrmon_TiersInfo.html

http://atldq2pro.cern.ch:8000/ft/mon/fdrmon_T1-T2_matrix_day.html

; Dashboard
http://dashb-atlas-data.cern.ch/dashboard/request.py/site?statsInterval=4&name=LYON
*Throughput 
:http://dashb-atlas-data.cern.ch/dashboard/templates/plots/LYON.T2.throughput.14400.png
*Errors
:http://dashb-atlas-data.cern.ch/dashboard/templates/plots/LYON.T2.num_file_xs_error.14400.png

=== Network Graphs ===
; IN2P3 Weathermap
* http://netstat.in2p3.fr/weathermap/
* http://netstat.in2p3.fr/weathermap/graphiques/

; cc-in2p3 : [http://netstat.in2p3.fr/weathermap/graphiques/lyo-cern.html lyo-cern] (lhcopn-in2p3.cern.ch) http://netstat.in2p3.fr/weathermap/graphiques/lyo-cern-daily.gif
; lyon-nord : [http://netstat.in2p3.fr/weathermap/graphiques/lyo-nrd.html lyo-nrd] http://netstat.in2p3.fr/weathermap/graphiques/lyo-nrd-daily.gif
; orsay : [http://netstat.in2p3.fr/weathermap/graphiques/orsay.html orsay] (in2p3-orsay.cssi.renater.fr http://netstat.in2p3.fr/weathermap/graphiques/orsay-daily.gif
; lal-cc : [http://netstat.in2p3.fr/weathermap/graphiques/lal-cc.html Liaison LAL - CC] http://netstat.in2p3.fr/weathermap/graphiques/lal-cc-daily.gif
; lal : [http://netstat.in2p3.fr/weathermap/graphiques/lal.html lal] http://netstat.in2p3.fr/weathermap/graphiques/lal-daily.gif
; lpnhe: [http://netstat.in2p3.fr/weathermap/graphiques/lpnhe-nrd.html lpnhe-nrd] (in2p3-jussieu.cssi.renater.fr) http://netstat.in2p3.fr/weathermap/graphiques/lpnhe-nrd-daily.gif [http://netstat.in2p3.fr/weathermap/graphiques/lpnhe.html lpnhe] (Paris-LPNHE.in2p3.fr) http://netstat.in2p3.fr/weathermap/graphiques/lpnhe-daily.gif
; lapp : [http://netstat.in2p3.fr/weathermap/graphiques/ann-nrd.html ann-nrd] http://netstat.in2p3.fr/weathermap/graphiques/ann-nrd-daily.gif
; lpc: [http://netstat.in2p3.fr/weathermap/graphiques/lpc-cf.html lpc-cf]
http://netstat.in2p3.fr/weathermap/graphiques/lpc-cf-daily.gif
; lpsc : [http://netstat.in2p3.fr/weathermap/graphiques/lpsc.html lpsc] http://netstat.in2p3.fr/weathermap/graphiques/lpsc-daily.gif
; cppm : [http://netstat.in2p3.fr/weathermap/graphiques/cppm.html cppm] 
http://netstat.in2p3.fr/weathermap/graphiques/cppm-daily.gif
; international (tokyo etc.) : [http://netstat.in2p3.fr/weathermap/graphiques/parisnrd.html parisnrd] 
http://netstat.in2p3.fr/weathermap/graphiques/parisnrd-daily.gif
; GEANT - NYC : [http://dc-snmp.wcc.grnoc.iu.edu/manlan/show-graph.cgi?title=sw.newy32aoa.manlan.internet2.edu--te10/1&rrdname=sw.newy32aoa.manlan.internet2.edu--te10_1.rrd MANLAN] 
http://dc-snmp.wcc.grnoc.iu.edu/manlan/img/sw.newy32aoa.manlan.internet2.edu--te10_1-std5.gif
; NYC - TOKYO : [http://dc-snmp.wcc.grnoc.iu.edu/manlan/show-graph.cgi?title=sw.newy32aoa.manlan.internet2.edu--te11/1&rrdname=sw.newy32aoa.manlan.internet2.edu--te11_1.rrd MANLAN] 
http://dc-snmp.wcc.grnoc.iu.edu/manlan/img/sw.newy32aoa.manlan.internet2.edu--te11_1-std5.gif
; tokyo : [http://www.icepp.jp/monitor/gridftp/monitor.html gridftp monitor] http://www.icepp.jp/monitor/gridftp/img/all.png

= T0-T1 =

= T1-T2 =
==== General informations ====
;FDR page:
https://twiki.cern.ch/twiki/bin/view/Atlas/FullDressRehearsal

;DDM_in_FRD2: https://twiki.cern.ch/twiki/bin/view/Atlas/DDMOperationsGroup#DDM_in_FRD2

;Replication status(checked ~ every 2h?)
http://pandamon.usatlas.bnl.gov:25880/server/pandamon/query?mode=listFDR

;Lyon FTS monitor: http://cctoolsafs.in2p3.fr/fts/monitoring/prod/ftsmonitor.php?vo=atlas

;Contacts:

* IN2P3-CPPM_DATADISK :
* IN2P3-LPSC_DATADISK :
* IN2P3-LPC_DATADISK :
* GRIF-LAL_DATADISK : (grid.admin a lal.in2p3.fr)
* GRIF-LPNHE_DATADISK :
* GRIF-SACLAY_DATADISK :
* BEIJING-LCG2_DATADISK : (yanxf a ihep.ac.cn, Erming.Pei a cern.ch)
* RO-07-NIPNE_DATADISK : (ciubancan a nipne.ro)
* RO-02-NIPNE_DATADISK : (tpreda a nipne.ro)
* TOKYO-LCG2_DATADISK : (lcg-admin a icepp.s.u-tokyo.ac.jp)

===== Shares =====
Only srmv2+ space tokens sites will collect data. The list of streams is : Egamma, Bphysics, MinBias, Muon and Jet. If no precise stream name is associated to the share, the site will get any stream. The site in parenthesis will collect the complementary datasets. The T2 ressources at T1 means that users without production role can access CPUs at the T1 without the Production role.
* US :
o AGLT2_DATADISK : 100 %
o MWT2_DATADISK: 100 %
o NET2_DATADISK: 100 %
o SLACXRD_DATADISK: 100 %
o SWT2_CPB_DATADISK: 100 %
o T2 CPU ressources in BNL to access AOD
* CA :
o SFU-LCG2_DATADISK : 40%
o VICTORIA-LCG2_DATADISK : 30%
o ALBERTA-LCG2_DATADISK : 30%
o TORONTO-LCG2_DATADISK : 50 %
* IT :
o INFN-MILANO_DATADISK : 25%
o INFN-FRASCATI_DATADISK : 25%
o INFN-ROMA1_DATADISK : 100%
o INFN-NAPOLI-ATLAS_DATADISK : 100 %
* TW :
o TW-FTT_DATADISK : 50 %
* NL
o JINR-LCG2_DATADISK : 50% Jet (RU-PROTVINO-IHEP_DATADISK)
o JINR-LCG2_DATADISK : 50% MinBias (RU-PROTVINO-IHEP_DATADISK)
o JINR-LCG2_DATADISK : 50% Muon (RU-PROTVINO-IHEP_DATADISK)
o JINR-LCG2_DATADISK : 50% Bphysics (RU-PNPI_DATADISK)
o RU-PROTVINO-IHEP_DATADISK : 50% Jet (JINR-LCG2_DATADISK)
o RU-PROTVINO-IHEP_DATADISK : 50% Muon (JINR-LCG2_DATADISK)
o RU-PROTVINO-IHEP_DATADISK : 50% MinBias (JINR-LCG2_DATADISK)
o RU-PROTVINO-IHEP_DATADISK : 50% Egamma (RU-PNPI_DATADISK)
o RU-PNPI_DATADISK : 50% Egamma (RU-PROTVINO-IHEP_DATADISK)
o RU-PNPI_DATADISK : 50% Bphysics (JINR-LCG2_DATADISK)
o RU-PNPI_DATADISK : 50% Jet (RRC-KI_DATADISK)
o RU-PNPI_DATADISK : 50% MinBias (RRC-KI_DATADISK)
o RRC-KI_DATADISK : 50% Jet (RU-PNPI_DATADISK)
o RRC-KI_DATADISK : 50% MinBias (RU-PNPI_DATADISK)
o RRC-KI_DATADISK : 50% Egamma
o RRC-KI_DATADISK : 50% Bphysics
o CSTCDIE_DATADISK : 20%
* FR :
o IN2P3-LAPP_DATADISK : 100 % Egamma
o IN2P3-CPPM_DATADISK : 100 % Egamma
o IN2P3-LPSC_DATADISK : 5 %
o IN2P3-LPC_DATADISK : 100% Muon, 50% Egamma
o BEIJING-LCG2_DATADISK : 100 % Egamma, 50% Muon, 35% MinBias
o RO-07-NIPNE_DATADISK : 10%
o RO-02-NIPNE_DATADISK : 10%
o GRIF-LAL_DATADISK : 45%
o GRIF-LPNHE_DATADISK : 25%
o GRIF-SACLAY_DATADISK : 30 %
o TOKYO-LCG2_DATADISK : 100%
o T2 CPU ressources in LYON to access AOD
* DE :
o DESY-HH_DATADISK : 100%
o CSCS-LCG2_DATADISK: 100%
o DESY-ZN_DATADISK : 50%
o GOEGRID_DATADISK : 50%
o WUPPERTALPROD_DATADISK: 25%
o CYFRONET-LCG2_DATADISK : 15%
o PRAGUELCG2_DATADISK : 25%
o LRZ-LMU_DATADISK : 35%
o UNI-FREIBURG_DATADISK : 100% Jet, 100% Muon
* UK :
o UKI-SOUTHGRID-CAM-HEP_DATADISK : 50 % Muon (UKI-NORTHGRID-LANCS-HEP_DATADISK
o UKI-NORTHGRID-LANCS-HEP_DATADISK : 50 % Muon (UKI-SOUTHGRID-CAM-HEP_DATADISK)
o UKI-NORTHGRID-SHEF-HEP_DATADISK : 50 % MinBias (UKI-SCOTGRID-GLASGOW_DATADISK)
o UKI-SCOTGRID-GLASGOW_DATADISK : 50 % MinBias (UKI-NORTHGRID-SHEF-HEP_DATADISK)
o UKI-LT2-RHUL_DATADISK : 50 % Egamma (UKI-NORTHGRID-LIV-HEP_DATADISK)
o UKI-NORTHGRID-LIV-HEP_DATADISK : 50 % Egamma (UKI-LT2-RHUL_DATADISK)
o UKI-SOUTHGRID-RALPP_DATADISK : 50 Egamma (UKI-LT2-RHUL_DATADISK)
o UKI-SOUTHGRID-BHAM-HEP_DATADISK : 50 Jet (UKI-SOUTHGRID-OX-HEP_DATADISK)
* ES :
o IFIC-LCG2_DATADISK: 50%
o UAM-LCG2_DATADISK: 25%
o IFAE_DATADISK: 25%
* NDGF :
o SIGNET_DATADISK : 50 %

== Logbook ==

=== 2nd June ===

* Since Lyon is in donwtime, so there's no transfers from LYON to T2s today.
http://cc.in2p3.fr/cc_accueil.php3?lang=fr

*Free Space

Token Total Guaranteed Unused
------------------------------------------
BEIJING-LCG2_DATADISK 6.366 6.366 6.343
GRIF-LAL_DATADISK 6.0 6.0 5.732
GRIF-LPNHE_DATADISK 3.906 3.906 3.89
GRIF-SACLAY_DATADISK 6.0 6.0 5.994
IN2P3-CPPM_DATADISK 1.0 1.0 0.977
IN2P3-LAPP_DATADISK 4.0 4.0 4.0
IN2P3-LPC_DATADISK 3.0 3.0 3.0
IN2P3-LPSC_DATADISK 0.488 0.488 0.482
RO-02-NIPNE_DATADISK 3.0 3.0 3.0
RO-07-NIPNE_DATADISK 2.0 2.0 2.0
TOKYO-LCG2_DATADISK 15.0 15.0 13.643

=== 3rd June ===

* Lyon is still in donwtime, no transfers from LYON to T2s in this morning.

; Throughput(snapshot) (MB/s) : [[Image:LYON.throughput.14400_20080603.png]]
; Number of transfer errors(snapshot) : [[Image:LYON.num_file_xs_error.14400_20080603.png]]

=== 4th June ===
;22h02
* Still no transfers, desipte that all the services in LYON shows OK.

;Throughput(snapshot)
[[Image:LYON.throughput.3600_20080604.png]]

;Services in LYON
[[Image:services_cc-in2p3_2008060422h15.PNG]]

;Free space

BEIJING-LCG2_DATADISK 6.366 6.366 6.343
BEIJING-LCG2_MCDISK 5.457 5.457 5.457
GRIF-LAL_DATADISK 6.0 6.0 5.732
GRIF-LAL_MCDISK 7.0 7.0 3.724
GRIF-LPNHE_DATADISK 3.906 3.906 3.89
GRIF-LPNHE_MCDISK 1.0 1.0 0.194
GRIF-SACLAY_DATADISK 6.0 6.0 5.994
GRIF-SACLAY_MCDISK 2.0 2.0 1.713
IN2P3-CPPM_DATADISK 1.0 1.0 0.977
IN2P3-CPPM_MCDISK 0.781 0.781 0.765
IN2P3-LAPP_DATADISK 4.0 4.0 4.0
IN2P3-LAPP_MCDISK 2.0 2.0 0.745
IN2P3-LPC_DATADISK 3.0 3.0 3.0
IN2P3-LPC_MCDISK 2.0 2.0 1.936
IN2P3-LPSC_DATADISK 0.488 0.488 0.482
IN2P3-LPSC_MCDISK 2.0 2.0 1.518
RO-02-NIPNE_DATADISK 3.0 3.0 3.0
RO-02-NIPNE_MCDISK 1.0 1.0 0.995
RO-07-NIPNE_DATADISK 2.0 2.0 2.0
RO-07-NIPNE_MCDISK 1.0 1.0 0.654
TOKYO-LCG2_DATADISK 15.0 15.0 13.643
TOKYO-LCG2_MCDISK 10.0 10.0 10.0

=== 5th June ===

* Data produced on Tuesday are supposed to be shipped in this night.

;14h50
* Now the RAW data has begun to shipped to LYON.

;Throughput(snapshot)(MB/s)
[[Image:LYON.throughput.14400_RAW_20080605_1816.png]]

;Number of transfer errors (snapshot)
[[Image:LYON.num_file_xs_error.14400_20080605_1823.png]]

;21h30
* IN2P3-CC_MCDISK has some problem of staging file. But it should have nothing to do with T0 export.

* 47 datasets have been exported to T1s. The total is 90.

;Throughput(snapshot)(MB/s)
[[Image:LYON.throughput.14400 20080605 2140.png]]

;Number of transfer errors (snapshot) -- It's the problem of staging file in T1 and nothing to do with FDR transfers.
[[Image:LYON.num_file_xs_error.14400_20080605_2140.png]]

=== 6th June ===
;09h50

;Throughput(snapshot) (MB/s)
[[Image:LYON.throughput.14400 200806060 0958.png]]

;Number of transfer errors (snapshot)
[[Image:LYON.num file xs error.14400 20080606 0959.png]]

;10h03
* Now all the RAW datasets have been exported to T1. The total number is 90.
http://atlas.web.cern.ch/Atlas/tzero/prod2/monitoring/tables/ddm_RAW_status.html

=== 7th June ===
* at CERN, will re-start the bulk processing with using the upcoming AtlasPoint1-14.1.0.12 release cache.

;11h30
* SRM in LYON is still in bad shape. Experts are working on it.

=== 8th June ===
;14h20
* FDR2 AODs/DPDs/TAGs/ESDs are being exported to LYON (IN2P3-CC_DATADISK), but haven't been shipped to T2s.

;Number of datasets transfered by far
$ dq2-list-dataset-site IN2P3-CC_DATADISK|grep fdr08_run2.*|wc -l
68

;Throughput(T0-T1,snapshot)(MB/s)
[[Image:LYON.throughput.14400 200806081420.png]]

;Number of transfer errors
[[Image:LYON.num file xs error.14400 200806081420.png]]

;Throughput(T1-T2,snapshot)(MB/s)
[[Image:LYON.T2.throughput.14400 T1-T2 200806081420.png]]

=== 9th June ===
;0h20
* FDR datasets has begun to be shipped to T2s.
* Some T2s (e.g. BEIJING) subscribed dedicated datasets, which will be done later in today's morning.

;Shipped datasets

IN2P3-CC_DATADISK 315
BEIJING-LCG2_DATADISK 0
GRIF-LAL_DATADISK 79
GRIF-LPNHE_DATADISK 39
GRIF-SACLAY_DATADISK 38
IN2P3-CPPM_DATADISK 0
IN2P3-LAPP_DATADISK 12
IN2P3-LPC_DATADISK 12
IN2P3-LPSC_DATADISK 32
RO-02-NIPNE_DATADISK 29
RO-07-NIPNE_DATADISK 46
TOKYO-LCG2_DATADISK 134

;Free space

BEIJING-LCG2_DATADISK 6.366 6.366 6.343
GRIF-LAL_DATADISK 6.0 6.0 5.579
GRIF-LPNHE_DATADISK 3.906 3.906 3.817
GRIF-SACLAY_DATADISK 6.0 6.0 5.927
IN2P3-CPPM_DATADISK 1.0 1.0 0.977
IN2P3-LAPP_DATADISK 4.0 4.0 4.0
IN2P3-LPC_DATADISK 0 0 0
IN2P3-LPSC_DATADISK 0.488 0.488 0.439
RO-02-NIPNE_DATADISK 3.0 3.0 2.986
RO-07-NIPNE_DATADISK 2.0 2.0 1.896
TOKYO-LCG2_DATADISK 15.0 15.0 13.313

;Troughputs(snapshot)(MB/s)
[[Image:LYON.throughput.14400 200806090046.png]]

;Number of transfer erros(snapshot)
* Note: No errors found in FDR2 transfers. Errors showed in this snapshot are since other production data transfers.
[[Image:LYON.num file xs error.14400 200806090047.png]]

;09h24

;Troughputs(snapshot)(MB/s)
[[Image:LYON.throughput.14400 200806090922.png]]

;Number of transfer erros(snapshot)
* Note: No errors found in FDR2 transfers. Errors showed in this snapshot are since other production data transfers.
[[Image:LYON.num file xs error.14400 20806090922.png]]

;Shipped datasets

IN2P3-CC_DATADISK 319
BEIJING-LCG2_DATADISK 0
GRIF-LAL_DATADISK 79
GRIF-LPNHE_DATADISK 39
GRIF-SACLAY_DATADISK 38
IN2P3-CPPM_DATADISK 0
IN2P3-LAPP_DATADISK 12
IN2P3-LPC_DATADISK 12
IN2P3-LPSC_DATADISK 32
RO-02-NIPNE_DATADISK 59
RO-07-NIPNE_DATADISK 49
TOKYO-LCG2_DATADISK 134

;Free space
BEIJING-LCG2_DATADISK 6.366 6.366 6.343
GRIF-LAL_DATADISK 6.0 6.0 5.579
GRIF-LPNHE_DATADISK 3.906 3.906 3.817
GRIF-SACLAY_DATADISK 6.0 6.0 5.927
IN2P3-CPPM_DATADISK 1.0 1.0 0.977
IN2P3-LAPP_DATADISK 4.0 4.0 4.0
IN2P3-LPC_DATADISK 3.0 3.0 3.0
IN2P3-LPSC_DATADISK 0.488 0.488 0.439
RO-02-NIPNE_DATADISK 3.0 3.0 2.945
RO-07-NIPNE_DATADISK 2.0 2.0 1.885
TOKYO-LCG2_DATADISK 15.0 15.0 13.313

;14h00
Many failed with RO-02-NIPNE_DATADISK.

[FTS] FTS State [Failed] FTS Retries [1] Reason [TRANSFER error during TRANSFER phase: [PERMISSION] the server sent an error response: 550 550 rfio write failure: Permission denied.] Source Host [ccsrm.in2p3.fr] 16
[FTS] FTS State [Failed] FTS Retries [1] Reason [TRANSFER error during TRANSFER phase: [TRANSFER_TIMEOUT] gridftp_copy_wait: Connection timed out] Source Host [ccsrm.in2p3.fr] 9
[FTS] FTS State [Failed] FTS Retries [1] Reason [TRANSFER error during TRANSFER phase: [GRIDFTP] the server sent an error response: 426 426 Transfer aborted (Unexpected Exception : java.io.IOException: Broken pipe)] Source Host [ccsrm.in2p3.fr] 2
[FTS] FTS State [Failed] FTS Retries [1] Reason [The process serving the transfer (status = PREPARING) is no longer active (could not open file /proc/3775/cmdline)] Source Host [ccsrm.in2p3.fr] 1
[FTS] FTS State [Failed] FTS Retries [1] Reason [The process serving the transfer (status = PREPARING) is no longer active (could not open file /proc/30076/cmdline)] Source Host [ccsrm.in2p3.fr] 1

;20h10
* FDR datasets have finished transferred to all T2s except BEIJING-LCG2_DATADISK and RO-02-NIPNE_DATADISK.
* Many "Connection timed out" errors found at BEIJING-LCG2_DATADISK and RO-02-NIPNE_DATADISK.

[FTS] FTS State [Failed] FTS Retries [1] Reason [SOURCE error during PREPARATION phase: [CONNECTION] failed to contact on remote SRM [httpg://ccsrm.in2p3.fr:8443/srm/managerv2]. Givin' up after 3 tries] Source Host [ccsrm.in2p3.fr] 111
[FTS] FTS State [Failed] FTS Retries [1] Reason [TRANSFER error during TRANSFER phase: [TRANSFER_TIMEOUT] gridftp_copy_wait: Connection timed out] Source Host [ccsrm.in2p3.fr] 95
[FTS] FTS State [Failed] FTS Retries [1] Reason [SOURCE error during PREPARATION phase: [REQUEST_TIMEOUT] failed to prepare source file in 180 seconds] Source Host [ccsrm.in2p3.fr] 3

; Shipped datasets

IN2P3-CC_DATADISK 134
BEIJING-LCG2_DATADISK 79
GRIF-LAL_DATADISK 67
GRIF-LPNHE_DATADISK 32
GRIF-SACLAY_DATADISK 26
IN2P3-CPPM_DATADISK 52
IN2P3-LAPP_DATADISK 52
IN2P3-LPC_DATADISK 52
IN2P3-LPSC_DATADISK 32
RO-02-NIPNE_DATADISK 49
RO-07-NIPNE_DATADISK 49
TOKYO-LCG2_DATADISK 126

=== 10th June ===

;Summary

All T2s except BEIJING-LCG2_DATADISK and RO-02-NIPNE_DATADISK were in good status and received all the exported datasets when FDR transferring.
The problem of "Connection timed out" with transfers to BEIJING will be hopefully solved by tuned the transfer duration >1800s by Lionel.
RO-02-NIPNE is said being upgraded from SL3 to SL4.

=== 11st June ===

; Anouncement
All FDR2 datasets will be deleted from all sites since part of the detector was not included in the reconstruction. It was not detected by the Data Quality because it is run on the express stream which is correct.

Old datasets will be deleted centrally by Stephane and new datasets that would be named begin with "ccrc08-run.0024*" will be re-generated and re-exported soon in today.

T2s will receive the same share rate as requested for FDR (except that DPD are not produced).

=== 12nd June ===
;01h20
* Function Test datasets(named beginning with "ccrc08_run2.*") to T2s.
For this moment, all T2 sites seems in good shape to accept datasets except RO-07-NIPNE and RO-02-NIPNE that are in downtime.

* Datasets shipped (begins with "ccrc08_run2.*")

IN2P3-CC_DATADISK 387
BEIJING-LCG2_DATADISK 46
GRIF-LAL_DATADISK 141
GRIF-LPNHE_DATADISK 79
GRIF-SACLAY_DATADISK 90
IN2P3-CPPM_DATADISK 22
IN2P3-LAPP_DATADISK 22
IN2P3-LPC_DATADISK 36
IN2P3-LPSC_DATADISK 17
RO-02-NIPNE_DATADISK 20
RO-07-NIPNE_DATADISK 33
TOKYO-LCG2_DATADISK 310

;Troughput
[[Image:LYON.T2.throughput.14400 200806120120.png]]

;Transfer errors
* Note that the errors showed in this snapshot involves all transfers mixed with other transfers(e.g MC/Panda). So it may not reflect the real situation FT transfers.
[[Image:LYON.T2.num file xs error.14400 200806120120.png]]

=== 13rd June ===
=== 14th June ===
* The re-produced FDR2 datasets have been re-exported to T1s.
* Now 3 RAW datasets have been exported to IN2P3-CC_DATATAPE.

=== 15th June ===
* Now 5 RAW datasets have been exported to IN2P3-CC_DATATAPE.
* There's something wrong with the configurations resulted in low transfer rate.

=== 16th June ===
* Configuration problems found in dCahce at LYON resulted in failure of transfers to T2s.

=== 17th June ===

* FDR2 datasets have been exported to T2s.

IN2P3-CC_DATADISK Total:218 Completed:218 AOD:30 TAG:30 DPD:150 ESD:8
BEIJING-LCG2_DATADISK Total:89 Completed:89 AOD:15 TAG:15 DPD:59 ESD:0
GRIF-LAL_DATADISK Total:94 Completed:94 AOD:15 TAG:15 DPD:64 ESD:0
GRIF-LPNHE_DATADISK Total:57 Completed:57 AOD:9 TAG:9 DPD:39 ESD:0
GRIF-SACLAY_DATADISK Total:59 Completed:59 AOD:6 TAG:6 DPD:47 ESD:0
IN2P3-CPPM_DATADISK Total:42 Completed:42 AOD:6 TAG:6 DPD:30 ESD:0
IN2P3-LAPP_DATADISK Total:42 Completed:42 AOD:6 TAG:6 DPD:30 ESD:0
IN2P3-LPC_DATADISK Total:69 Completed:69 AOD:11 TAG:11 DPD:47 ESD:0
IN2P3-LPSC_DATADISK Total:15 Completed:15 AOD:3 TAG:3 DPD:9 ESD:0
RO-02-NIPNE_DATADISK Total:27 Completed:10 AOD:2 TAG:2 DPD:6 ESD:0
RO-07-NIPNE_DATADISK Total:27 Completed:27 AOD:5 TAG:5 DPD:17 ESD:0
TOKYO-LCG2_DATADISK Total:210 Completed:210 AOD:30 TAG:30 DPD:150 ESD:0

Atlas:FDR2

2008-06-17T08:09:19Z

Erming: /* 17th June */

= Monitoring pages/graphs =
=== FTS ===
http://cctoolsafs.in2p3.fr/fts/monitoring/prod/ftsmonitor.php?vo=atlas

=== T0-T1 (ALL) ===
; FDR monitoring page - T1s
http://atldq2pro.cern.ch:8000/ft/mon/fdrmon_tier1s.html

http://dashb-atlas-data-tier0.cern.ch/dashboard/request.py/site
*Throughput 
:http://dashb-atlas-data-tier0.cern.ch/dashboard/templates/plots/OVERVIEW.throughput.14400.png
*Errors 
:http://dashb-atlas-data-tier0.cern.ch/dashboard/templates/plots/OVERVIEW.num_file_xs_error.14400.png

=== T0-T1 (Lyon) ===
http://dashb-atlas-data-tier0.cern.ch/dashboard/request.py/site?statsInterval=4&name=LYON
*Throughput T0-LYON 
:http://dashb-atlas-data-tier0.cern.ch/dashboard/templates/plots/LYON.throughput.14400.png
*Errors T0-LYON 
:http://dashb-atlas-data-tier0.cern.ch/dashboard/templates/plots/LYON.num_file_xs_error.14400.png
*Throughput Others-LYON 
:http://dashb-atlas-data.cern.ch/dashboard/templates/plots/LYON.T1.throughput.14400.png
*Errors Others-LYON :http://dashb-atlas-data.cern.ch/dashboard/templates/plots/LYON.T1.num_file_xs_error.14400.png

=== T1-T1 ===

=== T1-T2 (Lyon) ===
; FDR monitoring page
http://atldq2pro.cern.ch:8000/ft/mon/fdrmon_TiersInfo.html

http://atldq2pro.cern.ch:8000/ft/mon/fdrmon_T1-T2_matrix_day.html

; Dashboard
http://dashb-atlas-data.cern.ch/dashboard/request.py/site?statsInterval=4&name=LYON
*Throughput 
:http://dashb-atlas-data.cern.ch/dashboard/templates/plots/LYON.T2.throughput.14400.png
*Errors
:http://dashb-atlas-data.cern.ch/dashboard/templates/plots/LYON.T2.num_file_xs_error.14400.png

=== Network Graphs ===
; IN2P3 Weathermap
* http://netstat.in2p3.fr/weathermap/
* http://netstat.in2p3.fr/weathermap/graphiques/

; cc-in2p3 : [http://netstat.in2p3.fr/weathermap/graphiques/lyo-cern.html lyo-cern] (lhcopn-in2p3.cern.ch) http://netstat.in2p3.fr/weathermap/graphiques/lyo-cern-daily.gif
; lyon-nord : [http://netstat.in2p3.fr/weathermap/graphiques/lyo-nrd.html lyo-nrd] http://netstat.in2p3.fr/weathermap/graphiques/lyo-nrd-daily.gif
; orsay : [http://netstat.in2p3.fr/weathermap/graphiques/orsay.html orsay] (in2p3-orsay.cssi.renater.fr http://netstat.in2p3.fr/weathermap/graphiques/orsay-daily.gif
; lal-cc : [http://netstat.in2p3.fr/weathermap/graphiques/lal-cc.html Liaison LAL - CC] http://netstat.in2p3.fr/weathermap/graphiques/lal-cc-daily.gif
; lal : [http://netstat.in2p3.fr/weathermap/graphiques/lal.html lal] http://netstat.in2p3.fr/weathermap/graphiques/lal-daily.gif
; lpnhe: [http://netstat.in2p3.fr/weathermap/graphiques/lpnhe-nrd.html lpnhe-nrd] (in2p3-jussieu.cssi.renater.fr) http://netstat.in2p3.fr/weathermap/graphiques/lpnhe-nrd-daily.gif [http://netstat.in2p3.fr/weathermap/graphiques/lpnhe.html lpnhe] (Paris-LPNHE.in2p3.fr) http://netstat.in2p3.fr/weathermap/graphiques/lpnhe-daily.gif
; lapp : [http://netstat.in2p3.fr/weathermap/graphiques/ann-nrd.html ann-nrd] http://netstat.in2p3.fr/weathermap/graphiques/ann-nrd-daily.gif
; lpc: [http://netstat.in2p3.fr/weathermap/graphiques/lpc-cf.html lpc-cf]
http://netstat.in2p3.fr/weathermap/graphiques/lpc-cf-daily.gif
; lpsc : [http://netstat.in2p3.fr/weathermap/graphiques/lpsc.html lpsc] http://netstat.in2p3.fr/weathermap/graphiques/lpsc-daily.gif
; cppm : [http://netstat.in2p3.fr/weathermap/graphiques/cppm.html cppm] 
http://netstat.in2p3.fr/weathermap/graphiques/cppm-daily.gif
; international (tokyo etc.) : [http://netstat.in2p3.fr/weathermap/graphiques/parisnrd.html parisnrd] 
http://netstat.in2p3.fr/weathermap/graphiques/parisnrd-daily.gif
; GEANT - NYC : [http://dc-snmp.wcc.grnoc.iu.edu/manlan/show-graph.cgi?title=sw.newy32aoa.manlan.internet2.edu--te10/1&rrdname=sw.newy32aoa.manlan.internet2.edu--te10_1.rrd MANLAN] 
http://dc-snmp.wcc.grnoc.iu.edu/manlan/img/sw.newy32aoa.manlan.internet2.edu--te10_1-std5.gif
; NYC - TOKYO : [http://dc-snmp.wcc.grnoc.iu.edu/manlan/show-graph.cgi?title=sw.newy32aoa.manlan.internet2.edu--te11/1&rrdname=sw.newy32aoa.manlan.internet2.edu--te11_1.rrd MANLAN] 
http://dc-snmp.wcc.grnoc.iu.edu/manlan/img/sw.newy32aoa.manlan.internet2.edu--te11_1-std5.gif
; tokyo : [http://www.icepp.jp/monitor/gridftp/monitor.html gridftp monitor] http://www.icepp.jp/monitor/gridftp/img/all.png

= T0-T1 =

= T1-T2 =
==== General informations ====
;FDR page:
https://twiki.cern.ch/twiki/bin/view/Atlas/FullDressRehearsal

;DDM_in_FRD2: https://twiki.cern.ch/twiki/bin/view/Atlas/DDMOperationsGroup#DDM_in_FRD2

;Replication status(checked ~ every 2h?)
http://pandamon.usatlas.bnl.gov:25880/server/pandamon/query?mode=listFDR

;Lyon FTS monitor: http://cctoolsafs.in2p3.fr/fts/monitoring/prod/ftsmonitor.php?vo=atlas

;Contacts:

* IN2P3-CPPM_DATADISK :
* IN2P3-LPSC_DATADISK :
* IN2P3-LPC_DATADISK :
* GRIF-LAL_DATADISK : (grid.admin a lal.in2p3.fr)
* GRIF-LPNHE_DATADISK :
* GRIF-SACLAY_DATADISK :
* BEIJING-LCG2_DATADISK : (yanxf a ihep.ac.cn, Erming.Pei a cern.ch)
* RO-07-NIPNE_DATADISK : (ciubancan a nipne.ro)
* RO-02-NIPNE_DATADISK : (tpreda a nipne.ro)
* TOKYO-LCG2_DATADISK : (lcg-admin a icepp.s.u-tokyo.ac.jp)

===== Shares =====
Only srmv2+ space tokens sites will collect data. The list of streams is : Egamma, Bphysics, MinBias, Muon and Jet. If no precise stream name is associated to the share, the site will get any stream. The site in parenthesis will collect the complementary datasets. The T2 ressources at T1 means that users without production role can access CPUs at the T1 without the Production role.
* US :
o AGLT2_DATADISK : 100 %
o MWT2_DATADISK: 100 %
o NET2_DATADISK: 100 %
o SLACXRD_DATADISK: 100 %
o SWT2_CPB_DATADISK: 100 %
o T2 CPU ressources in BNL to access AOD
* CA :
o SFU-LCG2_DATADISK : 40%
o VICTORIA-LCG2_DATADISK : 30%
o ALBERTA-LCG2_DATADISK : 30%
o TORONTO-LCG2_DATADISK : 50 %
* IT :
o INFN-MILANO_DATADISK : 25%
o INFN-FRASCATI_DATADISK : 25%
o INFN-ROMA1_DATADISK : 100%
o INFN-NAPOLI-ATLAS_DATADISK : 100 %
* TW :
o TW-FTT_DATADISK : 50 %
* NL
o JINR-LCG2_DATADISK : 50% Jet (RU-PROTVINO-IHEP_DATADISK)
o JINR-LCG2_DATADISK : 50% MinBias (RU-PROTVINO-IHEP_DATADISK)
o JINR-LCG2_DATADISK : 50% Muon (RU-PROTVINO-IHEP_DATADISK)
o JINR-LCG2_DATADISK : 50% Bphysics (RU-PNPI_DATADISK)
o RU-PROTVINO-IHEP_DATADISK : 50% Jet (JINR-LCG2_DATADISK)
o RU-PROTVINO-IHEP_DATADISK : 50% Muon (JINR-LCG2_DATADISK)
o RU-PROTVINO-IHEP_DATADISK : 50% MinBias (JINR-LCG2_DATADISK)
o RU-PROTVINO-IHEP_DATADISK : 50% Egamma (RU-PNPI_DATADISK)
o RU-PNPI_DATADISK : 50% Egamma (RU-PROTVINO-IHEP_DATADISK)
o RU-PNPI_DATADISK : 50% Bphysics (JINR-LCG2_DATADISK)
o RU-PNPI_DATADISK : 50% Jet (RRC-KI_DATADISK)
o RU-PNPI_DATADISK : 50% MinBias (RRC-KI_DATADISK)
o RRC-KI_DATADISK : 50% Jet (RU-PNPI_DATADISK)
o RRC-KI_DATADISK : 50% MinBias (RU-PNPI_DATADISK)
o RRC-KI_DATADISK : 50% Egamma
o RRC-KI_DATADISK : 50% Bphysics
o CSTCDIE_DATADISK : 20%
* FR :
o IN2P3-LAPP_DATADISK : 100 % Egamma
o IN2P3-CPPM_DATADISK : 100 % Egamma
o IN2P3-LPSC_DATADISK : 5 %
o IN2P3-LPC_DATADISK : 100% Muon, 50% Egamma
o BEIJING-LCG2_DATADISK : 100 % Egamma, 50% Muon, 35% MinBias
o RO-07-NIPNE_DATADISK : 10%
o RO-02-NIPNE_DATADISK : 10%
o GRIF-LAL_DATADISK : 45%
o GRIF-LPNHE_DATADISK : 25%
o GRIF-SACLAY_DATADISK : 30 %
o TOKYO-LCG2_DATADISK : 100%
o T2 CPU ressources in LYON to access AOD
* DE :
o DESY-HH_DATADISK : 100%
o CSCS-LCG2_DATADISK: 100%
o DESY-ZN_DATADISK : 50%
o GOEGRID_DATADISK : 50%
o WUPPERTALPROD_DATADISK: 25%
o CYFRONET-LCG2_DATADISK : 15%
o PRAGUELCG2_DATADISK : 25%
o LRZ-LMU_DATADISK : 35%
o UNI-FREIBURG_DATADISK : 100% Jet, 100% Muon
* UK :
o UKI-SOUTHGRID-CAM-HEP_DATADISK : 50 % Muon (UKI-NORTHGRID-LANCS-HEP_DATADISK
o UKI-NORTHGRID-LANCS-HEP_DATADISK : 50 % Muon (UKI-SOUTHGRID-CAM-HEP_DATADISK)
o UKI-NORTHGRID-SHEF-HEP_DATADISK : 50 % MinBias (UKI-SCOTGRID-GLASGOW_DATADISK)
o UKI-SCOTGRID-GLASGOW_DATADISK : 50 % MinBias (UKI-NORTHGRID-SHEF-HEP_DATADISK)
o UKI-LT2-RHUL_DATADISK : 50 % Egamma (UKI-NORTHGRID-LIV-HEP_DATADISK)
o UKI-NORTHGRID-LIV-HEP_DATADISK : 50 % Egamma (UKI-LT2-RHUL_DATADISK)
o UKI-SOUTHGRID-RALPP_DATADISK : 50 Egamma (UKI-LT2-RHUL_DATADISK)
o UKI-SOUTHGRID-BHAM-HEP_DATADISK : 50 Jet (UKI-SOUTHGRID-OX-HEP_DATADISK)
* ES :
o IFIC-LCG2_DATADISK: 50%
o UAM-LCG2_DATADISK: 25%
o IFAE_DATADISK: 25%
* NDGF :
o SIGNET_DATADISK : 50 %

== Logbook ==

=== 2nd June ===

* Since Lyon is in donwtime, so there's no transfers from LYON to T2s today.
http://cc.in2p3.fr/cc_accueil.php3?lang=fr

*Free Space

Token Total Guaranteed Unused
------------------------------------------
BEIJING-LCG2_DATADISK 6.366 6.366 6.343
GRIF-LAL_DATADISK 6.0 6.0 5.732
GRIF-LPNHE_DATADISK 3.906 3.906 3.89
GRIF-SACLAY_DATADISK 6.0 6.0 5.994
IN2P3-CPPM_DATADISK 1.0 1.0 0.977
IN2P3-LAPP_DATADISK 4.0 4.0 4.0
IN2P3-LPC_DATADISK 3.0 3.0 3.0
IN2P3-LPSC_DATADISK 0.488 0.488 0.482
RO-02-NIPNE_DATADISK 3.0 3.0 3.0
RO-07-NIPNE_DATADISK 2.0 2.0 2.0
TOKYO-LCG2_DATADISK 15.0 15.0 13.643

=== 3rd June ===

* Lyon is still in donwtime, no transfers from LYON to T2s in this morning.

; Throughput(snapshot) (MB/s) : [[Image:LYON.throughput.14400_20080603.png]]
; Number of transfer errors(snapshot) : [[Image:LYON.num_file_xs_error.14400_20080603.png]]

=== 4th June ===
;22h02
* Still no transfers, desipte that all the services in LYON shows OK.

;Throughput(snapshot)
[[Image:LYON.throughput.3600_20080604.png]]

;Services in LYON
[[Image:services_cc-in2p3_2008060422h15.PNG]]

;Free space

BEIJING-LCG2_DATADISK 6.366 6.366 6.343
BEIJING-LCG2_MCDISK 5.457 5.457 5.457
GRIF-LAL_DATADISK 6.0 6.0 5.732
GRIF-LAL_MCDISK 7.0 7.0 3.724
GRIF-LPNHE_DATADISK 3.906 3.906 3.89
GRIF-LPNHE_MCDISK 1.0 1.0 0.194
GRIF-SACLAY_DATADISK 6.0 6.0 5.994
GRIF-SACLAY_MCDISK 2.0 2.0 1.713
IN2P3-CPPM_DATADISK 1.0 1.0 0.977
IN2P3-CPPM_MCDISK 0.781 0.781 0.765
IN2P3-LAPP_DATADISK 4.0 4.0 4.0
IN2P3-LAPP_MCDISK 2.0 2.0 0.745
IN2P3-LPC_DATADISK 3.0 3.0 3.0
IN2P3-LPC_MCDISK 2.0 2.0 1.936
IN2P3-LPSC_DATADISK 0.488 0.488 0.482
IN2P3-LPSC_MCDISK 2.0 2.0 1.518
RO-02-NIPNE_DATADISK 3.0 3.0 3.0
RO-02-NIPNE_MCDISK 1.0 1.0 0.995
RO-07-NIPNE_DATADISK 2.0 2.0 2.0
RO-07-NIPNE_MCDISK 1.0 1.0 0.654
TOKYO-LCG2_DATADISK 15.0 15.0 13.643
TOKYO-LCG2_MCDISK 10.0 10.0 10.0

=== 5th June ===

* Data produced on Tuesday are supposed to be shipped in this night.

;14h50
* Now the RAW data has begun to shipped to LYON.

;Throughput(snapshot)(MB/s)
[[Image:LYON.throughput.14400_RAW_20080605_1816.png]]

;Number of transfer errors (snapshot)
[[Image:LYON.num_file_xs_error.14400_20080605_1823.png]]

;21h30
* IN2P3-CC_MCDISK has some problem of staging file. But it should have nothing to do with T0 export.

* 47 datasets have been exported to T1s. The total is 90.

;Throughput(snapshot)(MB/s)
[[Image:LYON.throughput.14400 20080605 2140.png]]

;Number of transfer errors (snapshot) -- It's the problem of staging file in T1 and nothing to do with FDR transfers.
[[Image:LYON.num_file_xs_error.14400_20080605_2140.png]]

=== 6th June ===
;09h50

;Throughput(snapshot) (MB/s)
[[Image:LYON.throughput.14400 200806060 0958.png]]

;Number of transfer errors (snapshot)
[[Image:LYON.num file xs error.14400 20080606 0959.png]]

;10h03
* Now all the RAW datasets have been exported to T1. The total number is 90.
http://atlas.web.cern.ch/Atlas/tzero/prod2/monitoring/tables/ddm_RAW_status.html

=== 7th June ===
* at CERN, will re-start the bulk processing with using the upcoming AtlasPoint1-14.1.0.12 release cache.

;11h30
* SRM in LYON is still in bad shape. Experts are working on it.

=== 8th June ===
;14h20
* FDR2 AODs/DPDs/TAGs/ESDs are being exported to LYON (IN2P3-CC_DATADISK), but haven't been shipped to T2s.

;Number of datasets transfered by far
$ dq2-list-dataset-site IN2P3-CC_DATADISK|grep fdr08_run2.*|wc -l
68

;Throughput(T0-T1,snapshot)(MB/s)
[[Image:LYON.throughput.14400 200806081420.png]]

;Number of transfer errors
[[Image:LYON.num file xs error.14400 200806081420.png]]

;Throughput(T1-T2,snapshot)(MB/s)
[[Image:LYON.T2.throughput.14400 T1-T2 200806081420.png]]

=== 9th June ===
;0h20
* FDR datasets has begun to be shipped to T2s.
* Some T2s (e.g. BEIJING) subscribed dedicated datasets, which will be done later in today's morning.

;Shipped datasets

IN2P3-CC_DATADISK 315
BEIJING-LCG2_DATADISK 0
GRIF-LAL_DATADISK 79
GRIF-LPNHE_DATADISK 39
GRIF-SACLAY_DATADISK 38
IN2P3-CPPM_DATADISK 0
IN2P3-LAPP_DATADISK 12
IN2P3-LPC_DATADISK 12
IN2P3-LPSC_DATADISK 32
RO-02-NIPNE_DATADISK 29
RO-07-NIPNE_DATADISK 46
TOKYO-LCG2_DATADISK 134

;Free space

BEIJING-LCG2_DATADISK 6.366 6.366 6.343
GRIF-LAL_DATADISK 6.0 6.0 5.579
GRIF-LPNHE_DATADISK 3.906 3.906 3.817
GRIF-SACLAY_DATADISK 6.0 6.0 5.927
IN2P3-CPPM_DATADISK 1.0 1.0 0.977
IN2P3-LAPP_DATADISK 4.0 4.0 4.0
IN2P3-LPC_DATADISK 0 0 0
IN2P3-LPSC_DATADISK 0.488 0.488 0.439
RO-02-NIPNE_DATADISK 3.0 3.0 2.986
RO-07-NIPNE_DATADISK 2.0 2.0 1.896
TOKYO-LCG2_DATADISK 15.0 15.0 13.313

;Troughputs(snapshot)(MB/s)
[[Image:LYON.throughput.14400 200806090046.png]]

;Number of transfer erros(snapshot)
* Note: No errors found in FDR2 transfers. Errors showed in this snapshot are since other production data transfers.
[[Image:LYON.num file xs error.14400 200806090047.png]]

;09h24

;Troughputs(snapshot)(MB/s)
[[Image:LYON.throughput.14400 200806090922.png]]

;Number of transfer erros(snapshot)
* Note: No errors found in FDR2 transfers. Errors showed in this snapshot are since other production data transfers.
[[Image:LYON.num file xs error.14400 20806090922.png]]

;Shipped datasets

IN2P3-CC_DATADISK 319
BEIJING-LCG2_DATADISK 0
GRIF-LAL_DATADISK 79
GRIF-LPNHE_DATADISK 39
GRIF-SACLAY_DATADISK 38
IN2P3-CPPM_DATADISK 0
IN2P3-LAPP_DATADISK 12
IN2P3-LPC_DATADISK 12
IN2P3-LPSC_DATADISK 32
RO-02-NIPNE_DATADISK 59
RO-07-NIPNE_DATADISK 49
TOKYO-LCG2_DATADISK 134

;Free space
BEIJING-LCG2_DATADISK 6.366 6.366 6.343
GRIF-LAL_DATADISK 6.0 6.0 5.579
GRIF-LPNHE_DATADISK 3.906 3.906 3.817
GRIF-SACLAY_DATADISK 6.0 6.0 5.927
IN2P3-CPPM_DATADISK 1.0 1.0 0.977
IN2P3-LAPP_DATADISK 4.0 4.0 4.0
IN2P3-LPC_DATADISK 3.0 3.0 3.0
IN2P3-LPSC_DATADISK 0.488 0.488 0.439
RO-02-NIPNE_DATADISK 3.0 3.0 2.945
RO-07-NIPNE_DATADISK 2.0 2.0 1.885
TOKYO-LCG2_DATADISK 15.0 15.0 13.313

;14h00
Many failed with RO-02-NIPNE_DATADISK.

[FTS] FTS State [Failed] FTS Retries [1] Reason [TRANSFER error during TRANSFER phase: [PERMISSION] the server sent an error response: 550 550 rfio write failure: Permission denied.] Source Host [ccsrm.in2p3.fr] 16
[FTS] FTS State [Failed] FTS Retries [1] Reason [TRANSFER error during TRANSFER phase: [TRANSFER_TIMEOUT] gridftp_copy_wait: Connection timed out] Source Host [ccsrm.in2p3.fr] 9
[FTS] FTS State [Failed] FTS Retries [1] Reason [TRANSFER error during TRANSFER phase: [GRIDFTP] the server sent an error response: 426 426 Transfer aborted (Unexpected Exception : java.io.IOException: Broken pipe)] Source Host [ccsrm.in2p3.fr] 2
[FTS] FTS State [Failed] FTS Retries [1] Reason [The process serving the transfer (status = PREPARING) is no longer active (could not open file /proc/3775/cmdline)] Source Host [ccsrm.in2p3.fr] 1
[FTS] FTS State [Failed] FTS Retries [1] Reason [The process serving the transfer (status = PREPARING) is no longer active (could not open file /proc/30076/cmdline)] Source Host [ccsrm.in2p3.fr] 1

;20h10
* FDR datasets have finished transferred to all T2s except BEIJING-LCG2_DATADISK and RO-02-NIPNE_DATADISK.
* Many "Connection timed out" errors found at BEIJING-LCG2_DATADISK and RO-02-NIPNE_DATADISK.

[FTS] FTS State [Failed] FTS Retries [1] Reason [SOURCE error during PREPARATION phase: [CONNECTION] failed to contact on remote SRM [httpg://ccsrm.in2p3.fr:8443/srm/managerv2]. Givin' up after 3 tries] Source Host [ccsrm.in2p3.fr] 111
[FTS] FTS State [Failed] FTS Retries [1] Reason [TRANSFER error during TRANSFER phase: [TRANSFER_TIMEOUT] gridftp_copy_wait: Connection timed out] Source Host [ccsrm.in2p3.fr] 95
[FTS] FTS State [Failed] FTS Retries [1] Reason [SOURCE error during PREPARATION phase: [REQUEST_TIMEOUT] failed to prepare source file in 180 seconds] Source Host [ccsrm.in2p3.fr] 3

; Shipped datasets

IN2P3-CC_DATADISK 134
BEIJING-LCG2_DATADISK 79
GRIF-LAL_DATADISK 67
GRIF-LPNHE_DATADISK 32
GRIF-SACLAY_DATADISK 26
IN2P3-CPPM_DATADISK 52
IN2P3-LAPP_DATADISK 52
IN2P3-LPC_DATADISK 52
IN2P3-LPSC_DATADISK 32
RO-02-NIPNE_DATADISK 49
RO-07-NIPNE_DATADISK 49
TOKYO-LCG2_DATADISK 126

=== 10th June ===

;Summary

All T2s except BEIJING-LCG2_DATADISK and RO-02-NIPNE_DATADISK were in good status and received all the exported datasets when FDR transferring.
The problem of "Connection timed out" with transfers to BEIJING will be hopefully solved by tuned the transfer duration >1800s by Lionel.
RO-02-NIPNE is said being upgraded from SL3 to SL4.

=== 11st June ===

; Anouncement
All FDR2 datasets will be deleted from all sites since part of the detector was not included in the reconstruction. It was not detected by the Data Quality because it is run on the express stream which is correct.

Old datasets will be deleted centrally by Stephane and new datasets that would be named begin with "ccrc08-run.0024*" will be re-generated and re-exported soon in today.

T2s will receive the same share rate as requested for FDR (except that DPD are not produced).

=== 12nd June ===
;01h20
* Function Test datasets(named beginning with "ccrc08_run2.*") to T2s.
For this moment, all T2 sites seems in good shape to accept datasets except RO-07-NIPNE and RO-02-NIPNE that are in downtime.

* Datasets shipped (begins with "ccrc08_run2.*")

IN2P3-CC_DATADISK 387
BEIJING-LCG2_DATADISK 46
GRIF-LAL_DATADISK 141
GRIF-LPNHE_DATADISK 79
GRIF-SACLAY_DATADISK 90
IN2P3-CPPM_DATADISK 22
IN2P3-LAPP_DATADISK 22
IN2P3-LPC_DATADISK 36
IN2P3-LPSC_DATADISK 17
RO-02-NIPNE_DATADISK 20
RO-07-NIPNE_DATADISK 33
TOKYO-LCG2_DATADISK 310

;Troughput
[[Image:LYON.T2.throughput.14400 200806120120.png]]

;Transfer errors
* Note that the errors showed in this snapshot involves all transfers mixed with other transfers(e.g MC/Panda). So it may not reflect the real situation FT transfers.
[[Image:LYON.T2.num file xs error.14400 200806120120.png]]

=== 13rd June ===
=== 14th June ===
* The re-produced FDR2 datasets have been re-exported to T1s.
* Now 3 RAW datasets have been exported to IN2P3-CC_DATATAPE.

=== 15th June ===
* Now 5 RAW datasets have been exported to IN2P3-CC_DATATAPE.
* There's something wrong with the configurations resulted in low transfer rate.

=== 16th June ===
* Configuration problems found in dCahce at LYON resulted in failure of transfers to T2s.

=== 17th June ===

* FDR2 datasets have exported to T2s.

IN2P3-CC_DATADISK Total:218 Completed:218 AOD:30 TAG:30 DPD:150 ESD:8
BEIJING-LCG2_DATADISK Total:89 Completed:89 AOD:15 TAG:15 DPD:59 ESD:0
GRIF-LAL_DATADISK Total:94 Completed:94 AOD:15 TAG:15 DPD:64 ESD:0
GRIF-LPNHE_DATADISK Total:57 Completed:57 AOD:9 TAG:9 DPD:39 ESD:0
GRIF-SACLAY_DATADISK Total:59 Completed:59 AOD:6 TAG:6 DPD:47 ESD:0
IN2P3-CPPM_DATADISK Total:42 Completed:42 AOD:6 TAG:6 DPD:30 ESD:0
IN2P3-LAPP_DATADISK Total:42 Completed:42 AOD:6 TAG:6 DPD:30 ESD:0
IN2P3-LPC_DATADISK Total:69 Completed:69 AOD:11 TAG:11 DPD:47 ESD:0
IN2P3-LPSC_DATADISK Total:15 Completed:15 AOD:3 TAG:3 DPD:9 ESD:0
RO-02-NIPNE_DATADISK Total:27 Completed:10 AOD:2 TAG:2 DPD:6 ESD:0
RO-07-NIPNE_DATADISK Total:27 Completed:27 AOD:5 TAG:5 DPD:17 ESD:0
TOKYO-LCG2_DATADISK Total:210 Completed:210 AOD:30 TAG:30 DPD:150 ESD:0

Atlas:FDR2

2008-06-17T08:08:35Z

Erming: /* 17th June */

= Monitoring pages/graphs =
=== FTS ===
http://cctoolsafs.in2p3.fr/fts/monitoring/prod/ftsmonitor.php?vo=atlas

=== T0-T1 (ALL) ===
; FDR monitoring page - T1s
http://atldq2pro.cern.ch:8000/ft/mon/fdrmon_tier1s.html

http://dashb-atlas-data-tier0.cern.ch/dashboard/request.py/site
*Throughput 
:http://dashb-atlas-data-tier0.cern.ch/dashboard/templates/plots/OVERVIEW.throughput.14400.png
*Errors 
:http://dashb-atlas-data-tier0.cern.ch/dashboard/templates/plots/OVERVIEW.num_file_xs_error.14400.png

=== T0-T1 (Lyon) ===
http://dashb-atlas-data-tier0.cern.ch/dashboard/request.py/site?statsInterval=4&name=LYON
*Throughput T0-LYON 
:http://dashb-atlas-data-tier0.cern.ch/dashboard/templates/plots/LYON.throughput.14400.png
*Errors T0-LYON 
:http://dashb-atlas-data-tier0.cern.ch/dashboard/templates/plots/LYON.num_file_xs_error.14400.png
*Throughput Others-LYON 
:http://dashb-atlas-data.cern.ch/dashboard/templates/plots/LYON.T1.throughput.14400.png
*Errors Others-LYON :http://dashb-atlas-data.cern.ch/dashboard/templates/plots/LYON.T1.num_file_xs_error.14400.png

=== T1-T1 ===

=== T1-T2 (Lyon) ===
; FDR monitoring page
http://atldq2pro.cern.ch:8000/ft/mon/fdrmon_TiersInfo.html

http://atldq2pro.cern.ch:8000/ft/mon/fdrmon_T1-T2_matrix_day.html

; Dashboard
http://dashb-atlas-data.cern.ch/dashboard/request.py/site?statsInterval=4&name=LYON
*Throughput 
:http://dashb-atlas-data.cern.ch/dashboard/templates/plots/LYON.T2.throughput.14400.png
*Errors
:http://dashb-atlas-data.cern.ch/dashboard/templates/plots/LYON.T2.num_file_xs_error.14400.png

=== Network Graphs ===
; IN2P3 Weathermap
* http://netstat.in2p3.fr/weathermap/
* http://netstat.in2p3.fr/weathermap/graphiques/

; cc-in2p3 : [http://netstat.in2p3.fr/weathermap/graphiques/lyo-cern.html lyo-cern] (lhcopn-in2p3.cern.ch) http://netstat.in2p3.fr/weathermap/graphiques/lyo-cern-daily.gif
; lyon-nord : [http://netstat.in2p3.fr/weathermap/graphiques/lyo-nrd.html lyo-nrd] http://netstat.in2p3.fr/weathermap/graphiques/lyo-nrd-daily.gif
; orsay : [http://netstat.in2p3.fr/weathermap/graphiques/orsay.html orsay] (in2p3-orsay.cssi.renater.fr http://netstat.in2p3.fr/weathermap/graphiques/orsay-daily.gif
; lal-cc : [http://netstat.in2p3.fr/weathermap/graphiques/lal-cc.html Liaison LAL - CC] http://netstat.in2p3.fr/weathermap/graphiques/lal-cc-daily.gif
; lal : [http://netstat.in2p3.fr/weathermap/graphiques/lal.html lal] http://netstat.in2p3.fr/weathermap/graphiques/lal-daily.gif
; lpnhe: [http://netstat.in2p3.fr/weathermap/graphiques/lpnhe-nrd.html lpnhe-nrd] (in2p3-jussieu.cssi.renater.fr) http://netstat.in2p3.fr/weathermap/graphiques/lpnhe-nrd-daily.gif [http://netstat.in2p3.fr/weathermap/graphiques/lpnhe.html lpnhe] (Paris-LPNHE.in2p3.fr) http://netstat.in2p3.fr/weathermap/graphiques/lpnhe-daily.gif
; lapp : [http://netstat.in2p3.fr/weathermap/graphiques/ann-nrd.html ann-nrd] http://netstat.in2p3.fr/weathermap/graphiques/ann-nrd-daily.gif
; lpc: [http://netstat.in2p3.fr/weathermap/graphiques/lpc-cf.html lpc-cf]
http://netstat.in2p3.fr/weathermap/graphiques/lpc-cf-daily.gif
; lpsc : [http://netstat.in2p3.fr/weathermap/graphiques/lpsc.html lpsc] http://netstat.in2p3.fr/weathermap/graphiques/lpsc-daily.gif
; cppm : [http://netstat.in2p3.fr/weathermap/graphiques/cppm.html cppm] 
http://netstat.in2p3.fr/weathermap/graphiques/cppm-daily.gif
; international (tokyo etc.) : [http://netstat.in2p3.fr/weathermap/graphiques/parisnrd.html parisnrd] 
http://netstat.in2p3.fr/weathermap/graphiques/parisnrd-daily.gif
; GEANT - NYC : [http://dc-snmp.wcc.grnoc.iu.edu/manlan/show-graph.cgi?title=sw.newy32aoa.manlan.internet2.edu--te10/1&rrdname=sw.newy32aoa.manlan.internet2.edu--te10_1.rrd MANLAN] 
http://dc-snmp.wcc.grnoc.iu.edu/manlan/img/sw.newy32aoa.manlan.internet2.edu--te10_1-std5.gif
; NYC - TOKYO : [http://dc-snmp.wcc.grnoc.iu.edu/manlan/show-graph.cgi?title=sw.newy32aoa.manlan.internet2.edu--te11/1&rrdname=sw.newy32aoa.manlan.internet2.edu--te11_1.rrd MANLAN] 
http://dc-snmp.wcc.grnoc.iu.edu/manlan/img/sw.newy32aoa.manlan.internet2.edu--te11_1-std5.gif
; tokyo : [http://www.icepp.jp/monitor/gridftp/monitor.html gridftp monitor] http://www.icepp.jp/monitor/gridftp/img/all.png

= T0-T1 =

= T1-T2 =
==== General informations ====
;FDR page:
https://twiki.cern.ch/twiki/bin/view/Atlas/FullDressRehearsal

;DDM_in_FRD2: https://twiki.cern.ch/twiki/bin/view/Atlas/DDMOperationsGroup#DDM_in_FRD2

;Replication status(checked ~ every 2h?)
http://pandamon.usatlas.bnl.gov:25880/server/pandamon/query?mode=listFDR

;Lyon FTS monitor: http://cctoolsafs.in2p3.fr/fts/monitoring/prod/ftsmonitor.php?vo=atlas

;Contacts:

* IN2P3-CPPM_DATADISK :
* IN2P3-LPSC_DATADISK :
* IN2P3-LPC_DATADISK :
* GRIF-LAL_DATADISK : (grid.admin a lal.in2p3.fr)
* GRIF-LPNHE_DATADISK :
* GRIF-SACLAY_DATADISK :
* BEIJING-LCG2_DATADISK : (yanxf a ihep.ac.cn, Erming.Pei a cern.ch)
* RO-07-NIPNE_DATADISK : (ciubancan a nipne.ro)
* RO-02-NIPNE_DATADISK : (tpreda a nipne.ro)
* TOKYO-LCG2_DATADISK : (lcg-admin a icepp.s.u-tokyo.ac.jp)

===== Shares =====
Only srmv2+ space tokens sites will collect data. The list of streams is : Egamma, Bphysics, MinBias, Muon and Jet. If no precise stream name is associated to the share, the site will get any stream. The site in parenthesis will collect the complementary datasets. The T2 ressources at T1 means that users without production role can access CPUs at the T1 without the Production role.
* US :
o AGLT2_DATADISK : 100 %
o MWT2_DATADISK: 100 %
o NET2_DATADISK: 100 %
o SLACXRD_DATADISK: 100 %
o SWT2_CPB_DATADISK: 100 %
o T2 CPU ressources in BNL to access AOD
* CA :
o SFU-LCG2_DATADISK : 40%
o VICTORIA-LCG2_DATADISK : 30%
o ALBERTA-LCG2_DATADISK : 30%
o TORONTO-LCG2_DATADISK : 50 %
* IT :
o INFN-MILANO_DATADISK : 25%
o INFN-FRASCATI_DATADISK : 25%
o INFN-ROMA1_DATADISK : 100%
o INFN-NAPOLI-ATLAS_DATADISK : 100 %
* TW :
o TW-FTT_DATADISK : 50 %
* NL
o JINR-LCG2_DATADISK : 50% Jet (RU-PROTVINO-IHEP_DATADISK)
o JINR-LCG2_DATADISK : 50% MinBias (RU-PROTVINO-IHEP_DATADISK)
o JINR-LCG2_DATADISK : 50% Muon (RU-PROTVINO-IHEP_DATADISK)
o JINR-LCG2_DATADISK : 50% Bphysics (RU-PNPI_DATADISK)
o RU-PROTVINO-IHEP_DATADISK : 50% Jet (JINR-LCG2_DATADISK)
o RU-PROTVINO-IHEP_DATADISK : 50% Muon (JINR-LCG2_DATADISK)
o RU-PROTVINO-IHEP_DATADISK : 50% MinBias (JINR-LCG2_DATADISK)
o RU-PROTVINO-IHEP_DATADISK : 50% Egamma (RU-PNPI_DATADISK)
o RU-PNPI_DATADISK : 50% Egamma (RU-PROTVINO-IHEP_DATADISK)
o RU-PNPI_DATADISK : 50% Bphysics (JINR-LCG2_DATADISK)
o RU-PNPI_DATADISK : 50% Jet (RRC-KI_DATADISK)
o RU-PNPI_DATADISK : 50% MinBias (RRC-KI_DATADISK)
o RRC-KI_DATADISK : 50% Jet (RU-PNPI_DATADISK)
o RRC-KI_DATADISK : 50% MinBias (RU-PNPI_DATADISK)
o RRC-KI_DATADISK : 50% Egamma
o RRC-KI_DATADISK : 50% Bphysics
o CSTCDIE_DATADISK : 20%
* FR :
o IN2P3-LAPP_DATADISK : 100 % Egamma
o IN2P3-CPPM_DATADISK : 100 % Egamma
o IN2P3-LPSC_DATADISK : 5 %
o IN2P3-LPC_DATADISK : 100% Muon, 50% Egamma
o BEIJING-LCG2_DATADISK : 100 % Egamma, 50% Muon, 35% MinBias
o RO-07-NIPNE_DATADISK : 10%
o RO-02-NIPNE_DATADISK : 10%
o GRIF-LAL_DATADISK : 45%
o GRIF-LPNHE_DATADISK : 25%
o GRIF-SACLAY_DATADISK : 30 %
o TOKYO-LCG2_DATADISK : 100%
o T2 CPU ressources in LYON to access AOD
* DE :
o DESY-HH_DATADISK : 100%
o CSCS-LCG2_DATADISK: 100%
o DESY-ZN_DATADISK : 50%
o GOEGRID_DATADISK : 50%
o WUPPERTALPROD_DATADISK: 25%
o CYFRONET-LCG2_DATADISK : 15%
o PRAGUELCG2_DATADISK : 25%
o LRZ-LMU_DATADISK : 35%
o UNI-FREIBURG_DATADISK : 100% Jet, 100% Muon
* UK :
o UKI-SOUTHGRID-CAM-HEP_DATADISK : 50 % Muon (UKI-NORTHGRID-LANCS-HEP_DATADISK
o UKI-NORTHGRID-LANCS-HEP_DATADISK : 50 % Muon (UKI-SOUTHGRID-CAM-HEP_DATADISK)
o UKI-NORTHGRID-SHEF-HEP_DATADISK : 50 % MinBias (UKI-SCOTGRID-GLASGOW_DATADISK)
o UKI-SCOTGRID-GLASGOW_DATADISK : 50 % MinBias (UKI-NORTHGRID-SHEF-HEP_DATADISK)
o UKI-LT2-RHUL_DATADISK : 50 % Egamma (UKI-NORTHGRID-LIV-HEP_DATADISK)
o UKI-NORTHGRID-LIV-HEP_DATADISK : 50 % Egamma (UKI-LT2-RHUL_DATADISK)
o UKI-SOUTHGRID-RALPP_DATADISK : 50 Egamma (UKI-LT2-RHUL_DATADISK)
o UKI-SOUTHGRID-BHAM-HEP_DATADISK : 50 Jet (UKI-SOUTHGRID-OX-HEP_DATADISK)
* ES :
o IFIC-LCG2_DATADISK: 50%
o UAM-LCG2_DATADISK: 25%
o IFAE_DATADISK: 25%
* NDGF :
o SIGNET_DATADISK : 50 %

== Logbook ==

=== 2nd June ===

* Since Lyon is in donwtime, so there's no transfers from LYON to T2s today.
http://cc.in2p3.fr/cc_accueil.php3?lang=fr

*Free Space

Token Total Guaranteed Unused
------------------------------------------
BEIJING-LCG2_DATADISK 6.366 6.366 6.343
GRIF-LAL_DATADISK 6.0 6.0 5.732
GRIF-LPNHE_DATADISK 3.906 3.906 3.89
GRIF-SACLAY_DATADISK 6.0 6.0 5.994
IN2P3-CPPM_DATADISK 1.0 1.0 0.977
IN2P3-LAPP_DATADISK 4.0 4.0 4.0
IN2P3-LPC_DATADISK 3.0 3.0 3.0
IN2P3-LPSC_DATADISK 0.488 0.488 0.482
RO-02-NIPNE_DATADISK 3.0 3.0 3.0
RO-07-NIPNE_DATADISK 2.0 2.0 2.0
TOKYO-LCG2_DATADISK 15.0 15.0 13.643

=== 3rd June ===

* Lyon is still in donwtime, no transfers from LYON to T2s in this morning.

; Throughput(snapshot) (MB/s) : [[Image:LYON.throughput.14400_20080603.png]]
; Number of transfer errors(snapshot) : [[Image:LYON.num_file_xs_error.14400_20080603.png]]

=== 4th June ===
;22h02
* Still no transfers, desipte that all the services in LYON shows OK.

;Throughput(snapshot)
[[Image:LYON.throughput.3600_20080604.png]]

;Services in LYON
[[Image:services_cc-in2p3_2008060422h15.PNG]]

;Free space

BEIJING-LCG2_DATADISK 6.366 6.366 6.343
BEIJING-LCG2_MCDISK 5.457 5.457 5.457
GRIF-LAL_DATADISK 6.0 6.0 5.732
GRIF-LAL_MCDISK 7.0 7.0 3.724
GRIF-LPNHE_DATADISK 3.906 3.906 3.89
GRIF-LPNHE_MCDISK 1.0 1.0 0.194
GRIF-SACLAY_DATADISK 6.0 6.0 5.994
GRIF-SACLAY_MCDISK 2.0 2.0 1.713
IN2P3-CPPM_DATADISK 1.0 1.0 0.977
IN2P3-CPPM_MCDISK 0.781 0.781 0.765
IN2P3-LAPP_DATADISK 4.0 4.0 4.0
IN2P3-LAPP_MCDISK 2.0 2.0 0.745
IN2P3-LPC_DATADISK 3.0 3.0 3.0
IN2P3-LPC_MCDISK 2.0 2.0 1.936
IN2P3-LPSC_DATADISK 0.488 0.488 0.482
IN2P3-LPSC_MCDISK 2.0 2.0 1.518
RO-02-NIPNE_DATADISK 3.0 3.0 3.0
RO-02-NIPNE_MCDISK 1.0 1.0 0.995
RO-07-NIPNE_DATADISK 2.0 2.0 2.0
RO-07-NIPNE_MCDISK 1.0 1.0 0.654
TOKYO-LCG2_DATADISK 15.0 15.0 13.643
TOKYO-LCG2_MCDISK 10.0 10.0 10.0

=== 5th June ===

* Data produced on Tuesday are supposed to be shipped in this night.

;14h50
* Now the RAW data has begun to shipped to LYON.

;Throughput(snapshot)(MB/s)
[[Image:LYON.throughput.14400_RAW_20080605_1816.png]]

;Number of transfer errors (snapshot)
[[Image:LYON.num_file_xs_error.14400_20080605_1823.png]]

;21h30
* IN2P3-CC_MCDISK has some problem of staging file. But it should have nothing to do with T0 export.

* 47 datasets have been exported to T1s. The total is 90.

;Throughput(snapshot)(MB/s)
[[Image:LYON.throughput.14400 20080605 2140.png]]

;Number of transfer errors (snapshot) -- It's the problem of staging file in T1 and nothing to do with FDR transfers.
[[Image:LYON.num_file_xs_error.14400_20080605_2140.png]]

=== 6th June ===
;09h50

;Throughput(snapshot) (MB/s)
[[Image:LYON.throughput.14400 200806060 0958.png]]

;Number of transfer errors (snapshot)
[[Image:LYON.num file xs error.14400 20080606 0959.png]]

;10h03
* Now all the RAW datasets have been exported to T1. The total number is 90.
http://atlas.web.cern.ch/Atlas/tzero/prod2/monitoring/tables/ddm_RAW_status.html

=== 7th June ===
* at CERN, will re-start the bulk processing with using the upcoming AtlasPoint1-14.1.0.12 release cache.

;11h30
* SRM in LYON is still in bad shape. Experts are working on it.

=== 8th June ===
;14h20
* FDR2 AODs/DPDs/TAGs/ESDs are being exported to LYON (IN2P3-CC_DATADISK), but haven't been shipped to T2s.

;Number of datasets transfered by far
$ dq2-list-dataset-site IN2P3-CC_DATADISK|grep fdr08_run2.*|wc -l
68

;Throughput(T0-T1,snapshot)(MB/s)
[[Image:LYON.throughput.14400 200806081420.png]]

;Number of transfer errors
[[Image:LYON.num file xs error.14400 200806081420.png]]

;Throughput(T1-T2,snapshot)(MB/s)
[[Image:LYON.T2.throughput.14400 T1-T2 200806081420.png]]

=== 9th June ===
;0h20
* FDR datasets has begun to be shipped to T2s.
* Some T2s (e.g. BEIJING) subscribed dedicated datasets, which will be done later in today's morning.

;Shipped datasets

IN2P3-CC_DATADISK 315
BEIJING-LCG2_DATADISK 0
GRIF-LAL_DATADISK 79
GRIF-LPNHE_DATADISK 39
GRIF-SACLAY_DATADISK 38
IN2P3-CPPM_DATADISK 0
IN2P3-LAPP_DATADISK 12
IN2P3-LPC_DATADISK 12
IN2P3-LPSC_DATADISK 32
RO-02-NIPNE_DATADISK 29
RO-07-NIPNE_DATADISK 46
TOKYO-LCG2_DATADISK 134

;Free space

BEIJING-LCG2_DATADISK 6.366 6.366 6.343
GRIF-LAL_DATADISK 6.0 6.0 5.579
GRIF-LPNHE_DATADISK 3.906 3.906 3.817
GRIF-SACLAY_DATADISK 6.0 6.0 5.927
IN2P3-CPPM_DATADISK 1.0 1.0 0.977
IN2P3-LAPP_DATADISK 4.0 4.0 4.0
IN2P3-LPC_DATADISK 0 0 0
IN2P3-LPSC_DATADISK 0.488 0.488 0.439
RO-02-NIPNE_DATADISK 3.0 3.0 2.986
RO-07-NIPNE_DATADISK 2.0 2.0 1.896
TOKYO-LCG2_DATADISK 15.0 15.0 13.313

;Troughputs(snapshot)(MB/s)
[[Image:LYON.throughput.14400 200806090046.png]]

;Number of transfer erros(snapshot)
* Note: No errors found in FDR2 transfers. Errors showed in this snapshot are since other production data transfers.
[[Image:LYON.num file xs error.14400 200806090047.png]]

;09h24

;Troughputs(snapshot)(MB/s)
[[Image:LYON.throughput.14400 200806090922.png]]

;Number of transfer erros(snapshot)
* Note: No errors found in FDR2 transfers. Errors showed in this snapshot are since other production data transfers.
[[Image:LYON.num file xs error.14400 20806090922.png]]

;Shipped datasets

IN2P3-CC_DATADISK 319
BEIJING-LCG2_DATADISK 0
GRIF-LAL_DATADISK 79
GRIF-LPNHE_DATADISK 39
GRIF-SACLAY_DATADISK 38
IN2P3-CPPM_DATADISK 0
IN2P3-LAPP_DATADISK 12
IN2P3-LPC_DATADISK 12
IN2P3-LPSC_DATADISK 32
RO-02-NIPNE_DATADISK 59
RO-07-NIPNE_DATADISK 49
TOKYO-LCG2_DATADISK 134

;Free space
BEIJING-LCG2_DATADISK 6.366 6.366 6.343
GRIF-LAL_DATADISK 6.0 6.0 5.579
GRIF-LPNHE_DATADISK 3.906 3.906 3.817
GRIF-SACLAY_DATADISK 6.0 6.0 5.927
IN2P3-CPPM_DATADISK 1.0 1.0 0.977
IN2P3-LAPP_DATADISK 4.0 4.0 4.0
IN2P3-LPC_DATADISK 3.0 3.0 3.0
IN2P3-LPSC_DATADISK 0.488 0.488 0.439
RO-02-NIPNE_DATADISK 3.0 3.0 2.945
RO-07-NIPNE_DATADISK 2.0 2.0 1.885
TOKYO-LCG2_DATADISK 15.0 15.0 13.313

;14h00
Many failed with RO-02-NIPNE_DATADISK.

[FTS] FTS State [Failed] FTS Retries [1] Reason [TRANSFER error during TRANSFER phase: [PERMISSION] the server sent an error response: 550 550 rfio write failure: Permission denied.] Source Host [ccsrm.in2p3.fr] 16
[FTS] FTS State [Failed] FTS Retries [1] Reason [TRANSFER error during TRANSFER phase: [TRANSFER_TIMEOUT] gridftp_copy_wait: Connection timed out] Source Host [ccsrm.in2p3.fr] 9
[FTS] FTS State [Failed] FTS Retries [1] Reason [TRANSFER error during TRANSFER phase: [GRIDFTP] the server sent an error response: 426 426 Transfer aborted (Unexpected Exception : java.io.IOException: Broken pipe)] Source Host [ccsrm.in2p3.fr] 2
[FTS] FTS State [Failed] FTS Retries [1] Reason [The process serving the transfer (status = PREPARING) is no longer active (could not open file /proc/3775/cmdline)] Source Host [ccsrm.in2p3.fr] 1
[FTS] FTS State [Failed] FTS Retries [1] Reason [The process serving the transfer (status = PREPARING) is no longer active (could not open file /proc/30076/cmdline)] Source Host [ccsrm.in2p3.fr] 1

;20h10
* FDR datasets have finished transferred to all T2s except BEIJING-LCG2_DATADISK and RO-02-NIPNE_DATADISK.
* Many "Connection timed out" errors found at BEIJING-LCG2_DATADISK and RO-02-NIPNE_DATADISK.

[FTS] FTS State [Failed] FTS Retries [1] Reason [SOURCE error during PREPARATION phase: [CONNECTION] failed to contact on remote SRM [httpg://ccsrm.in2p3.fr:8443/srm/managerv2]. Givin' up after 3 tries] Source Host [ccsrm.in2p3.fr] 111
[FTS] FTS State [Failed] FTS Retries [1] Reason [TRANSFER error during TRANSFER phase: [TRANSFER_TIMEOUT] gridftp_copy_wait: Connection timed out] Source Host [ccsrm.in2p3.fr] 95
[FTS] FTS State [Failed] FTS Retries [1] Reason [SOURCE error during PREPARATION phase: [REQUEST_TIMEOUT] failed to prepare source file in 180 seconds] Source Host [ccsrm.in2p3.fr] 3

; Shipped datasets

IN2P3-CC_DATADISK 134
BEIJING-LCG2_DATADISK 79
GRIF-LAL_DATADISK 67
GRIF-LPNHE_DATADISK 32
GRIF-SACLAY_DATADISK 26
IN2P3-CPPM_DATADISK 52
IN2P3-LAPP_DATADISK 52
IN2P3-LPC_DATADISK 52
IN2P3-LPSC_DATADISK 32
RO-02-NIPNE_DATADISK 49
RO-07-NIPNE_DATADISK 49
TOKYO-LCG2_DATADISK 126

=== 10th June ===

;Summary

All T2s except BEIJING-LCG2_DATADISK and RO-02-NIPNE_DATADISK were in good status and received all the exported datasets when FDR transferring.
The problem of "Connection timed out" with transfers to BEIJING will be hopefully solved by tuned the transfer duration >1800s by Lionel.
RO-02-NIPNE is said being upgraded from SL3 to SL4.

=== 11st June ===

; Anouncement
All FDR2 datasets will be deleted from all sites since part of the detector was not included in the reconstruction. It was not detected by the Data Quality because it is run on the express stream which is correct.

Old datasets will be deleted centrally by Stephane and new datasets that would be named begin with "ccrc08-run.0024*" will be re-generated and re-exported soon in today.

T2s will receive the same share rate as requested for FDR (except that DPD are not produced).

=== 12nd June ===
;01h20
* Function Test datasets(named beginning with "ccrc08_run2.*") to T2s.
For this moment, all T2 sites seems in good shape to accept datasets except RO-07-NIPNE and RO-02-NIPNE that are in downtime.

* Datasets shipped (begins with "ccrc08_run2.*")

IN2P3-CC_DATADISK 387
BEIJING-LCG2_DATADISK 46
GRIF-LAL_DATADISK 141
GRIF-LPNHE_DATADISK 79
GRIF-SACLAY_DATADISK 90
IN2P3-CPPM_DATADISK 22
IN2P3-LAPP_DATADISK 22
IN2P3-LPC_DATADISK 36
IN2P3-LPSC_DATADISK 17
RO-02-NIPNE_DATADISK 20
RO-07-NIPNE_DATADISK 33
TOKYO-LCG2_DATADISK 310

;Troughput
[[Image:LYON.T2.throughput.14400 200806120120.png]]

;Transfer errors
* Note that the errors showed in this snapshot involves all transfers mixed with other transfers(e.g MC/Panda). So it may not reflect the real situation FT transfers.
[[Image:LYON.T2.num file xs error.14400 200806120120.png]]

=== 13rd June ===
=== 14th June ===
* The re-produced FDR2 datasets have been re-exported to T1s.
* Now 3 RAW datasets have been exported to IN2P3-CC_DATATAPE.

=== 15th June ===
* Now 5 RAW datasets have been exported to IN2P3-CC_DATATAPE.
* There's something wrong with the configurations resulted in low transfer rate.

=== 16th June ===
* Configuration problems found in dCahce at LYON resulted in failure of transfers to T2s.

=== 17th June ===

* FDR2 datasets have exported to T2s.

IN2P3-CC_DATADISK Total:218 Completed:218 AOD:30 TAG:30 DPD:150 ESD:8
BEIJING-LCG2_DATADISK Total:89 Completed:89 AOD:15 TAG:15 DPD:59 ESD:0
GRIF-LAL_DATADISK Total:94 Completed:94 AOD:15 TAG:15 DPD:64 ESD:0
GRIF-LPNHE_DATADISK Total:57 Completed:57 AOD:9 TAG:9 DPD:39 ESD:0
GRIF-SACLAY_DATADISK Total:59 Completed:59 AOD:6 TAG:6 DPD:47 ESD:0
IN2P3-CPPM_DATADISK Total:42 Completed:42 AOD:6 TAG:6 DPD:30 ESD:0
IN2P3-LAPP_DATADISK Total:42 Completed:42 AOD:6 TAG:6 DPD:30 ESD:0
IN2P3-LPC_DATADISK Total:69 Completed:69 AOD:11 TAG:11 DPD:47 ESD:0
IN2P3-LPSC_DATADISK Total:15 Completed:15 AOD:3 TAG:3 DPD:9 ESD:0
RO-02-NIPNE_DATADISK Total:27 Completed:10 AOD:2 TAG:2 DPD:6 ESD:0
RO-07-NIPNE_DATADISK Total:27 Completed:27 AOD:5 TAG:5 DPD:17 ESD:0
TOKYO-LCG2_DATADISK Total:210 Completed:210 AOD:30 TAG:30 DPD:150 ESD:0

Atlas:FDR2

2008-06-17T08:08:10Z

Erming: /* 17th June */

= Monitoring pages/graphs =
=== FTS ===
http://cctoolsafs.in2p3.fr/fts/monitoring/prod/ftsmonitor.php?vo=atlas

=== T0-T1 (ALL) ===
; FDR monitoring page - T1s
http://atldq2pro.cern.ch:8000/ft/mon/fdrmon_tier1s.html

http://dashb-atlas-data-tier0.cern.ch/dashboard/request.py/site
*Throughput 
:http://dashb-atlas-data-tier0.cern.ch/dashboard/templates/plots/OVERVIEW.throughput.14400.png
*Errors 
:http://dashb-atlas-data-tier0.cern.ch/dashboard/templates/plots/OVERVIEW.num_file_xs_error.14400.png

=== T0-T1 (Lyon) ===
http://dashb-atlas-data-tier0.cern.ch/dashboard/request.py/site?statsInterval=4&name=LYON
*Throughput T0-LYON 
:http://dashb-atlas-data-tier0.cern.ch/dashboard/templates/plots/LYON.throughput.14400.png
*Errors T0-LYON 
:http://dashb-atlas-data-tier0.cern.ch/dashboard/templates/plots/LYON.num_file_xs_error.14400.png
*Throughput Others-LYON 
:http://dashb-atlas-data.cern.ch/dashboard/templates/plots/LYON.T1.throughput.14400.png
*Errors Others-LYON :http://dashb-atlas-data.cern.ch/dashboard/templates/plots/LYON.T1.num_file_xs_error.14400.png

=== T1-T1 ===

=== T1-T2 (Lyon) ===
; FDR monitoring page
http://atldq2pro.cern.ch:8000/ft/mon/fdrmon_TiersInfo.html

http://atldq2pro.cern.ch:8000/ft/mon/fdrmon_T1-T2_matrix_day.html

; Dashboard
http://dashb-atlas-data.cern.ch/dashboard/request.py/site?statsInterval=4&name=LYON
*Throughput 
:http://dashb-atlas-data.cern.ch/dashboard/templates/plots/LYON.T2.throughput.14400.png
*Errors
:http://dashb-atlas-data.cern.ch/dashboard/templates/plots/LYON.T2.num_file_xs_error.14400.png

=== Network Graphs ===
; IN2P3 Weathermap
* http://netstat.in2p3.fr/weathermap/
* http://netstat.in2p3.fr/weathermap/graphiques/

; cc-in2p3 : [http://netstat.in2p3.fr/weathermap/graphiques/lyo-cern.html lyo-cern] (lhcopn-in2p3.cern.ch) http://netstat.in2p3.fr/weathermap/graphiques/lyo-cern-daily.gif
; lyon-nord : [http://netstat.in2p3.fr/weathermap/graphiques/lyo-nrd.html lyo-nrd] http://netstat.in2p3.fr/weathermap/graphiques/lyo-nrd-daily.gif
; orsay : [http://netstat.in2p3.fr/weathermap/graphiques/orsay.html orsay] (in2p3-orsay.cssi.renater.fr http://netstat.in2p3.fr/weathermap/graphiques/orsay-daily.gif
; lal-cc : [http://netstat.in2p3.fr/weathermap/graphiques/lal-cc.html Liaison LAL - CC] http://netstat.in2p3.fr/weathermap/graphiques/lal-cc-daily.gif
; lal : [http://netstat.in2p3.fr/weathermap/graphiques/lal.html lal] http://netstat.in2p3.fr/weathermap/graphiques/lal-daily.gif
; lpnhe: [http://netstat.in2p3.fr/weathermap/graphiques/lpnhe-nrd.html lpnhe-nrd] (in2p3-jussieu.cssi.renater.fr) http://netstat.in2p3.fr/weathermap/graphiques/lpnhe-nrd-daily.gif [http://netstat.in2p3.fr/weathermap/graphiques/lpnhe.html lpnhe] (Paris-LPNHE.in2p3.fr) http://netstat.in2p3.fr/weathermap/graphiques/lpnhe-daily.gif
; lapp : [http://netstat.in2p3.fr/weathermap/graphiques/ann-nrd.html ann-nrd] http://netstat.in2p3.fr/weathermap/graphiques/ann-nrd-daily.gif
; lpc: [http://netstat.in2p3.fr/weathermap/graphiques/lpc-cf.html lpc-cf]
http://netstat.in2p3.fr/weathermap/graphiques/lpc-cf-daily.gif
; lpsc : [http://netstat.in2p3.fr/weathermap/graphiques/lpsc.html lpsc] http://netstat.in2p3.fr/weathermap/graphiques/lpsc-daily.gif
; cppm : [http://netstat.in2p3.fr/weathermap/graphiques/cppm.html cppm] 
http://netstat.in2p3.fr/weathermap/graphiques/cppm-daily.gif
; international (tokyo etc.) : [http://netstat.in2p3.fr/weathermap/graphiques/parisnrd.html parisnrd] 
http://netstat.in2p3.fr/weathermap/graphiques/parisnrd-daily.gif
; GEANT - NYC : [http://dc-snmp.wcc.grnoc.iu.edu/manlan/show-graph.cgi?title=sw.newy32aoa.manlan.internet2.edu--te10/1&rrdname=sw.newy32aoa.manlan.internet2.edu--te10_1.rrd MANLAN] 
http://dc-snmp.wcc.grnoc.iu.edu/manlan/img/sw.newy32aoa.manlan.internet2.edu--te10_1-std5.gif
; NYC - TOKYO : [http://dc-snmp.wcc.grnoc.iu.edu/manlan/show-graph.cgi?title=sw.newy32aoa.manlan.internet2.edu--te11/1&rrdname=sw.newy32aoa.manlan.internet2.edu--te11_1.rrd MANLAN] 
http://dc-snmp.wcc.grnoc.iu.edu/manlan/img/sw.newy32aoa.manlan.internet2.edu--te11_1-std5.gif
; tokyo : [http://www.icepp.jp/monitor/gridftp/monitor.html gridftp monitor] http://www.icepp.jp/monitor/gridftp/img/all.png

= T0-T1 =

= T1-T2 =
==== General informations ====
;FDR page:
https://twiki.cern.ch/twiki/bin/view/Atlas/FullDressRehearsal

;DDM_in_FRD2: https://twiki.cern.ch/twiki/bin/view/Atlas/DDMOperationsGroup#DDM_in_FRD2

;Replication status(checked ~ every 2h?)
http://pandamon.usatlas.bnl.gov:25880/server/pandamon/query?mode=listFDR

;Lyon FTS monitor: http://cctoolsafs.in2p3.fr/fts/monitoring/prod/ftsmonitor.php?vo=atlas

;Contacts:

* IN2P3-CPPM_DATADISK :
* IN2P3-LPSC_DATADISK :
* IN2P3-LPC_DATADISK :
* GRIF-LAL_DATADISK : (grid.admin a lal.in2p3.fr)
* GRIF-LPNHE_DATADISK :
* GRIF-SACLAY_DATADISK :
* BEIJING-LCG2_DATADISK : (yanxf a ihep.ac.cn, Erming.Pei a cern.ch)
* RO-07-NIPNE_DATADISK : (ciubancan a nipne.ro)
* RO-02-NIPNE_DATADISK : (tpreda a nipne.ro)
* TOKYO-LCG2_DATADISK : (lcg-admin a icepp.s.u-tokyo.ac.jp)

===== Shares =====
Only srmv2+ space tokens sites will collect data. The list of streams is : Egamma, Bphysics, MinBias, Muon and Jet. If no precise stream name is associated to the share, the site will get any stream. The site in parenthesis will collect the complementary datasets. The T2 ressources at T1 means that users without production role can access CPUs at the T1 without the Production role.
* US :
o AGLT2_DATADISK : 100 %
o MWT2_DATADISK: 100 %
o NET2_DATADISK: 100 %
o SLACXRD_DATADISK: 100 %
o SWT2_CPB_DATADISK: 100 %
o T2 CPU ressources in BNL to access AOD
* CA :
o SFU-LCG2_DATADISK : 40%
o VICTORIA-LCG2_DATADISK : 30%
o ALBERTA-LCG2_DATADISK : 30%
o TORONTO-LCG2_DATADISK : 50 %
* IT :
o INFN-MILANO_DATADISK : 25%
o INFN-FRASCATI_DATADISK : 25%
o INFN-ROMA1_DATADISK : 100%
o INFN-NAPOLI-ATLAS_DATADISK : 100 %
* TW :
o TW-FTT_DATADISK : 50 %
* NL
o JINR-LCG2_DATADISK : 50% Jet (RU-PROTVINO-IHEP_DATADISK)
o JINR-LCG2_DATADISK : 50% MinBias (RU-PROTVINO-IHEP_DATADISK)
o JINR-LCG2_DATADISK : 50% Muon (RU-PROTVINO-IHEP_DATADISK)
o JINR-LCG2_DATADISK : 50% Bphysics (RU-PNPI_DATADISK)
o RU-PROTVINO-IHEP_DATADISK : 50% Jet (JINR-LCG2_DATADISK)
o RU-PROTVINO-IHEP_DATADISK : 50% Muon (JINR-LCG2_DATADISK)
o RU-PROTVINO-IHEP_DATADISK : 50% MinBias (JINR-LCG2_DATADISK)
o RU-PROTVINO-IHEP_DATADISK : 50% Egamma (RU-PNPI_DATADISK)
o RU-PNPI_DATADISK : 50% Egamma (RU-PROTVINO-IHEP_DATADISK)
o RU-PNPI_DATADISK : 50% Bphysics (JINR-LCG2_DATADISK)
o RU-PNPI_DATADISK : 50% Jet (RRC-KI_DATADISK)
o RU-PNPI_DATADISK : 50% MinBias (RRC-KI_DATADISK)
o RRC-KI_DATADISK : 50% Jet (RU-PNPI_DATADISK)
o RRC-KI_DATADISK : 50% MinBias (RU-PNPI_DATADISK)
o RRC-KI_DATADISK : 50% Egamma
o RRC-KI_DATADISK : 50% Bphysics
o CSTCDIE_DATADISK : 20%
* FR :
o IN2P3-LAPP_DATADISK : 100 % Egamma
o IN2P3-CPPM_DATADISK : 100 % Egamma
o IN2P3-LPSC_DATADISK : 5 %
o IN2P3-LPC_DATADISK : 100% Muon, 50% Egamma
o BEIJING-LCG2_DATADISK : 100 % Egamma, 50% Muon, 35% MinBias
o RO-07-NIPNE_DATADISK : 10%
o RO-02-NIPNE_DATADISK : 10%
o GRIF-LAL_DATADISK : 45%
o GRIF-LPNHE_DATADISK : 25%
o GRIF-SACLAY_DATADISK : 30 %
o TOKYO-LCG2_DATADISK : 100%
o T2 CPU ressources in LYON to access AOD
* DE :
o DESY-HH_DATADISK : 100%
o CSCS-LCG2_DATADISK: 100%
o DESY-ZN_DATADISK : 50%
o GOEGRID_DATADISK : 50%
o WUPPERTALPROD_DATADISK: 25%
o CYFRONET-LCG2_DATADISK : 15%
o PRAGUELCG2_DATADISK : 25%
o LRZ-LMU_DATADISK : 35%
o UNI-FREIBURG_DATADISK : 100% Jet, 100% Muon
* UK :
o UKI-SOUTHGRID-CAM-HEP_DATADISK : 50 % Muon (UKI-NORTHGRID-LANCS-HEP_DATADISK
o UKI-NORTHGRID-LANCS-HEP_DATADISK : 50 % Muon (UKI-SOUTHGRID-CAM-HEP_DATADISK)
o UKI-NORTHGRID-SHEF-HEP_DATADISK : 50 % MinBias (UKI-SCOTGRID-GLASGOW_DATADISK)
o UKI-SCOTGRID-GLASGOW_DATADISK : 50 % MinBias (UKI-NORTHGRID-SHEF-HEP_DATADISK)
o UKI-LT2-RHUL_DATADISK : 50 % Egamma (UKI-NORTHGRID-LIV-HEP_DATADISK)
o UKI-NORTHGRID-LIV-HEP_DATADISK : 50 % Egamma (UKI-LT2-RHUL_DATADISK)
o UKI-SOUTHGRID-RALPP_DATADISK : 50 Egamma (UKI-LT2-RHUL_DATADISK)
o UKI-SOUTHGRID-BHAM-HEP_DATADISK : 50 Jet (UKI-SOUTHGRID-OX-HEP_DATADISK)
* ES :
o IFIC-LCG2_DATADISK: 50%
o UAM-LCG2_DATADISK: 25%
o IFAE_DATADISK: 25%
* NDGF :
o SIGNET_DATADISK : 50 %

== Logbook ==

=== 2nd June ===

* Since Lyon is in donwtime, so there's no transfers from LYON to T2s today.
http://cc.in2p3.fr/cc_accueil.php3?lang=fr

*Free Space

Token Total Guaranteed Unused
------------------------------------------
BEIJING-LCG2_DATADISK 6.366 6.366 6.343
GRIF-LAL_DATADISK 6.0 6.0 5.732
GRIF-LPNHE_DATADISK 3.906 3.906 3.89
GRIF-SACLAY_DATADISK 6.0 6.0 5.994
IN2P3-CPPM_DATADISK 1.0 1.0 0.977
IN2P3-LAPP_DATADISK 4.0 4.0 4.0
IN2P3-LPC_DATADISK 3.0 3.0 3.0
IN2P3-LPSC_DATADISK 0.488 0.488 0.482
RO-02-NIPNE_DATADISK 3.0 3.0 3.0
RO-07-NIPNE_DATADISK 2.0 2.0 2.0
TOKYO-LCG2_DATADISK 15.0 15.0 13.643

=== 3rd June ===

* Lyon is still in donwtime, no transfers from LYON to T2s in this morning.

; Throughput(snapshot) (MB/s) : [[Image:LYON.throughput.14400_20080603.png]]
; Number of transfer errors(snapshot) : [[Image:LYON.num_file_xs_error.14400_20080603.png]]

=== 4th June ===
;22h02
* Still no transfers, desipte that all the services in LYON shows OK.

;Throughput(snapshot)
[[Image:LYON.throughput.3600_20080604.png]]

;Services in LYON
[[Image:services_cc-in2p3_2008060422h15.PNG]]

;Free space

BEIJING-LCG2_DATADISK 6.366 6.366 6.343
BEIJING-LCG2_MCDISK 5.457 5.457 5.457
GRIF-LAL_DATADISK 6.0 6.0 5.732
GRIF-LAL_MCDISK 7.0 7.0 3.724
GRIF-LPNHE_DATADISK 3.906 3.906 3.89
GRIF-LPNHE_MCDISK 1.0 1.0 0.194
GRIF-SACLAY_DATADISK 6.0 6.0 5.994
GRIF-SACLAY_MCDISK 2.0 2.0 1.713
IN2P3-CPPM_DATADISK 1.0 1.0 0.977
IN2P3-CPPM_MCDISK 0.781 0.781 0.765
IN2P3-LAPP_DATADISK 4.0 4.0 4.0
IN2P3-LAPP_MCDISK 2.0 2.0 0.745
IN2P3-LPC_DATADISK 3.0 3.0 3.0
IN2P3-LPC_MCDISK 2.0 2.0 1.936
IN2P3-LPSC_DATADISK 0.488 0.488 0.482
IN2P3-LPSC_MCDISK 2.0 2.0 1.518
RO-02-NIPNE_DATADISK 3.0 3.0 3.0
RO-02-NIPNE_MCDISK 1.0 1.0 0.995
RO-07-NIPNE_DATADISK 2.0 2.0 2.0
RO-07-NIPNE_MCDISK 1.0 1.0 0.654
TOKYO-LCG2_DATADISK 15.0 15.0 13.643
TOKYO-LCG2_MCDISK 10.0 10.0 10.0

=== 5th June ===

* Data produced on Tuesday are supposed to be shipped in this night.

;14h50
* Now the RAW data has begun to shipped to LYON.

;Throughput(snapshot)(MB/s)
[[Image:LYON.throughput.14400_RAW_20080605_1816.png]]

;Number of transfer errors (snapshot)
[[Image:LYON.num_file_xs_error.14400_20080605_1823.png]]

;21h30
* IN2P3-CC_MCDISK has some problem of staging file. But it should have nothing to do with T0 export.

* 47 datasets have been exported to T1s. The total is 90.

;Throughput(snapshot)(MB/s)
[[Image:LYON.throughput.14400 20080605 2140.png]]

;Number of transfer errors (snapshot) -- It's the problem of staging file in T1 and nothing to do with FDR transfers.
[[Image:LYON.num_file_xs_error.14400_20080605_2140.png]]

=== 6th June ===
;09h50

;Throughput(snapshot) (MB/s)
[[Image:LYON.throughput.14400 200806060 0958.png]]

;Number of transfer errors (snapshot)
[[Image:LYON.num file xs error.14400 20080606 0959.png]]

;10h03
* Now all the RAW datasets have been exported to T1. The total number is 90.
http://atlas.web.cern.ch/Atlas/tzero/prod2/monitoring/tables/ddm_RAW_status.html

=== 7th June ===
* at CERN, will re-start the bulk processing with using the upcoming AtlasPoint1-14.1.0.12 release cache.

;11h30
* SRM in LYON is still in bad shape. Experts are working on it.

=== 8th June ===
;14h20
* FDR2 AODs/DPDs/TAGs/ESDs are being exported to LYON (IN2P3-CC_DATADISK), but haven't been shipped to T2s.

;Number of datasets transfered by far
$ dq2-list-dataset-site IN2P3-CC_DATADISK|grep fdr08_run2.*|wc -l
68

;Throughput(T0-T1,snapshot)(MB/s)
[[Image:LYON.throughput.14400 200806081420.png]]

;Number of transfer errors
[[Image:LYON.num file xs error.14400 200806081420.png]]

;Throughput(T1-T2,snapshot)(MB/s)
[[Image:LYON.T2.throughput.14400 T1-T2 200806081420.png]]

=== 9th June ===
;0h20
* FDR datasets has begun to be shipped to T2s.
* Some T2s (e.g. BEIJING) subscribed dedicated datasets, which will be done later in today's morning.

;Shipped datasets

IN2P3-CC_DATADISK 315
BEIJING-LCG2_DATADISK 0
GRIF-LAL_DATADISK 79
GRIF-LPNHE_DATADISK 39
GRIF-SACLAY_DATADISK 38
IN2P3-CPPM_DATADISK 0
IN2P3-LAPP_DATADISK 12
IN2P3-LPC_DATADISK 12
IN2P3-LPSC_DATADISK 32
RO-02-NIPNE_DATADISK 29
RO-07-NIPNE_DATADISK 46
TOKYO-LCG2_DATADISK 134

;Free space

BEIJING-LCG2_DATADISK 6.366 6.366 6.343
GRIF-LAL_DATADISK 6.0 6.0 5.579
GRIF-LPNHE_DATADISK 3.906 3.906 3.817
GRIF-SACLAY_DATADISK 6.0 6.0 5.927
IN2P3-CPPM_DATADISK 1.0 1.0 0.977
IN2P3-LAPP_DATADISK 4.0 4.0 4.0
IN2P3-LPC_DATADISK 0 0 0
IN2P3-LPSC_DATADISK 0.488 0.488 0.439
RO-02-NIPNE_DATADISK 3.0 3.0 2.986
RO-07-NIPNE_DATADISK 2.0 2.0 1.896
TOKYO-LCG2_DATADISK 15.0 15.0 13.313

;Troughputs(snapshot)(MB/s)
[[Image:LYON.throughput.14400 200806090046.png]]

;Number of transfer erros(snapshot)
* Note: No errors found in FDR2 transfers. Errors showed in this snapshot are since other production data transfers.
[[Image:LYON.num file xs error.14400 200806090047.png]]

;09h24

;Troughputs(snapshot)(MB/s)
[[Image:LYON.throughput.14400 200806090922.png]]

;Number of transfer erros(snapshot)
* Note: No errors found in FDR2 transfers. Errors showed in this snapshot are since other production data transfers.
[[Image:LYON.num file xs error.14400 20806090922.png]]

;Shipped datasets

IN2P3-CC_DATADISK 319
BEIJING-LCG2_DATADISK 0
GRIF-LAL_DATADISK 79
GRIF-LPNHE_DATADISK 39
GRIF-SACLAY_DATADISK 38
IN2P3-CPPM_DATADISK 0
IN2P3-LAPP_DATADISK 12
IN2P3-LPC_DATADISK 12
IN2P3-LPSC_DATADISK 32
RO-02-NIPNE_DATADISK 59
RO-07-NIPNE_DATADISK 49
TOKYO-LCG2_DATADISK 134

;Free space
BEIJING-LCG2_DATADISK 6.366 6.366 6.343
GRIF-LAL_DATADISK 6.0 6.0 5.579
GRIF-LPNHE_DATADISK 3.906 3.906 3.817
GRIF-SACLAY_DATADISK 6.0 6.0 5.927
IN2P3-CPPM_DATADISK 1.0 1.0 0.977
IN2P3-LAPP_DATADISK 4.0 4.0 4.0
IN2P3-LPC_DATADISK 3.0 3.0 3.0
IN2P3-LPSC_DATADISK 0.488 0.488 0.439
RO-02-NIPNE_DATADISK 3.0 3.0 2.945
RO-07-NIPNE_DATADISK 2.0 2.0 1.885
TOKYO-LCG2_DATADISK 15.0 15.0 13.313

;14h00
Many failed with RO-02-NIPNE_DATADISK.

[FTS] FTS State [Failed] FTS Retries [1] Reason [TRANSFER error during TRANSFER phase: [PERMISSION] the server sent an error response: 550 550 rfio write failure: Permission denied.] Source Host [ccsrm.in2p3.fr] 16
[FTS] FTS State [Failed] FTS Retries [1] Reason [TRANSFER error during TRANSFER phase: [TRANSFER_TIMEOUT] gridftp_copy_wait: Connection timed out] Source Host [ccsrm.in2p3.fr] 9
[FTS] FTS State [Failed] FTS Retries [1] Reason [TRANSFER error during TRANSFER phase: [GRIDFTP] the server sent an error response: 426 426 Transfer aborted (Unexpected Exception : java.io.IOException: Broken pipe)] Source Host [ccsrm.in2p3.fr] 2
[FTS] FTS State [Failed] FTS Retries [1] Reason [The process serving the transfer (status = PREPARING) is no longer active (could not open file /proc/3775/cmdline)] Source Host [ccsrm.in2p3.fr] 1
[FTS] FTS State [Failed] FTS Retries [1] Reason [The process serving the transfer (status = PREPARING) is no longer active (could not open file /proc/30076/cmdline)] Source Host [ccsrm.in2p3.fr] 1

;20h10
* FDR datasets have finished transferred to all T2s except BEIJING-LCG2_DATADISK and RO-02-NIPNE_DATADISK.
* Many "Connection timed out" errors found at BEIJING-LCG2_DATADISK and RO-02-NIPNE_DATADISK.

[FTS] FTS State [Failed] FTS Retries [1] Reason [SOURCE error during PREPARATION phase: [CONNECTION] failed to contact on remote SRM [httpg://ccsrm.in2p3.fr:8443/srm/managerv2]. Givin' up after 3 tries] Source Host [ccsrm.in2p3.fr] 111
[FTS] FTS State [Failed] FTS Retries [1] Reason [TRANSFER error during TRANSFER phase: [TRANSFER_TIMEOUT] gridftp_copy_wait: Connection timed out] Source Host [ccsrm.in2p3.fr] 95
[FTS] FTS State [Failed] FTS Retries [1] Reason [SOURCE error during PREPARATION phase: [REQUEST_TIMEOUT] failed to prepare source file in 180 seconds] Source Host [ccsrm.in2p3.fr] 3

; Shipped datasets

IN2P3-CC_DATADISK 134
BEIJING-LCG2_DATADISK 79
GRIF-LAL_DATADISK 67
GRIF-LPNHE_DATADISK 32
GRIF-SACLAY_DATADISK 26
IN2P3-CPPM_DATADISK 52
IN2P3-LAPP_DATADISK 52
IN2P3-LPC_DATADISK 52
IN2P3-LPSC_DATADISK 32
RO-02-NIPNE_DATADISK 49
RO-07-NIPNE_DATADISK 49
TOKYO-LCG2_DATADISK 126

=== 10th June ===

;Summary

All T2s except BEIJING-LCG2_DATADISK and RO-02-NIPNE_DATADISK were in good status and received all the exported datasets when FDR transferring.
The problem of "Connection timed out" with transfers to BEIJING will be hopefully solved by tuned the transfer duration >1800s by Lionel.
RO-02-NIPNE is said being upgraded from SL3 to SL4.

=== 11st June ===

; Anouncement
All FDR2 datasets will be deleted from all sites since part of the detector was not included in the reconstruction. It was not detected by the Data Quality because it is run on the express stream which is correct.

Old datasets will be deleted centrally by Stephane and new datasets that would be named begin with "ccrc08-run.0024*" will be re-generated and re-exported soon in today.

T2s will receive the same share rate as requested for FDR (except that DPD are not produced).

=== 12nd June ===
;01h20
* Function Test datasets(named beginning with "ccrc08_run2.*") to T2s.
For this moment, all T2 sites seems in good shape to accept datasets except RO-07-NIPNE and RO-02-NIPNE that are in downtime.

* Datasets shipped (begins with "ccrc08_run2.*")

IN2P3-CC_DATADISK 387
BEIJING-LCG2_DATADISK 46
GRIF-LAL_DATADISK 141
GRIF-LPNHE_DATADISK 79
GRIF-SACLAY_DATADISK 90
IN2P3-CPPM_DATADISK 22
IN2P3-LAPP_DATADISK 22
IN2P3-LPC_DATADISK 36
IN2P3-LPSC_DATADISK 17
RO-02-NIPNE_DATADISK 20
RO-07-NIPNE_DATADISK 33
TOKYO-LCG2_DATADISK 310

;Troughput
[[Image:LYON.T2.throughput.14400 200806120120.png]]

;Transfer errors
* Note that the errors showed in this snapshot involves all transfers mixed with other transfers(e.g MC/Panda). So it may not reflect the real situation FT transfers.
[[Image:LYON.T2.num file xs error.14400 200806120120.png]]

=== 13rd June ===
=== 14th June ===
* The re-produced FDR2 datasets have been re-exported to T1s.
* Now 3 RAW datasets have been exported to IN2P3-CC_DATATAPE.

=== 15th June ===
* Now 5 RAW datasets have been exported to IN2P3-CC_DATATAPE.
* There's something wrong with the configurations resulted in low transfer rate.

=== 16th June ===
* Configuration problems found in dCahce at LYON resulted in failure of transfers to T2s.

=== 17th June ===

* FDR2 datasets have exported to T2s.

IN2P3-CC_DATADISK Total:218 Completed:218 AOD:30 TAG:30 DPD:150 ESD:8
BEIJING-LCG2_DATADISK Total:89 Completed:89 AOD:15 TAG:15 DPD:59 ESD:0
GRIF-LAL_DATADISK Total:94 Completed:94 AOD:15 TAG:15 DPD:64 ESD:0
GRIF-LPNHE_DATADISK Total:57 Completed:57 AOD:9 TAG:9 DPD:39 ESD:0
GRIF-SACLAY_DATADISK Total:59 Completed:59 AOD:6 TAG:6 DPD:47 ESD:0
IN2P3-CPPM_DATADISK Total:42 Completed:42 AOD:6 TAG:6 DPD:30 ESD:0
IN2P3-LAPP_DATADISK Total:42 Completed:42 AOD:6 TAG:6 DPD:30 ESD:0
IN2P3-LPC_DATADISK Total:69 Completed:69 AOD:11 TAG:11 DPD:47 ESD:0
IN2P3-LPSC_DATADISK Total:15 Completed:15 AOD:3 TAG:3 DPD:9 ESD:0
RO-02-NIPNE_DATADISK Total:27 Completed:10 AOD:2 TAG:2 DPD:6 ESD:0
RO-07-NIPNE_DATADISK Total:27 Completed:27 AOD:5 TAG:5 DPD:17 ESD:0
TOKYO-LCG2_DATADISK Total:210 Completed:210 AOD:30 TAG:30 DPD:150 ESD:0

Atlas:FDR2

2008-06-17T07:43:35Z

Erming: /* 16th June */

= Monitoring pages/graphs =
=== FTS ===
http://cctoolsafs.in2p3.fr/fts/monitoring/prod/ftsmonitor.php?vo=atlas

=== T0-T1 (ALL) ===
; FDR monitoring page - T1s
http://atldq2pro.cern.ch:8000/ft/mon/fdrmon_tier1s.html

http://dashb-atlas-data-tier0.cern.ch/dashboard/request.py/site
*Throughput 
:http://dashb-atlas-data-tier0.cern.ch/dashboard/templates/plots/OVERVIEW.throughput.14400.png
*Errors 
:http://dashb-atlas-data-tier0.cern.ch/dashboard/templates/plots/OVERVIEW.num_file_xs_error.14400.png

=== T0-T1 (Lyon) ===
http://dashb-atlas-data-tier0.cern.ch/dashboard/request.py/site?statsInterval=4&name=LYON
*Throughput T0-LYON 
:http://dashb-atlas-data-tier0.cern.ch/dashboard/templates/plots/LYON.throughput.14400.png
*Errors T0-LYON 
:http://dashb-atlas-data-tier0.cern.ch/dashboard/templates/plots/LYON.num_file_xs_error.14400.png
*Throughput Others-LYON 
:http://dashb-atlas-data.cern.ch/dashboard/templates/plots/LYON.T1.throughput.14400.png
*Errors Others-LYON :http://dashb-atlas-data.cern.ch/dashboard/templates/plots/LYON.T1.num_file_xs_error.14400.png

=== T1-T1 ===

=== T1-T2 (Lyon) ===
; FDR monitoring page
http://atldq2pro.cern.ch:8000/ft/mon/fdrmon_TiersInfo.html

http://atldq2pro.cern.ch:8000/ft/mon/fdrmon_T1-T2_matrix_day.html

; Dashboard
http://dashb-atlas-data.cern.ch/dashboard/request.py/site?statsInterval=4&name=LYON
*Throughput 
:http://dashb-atlas-data.cern.ch/dashboard/templates/plots/LYON.T2.throughput.14400.png
*Errors
:http://dashb-atlas-data.cern.ch/dashboard/templates/plots/LYON.T2.num_file_xs_error.14400.png

=== Network Graphs ===
; IN2P3 Weathermap
* http://netstat.in2p3.fr/weathermap/
* http://netstat.in2p3.fr/weathermap/graphiques/

; cc-in2p3 : [http://netstat.in2p3.fr/weathermap/graphiques/lyo-cern.html lyo-cern] (lhcopn-in2p3.cern.ch) http://netstat.in2p3.fr/weathermap/graphiques/lyo-cern-daily.gif
; lyon-nord : [http://netstat.in2p3.fr/weathermap/graphiques/lyo-nrd.html lyo-nrd] http://netstat.in2p3.fr/weathermap/graphiques/lyo-nrd-daily.gif
; orsay : [http://netstat.in2p3.fr/weathermap/graphiques/orsay.html orsay] (in2p3-orsay.cssi.renater.fr http://netstat.in2p3.fr/weathermap/graphiques/orsay-daily.gif
; lal-cc : [http://netstat.in2p3.fr/weathermap/graphiques/lal-cc.html Liaison LAL - CC] http://netstat.in2p3.fr/weathermap/graphiques/lal-cc-daily.gif
; lal : [http://netstat.in2p3.fr/weathermap/graphiques/lal.html lal] http://netstat.in2p3.fr/weathermap/graphiques/lal-daily.gif
; lpnhe: [http://netstat.in2p3.fr/weathermap/graphiques/lpnhe-nrd.html lpnhe-nrd] (in2p3-jussieu.cssi.renater.fr) http://netstat.in2p3.fr/weathermap/graphiques/lpnhe-nrd-daily.gif [http://netstat.in2p3.fr/weathermap/graphiques/lpnhe.html lpnhe] (Paris-LPNHE.in2p3.fr) http://netstat.in2p3.fr/weathermap/graphiques/lpnhe-daily.gif
; lapp : [http://netstat.in2p3.fr/weathermap/graphiques/ann-nrd.html ann-nrd] http://netstat.in2p3.fr/weathermap/graphiques/ann-nrd-daily.gif
; lpc: [http://netstat.in2p3.fr/weathermap/graphiques/lpc-cf.html lpc-cf]
http://netstat.in2p3.fr/weathermap/graphiques/lpc-cf-daily.gif
; lpsc : [http://netstat.in2p3.fr/weathermap/graphiques/lpsc.html lpsc] http://netstat.in2p3.fr/weathermap/graphiques/lpsc-daily.gif
; cppm : [http://netstat.in2p3.fr/weathermap/graphiques/cppm.html cppm] 
http://netstat.in2p3.fr/weathermap/graphiques/cppm-daily.gif
; international (tokyo etc.) : [http://netstat.in2p3.fr/weathermap/graphiques/parisnrd.html parisnrd] 
http://netstat.in2p3.fr/weathermap/graphiques/parisnrd-daily.gif
; GEANT - NYC : [http://dc-snmp.wcc.grnoc.iu.edu/manlan/show-graph.cgi?title=sw.newy32aoa.manlan.internet2.edu--te10/1&rrdname=sw.newy32aoa.manlan.internet2.edu--te10_1.rrd MANLAN] 
http://dc-snmp.wcc.grnoc.iu.edu/manlan/img/sw.newy32aoa.manlan.internet2.edu--te10_1-std5.gif
; NYC - TOKYO : [http://dc-snmp.wcc.grnoc.iu.edu/manlan/show-graph.cgi?title=sw.newy32aoa.manlan.internet2.edu--te11/1&rrdname=sw.newy32aoa.manlan.internet2.edu--te11_1.rrd MANLAN] 
http://dc-snmp.wcc.grnoc.iu.edu/manlan/img/sw.newy32aoa.manlan.internet2.edu--te11_1-std5.gif
; tokyo : [http://www.icepp.jp/monitor/gridftp/monitor.html gridftp monitor] http://www.icepp.jp/monitor/gridftp/img/all.png

= T0-T1 =

= T1-T2 =
==== General informations ====
;FDR page:
https://twiki.cern.ch/twiki/bin/view/Atlas/FullDressRehearsal

;DDM_in_FRD2: https://twiki.cern.ch/twiki/bin/view/Atlas/DDMOperationsGroup#DDM_in_FRD2

;Replication status(checked ~ every 2h?)
http://pandamon.usatlas.bnl.gov:25880/server/pandamon/query?mode=listFDR

;Lyon FTS monitor: http://cctoolsafs.in2p3.fr/fts/monitoring/prod/ftsmonitor.php?vo=atlas

;Contacts:

* IN2P3-CPPM_DATADISK :
* IN2P3-LPSC_DATADISK :
* IN2P3-LPC_DATADISK :
* GRIF-LAL_DATADISK : (grid.admin a lal.in2p3.fr)
* GRIF-LPNHE_DATADISK :
* GRIF-SACLAY_DATADISK :
* BEIJING-LCG2_DATADISK : (yanxf a ihep.ac.cn, Erming.Pei a cern.ch)
* RO-07-NIPNE_DATADISK : (ciubancan a nipne.ro)
* RO-02-NIPNE_DATADISK : (tpreda a nipne.ro)
* TOKYO-LCG2_DATADISK : (lcg-admin a icepp.s.u-tokyo.ac.jp)

===== Shares =====
Only srmv2+ space tokens sites will collect data. The list of streams is : Egamma, Bphysics, MinBias, Muon and Jet. If no precise stream name is associated to the share, the site will get any stream. The site in parenthesis will collect the complementary datasets. The T2 ressources at T1 means that users without production role can access CPUs at the T1 without the Production role.
* US :
o AGLT2_DATADISK : 100 %
o MWT2_DATADISK: 100 %
o NET2_DATADISK: 100 %
o SLACXRD_DATADISK: 100 %
o SWT2_CPB_DATADISK: 100 %
o T2 CPU ressources in BNL to access AOD
* CA :
o SFU-LCG2_DATADISK : 40%
o VICTORIA-LCG2_DATADISK : 30%
o ALBERTA-LCG2_DATADISK : 30%
o TORONTO-LCG2_DATADISK : 50 %
* IT :
o INFN-MILANO_DATADISK : 25%
o INFN-FRASCATI_DATADISK : 25%
o INFN-ROMA1_DATADISK : 100%
o INFN-NAPOLI-ATLAS_DATADISK : 100 %
* TW :
o TW-FTT_DATADISK : 50 %
* NL
o JINR-LCG2_DATADISK : 50% Jet (RU-PROTVINO-IHEP_DATADISK)
o JINR-LCG2_DATADISK : 50% MinBias (RU-PROTVINO-IHEP_DATADISK)
o JINR-LCG2_DATADISK : 50% Muon (RU-PROTVINO-IHEP_DATADISK)
o JINR-LCG2_DATADISK : 50% Bphysics (RU-PNPI_DATADISK)
o RU-PROTVINO-IHEP_DATADISK : 50% Jet (JINR-LCG2_DATADISK)
o RU-PROTVINO-IHEP_DATADISK : 50% Muon (JINR-LCG2_DATADISK)
o RU-PROTVINO-IHEP_DATADISK : 50% MinBias (JINR-LCG2_DATADISK)
o RU-PROTVINO-IHEP_DATADISK : 50% Egamma (RU-PNPI_DATADISK)
o RU-PNPI_DATADISK : 50% Egamma (RU-PROTVINO-IHEP_DATADISK)
o RU-PNPI_DATADISK : 50% Bphysics (JINR-LCG2_DATADISK)
o RU-PNPI_DATADISK : 50% Jet (RRC-KI_DATADISK)
o RU-PNPI_DATADISK : 50% MinBias (RRC-KI_DATADISK)
o RRC-KI_DATADISK : 50% Jet (RU-PNPI_DATADISK)
o RRC-KI_DATADISK : 50% MinBias (RU-PNPI_DATADISK)
o RRC-KI_DATADISK : 50% Egamma
o RRC-KI_DATADISK : 50% Bphysics
o CSTCDIE_DATADISK : 20%
* FR :
o IN2P3-LAPP_DATADISK : 100 % Egamma
o IN2P3-CPPM_DATADISK : 100 % Egamma
o IN2P3-LPSC_DATADISK : 5 %
o IN2P3-LPC_DATADISK : 100% Muon, 50% Egamma
o BEIJING-LCG2_DATADISK : 100 % Egamma, 50% Muon, 35% MinBias
o RO-07-NIPNE_DATADISK : 10%
o RO-02-NIPNE_DATADISK : 10%
o GRIF-LAL_DATADISK : 45%
o GRIF-LPNHE_DATADISK : 25%
o GRIF-SACLAY_DATADISK : 30 %
o TOKYO-LCG2_DATADISK : 100%
o T2 CPU ressources in LYON to access AOD
* DE :
o DESY-HH_DATADISK : 100%
o CSCS-LCG2_DATADISK: 100%
o DESY-ZN_DATADISK : 50%
o GOEGRID_DATADISK : 50%
o WUPPERTALPROD_DATADISK: 25%
o CYFRONET-LCG2_DATADISK : 15%
o PRAGUELCG2_DATADISK : 25%
o LRZ-LMU_DATADISK : 35%
o UNI-FREIBURG_DATADISK : 100% Jet, 100% Muon
* UK :
o UKI-SOUTHGRID-CAM-HEP_DATADISK : 50 % Muon (UKI-NORTHGRID-LANCS-HEP_DATADISK
o UKI-NORTHGRID-LANCS-HEP_DATADISK : 50 % Muon (UKI-SOUTHGRID-CAM-HEP_DATADISK)
o UKI-NORTHGRID-SHEF-HEP_DATADISK : 50 % MinBias (UKI-SCOTGRID-GLASGOW_DATADISK)
o UKI-SCOTGRID-GLASGOW_DATADISK : 50 % MinBias (UKI-NORTHGRID-SHEF-HEP_DATADISK)
o UKI-LT2-RHUL_DATADISK : 50 % Egamma (UKI-NORTHGRID-LIV-HEP_DATADISK)
o UKI-NORTHGRID-LIV-HEP_DATADISK : 50 % Egamma (UKI-LT2-RHUL_DATADISK)
o UKI-SOUTHGRID-RALPP_DATADISK : 50 Egamma (UKI-LT2-RHUL_DATADISK)
o UKI-SOUTHGRID-BHAM-HEP_DATADISK : 50 Jet (UKI-SOUTHGRID-OX-HEP_DATADISK)
* ES :
o IFIC-LCG2_DATADISK: 50%
o UAM-LCG2_DATADISK: 25%
o IFAE_DATADISK: 25%
* NDGF :
o SIGNET_DATADISK : 50 %

== Logbook ==

=== 2nd June ===

* Since Lyon is in donwtime, so there's no transfers from LYON to T2s today.
http://cc.in2p3.fr/cc_accueil.php3?lang=fr

*Free Space

Token Total Guaranteed Unused
------------------------------------------
BEIJING-LCG2_DATADISK 6.366 6.366 6.343
GRIF-LAL_DATADISK 6.0 6.0 5.732
GRIF-LPNHE_DATADISK 3.906 3.906 3.89
GRIF-SACLAY_DATADISK 6.0 6.0 5.994
IN2P3-CPPM_DATADISK 1.0 1.0 0.977
IN2P3-LAPP_DATADISK 4.0 4.0 4.0
IN2P3-LPC_DATADISK 3.0 3.0 3.0
IN2P3-LPSC_DATADISK 0.488 0.488 0.482
RO-02-NIPNE_DATADISK 3.0 3.0 3.0
RO-07-NIPNE_DATADISK 2.0 2.0 2.0
TOKYO-LCG2_DATADISK 15.0 15.0 13.643

=== 3rd June ===

* Lyon is still in donwtime, no transfers from LYON to T2s in this morning.

; Throughput(snapshot) (MB/s) : [[Image:LYON.throughput.14400_20080603.png]]
; Number of transfer errors(snapshot) : [[Image:LYON.num_file_xs_error.14400_20080603.png]]

=== 4th June ===
;22h02
* Still no transfers, desipte that all the services in LYON shows OK.

;Throughput(snapshot)
[[Image:LYON.throughput.3600_20080604.png]]

;Services in LYON
[[Image:services_cc-in2p3_2008060422h15.PNG]]

;Free space

BEIJING-LCG2_DATADISK 6.366 6.366 6.343
BEIJING-LCG2_MCDISK 5.457 5.457 5.457
GRIF-LAL_DATADISK 6.0 6.0 5.732
GRIF-LAL_MCDISK 7.0 7.0 3.724
GRIF-LPNHE_DATADISK 3.906 3.906 3.89
GRIF-LPNHE_MCDISK 1.0 1.0 0.194
GRIF-SACLAY_DATADISK 6.0 6.0 5.994
GRIF-SACLAY_MCDISK 2.0 2.0 1.713
IN2P3-CPPM_DATADISK 1.0 1.0 0.977
IN2P3-CPPM_MCDISK 0.781 0.781 0.765
IN2P3-LAPP_DATADISK 4.0 4.0 4.0
IN2P3-LAPP_MCDISK 2.0 2.0 0.745
IN2P3-LPC_DATADISK 3.0 3.0 3.0
IN2P3-LPC_MCDISK 2.0 2.0 1.936
IN2P3-LPSC_DATADISK 0.488 0.488 0.482
IN2P3-LPSC_MCDISK 2.0 2.0 1.518
RO-02-NIPNE_DATADISK 3.0 3.0 3.0
RO-02-NIPNE_MCDISK 1.0 1.0 0.995
RO-07-NIPNE_DATADISK 2.0 2.0 2.0
RO-07-NIPNE_MCDISK 1.0 1.0 0.654
TOKYO-LCG2_DATADISK 15.0 15.0 13.643
TOKYO-LCG2_MCDISK 10.0 10.0 10.0

=== 5th June ===

* Data produced on Tuesday are supposed to be shipped in this night.

;14h50
* Now the RAW data has begun to shipped to LYON.

;Throughput(snapshot)(MB/s)
[[Image:LYON.throughput.14400_RAW_20080605_1816.png]]

;Number of transfer errors (snapshot)
[[Image:LYON.num_file_xs_error.14400_20080605_1823.png]]

;21h30
* IN2P3-CC_MCDISK has some problem of staging file. But it should have nothing to do with T0 export.

* 47 datasets have been exported to T1s. The total is 90.

;Throughput(snapshot)(MB/s)
[[Image:LYON.throughput.14400 20080605 2140.png]]

;Number of transfer errors (snapshot) -- It's the problem of staging file in T1 and nothing to do with FDR transfers.
[[Image:LYON.num_file_xs_error.14400_20080605_2140.png]]

=== 6th June ===
;09h50

;Throughput(snapshot) (MB/s)
[[Image:LYON.throughput.14400 200806060 0958.png]]

;Number of transfer errors (snapshot)
[[Image:LYON.num file xs error.14400 20080606 0959.png]]

;10h03
* Now all the RAW datasets have been exported to T1. The total number is 90.
http://atlas.web.cern.ch/Atlas/tzero/prod2/monitoring/tables/ddm_RAW_status.html

=== 7th June ===
* at CERN, will re-start the bulk processing with using the upcoming AtlasPoint1-14.1.0.12 release cache.

;11h30
* SRM in LYON is still in bad shape. Experts are working on it.

=== 8th June ===
;14h20
* FDR2 AODs/DPDs/TAGs/ESDs are being exported to LYON (IN2P3-CC_DATADISK), but haven't been shipped to T2s.

;Number of datasets transfered by far
$ dq2-list-dataset-site IN2P3-CC_DATADISK|grep fdr08_run2.*|wc -l
68

;Throughput(T0-T1,snapshot)(MB/s)
[[Image:LYON.throughput.14400 200806081420.png]]

;Number of transfer errors
[[Image:LYON.num file xs error.14400 200806081420.png]]

;Throughput(T1-T2,snapshot)(MB/s)
[[Image:LYON.T2.throughput.14400 T1-T2 200806081420.png]]

=== 9th June ===
;0h20
* FDR datasets has begun to be shipped to T2s.
* Some T2s (e.g. BEIJING) subscribed dedicated datasets, which will be done later in today's morning.

;Shipped datasets

IN2P3-CC_DATADISK 315
BEIJING-LCG2_DATADISK 0
GRIF-LAL_DATADISK 79
GRIF-LPNHE_DATADISK 39
GRIF-SACLAY_DATADISK 38
IN2P3-CPPM_DATADISK 0
IN2P3-LAPP_DATADISK 12
IN2P3-LPC_DATADISK 12
IN2P3-LPSC_DATADISK 32
RO-02-NIPNE_DATADISK 29
RO-07-NIPNE_DATADISK 46
TOKYO-LCG2_DATADISK 134

;Free space

BEIJING-LCG2_DATADISK 6.366 6.366 6.343
GRIF-LAL_DATADISK 6.0 6.0 5.579
GRIF-LPNHE_DATADISK 3.906 3.906 3.817
GRIF-SACLAY_DATADISK 6.0 6.0 5.927
IN2P3-CPPM_DATADISK 1.0 1.0 0.977
IN2P3-LAPP_DATADISK 4.0 4.0 4.0
IN2P3-LPC_DATADISK 0 0 0
IN2P3-LPSC_DATADISK 0.488 0.488 0.439
RO-02-NIPNE_DATADISK 3.0 3.0 2.986
RO-07-NIPNE_DATADISK 2.0 2.0 1.896
TOKYO-LCG2_DATADISK 15.0 15.0 13.313

;Troughputs(snapshot)(MB/s)
[[Image:LYON.throughput.14400 200806090046.png]]

;Number of transfer erros(snapshot)
* Note: No errors found in FDR2 transfers. Errors showed in this snapshot are since other production data transfers.
[[Image:LYON.num file xs error.14400 200806090047.png]]

;09h24

;Troughputs(snapshot)(MB/s)
[[Image:LYON.throughput.14400 200806090922.png]]

;Number of transfer erros(snapshot)
* Note: No errors found in FDR2 transfers. Errors showed in this snapshot are since other production data transfers.
[[Image:LYON.num file xs error.14400 20806090922.png]]

;Shipped datasets

IN2P3-CC_DATADISK 319
BEIJING-LCG2_DATADISK 0
GRIF-LAL_DATADISK 79
GRIF-LPNHE_DATADISK 39
GRIF-SACLAY_DATADISK 38
IN2P3-CPPM_DATADISK 0
IN2P3-LAPP_DATADISK 12
IN2P3-LPC_DATADISK 12
IN2P3-LPSC_DATADISK 32
RO-02-NIPNE_DATADISK 59
RO-07-NIPNE_DATADISK 49
TOKYO-LCG2_DATADISK 134

;Free space
BEIJING-LCG2_DATADISK 6.366 6.366 6.343
GRIF-LAL_DATADISK 6.0 6.0 5.579
GRIF-LPNHE_DATADISK 3.906 3.906 3.817
GRIF-SACLAY_DATADISK 6.0 6.0 5.927
IN2P3-CPPM_DATADISK 1.0 1.0 0.977
IN2P3-LAPP_DATADISK 4.0 4.0 4.0
IN2P3-LPC_DATADISK 3.0 3.0 3.0
IN2P3-LPSC_DATADISK 0.488 0.488 0.439
RO-02-NIPNE_DATADISK 3.0 3.0 2.945
RO-07-NIPNE_DATADISK 2.0 2.0 1.885
TOKYO-LCG2_DATADISK 15.0 15.0 13.313

;14h00
Many failed with RO-02-NIPNE_DATADISK.

[FTS] FTS State [Failed] FTS Retries [1] Reason [TRANSFER error during TRANSFER phase: [PERMISSION] the server sent an error response: 550 550 rfio write failure: Permission denied.] Source Host [ccsrm.in2p3.fr] 16
[FTS] FTS State [Failed] FTS Retries [1] Reason [TRANSFER error during TRANSFER phase: [TRANSFER_TIMEOUT] gridftp_copy_wait: Connection timed out] Source Host [ccsrm.in2p3.fr] 9
[FTS] FTS State [Failed] FTS Retries [1] Reason [TRANSFER error during TRANSFER phase: [GRIDFTP] the server sent an error response: 426 426 Transfer aborted (Unexpected Exception : java.io.IOException: Broken pipe)] Source Host [ccsrm.in2p3.fr] 2
[FTS] FTS State [Failed] FTS Retries [1] Reason [The process serving the transfer (status = PREPARING) is no longer active (could not open file /proc/3775/cmdline)] Source Host [ccsrm.in2p3.fr] 1
[FTS] FTS State [Failed] FTS Retries [1] Reason [The process serving the transfer (status = PREPARING) is no longer active (could not open file /proc/30076/cmdline)] Source Host [ccsrm.in2p3.fr] 1

;20h10
* FDR datasets have finished transferred to all T2s except BEIJING-LCG2_DATADISK and RO-02-NIPNE_DATADISK.
* Many "Connection timed out" errors found at BEIJING-LCG2_DATADISK and RO-02-NIPNE_DATADISK.

[FTS] FTS State [Failed] FTS Retries [1] Reason [SOURCE error during PREPARATION phase: [CONNECTION] failed to contact on remote SRM [httpg://ccsrm.in2p3.fr:8443/srm/managerv2]. Givin' up after 3 tries] Source Host [ccsrm.in2p3.fr] 111
[FTS] FTS State [Failed] FTS Retries [1] Reason [TRANSFER error during TRANSFER phase: [TRANSFER_TIMEOUT] gridftp_copy_wait: Connection timed out] Source Host [ccsrm.in2p3.fr] 95
[FTS] FTS State [Failed] FTS Retries [1] Reason [SOURCE error during PREPARATION phase: [REQUEST_TIMEOUT] failed to prepare source file in 180 seconds] Source Host [ccsrm.in2p3.fr] 3

; Shipped datasets

IN2P3-CC_DATADISK 134
BEIJING-LCG2_DATADISK 79
GRIF-LAL_DATADISK 67
GRIF-LPNHE_DATADISK 32
GRIF-SACLAY_DATADISK 26
IN2P3-CPPM_DATADISK 52
IN2P3-LAPP_DATADISK 52
IN2P3-LPC_DATADISK 52
IN2P3-LPSC_DATADISK 32
RO-02-NIPNE_DATADISK 49
RO-07-NIPNE_DATADISK 49
TOKYO-LCG2_DATADISK 126

=== 10th June ===

;Summary

All T2s except BEIJING-LCG2_DATADISK and RO-02-NIPNE_DATADISK were in good status and received all the exported datasets when FDR transferring.
The problem of "Connection timed out" with transfers to BEIJING will be hopefully solved by tuned the transfer duration >1800s by Lionel.
RO-02-NIPNE is said being upgraded from SL3 to SL4.

=== 11st June ===

; Anouncement
All FDR2 datasets will be deleted from all sites since part of the detector was not included in the reconstruction. It was not detected by the Data Quality because it is run on the express stream which is correct.

Old datasets will be deleted centrally by Stephane and new datasets that would be named begin with "ccrc08-run.0024*" will be re-generated and re-exported soon in today.

T2s will receive the same share rate as requested for FDR (except that DPD are not produced).

=== 12nd June ===
;01h20
* Function Test datasets(named beginning with "ccrc08_run2.*") to T2s.
For this moment, all T2 sites seems in good shape to accept datasets except RO-07-NIPNE and RO-02-NIPNE that are in downtime.

* Datasets shipped (begins with "ccrc08_run2.*")

IN2P3-CC_DATADISK 387
BEIJING-LCG2_DATADISK 46
GRIF-LAL_DATADISK 141
GRIF-LPNHE_DATADISK 79
GRIF-SACLAY_DATADISK 90
IN2P3-CPPM_DATADISK 22
IN2P3-LAPP_DATADISK 22
IN2P3-LPC_DATADISK 36
IN2P3-LPSC_DATADISK 17
RO-02-NIPNE_DATADISK 20
RO-07-NIPNE_DATADISK 33
TOKYO-LCG2_DATADISK 310

;Troughput
[[Image:LYON.T2.throughput.14400 200806120120.png]]

;Transfer errors
* Note that the errors showed in this snapshot involves all transfers mixed with other transfers(e.g MC/Panda). So it may not reflect the real situation FT transfers.
[[Image:LYON.T2.num file xs error.14400 200806120120.png]]

=== 13rd June ===
=== 14th June ===
* The re-produced FDR2 datasets have been re-exported to T1s.
* Now 3 RAW datasets have been exported to IN2P3-CC_DATATAPE.

=== 15th June ===
* Now 5 RAW datasets have been exported to IN2P3-CC_DATATAPE.
* There's something wrong with the configurations resulted in low transfer rate.

=== 16th June ===
* Configuration problems found in dCahce at LYON resulted in failure of transfers to T2s.

=== 17th June ===

* FDR2 datasets have exported to T2s.

IN2P3-CC_DATADISK TOTAL:218 AOD:30 TAG:30 DPD:150 ESD:8
BEIJING-LCG2_DATADISK TOTAL:89 AOD:15 TAG:15 DPD:59 ESD:0
GRIF-LAL_DATADISK TOTAL:94 AOD:15 TAG:15 DPD:64 ESD:0
GRIF-LPNHE_DATADISK TOTAL:57 AOD:9 TAG:9 DPD:39 ESD:0
GRIF-SACLAY_DATADISK TOTAL:59 AOD:6 TAG:6 DPD:47 ESD:0
IN2P3-CPPM_DATADISK TOTAL:42 AOD:6 TAG:6 DPD:30 ESD:0
IN2P3-LAPP_DATADISK TOTAL:42 AOD:6 TAG:6 DPD:30 ESD:0
IN2P3-LPC_DATADISK TOTAL:69 AOD:11 TAG:11 DPD:47 ESD:0
IN2P3-LPSC_DATADISK TOTAL:15 AOD:3 TAG:3 DPD:9 ESD:0
RO-02-NIPNE_DATADISK TOTAL:10 AOD:2 TAG:2 DPD:6 ESD:0
RO-07-NIPNE_DATADISK TOTAL:27 AOD:5 TAG:5 DPD:17 ESD:0
TOKYO-LCG2_DATADISK TOTAL:210 AOD:30 TAG:30 DPD:150 ESD:0

Atlas:FDR2

2008-06-16T11:56:10Z

Erming: /* 14th June */

Atlas:FDR2

2008-06-16T11:53:16Z

Erming: /* 12nd June */

= Monitoring pages/graphs =
=== FTS ===
http://cctoolsafs.in2p3.fr/fts/monitoring/prod/ftsmonitor.php?vo=atlas

=== T0-T1 (ALL) ===
; FDR monitoring page - T1s
http://atldq2pro.cern.ch:8000/ft/mon/fdrmon_tier1s.html

http://dashb-atlas-data-tier0.cern.ch/dashboard/request.py/site
*Throughput 
:http://dashb-atlas-data-tier0.cern.ch/dashboard/templates/plots/OVERVIEW.throughput.14400.png
*Errors 
:http://dashb-atlas-data-tier0.cern.ch/dashboard/templates/plots/OVERVIEW.num_file_xs_error.14400.png

=== T0-T1 (Lyon) ===
http://dashb-atlas-data-tier0.cern.ch/dashboard/request.py/site?statsInterval=4&name=LYON
*Throughput T0-LYON 
:http://dashb-atlas-data-tier0.cern.ch/dashboard/templates/plots/LYON.throughput.14400.png
*Errors T0-LYON 
:http://dashb-atlas-data-tier0.cern.ch/dashboard/templates/plots/LYON.num_file_xs_error.14400.png
*Throughput Others-LYON 
:http://dashb-atlas-data.cern.ch/dashboard/templates/plots/LYON.T1.throughput.14400.png
*Errors Others-LYON :http://dashb-atlas-data.cern.ch/dashboard/templates/plots/LYON.T1.num_file_xs_error.14400.png

=== T1-T1 ===

=== T1-T2 (Lyon) ===
; FDR monitoring page
http://atldq2pro.cern.ch:8000/ft/mon/fdrmon_TiersInfo.html

http://atldq2pro.cern.ch:8000/ft/mon/fdrmon_T1-T2_matrix_day.html

; Dashboard
http://dashb-atlas-data.cern.ch/dashboard/request.py/site?statsInterval=4&name=LYON
*Throughput 
:http://dashb-atlas-data.cern.ch/dashboard/templates/plots/LYON.T2.throughput.14400.png
*Errors
:http://dashb-atlas-data.cern.ch/dashboard/templates/plots/LYON.T2.num_file_xs_error.14400.png

=== Network Graphs ===
; IN2P3 Weathermap
* http://netstat.in2p3.fr/weathermap/
* http://netstat.in2p3.fr/weathermap/graphiques/

; cc-in2p3 : [http://netstat.in2p3.fr/weathermap/graphiques/lyo-cern.html lyo-cern] (lhcopn-in2p3.cern.ch) http://netstat.in2p3.fr/weathermap/graphiques/lyo-cern-daily.gif
; lyon-nord : [http://netstat.in2p3.fr/weathermap/graphiques/lyo-nrd.html lyo-nrd] http://netstat.in2p3.fr/weathermap/graphiques/lyo-nrd-daily.gif
; orsay : [http://netstat.in2p3.fr/weathermap/graphiques/orsay.html orsay] (in2p3-orsay.cssi.renater.fr http://netstat.in2p3.fr/weathermap/graphiques/orsay-daily.gif
; lal-cc : [http://netstat.in2p3.fr/weathermap/graphiques/lal-cc.html Liaison LAL - CC] http://netstat.in2p3.fr/weathermap/graphiques/lal-cc-daily.gif
; lal : [http://netstat.in2p3.fr/weathermap/graphiques/lal.html lal] http://netstat.in2p3.fr/weathermap/graphiques/lal-daily.gif
; lpnhe: [http://netstat.in2p3.fr/weathermap/graphiques/lpnhe-nrd.html lpnhe-nrd] (in2p3-jussieu.cssi.renater.fr) http://netstat.in2p3.fr/weathermap/graphiques/lpnhe-nrd-daily.gif [http://netstat.in2p3.fr/weathermap/graphiques/lpnhe.html lpnhe] (Paris-LPNHE.in2p3.fr) http://netstat.in2p3.fr/weathermap/graphiques/lpnhe-daily.gif
; lapp : [http://netstat.in2p3.fr/weathermap/graphiques/ann-nrd.html ann-nrd] http://netstat.in2p3.fr/weathermap/graphiques/ann-nrd-daily.gif
; lpc: [http://netstat.in2p3.fr/weathermap/graphiques/lpc-cf.html lpc-cf]
http://netstat.in2p3.fr/weathermap/graphiques/lpc-cf-daily.gif
; lpsc : [http://netstat.in2p3.fr/weathermap/graphiques/lpsc.html lpsc] http://netstat.in2p3.fr/weathermap/graphiques/lpsc-daily.gif
; cppm : [http://netstat.in2p3.fr/weathermap/graphiques/cppm.html cppm] 
http://netstat.in2p3.fr/weathermap/graphiques/cppm-daily.gif
; international (tokyo etc.) : [http://netstat.in2p3.fr/weathermap/graphiques/parisnrd.html parisnrd] 
http://netstat.in2p3.fr/weathermap/graphiques/parisnrd-daily.gif
; GEANT - NYC : [http://dc-snmp.wcc.grnoc.iu.edu/manlan/show-graph.cgi?title=sw.newy32aoa.manlan.internet2.edu--te10/1&rrdname=sw.newy32aoa.manlan.internet2.edu--te10_1.rrd MANLAN] 
http://dc-snmp.wcc.grnoc.iu.edu/manlan/img/sw.newy32aoa.manlan.internet2.edu--te10_1-std5.gif
; NYC - TOKYO : [http://dc-snmp.wcc.grnoc.iu.edu/manlan/show-graph.cgi?title=sw.newy32aoa.manlan.internet2.edu--te11/1&rrdname=sw.newy32aoa.manlan.internet2.edu--te11_1.rrd MANLAN] 
http://dc-snmp.wcc.grnoc.iu.edu/manlan/img/sw.newy32aoa.manlan.internet2.edu--te11_1-std5.gif
; tokyo : [http://www.icepp.jp/monitor/gridftp/monitor.html gridftp monitor] http://www.icepp.jp/monitor/gridftp/img/all.png

= T0-T1 =

= T1-T2 =
==== General informations ====
;FDR page:
https://twiki.cern.ch/twiki/bin/view/Atlas/FullDressRehearsal

;DDM_in_FRD2: https://twiki.cern.ch/twiki/bin/view/Atlas/DDMOperationsGroup#DDM_in_FRD2

;Replication status(checked ~ every 2h?)
http://pandamon.usatlas.bnl.gov:25880/server/pandamon/query?mode=listFDR

;Lyon FTS monitor: http://cctoolsafs.in2p3.fr/fts/monitoring/prod/ftsmonitor.php?vo=atlas

;Contacts:

* IN2P3-CPPM_DATADISK :
* IN2P3-LPSC_DATADISK :
* IN2P3-LPC_DATADISK :
* GRIF-LAL_DATADISK : (grid.admin a lal.in2p3.fr)
* GRIF-LPNHE_DATADISK :
* GRIF-SACLAY_DATADISK :
* BEIJING-LCG2_DATADISK : (yanxf a ihep.ac.cn, Erming.Pei a cern.ch)
* RO-07-NIPNE_DATADISK : (ciubancan a nipne.ro)
* RO-02-NIPNE_DATADISK : (tpreda a nipne.ro)
* TOKYO-LCG2_DATADISK : (lcg-admin a icepp.s.u-tokyo.ac.jp)

===== Shares =====
Only srmv2+ space tokens sites will collect data. The list of streams is : Egamma, Bphysics, MinBias, Muon and Jet. If no precise stream name is associated to the share, the site will get any stream. The site in parenthesis will collect the complementary datasets. The T2 ressources at T1 means that users without production role can access CPUs at the T1 without the Production role.
* US :
o AGLT2_DATADISK : 100 %
o MWT2_DATADISK: 100 %
o NET2_DATADISK: 100 %
o SLACXRD_DATADISK: 100 %
o SWT2_CPB_DATADISK: 100 %
o T2 CPU ressources in BNL to access AOD
* CA :
o SFU-LCG2_DATADISK : 40%
o VICTORIA-LCG2_DATADISK : 30%
o ALBERTA-LCG2_DATADISK : 30%
o TORONTO-LCG2_DATADISK : 50 %
* IT :
o INFN-MILANO_DATADISK : 25%
o INFN-FRASCATI_DATADISK : 25%
o INFN-ROMA1_DATADISK : 100%
o INFN-NAPOLI-ATLAS_DATADISK : 100 %
* TW :
o TW-FTT_DATADISK : 50 %
* NL
o JINR-LCG2_DATADISK : 50% Jet (RU-PROTVINO-IHEP_DATADISK)
o JINR-LCG2_DATADISK : 50% MinBias (RU-PROTVINO-IHEP_DATADISK)
o JINR-LCG2_DATADISK : 50% Muon (RU-PROTVINO-IHEP_DATADISK)
o JINR-LCG2_DATADISK : 50% Bphysics (RU-PNPI_DATADISK)
o RU-PROTVINO-IHEP_DATADISK : 50% Jet (JINR-LCG2_DATADISK)
o RU-PROTVINO-IHEP_DATADISK : 50% Muon (JINR-LCG2_DATADISK)
o RU-PROTVINO-IHEP_DATADISK : 50% MinBias (JINR-LCG2_DATADISK)
o RU-PROTVINO-IHEP_DATADISK : 50% Egamma (RU-PNPI_DATADISK)
o RU-PNPI_DATADISK : 50% Egamma (RU-PROTVINO-IHEP_DATADISK)
o RU-PNPI_DATADISK : 50% Bphysics (JINR-LCG2_DATADISK)
o RU-PNPI_DATADISK : 50% Jet (RRC-KI_DATADISK)
o RU-PNPI_DATADISK : 50% MinBias (RRC-KI_DATADISK)
o RRC-KI_DATADISK : 50% Jet (RU-PNPI_DATADISK)
o RRC-KI_DATADISK : 50% MinBias (RU-PNPI_DATADISK)
o RRC-KI_DATADISK : 50% Egamma
o RRC-KI_DATADISK : 50% Bphysics
o CSTCDIE_DATADISK : 20%
* FR :
o IN2P3-LAPP_DATADISK : 100 % Egamma
o IN2P3-CPPM_DATADISK : 100 % Egamma
o IN2P3-LPSC_DATADISK : 5 %
o IN2P3-LPC_DATADISK : 100% Muon, 50% Egamma
o BEIJING-LCG2_DATADISK : 100 % Egamma, 50% Muon, 35% MinBias
o RO-07-NIPNE_DATADISK : 10%
o RO-02-NIPNE_DATADISK : 10%
o GRIF-LAL_DATADISK : 45%
o GRIF-LPNHE_DATADISK : 25%
o GRIF-SACLAY_DATADISK : 30 %
o TOKYO-LCG2_DATADISK : 100%
o T2 CPU ressources in LYON to access AOD
* DE :
o DESY-HH_DATADISK : 100%
o CSCS-LCG2_DATADISK: 100%
o DESY-ZN_DATADISK : 50%
o GOEGRID_DATADISK : 50%
o WUPPERTALPROD_DATADISK: 25%
o CYFRONET-LCG2_DATADISK : 15%
o PRAGUELCG2_DATADISK : 25%
o LRZ-LMU_DATADISK : 35%
o UNI-FREIBURG_DATADISK : 100% Jet, 100% Muon
* UK :
o UKI-SOUTHGRID-CAM-HEP_DATADISK : 50 % Muon (UKI-NORTHGRID-LANCS-HEP_DATADISK
o UKI-NORTHGRID-LANCS-HEP_DATADISK : 50 % Muon (UKI-SOUTHGRID-CAM-HEP_DATADISK)
o UKI-NORTHGRID-SHEF-HEP_DATADISK : 50 % MinBias (UKI-SCOTGRID-GLASGOW_DATADISK)
o UKI-SCOTGRID-GLASGOW_DATADISK : 50 % MinBias (UKI-NORTHGRID-SHEF-HEP_DATADISK)
o UKI-LT2-RHUL_DATADISK : 50 % Egamma (UKI-NORTHGRID-LIV-HEP_DATADISK)
o UKI-NORTHGRID-LIV-HEP_DATADISK : 50 % Egamma (UKI-LT2-RHUL_DATADISK)
o UKI-SOUTHGRID-RALPP_DATADISK : 50 Egamma (UKI-LT2-RHUL_DATADISK)
o UKI-SOUTHGRID-BHAM-HEP_DATADISK : 50 Jet (UKI-SOUTHGRID-OX-HEP_DATADISK)
* ES :
o IFIC-LCG2_DATADISK: 50%
o UAM-LCG2_DATADISK: 25%
o IFAE_DATADISK: 25%
* NDGF :
o SIGNET_DATADISK : 50 %

== Logbook ==

=== 2nd June ===

* Since Lyon is in donwtime, so there's no transfers from LYON to T2s today.
http://cc.in2p3.fr/cc_accueil.php3?lang=fr

*Free Space

Token Total Guaranteed Unused
------------------------------------------
BEIJING-LCG2_DATADISK 6.366 6.366 6.343
GRIF-LAL_DATADISK 6.0 6.0 5.732
GRIF-LPNHE_DATADISK 3.906 3.906 3.89
GRIF-SACLAY_DATADISK 6.0 6.0 5.994
IN2P3-CPPM_DATADISK 1.0 1.0 0.977
IN2P3-LAPP_DATADISK 4.0 4.0 4.0
IN2P3-LPC_DATADISK 3.0 3.0 3.0
IN2P3-LPSC_DATADISK 0.488 0.488 0.482
RO-02-NIPNE_DATADISK 3.0 3.0 3.0
RO-07-NIPNE_DATADISK 2.0 2.0 2.0
TOKYO-LCG2_DATADISK 15.0 15.0 13.643

=== 3rd June ===

* Lyon is still in donwtime, no transfers from LYON to T2s in this morning.

; Throughput(snapshot) (MB/s) : [[Image:LYON.throughput.14400_20080603.png]]
; Number of transfer errors(snapshot) : [[Image:LYON.num_file_xs_error.14400_20080603.png]]

=== 4th June ===
;22h02
* Still no transfers, desipte that all the services in LYON shows OK.

;Throughput(snapshot)
[[Image:LYON.throughput.3600_20080604.png]]

;Services in LYON
[[Image:services_cc-in2p3_2008060422h15.PNG]]

;Free space

BEIJING-LCG2_DATADISK 6.366 6.366 6.343
BEIJING-LCG2_MCDISK 5.457 5.457 5.457
GRIF-LAL_DATADISK 6.0 6.0 5.732
GRIF-LAL_MCDISK 7.0 7.0 3.724
GRIF-LPNHE_DATADISK 3.906 3.906 3.89
GRIF-LPNHE_MCDISK 1.0 1.0 0.194
GRIF-SACLAY_DATADISK 6.0 6.0 5.994
GRIF-SACLAY_MCDISK 2.0 2.0 1.713
IN2P3-CPPM_DATADISK 1.0 1.0 0.977
IN2P3-CPPM_MCDISK 0.781 0.781 0.765
IN2P3-LAPP_DATADISK 4.0 4.0 4.0
IN2P3-LAPP_MCDISK 2.0 2.0 0.745
IN2P3-LPC_DATADISK 3.0 3.0 3.0
IN2P3-LPC_MCDISK 2.0 2.0 1.936
IN2P3-LPSC_DATADISK 0.488 0.488 0.482
IN2P3-LPSC_MCDISK 2.0 2.0 1.518
RO-02-NIPNE_DATADISK 3.0 3.0 3.0
RO-02-NIPNE_MCDISK 1.0 1.0 0.995
RO-07-NIPNE_DATADISK 2.0 2.0 2.0
RO-07-NIPNE_MCDISK 1.0 1.0 0.654
TOKYO-LCG2_DATADISK 15.0 15.0 13.643
TOKYO-LCG2_MCDISK 10.0 10.0 10.0

=== 5th June ===

* Data produced on Tuesday are supposed to be shipped in this night.

;14h50
* Now the RAW data has begun to shipped to LYON.

;Throughput(snapshot)(MB/s)
[[Image:LYON.throughput.14400_RAW_20080605_1816.png]]

;Number of transfer errors (snapshot)
[[Image:LYON.num_file_xs_error.14400_20080605_1823.png]]

;21h30
* IN2P3-CC_MCDISK has some problem of staging file. But it should have nothing to do with T0 export.

* 47 datasets have been exported to T1s. The total is 90.

;Throughput(snapshot)(MB/s)
[[Image:LYON.throughput.14400 20080605 2140.png]]

;Number of transfer errors (snapshot) -- It's the problem of staging file in T1 and nothing to do with FDR transfers.
[[Image:LYON.num_file_xs_error.14400_20080605_2140.png]]

=== 6th June ===
;09h50

;Throughput(snapshot) (MB/s)
[[Image:LYON.throughput.14400 200806060 0958.png]]

;Number of transfer errors (snapshot)
[[Image:LYON.num file xs error.14400 20080606 0959.png]]

;10h03
* Now all the RAW datasets have been exported to T1. The total number is 90.
http://atlas.web.cern.ch/Atlas/tzero/prod2/monitoring/tables/ddm_RAW_status.html

=== 7th June ===
* at CERN, will re-start the bulk processing with using the upcoming AtlasPoint1-14.1.0.12 release cache.

;11h30
* SRM in LYON is still in bad shape. Experts are working on it.

=== 8th June ===
;14h20
* FDR2 AODs/DPDs/TAGs/ESDs are being exported to LYON (IN2P3-CC_DATADISK), but haven't been shipped to T2s.

;Number of datasets transfered by far
$ dq2-list-dataset-site IN2P3-CC_DATADISK|grep fdr08_run2.*|wc -l
68

;Throughput(T0-T1,snapshot)(MB/s)
[[Image:LYON.throughput.14400 200806081420.png]]

;Number of transfer errors
[[Image:LYON.num file xs error.14400 200806081420.png]]

;Throughput(T1-T2,snapshot)(MB/s)
[[Image:LYON.T2.throughput.14400 T1-T2 200806081420.png]]

=== 9th June ===
;0h20
* FDR datasets has begun to be shipped to T2s.
* Some T2s (e.g. BEIJING) subscribed dedicated datasets, which will be done later in today's morning.

;Shipped datasets

IN2P3-CC_DATADISK 315
BEIJING-LCG2_DATADISK 0
GRIF-LAL_DATADISK 79
GRIF-LPNHE_DATADISK 39
GRIF-SACLAY_DATADISK 38
IN2P3-CPPM_DATADISK 0
IN2P3-LAPP_DATADISK 12
IN2P3-LPC_DATADISK 12
IN2P3-LPSC_DATADISK 32
RO-02-NIPNE_DATADISK 29
RO-07-NIPNE_DATADISK 46
TOKYO-LCG2_DATADISK 134

;Free space

BEIJING-LCG2_DATADISK 6.366 6.366 6.343
GRIF-LAL_DATADISK 6.0 6.0 5.579
GRIF-LPNHE_DATADISK 3.906 3.906 3.817
GRIF-SACLAY_DATADISK 6.0 6.0 5.927
IN2P3-CPPM_DATADISK 1.0 1.0 0.977
IN2P3-LAPP_DATADISK 4.0 4.0 4.0
IN2P3-LPC_DATADISK 0 0 0
IN2P3-LPSC_DATADISK 0.488 0.488 0.439
RO-02-NIPNE_DATADISK 3.0 3.0 2.986
RO-07-NIPNE_DATADISK 2.0 2.0 1.896
TOKYO-LCG2_DATADISK 15.0 15.0 13.313

;Troughputs(snapshot)(MB/s)
[[Image:LYON.throughput.14400 200806090046.png]]

;Number of transfer erros(snapshot)
* Note: No errors found in FDR2 transfers. Errors showed in this snapshot are since other production data transfers.
[[Image:LYON.num file xs error.14400 200806090047.png]]

;09h24

;Troughputs(snapshot)(MB/s)
[[Image:LYON.throughput.14400 200806090922.png]]

;Number of transfer erros(snapshot)
* Note: No errors found in FDR2 transfers. Errors showed in this snapshot are since other production data transfers.
[[Image:LYON.num file xs error.14400 20806090922.png]]

;Shipped datasets

IN2P3-CC_DATADISK 319
BEIJING-LCG2_DATADISK 0
GRIF-LAL_DATADISK 79
GRIF-LPNHE_DATADISK 39
GRIF-SACLAY_DATADISK 38
IN2P3-CPPM_DATADISK 0
IN2P3-LAPP_DATADISK 12
IN2P3-LPC_DATADISK 12
IN2P3-LPSC_DATADISK 32
RO-02-NIPNE_DATADISK 59
RO-07-NIPNE_DATADISK 49
TOKYO-LCG2_DATADISK 134

;Free space
BEIJING-LCG2_DATADISK 6.366 6.366 6.343
GRIF-LAL_DATADISK 6.0 6.0 5.579
GRIF-LPNHE_DATADISK 3.906 3.906 3.817
GRIF-SACLAY_DATADISK 6.0 6.0 5.927
IN2P3-CPPM_DATADISK 1.0 1.0 0.977
IN2P3-LAPP_DATADISK 4.0 4.0 4.0
IN2P3-LPC_DATADISK 3.0 3.0 3.0
IN2P3-LPSC_DATADISK 0.488 0.488 0.439
RO-02-NIPNE_DATADISK 3.0 3.0 2.945
RO-07-NIPNE_DATADISK 2.0 2.0 1.885
TOKYO-LCG2_DATADISK 15.0 15.0 13.313

;14h00
Many failed with RO-02-NIPNE_DATADISK.

[FTS] FTS State [Failed] FTS Retries [1] Reason [TRANSFER error during TRANSFER phase: [PERMISSION] the server sent an error response: 550 550 rfio write failure: Permission denied.] Source Host [ccsrm.in2p3.fr] 16
[FTS] FTS State [Failed] FTS Retries [1] Reason [TRANSFER error during TRANSFER phase: [TRANSFER_TIMEOUT] gridftp_copy_wait: Connection timed out] Source Host [ccsrm.in2p3.fr] 9
[FTS] FTS State [Failed] FTS Retries [1] Reason [TRANSFER error during TRANSFER phase: [GRIDFTP] the server sent an error response: 426 426 Transfer aborted (Unexpected Exception : java.io.IOException: Broken pipe)] Source Host [ccsrm.in2p3.fr] 2
[FTS] FTS State [Failed] FTS Retries [1] Reason [The process serving the transfer (status = PREPARING) is no longer active (could not open file /proc/3775/cmdline)] Source Host [ccsrm.in2p3.fr] 1
[FTS] FTS State [Failed] FTS Retries [1] Reason [The process serving the transfer (status = PREPARING) is no longer active (could not open file /proc/30076/cmdline)] Source Host [ccsrm.in2p3.fr] 1

;20h10
* FDR datasets have finished transferred to all T2s except BEIJING-LCG2_DATADISK and RO-02-NIPNE_DATADISK.
* Many "Connection timed out" errors found at BEIJING-LCG2_DATADISK and RO-02-NIPNE_DATADISK.

[FTS] FTS State [Failed] FTS Retries [1] Reason [SOURCE error during PREPARATION phase: [CONNECTION] failed to contact on remote SRM [httpg://ccsrm.in2p3.fr:8443/srm/managerv2]. Givin' up after 3 tries] Source Host [ccsrm.in2p3.fr] 111
[FTS] FTS State [Failed] FTS Retries [1] Reason [TRANSFER error during TRANSFER phase: [TRANSFER_TIMEOUT] gridftp_copy_wait: Connection timed out] Source Host [ccsrm.in2p3.fr] 95
[FTS] FTS State [Failed] FTS Retries [1] Reason [SOURCE error during PREPARATION phase: [REQUEST_TIMEOUT] failed to prepare source file in 180 seconds] Source Host [ccsrm.in2p3.fr] 3

; Shipped datasets

IN2P3-CC_DATADISK 134
BEIJING-LCG2_DATADISK 79
GRIF-LAL_DATADISK 67
GRIF-LPNHE_DATADISK 32
GRIF-SACLAY_DATADISK 26
IN2P3-CPPM_DATADISK 52
IN2P3-LAPP_DATADISK 52
IN2P3-LPC_DATADISK 52
IN2P3-LPSC_DATADISK 32
RO-02-NIPNE_DATADISK 49
RO-07-NIPNE_DATADISK 49
TOKYO-LCG2_DATADISK 126

=== 10th June ===

;Summary

All T2s except BEIJING-LCG2_DATADISK and RO-02-NIPNE_DATADISK were in good status and received all the exported datasets when FDR transferring.
The problem of "Connection timed out" with transfers to BEIJING will be hopefully solved by tuned the transfer duration >1800s by Lionel.
RO-02-NIPNE is said being upgraded from SL3 to SL4.

=== 11st June ===

; Anouncement
All FDR2 datasets will be deleted from all sites since part of the detector was not included in the reconstruction. It was not detected by the Data Quality because it is run on the express stream which is correct.

Old datasets will be deleted centrally by Stephane and new datasets that would be named begin with "ccrc08-run.0024*" will be re-generated and re-exported soon in today.

T2s will receive the same share rate as requested for FDR (except that DPD are not produced).

=== 12nd June ===
;01h20
* Function Test datasets(named beginning with "ccrc08_run2.*") to T2s.
For this moment, all T2 sites seems in good shape to accept datasets except RO-07-NIPNE and RO-02-NIPNE that are in downtime.

* Datasets shipped (begins with "ccrc08_run2.*")

IN2P3-CC_DATADISK 387
BEIJING-LCG2_DATADISK 46
GRIF-LAL_DATADISK 141
GRIF-LPNHE_DATADISK 79
GRIF-SACLAY_DATADISK 90
IN2P3-CPPM_DATADISK 22
IN2P3-LAPP_DATADISK 22
IN2P3-LPC_DATADISK 36
IN2P3-LPSC_DATADISK 17
RO-02-NIPNE_DATADISK 20
RO-07-NIPNE_DATADISK 33
TOKYO-LCG2_DATADISK 310

;Troughput
[[Image:LYON.T2.throughput.14400 200806120120.png]]

;Transfer errors
* Note that the errors showed in this snapshot involves all transfers mixed with other transfers(e.g MC/Panda). So it may not reflect the real situation FT transfers.
[[Image:LYON.T2.num file xs error.14400 200806120120.png]]

=== 13rd June ===
=== 14th June ===
* The re-produced FDR2 datasets have been re-exported to T1s.
* Now 5 RAW datasets have been exported to IN2P3-CC_DATATAPE.

Atlas:FDR2

2008-06-13T10:42:47Z

Erming: /* 12nd June */

Atlas:FDR2

2008-06-12T10:34:29Z

Erming: /* 12nd June */

= Monitoring pages/graphs =
=== FTS ===
http://cctoolsafs.in2p3.fr/fts/monitoring/prod/ftsmonitor.php?vo=atlas

=== T0-T1 (ALL) ===
; FDR monitoring page - T1s
http://atldq2pro.cern.ch:8000/ft/mon/fdrmon_tier1s.html

http://dashb-atlas-data-tier0.cern.ch/dashboard/request.py/site
*Throughput 
:http://dashb-atlas-data-tier0.cern.ch/dashboard/templates/plots/OVERVIEW.throughput.14400.png
*Errors 
:http://dashb-atlas-data-tier0.cern.ch/dashboard/templates/plots/OVERVIEW.num_file_xs_error.14400.png

=== T0-T1 (Lyon) ===
http://dashb-atlas-data-tier0.cern.ch/dashboard/request.py/site?statsInterval=4&name=LYON
*Throughput T0-LYON 
:http://dashb-atlas-data-tier0.cern.ch/dashboard/templates/plots/LYON.throughput.14400.png
*Errors T0-LYON 
:http://dashb-atlas-data-tier0.cern.ch/dashboard/templates/plots/LYON.num_file_xs_error.14400.png
*Throughput Others-LYON 
:http://dashb-atlas-data.cern.ch/dashboard/templates/plots/LYON.T1.throughput.14400.png
*Errors Others-LYON :http://dashb-atlas-data.cern.ch/dashboard/templates/plots/LYON.T1.num_file_xs_error.14400.png

=== T1-T1 ===

=== T1-T2 (Lyon) ===
; FDR monitoring page
http://atldq2pro.cern.ch:8000/ft/mon/fdrmon_TiersInfo.html

http://atldq2pro.cern.ch:8000/ft/mon/fdrmon_T1-T2_matrix_day.html

; Dashboard
http://dashb-atlas-data.cern.ch/dashboard/request.py/site?statsInterval=4&name=LYON
*Throughput 
:http://dashb-atlas-data.cern.ch/dashboard/templates/plots/LYON.T2.throughput.14400.png
*Errors
:http://dashb-atlas-data.cern.ch/dashboard/templates/plots/LYON.T2.num_file_xs_error.14400.png

=== Network Graphs ===
; IN2P3 Weathermap
* http://netstat.in2p3.fr/weathermap/
* http://netstat.in2p3.fr/weathermap/graphiques/

; cc-in2p3 : [http://netstat.in2p3.fr/weathermap/graphiques/lyo-cern.html lyo-cern] (lhcopn-in2p3.cern.ch) http://netstat.in2p3.fr/weathermap/graphiques/lyo-cern-daily.gif
; lyon-nord : [http://netstat.in2p3.fr/weathermap/graphiques/lyo-nrd.html lyo-nrd] http://netstat.in2p3.fr/weathermap/graphiques/lyo-nrd-daily.gif
; orsay : [http://netstat.in2p3.fr/weathermap/graphiques/orsay.html orsay] (in2p3-orsay.cssi.renater.fr http://netstat.in2p3.fr/weathermap/graphiques/orsay-daily.gif
; lal-cc : [http://netstat.in2p3.fr/weathermap/graphiques/lal-cc.html Liaison LAL - CC] http://netstat.in2p3.fr/weathermap/graphiques/lal-cc-daily.gif
; lal : [http://netstat.in2p3.fr/weathermap/graphiques/lal.html lal] http://netstat.in2p3.fr/weathermap/graphiques/lal-daily.gif
; lpnhe: [http://netstat.in2p3.fr/weathermap/graphiques/lpnhe-nrd.html lpnhe-nrd] (in2p3-jussieu.cssi.renater.fr) http://netstat.in2p3.fr/weathermap/graphiques/lpnhe-nrd-daily.gif [http://netstat.in2p3.fr/weathermap/graphiques/lpnhe.html lpnhe] (Paris-LPNHE.in2p3.fr) http://netstat.in2p3.fr/weathermap/graphiques/lpnhe-daily.gif
; lapp : [http://netstat.in2p3.fr/weathermap/graphiques/ann-nrd.html ann-nrd] http://netstat.in2p3.fr/weathermap/graphiques/ann-nrd-daily.gif
; lpc: [http://netstat.in2p3.fr/weathermap/graphiques/lpc-cf.html lpc-cf]
http://netstat.in2p3.fr/weathermap/graphiques/lpc-cf-daily.gif
; lpsc : [http://netstat.in2p3.fr/weathermap/graphiques/lpsc.html lpsc] http://netstat.in2p3.fr/weathermap/graphiques/lpsc-daily.gif
; cppm : [http://netstat.in2p3.fr/weathermap/graphiques/cppm.html cppm] 
http://netstat.in2p3.fr/weathermap/graphiques/cppm-daily.gif
; international (tokyo etc.) : [http://netstat.in2p3.fr/weathermap/graphiques/parisnrd.html parisnrd] 
http://netstat.in2p3.fr/weathermap/graphiques/parisnrd-daily.gif
; GEANT - NYC : [http://dc-snmp.wcc.grnoc.iu.edu/manlan/show-graph.cgi?title=sw.newy32aoa.manlan.internet2.edu--te10/1&rrdname=sw.newy32aoa.manlan.internet2.edu--te10_1.rrd MANLAN] 
http://dc-snmp.wcc.grnoc.iu.edu/manlan/img/sw.newy32aoa.manlan.internet2.edu--te10_1-std5.gif
; NYC - TOKYO : [http://dc-snmp.wcc.grnoc.iu.edu/manlan/show-graph.cgi?title=sw.newy32aoa.manlan.internet2.edu--te11/1&rrdname=sw.newy32aoa.manlan.internet2.edu--te11_1.rrd MANLAN] 
http://dc-snmp.wcc.grnoc.iu.edu/manlan/img/sw.newy32aoa.manlan.internet2.edu--te11_1-std5.gif
; tokyo : [http://www.icepp.jp/monitor/gridftp/monitor.html gridftp monitor] http://www.icepp.jp/monitor/gridftp/img/all.png

= T0-T1 =

= T1-T2 =
==== General informations ====
;FDR page:
https://twiki.cern.ch/twiki/bin/view/Atlas/FullDressRehearsal

;DDM_in_FRD2: https://twiki.cern.ch/twiki/bin/view/Atlas/DDMOperationsGroup#DDM_in_FRD2

;Replication status(checked ~ every 2h?)
http://pandamon.usatlas.bnl.gov:25880/server/pandamon/query?mode=listFDR

;Lyon FTS monitor: http://cctoolsafs.in2p3.fr/fts/monitoring/prod/ftsmonitor.php?vo=atlas

;Contacts:

* IN2P3-CPPM_DATADISK :
* IN2P3-LPSC_DATADISK :
* IN2P3-LPC_DATADISK :
* GRIF-LAL_DATADISK : (grid.admin a lal.in2p3.fr)
* GRIF-LPNHE_DATADISK :
* GRIF-SACLAY_DATADISK :
* BEIJING-LCG2_DATADISK : (yanxf a ihep.ac.cn, Erming.Pei a cern.ch)
* RO-07-NIPNE_DATADISK : (ciubancan a nipne.ro)
* RO-02-NIPNE_DATADISK : (tpreda a nipne.ro)
* TOKYO-LCG2_DATADISK : (lcg-admin a icepp.s.u-tokyo.ac.jp)

===== Shares =====
Only srmv2+ space tokens sites will collect data. The list of streams is : Egamma, Bphysics, MinBias, Muon and Jet. If no precise stream name is associated to the share, the site will get any stream. The site in parenthesis will collect the complementary datasets. The T2 ressources at T1 means that users without production role can access CPUs at the T1 without the Production role.
* US :
o AGLT2_DATADISK : 100 %
o MWT2_DATADISK: 100 %
o NET2_DATADISK: 100 %
o SLACXRD_DATADISK: 100 %
o SWT2_CPB_DATADISK: 100 %
o T2 CPU ressources in BNL to access AOD
* CA :
o SFU-LCG2_DATADISK : 40%
o VICTORIA-LCG2_DATADISK : 30%
o ALBERTA-LCG2_DATADISK : 30%
o TORONTO-LCG2_DATADISK : 50 %
* IT :
o INFN-MILANO_DATADISK : 25%
o INFN-FRASCATI_DATADISK : 25%
o INFN-ROMA1_DATADISK : 100%
o INFN-NAPOLI-ATLAS_DATADISK : 100 %
* TW :
o TW-FTT_DATADISK : 50 %
* NL
o JINR-LCG2_DATADISK : 50% Jet (RU-PROTVINO-IHEP_DATADISK)
o JINR-LCG2_DATADISK : 50% MinBias (RU-PROTVINO-IHEP_DATADISK)
o JINR-LCG2_DATADISK : 50% Muon (RU-PROTVINO-IHEP_DATADISK)
o JINR-LCG2_DATADISK : 50% Bphysics (RU-PNPI_DATADISK)
o RU-PROTVINO-IHEP_DATADISK : 50% Jet (JINR-LCG2_DATADISK)
o RU-PROTVINO-IHEP_DATADISK : 50% Muon (JINR-LCG2_DATADISK)
o RU-PROTVINO-IHEP_DATADISK : 50% MinBias (JINR-LCG2_DATADISK)
o RU-PROTVINO-IHEP_DATADISK : 50% Egamma (RU-PNPI_DATADISK)
o RU-PNPI_DATADISK : 50% Egamma (RU-PROTVINO-IHEP_DATADISK)
o RU-PNPI_DATADISK : 50% Bphysics (JINR-LCG2_DATADISK)
o RU-PNPI_DATADISK : 50% Jet (RRC-KI_DATADISK)
o RU-PNPI_DATADISK : 50% MinBias (RRC-KI_DATADISK)
o RRC-KI_DATADISK : 50% Jet (RU-PNPI_DATADISK)
o RRC-KI_DATADISK : 50% MinBias (RU-PNPI_DATADISK)
o RRC-KI_DATADISK : 50% Egamma
o RRC-KI_DATADISK : 50% Bphysics
o CSTCDIE_DATADISK : 20%
* FR :
o IN2P3-LAPP_DATADISK : 100 % Egamma
o IN2P3-CPPM_DATADISK : 100 % Egamma
o IN2P3-LPSC_DATADISK : 5 %
o IN2P3-LPC_DATADISK : 100% Muon, 50% Egamma
o BEIJING-LCG2_DATADISK : 100 % Egamma, 50% Muon, 35% MinBias
o RO-07-NIPNE_DATADISK : 10%
o RO-02-NIPNE_DATADISK : 10%
o GRIF-LAL_DATADISK : 45%
o GRIF-LPNHE_DATADISK : 25%
o GRIF-SACLAY_DATADISK : 30 %
o TOKYO-LCG2_DATADISK : 100%
o T2 CPU ressources in LYON to access AOD
* DE :
o DESY-HH_DATADISK : 100%
o CSCS-LCG2_DATADISK: 100%
o DESY-ZN_DATADISK : 50%
o GOEGRID_DATADISK : 50%
o WUPPERTALPROD_DATADISK: 25%
o CYFRONET-LCG2_DATADISK : 15%
o PRAGUELCG2_DATADISK : 25%
o LRZ-LMU_DATADISK : 35%
o UNI-FREIBURG_DATADISK : 100% Jet, 100% Muon
* UK :
o UKI-SOUTHGRID-CAM-HEP_DATADISK : 50 % Muon (UKI-NORTHGRID-LANCS-HEP_DATADISK
o UKI-NORTHGRID-LANCS-HEP_DATADISK : 50 % Muon (UKI-SOUTHGRID-CAM-HEP_DATADISK)
o UKI-NORTHGRID-SHEF-HEP_DATADISK : 50 % MinBias (UKI-SCOTGRID-GLASGOW_DATADISK)
o UKI-SCOTGRID-GLASGOW_DATADISK : 50 % MinBias (UKI-NORTHGRID-SHEF-HEP_DATADISK)
o UKI-LT2-RHUL_DATADISK : 50 % Egamma (UKI-NORTHGRID-LIV-HEP_DATADISK)
o UKI-NORTHGRID-LIV-HEP_DATADISK : 50 % Egamma (UKI-LT2-RHUL_DATADISK)
o UKI-SOUTHGRID-RALPP_DATADISK : 50 Egamma (UKI-LT2-RHUL_DATADISK)
o UKI-SOUTHGRID-BHAM-HEP_DATADISK : 50 Jet (UKI-SOUTHGRID-OX-HEP_DATADISK)
* ES :
o IFIC-LCG2_DATADISK: 50%
o UAM-LCG2_DATADISK: 25%
o IFAE_DATADISK: 25%
* NDGF :
o SIGNET_DATADISK : 50 %

== Logbook ==

=== 2nd June ===

* Since Lyon is in donwtime, so there's no transfers from LYON to T2s today.
http://cc.in2p3.fr/cc_accueil.php3?lang=fr

*Free Space

Token Total Guaranteed Unused
------------------------------------------
BEIJING-LCG2_DATADISK 6.366 6.366 6.343
GRIF-LAL_DATADISK 6.0 6.0 5.732
GRIF-LPNHE_DATADISK 3.906 3.906 3.89
GRIF-SACLAY_DATADISK 6.0 6.0 5.994
IN2P3-CPPM_DATADISK 1.0 1.0 0.977
IN2P3-LAPP_DATADISK 4.0 4.0 4.0
IN2P3-LPC_DATADISK 3.0 3.0 3.0
IN2P3-LPSC_DATADISK 0.488 0.488 0.482
RO-02-NIPNE_DATADISK 3.0 3.0 3.0
RO-07-NIPNE_DATADISK 2.0 2.0 2.0
TOKYO-LCG2_DATADISK 15.0 15.0 13.643

=== 3rd June ===

* Lyon is still in donwtime, no transfers from LYON to T2s in this morning.

; Throughput(snapshot) (MB/s) : [[Image:LYON.throughput.14400_20080603.png]]
; Number of transfer errors(snapshot) : [[Image:LYON.num_file_xs_error.14400_20080603.png]]

=== 4th June ===
;22h02
* Still no transfers, desipte that all the services in LYON shows OK.

;Throughput(snapshot)
[[Image:LYON.throughput.3600_20080604.png]]

;Services in LYON
[[Image:services_cc-in2p3_2008060422h15.PNG]]

;Free space

BEIJING-LCG2_DATADISK 6.366 6.366 6.343
BEIJING-LCG2_MCDISK 5.457 5.457 5.457
GRIF-LAL_DATADISK 6.0 6.0 5.732
GRIF-LAL_MCDISK 7.0 7.0 3.724
GRIF-LPNHE_DATADISK 3.906 3.906 3.89
GRIF-LPNHE_MCDISK 1.0 1.0 0.194
GRIF-SACLAY_DATADISK 6.0 6.0 5.994
GRIF-SACLAY_MCDISK 2.0 2.0 1.713
IN2P3-CPPM_DATADISK 1.0 1.0 0.977
IN2P3-CPPM_MCDISK 0.781 0.781 0.765
IN2P3-LAPP_DATADISK 4.0 4.0 4.0
IN2P3-LAPP_MCDISK 2.0 2.0 0.745
IN2P3-LPC_DATADISK 3.0 3.0 3.0
IN2P3-LPC_MCDISK 2.0 2.0 1.936
IN2P3-LPSC_DATADISK 0.488 0.488 0.482
IN2P3-LPSC_MCDISK 2.0 2.0 1.518
RO-02-NIPNE_DATADISK 3.0 3.0 3.0
RO-02-NIPNE_MCDISK 1.0 1.0 0.995
RO-07-NIPNE_DATADISK 2.0 2.0 2.0
RO-07-NIPNE_MCDISK 1.0 1.0 0.654
TOKYO-LCG2_DATADISK 15.0 15.0 13.643
TOKYO-LCG2_MCDISK 10.0 10.0 10.0

=== 5th June ===

* Data produced on Tuesday are supposed to be shipped in this night.

;14h50
* Now the RAW data has begun to shipped to LYON.

;Throughput(snapshot)(MB/s)
[[Image:LYON.throughput.14400_RAW_20080605_1816.png]]

;Number of transfer errors (snapshot)
[[Image:LYON.num_file_xs_error.14400_20080605_1823.png]]

;21h30
* IN2P3-CC_MCDISK has some problem of staging file. But it should have nothing to do with T0 export.

* 47 datasets have been exported to T1s. The total is 90.

;Throughput(snapshot)(MB/s)
[[Image:LYON.throughput.14400 20080605 2140.png]]

;Number of transfer errors (snapshot) -- It's the problem of staging file in T1 and nothing to do with FDR transfers.
[[Image:LYON.num_file_xs_error.14400_20080605_2140.png]]

=== 6th June ===
;09h50

;Throughput(snapshot) (MB/s)
[[Image:LYON.throughput.14400 200806060 0958.png]]

;Number of transfer errors (snapshot)
[[Image:LYON.num file xs error.14400 20080606 0959.png]]

;10h03
* Now all the RAW datasets have been exported to T1. The total number is 90.
http://atlas.web.cern.ch/Atlas/tzero/prod2/monitoring/tables/ddm_RAW_status.html

=== 7th June ===
* at CERN, will re-start the bulk processing with using the upcoming AtlasPoint1-14.1.0.12 release cache.

;11h30
* SRM in LYON is still in bad shape. Experts are working on it.

=== 8th June ===
;14h20
* FDR2 AODs/DPDs/TAGs/ESDs are being exported to LYON (IN2P3-CC_DATADISK), but haven't been shipped to T2s.

;Number of datasets transfered by far
$ dq2-list-dataset-site IN2P3-CC_DATADISK|grep fdr08_run2.*|wc -l
68

;Throughput(T0-T1,snapshot)(MB/s)
[[Image:LYON.throughput.14400 200806081420.png]]

;Number of transfer errors
[[Image:LYON.num file xs error.14400 200806081420.png]]

;Throughput(T1-T2,snapshot)(MB/s)
[[Image:LYON.T2.throughput.14400 T1-T2 200806081420.png]]

=== 9th June ===
;0h20
* FDR datasets has begun to be shipped to T2s.
* Some T2s (e.g. BEIJING) subscribed dedicated datasets, which will be done later in today's morning.

;Shipped datasets

IN2P3-CC_DATADISK 315
BEIJING-LCG2_DATADISK 0
GRIF-LAL_DATADISK 79
GRIF-LPNHE_DATADISK 39
GRIF-SACLAY_DATADISK 38
IN2P3-CPPM_DATADISK 0
IN2P3-LAPP_DATADISK 12
IN2P3-LPC_DATADISK 12
IN2P3-LPSC_DATADISK 32
RO-02-NIPNE_DATADISK 29
RO-07-NIPNE_DATADISK 46
TOKYO-LCG2_DATADISK 134

;Free space

BEIJING-LCG2_DATADISK 6.366 6.366 6.343
GRIF-LAL_DATADISK 6.0 6.0 5.579
GRIF-LPNHE_DATADISK 3.906 3.906 3.817
GRIF-SACLAY_DATADISK 6.0 6.0 5.927
IN2P3-CPPM_DATADISK 1.0 1.0 0.977
IN2P3-LAPP_DATADISK 4.0 4.0 4.0
IN2P3-LPC_DATADISK 0 0 0
IN2P3-LPSC_DATADISK 0.488 0.488 0.439
RO-02-NIPNE_DATADISK 3.0 3.0 2.986
RO-07-NIPNE_DATADISK 2.0 2.0 1.896
TOKYO-LCG2_DATADISK 15.0 15.0 13.313

;Troughputs(snapshot)(MB/s)
[[Image:LYON.throughput.14400 200806090046.png]]

;Number of transfer erros(snapshot)
* Note: No errors found in FDR2 transfers. Errors showed in this snapshot are since other production data transfers.
[[Image:LYON.num file xs error.14400 200806090047.png]]

;09h24

;Troughputs(snapshot)(MB/s)
[[Image:LYON.throughput.14400 200806090922.png]]

;Number of transfer erros(snapshot)
* Note: No errors found in FDR2 transfers. Errors showed in this snapshot are since other production data transfers.
[[Image:LYON.num file xs error.14400 20806090922.png]]

;Shipped datasets

IN2P3-CC_DATADISK 319
BEIJING-LCG2_DATADISK 0
GRIF-LAL_DATADISK 79
GRIF-LPNHE_DATADISK 39
GRIF-SACLAY_DATADISK 38
IN2P3-CPPM_DATADISK 0
IN2P3-LAPP_DATADISK 12
IN2P3-LPC_DATADISK 12
IN2P3-LPSC_DATADISK 32
RO-02-NIPNE_DATADISK 59
RO-07-NIPNE_DATADISK 49
TOKYO-LCG2_DATADISK 134

;Free space
BEIJING-LCG2_DATADISK 6.366 6.366 6.343
GRIF-LAL_DATADISK 6.0 6.0 5.579
GRIF-LPNHE_DATADISK 3.906 3.906 3.817
GRIF-SACLAY_DATADISK 6.0 6.0 5.927
IN2P3-CPPM_DATADISK 1.0 1.0 0.977
IN2P3-LAPP_DATADISK 4.0 4.0 4.0
IN2P3-LPC_DATADISK 3.0 3.0 3.0
IN2P3-LPSC_DATADISK 0.488 0.488 0.439
RO-02-NIPNE_DATADISK 3.0 3.0 2.945
RO-07-NIPNE_DATADISK 2.0 2.0 1.885
TOKYO-LCG2_DATADISK 15.0 15.0 13.313

;14h00
Many failed with RO-02-NIPNE_DATADISK.

[FTS] FTS State [Failed] FTS Retries [1] Reason [TRANSFER error during TRANSFER phase: [PERMISSION] the server sent an error response: 550 550 rfio write failure: Permission denied.] Source Host [ccsrm.in2p3.fr] 16
[FTS] FTS State [Failed] FTS Retries [1] Reason [TRANSFER error during TRANSFER phase: [TRANSFER_TIMEOUT] gridftp_copy_wait: Connection timed out] Source Host [ccsrm.in2p3.fr] 9
[FTS] FTS State [Failed] FTS Retries [1] Reason [TRANSFER error during TRANSFER phase: [GRIDFTP] the server sent an error response: 426 426 Transfer aborted (Unexpected Exception : java.io.IOException: Broken pipe)] Source Host [ccsrm.in2p3.fr] 2
[FTS] FTS State [Failed] FTS Retries [1] Reason [The process serving the transfer (status = PREPARING) is no longer active (could not open file /proc/3775/cmdline)] Source Host [ccsrm.in2p3.fr] 1
[FTS] FTS State [Failed] FTS Retries [1] Reason [The process serving the transfer (status = PREPARING) is no longer active (could not open file /proc/30076/cmdline)] Source Host [ccsrm.in2p3.fr] 1

;20h10
* FDR datasets have finished transferred to all T2s except BEIJING-LCG2_DATADISK and RO-02-NIPNE_DATADISK.
* Many "Connection timed out" errors found at BEIJING-LCG2_DATADISK and RO-02-NIPNE_DATADISK.

[FTS] FTS State [Failed] FTS Retries [1] Reason [SOURCE error during PREPARATION phase: [CONNECTION] failed to contact on remote SRM [httpg://ccsrm.in2p3.fr:8443/srm/managerv2]. Givin' up after 3 tries] Source Host [ccsrm.in2p3.fr] 111
[FTS] FTS State [Failed] FTS Retries [1] Reason [TRANSFER error during TRANSFER phase: [TRANSFER_TIMEOUT] gridftp_copy_wait: Connection timed out] Source Host [ccsrm.in2p3.fr] 95
[FTS] FTS State [Failed] FTS Retries [1] Reason [SOURCE error during PREPARATION phase: [REQUEST_TIMEOUT] failed to prepare source file in 180 seconds] Source Host [ccsrm.in2p3.fr] 3

; Shipped datasets

IN2P3-CC_DATADISK 134
BEIJING-LCG2_DATADISK 79
GRIF-LAL_DATADISK 67
GRIF-LPNHE_DATADISK 32
GRIF-SACLAY_DATADISK 26
IN2P3-CPPM_DATADISK 52
IN2P3-LAPP_DATADISK 52
IN2P3-LPC_DATADISK 52
IN2P3-LPSC_DATADISK 32
RO-02-NIPNE_DATADISK 49
RO-07-NIPNE_DATADISK 49
TOKYO-LCG2_DATADISK 126

=== 10th June ===

;Summary

All T2s except BEIJING-LCG2_DATADISK and RO-02-NIPNE_DATADISK were in good status and received all the exported datasets when FDR transferring.
The problem of "Connection timed out" with transfers to BEIJING will be hopefully solved by tuned the transfer duration >1800s by Lionel.
RO-02-NIPNE is said being upgraded from SL3 to SL4.

=== 11st June ===

; Anouncement
All FDR2 datasets will be deleted from all sites since part of the detector was not included in the reconstruction. It was not detected by the Data Quality because it is run on the express stream which is correct.

Old datasets will be deleted centrally by Stephane and new datasets that would be named begin with "ccrc08-run.0024*" will be re-generated and re-exported soon in today.

T2s will receive the same share rate as requested for FDR (except that DPD are not produced).

=== 12nd June ===
;01h20
* Function Test datasets(named beginning with "ccrc08_run2.*") have been re-exported to T2s. For this moment, all T2 sites seems in good shape to accept datasets except RO-07-NIPNE and RO-02-NIPNE that are in downtime.

* Datasets shipped:

IN2P3-CC_DATADISK 387
BEIJING-LCG2_DATADISK 46
GRIF-LAL_DATADISK 141
GRIF-LPNHE_DATADISK 79
GRIF-SACLAY_DATADISK 90
IN2P3-CPPM_DATADISK 22
IN2P3-LAPP_DATADISK 22
IN2P3-LPC_DATADISK 36
IN2P3-LPSC_DATADISK 17
RO-02-NIPNE_DATADISK 20
RO-07-NIPNE_DATADISK 33
TOKYO-LCG2_DATADISK 310

;Troughput
[[Image:LYON.T2.throughput.14400 200806120120.png]]

;Transfer errors
* Note that the errors showed in this snapshot involves all transfers mixed with other transfers(e.g MC/Panda). So it may not reflect the real situation FT transfers.
[[Image:LYON.T2.num file xs error.14400 200806120120.png]]

Atlas:FDR2

2008-06-12T10:33:22Z

Erming: /* 12nd June */

= Monitoring pages/graphs =
=== FTS ===
http://cctoolsafs.in2p3.fr/fts/monitoring/prod/ftsmonitor.php?vo=atlas

=== T0-T1 (ALL) ===
; FDR monitoring page - T1s
http://atldq2pro.cern.ch:8000/ft/mon/fdrmon_tier1s.html

http://dashb-atlas-data-tier0.cern.ch/dashboard/request.py/site
*Throughput 
:http://dashb-atlas-data-tier0.cern.ch/dashboard/templates/plots/OVERVIEW.throughput.14400.png
*Errors 
:http://dashb-atlas-data-tier0.cern.ch/dashboard/templates/plots/OVERVIEW.num_file_xs_error.14400.png

=== T0-T1 (Lyon) ===
http://dashb-atlas-data-tier0.cern.ch/dashboard/request.py/site?statsInterval=4&name=LYON
*Throughput T0-LYON 
:http://dashb-atlas-data-tier0.cern.ch/dashboard/templates/plots/LYON.throughput.14400.png
*Errors T0-LYON 
:http://dashb-atlas-data-tier0.cern.ch/dashboard/templates/plots/LYON.num_file_xs_error.14400.png
*Throughput Others-LYON 
:http://dashb-atlas-data.cern.ch/dashboard/templates/plots/LYON.T1.throughput.14400.png
*Errors Others-LYON :http://dashb-atlas-data.cern.ch/dashboard/templates/plots/LYON.T1.num_file_xs_error.14400.png

=== T1-T1 ===

=== T1-T2 (Lyon) ===
; FDR monitoring page
http://atldq2pro.cern.ch:8000/ft/mon/fdrmon_TiersInfo.html

http://atldq2pro.cern.ch:8000/ft/mon/fdrmon_T1-T2_matrix_day.html

; Dashboard
http://dashb-atlas-data.cern.ch/dashboard/request.py/site?statsInterval=4&name=LYON
*Throughput 
:http://dashb-atlas-data.cern.ch/dashboard/templates/plots/LYON.T2.throughput.14400.png
*Errors
:http://dashb-atlas-data.cern.ch/dashboard/templates/plots/LYON.T2.num_file_xs_error.14400.png

=== Network Graphs ===
; IN2P3 Weathermap
* http://netstat.in2p3.fr/weathermap/
* http://netstat.in2p3.fr/weathermap/graphiques/

; cc-in2p3 : [http://netstat.in2p3.fr/weathermap/graphiques/lyo-cern.html lyo-cern] (lhcopn-in2p3.cern.ch) http://netstat.in2p3.fr/weathermap/graphiques/lyo-cern-daily.gif
; lyon-nord : [http://netstat.in2p3.fr/weathermap/graphiques/lyo-nrd.html lyo-nrd] http://netstat.in2p3.fr/weathermap/graphiques/lyo-nrd-daily.gif
; orsay : [http://netstat.in2p3.fr/weathermap/graphiques/orsay.html orsay] (in2p3-orsay.cssi.renater.fr http://netstat.in2p3.fr/weathermap/graphiques/orsay-daily.gif
; lal-cc : [http://netstat.in2p3.fr/weathermap/graphiques/lal-cc.html Liaison LAL - CC] http://netstat.in2p3.fr/weathermap/graphiques/lal-cc-daily.gif
; lal : [http://netstat.in2p3.fr/weathermap/graphiques/lal.html lal] http://netstat.in2p3.fr/weathermap/graphiques/lal-daily.gif
; lpnhe: [http://netstat.in2p3.fr/weathermap/graphiques/lpnhe-nrd.html lpnhe-nrd] (in2p3-jussieu.cssi.renater.fr) http://netstat.in2p3.fr/weathermap/graphiques/lpnhe-nrd-daily.gif [http://netstat.in2p3.fr/weathermap/graphiques/lpnhe.html lpnhe] (Paris-LPNHE.in2p3.fr) http://netstat.in2p3.fr/weathermap/graphiques/lpnhe-daily.gif
; lapp : [http://netstat.in2p3.fr/weathermap/graphiques/ann-nrd.html ann-nrd] http://netstat.in2p3.fr/weathermap/graphiques/ann-nrd-daily.gif
; lpc: [http://netstat.in2p3.fr/weathermap/graphiques/lpc-cf.html lpc-cf]
http://netstat.in2p3.fr/weathermap/graphiques/lpc-cf-daily.gif
; lpsc : [http://netstat.in2p3.fr/weathermap/graphiques/lpsc.html lpsc] http://netstat.in2p3.fr/weathermap/graphiques/lpsc-daily.gif
; cppm : [http://netstat.in2p3.fr/weathermap/graphiques/cppm.html cppm] 
http://netstat.in2p3.fr/weathermap/graphiques/cppm-daily.gif
; international (tokyo etc.) : [http://netstat.in2p3.fr/weathermap/graphiques/parisnrd.html parisnrd] 
http://netstat.in2p3.fr/weathermap/graphiques/parisnrd-daily.gif
; GEANT - NYC : [http://dc-snmp.wcc.grnoc.iu.edu/manlan/show-graph.cgi?title=sw.newy32aoa.manlan.internet2.edu--te10/1&rrdname=sw.newy32aoa.manlan.internet2.edu--te10_1.rrd MANLAN] 
http://dc-snmp.wcc.grnoc.iu.edu/manlan/img/sw.newy32aoa.manlan.internet2.edu--te10_1-std5.gif
; NYC - TOKYO : [http://dc-snmp.wcc.grnoc.iu.edu/manlan/show-graph.cgi?title=sw.newy32aoa.manlan.internet2.edu--te11/1&rrdname=sw.newy32aoa.manlan.internet2.edu--te11_1.rrd MANLAN] 
http://dc-snmp.wcc.grnoc.iu.edu/manlan/img/sw.newy32aoa.manlan.internet2.edu--te11_1-std5.gif
; tokyo : [http://www.icepp.jp/monitor/gridftp/monitor.html gridftp monitor] http://www.icepp.jp/monitor/gridftp/img/all.png

= T0-T1 =

= T1-T2 =
==== General informations ====
;FDR page:
https://twiki.cern.ch/twiki/bin/view/Atlas/FullDressRehearsal

;DDM_in_FRD2: https://twiki.cern.ch/twiki/bin/view/Atlas/DDMOperationsGroup#DDM_in_FRD2

;Replication status(checked ~ every 2h?)
http://pandamon.usatlas.bnl.gov:25880/server/pandamon/query?mode=listFDR

;Lyon FTS monitor: http://cctoolsafs.in2p3.fr/fts/monitoring/prod/ftsmonitor.php?vo=atlas

;Contacts:

* IN2P3-CPPM_DATADISK :
* IN2P3-LPSC_DATADISK :
* IN2P3-LPC_DATADISK :
* GRIF-LAL_DATADISK : (grid.admin a lal.in2p3.fr)
* GRIF-LPNHE_DATADISK :
* GRIF-SACLAY_DATADISK :
* BEIJING-LCG2_DATADISK : (yanxf a ihep.ac.cn, Erming.Pei a cern.ch)
* RO-07-NIPNE_DATADISK : (ciubancan a nipne.ro)
* RO-02-NIPNE_DATADISK : (tpreda a nipne.ro)
* TOKYO-LCG2_DATADISK : (lcg-admin a icepp.s.u-tokyo.ac.jp)

===== Shares =====
Only srmv2+ space tokens sites will collect data. The list of streams is : Egamma, Bphysics, MinBias, Muon and Jet. If no precise stream name is associated to the share, the site will get any stream. The site in parenthesis will collect the complementary datasets. The T2 ressources at T1 means that users without production role can access CPUs at the T1 without the Production role.
* US :
o AGLT2_DATADISK : 100 %
o MWT2_DATADISK: 100 %
o NET2_DATADISK: 100 %
o SLACXRD_DATADISK: 100 %
o SWT2_CPB_DATADISK: 100 %
o T2 CPU ressources in BNL to access AOD
* CA :
o SFU-LCG2_DATADISK : 40%
o VICTORIA-LCG2_DATADISK : 30%
o ALBERTA-LCG2_DATADISK : 30%
o TORONTO-LCG2_DATADISK : 50 %
* IT :
o INFN-MILANO_DATADISK : 25%
o INFN-FRASCATI_DATADISK : 25%
o INFN-ROMA1_DATADISK : 100%
o INFN-NAPOLI-ATLAS_DATADISK : 100 %
* TW :
o TW-FTT_DATADISK : 50 %
* NL
o JINR-LCG2_DATADISK : 50% Jet (RU-PROTVINO-IHEP_DATADISK)
o JINR-LCG2_DATADISK : 50% MinBias (RU-PROTVINO-IHEP_DATADISK)
o JINR-LCG2_DATADISK : 50% Muon (RU-PROTVINO-IHEP_DATADISK)
o JINR-LCG2_DATADISK : 50% Bphysics (RU-PNPI_DATADISK)
o RU-PROTVINO-IHEP_DATADISK : 50% Jet (JINR-LCG2_DATADISK)
o RU-PROTVINO-IHEP_DATADISK : 50% Muon (JINR-LCG2_DATADISK)
o RU-PROTVINO-IHEP_DATADISK : 50% MinBias (JINR-LCG2_DATADISK)
o RU-PROTVINO-IHEP_DATADISK : 50% Egamma (RU-PNPI_DATADISK)
o RU-PNPI_DATADISK : 50% Egamma (RU-PROTVINO-IHEP_DATADISK)
o RU-PNPI_DATADISK : 50% Bphysics (JINR-LCG2_DATADISK)
o RU-PNPI_DATADISK : 50% Jet (RRC-KI_DATADISK)
o RU-PNPI_DATADISK : 50% MinBias (RRC-KI_DATADISK)
o RRC-KI_DATADISK : 50% Jet (RU-PNPI_DATADISK)
o RRC-KI_DATADISK : 50% MinBias (RU-PNPI_DATADISK)
o RRC-KI_DATADISK : 50% Egamma
o RRC-KI_DATADISK : 50% Bphysics
o CSTCDIE_DATADISK : 20%
* FR :
o IN2P3-LAPP_DATADISK : 100 % Egamma
o IN2P3-CPPM_DATADISK : 100 % Egamma
o IN2P3-LPSC_DATADISK : 5 %
o IN2P3-LPC_DATADISK : 100% Muon, 50% Egamma
o BEIJING-LCG2_DATADISK : 100 % Egamma, 50% Muon, 35% MinBias
o RO-07-NIPNE_DATADISK : 10%
o RO-02-NIPNE_DATADISK : 10%
o GRIF-LAL_DATADISK : 45%
o GRIF-LPNHE_DATADISK : 25%
o GRIF-SACLAY_DATADISK : 30 %
o TOKYO-LCG2_DATADISK : 100%
o T2 CPU ressources in LYON to access AOD
* DE :
o DESY-HH_DATADISK : 100%
o CSCS-LCG2_DATADISK: 100%
o DESY-ZN_DATADISK : 50%
o GOEGRID_DATADISK : 50%
o WUPPERTALPROD_DATADISK: 25%
o CYFRONET-LCG2_DATADISK : 15%
o PRAGUELCG2_DATADISK : 25%
o LRZ-LMU_DATADISK : 35%
o UNI-FREIBURG_DATADISK : 100% Jet, 100% Muon
* UK :
o UKI-SOUTHGRID-CAM-HEP_DATADISK : 50 % Muon (UKI-NORTHGRID-LANCS-HEP_DATADISK
o UKI-NORTHGRID-LANCS-HEP_DATADISK : 50 % Muon (UKI-SOUTHGRID-CAM-HEP_DATADISK)
o UKI-NORTHGRID-SHEF-HEP_DATADISK : 50 % MinBias (UKI-SCOTGRID-GLASGOW_DATADISK)
o UKI-SCOTGRID-GLASGOW_DATADISK : 50 % MinBias (UKI-NORTHGRID-SHEF-HEP_DATADISK)
o UKI-LT2-RHUL_DATADISK : 50 % Egamma (UKI-NORTHGRID-LIV-HEP_DATADISK)
o UKI-NORTHGRID-LIV-HEP_DATADISK : 50 % Egamma (UKI-LT2-RHUL_DATADISK)
o UKI-SOUTHGRID-RALPP_DATADISK : 50 Egamma (UKI-LT2-RHUL_DATADISK)
o UKI-SOUTHGRID-BHAM-HEP_DATADISK : 50 Jet (UKI-SOUTHGRID-OX-HEP_DATADISK)
* ES :
o IFIC-LCG2_DATADISK: 50%
o UAM-LCG2_DATADISK: 25%
o IFAE_DATADISK: 25%
* NDGF :
o SIGNET_DATADISK : 50 %

== Logbook ==

=== 2nd June ===

* Since Lyon is in donwtime, so there's no transfers from LYON to T2s today.
http://cc.in2p3.fr/cc_accueil.php3?lang=fr

*Free Space

Token Total Guaranteed Unused
------------------------------------------
BEIJING-LCG2_DATADISK 6.366 6.366 6.343
GRIF-LAL_DATADISK 6.0 6.0 5.732
GRIF-LPNHE_DATADISK 3.906 3.906 3.89
GRIF-SACLAY_DATADISK 6.0 6.0 5.994
IN2P3-CPPM_DATADISK 1.0 1.0 0.977
IN2P3-LAPP_DATADISK 4.0 4.0 4.0
IN2P3-LPC_DATADISK 3.0 3.0 3.0
IN2P3-LPSC_DATADISK 0.488 0.488 0.482
RO-02-NIPNE_DATADISK 3.0 3.0 3.0
RO-07-NIPNE_DATADISK 2.0 2.0 2.0
TOKYO-LCG2_DATADISK 15.0 15.0 13.643

=== 3rd June ===

* Lyon is still in donwtime, no transfers from LYON to T2s in this morning.

; Throughput(snapshot) (MB/s) : [[Image:LYON.throughput.14400_20080603.png]]
; Number of transfer errors(snapshot) : [[Image:LYON.num_file_xs_error.14400_20080603.png]]

=== 4th June ===
;22h02
* Still no transfers, desipte that all the services in LYON shows OK.

;Throughput(snapshot)
[[Image:LYON.throughput.3600_20080604.png]]

;Services in LYON
[[Image:services_cc-in2p3_2008060422h15.PNG]]

;Free space

BEIJING-LCG2_DATADISK 6.366 6.366 6.343
BEIJING-LCG2_MCDISK 5.457 5.457 5.457
GRIF-LAL_DATADISK 6.0 6.0 5.732
GRIF-LAL_MCDISK 7.0 7.0 3.724
GRIF-LPNHE_DATADISK 3.906 3.906 3.89
GRIF-LPNHE_MCDISK 1.0 1.0 0.194
GRIF-SACLAY_DATADISK 6.0 6.0 5.994
GRIF-SACLAY_MCDISK 2.0 2.0 1.713
IN2P3-CPPM_DATADISK 1.0 1.0 0.977
IN2P3-CPPM_MCDISK 0.781 0.781 0.765
IN2P3-LAPP_DATADISK 4.0 4.0 4.0
IN2P3-LAPP_MCDISK 2.0 2.0 0.745
IN2P3-LPC_DATADISK 3.0 3.0 3.0
IN2P3-LPC_MCDISK 2.0 2.0 1.936
IN2P3-LPSC_DATADISK 0.488 0.488 0.482
IN2P3-LPSC_MCDISK 2.0 2.0 1.518
RO-02-NIPNE_DATADISK 3.0 3.0 3.0
RO-02-NIPNE_MCDISK 1.0 1.0 0.995
RO-07-NIPNE_DATADISK 2.0 2.0 2.0
RO-07-NIPNE_MCDISK 1.0 1.0 0.654
TOKYO-LCG2_DATADISK 15.0 15.0 13.643
TOKYO-LCG2_MCDISK 10.0 10.0 10.0

=== 5th June ===

* Data produced on Tuesday are supposed to be shipped in this night.

;14h50
* Now the RAW data has begun to shipped to LYON.

;Throughput(snapshot)(MB/s)
[[Image:LYON.throughput.14400_RAW_20080605_1816.png]]

;Number of transfer errors (snapshot)
[[Image:LYON.num_file_xs_error.14400_20080605_1823.png]]

;21h30
* IN2P3-CC_MCDISK has some problem of staging file. But it should have nothing to do with T0 export.

* 47 datasets have been exported to T1s. The total is 90.

;Throughput(snapshot)(MB/s)
[[Image:LYON.throughput.14400 20080605 2140.png]]

;Number of transfer errors (snapshot) -- It's the problem of staging file in T1 and nothing to do with FDR transfers.
[[Image:LYON.num_file_xs_error.14400_20080605_2140.png]]

=== 6th June ===
;09h50

;Throughput(snapshot) (MB/s)
[[Image:LYON.throughput.14400 200806060 0958.png]]

;Number of transfer errors (snapshot)
[[Image:LYON.num file xs error.14400 20080606 0959.png]]

;10h03
* Now all the RAW datasets have been exported to T1. The total number is 90.
http://atlas.web.cern.ch/Atlas/tzero/prod2/monitoring/tables/ddm_RAW_status.html

=== 7th June ===
* at CERN, will re-start the bulk processing with using the upcoming AtlasPoint1-14.1.0.12 release cache.

;11h30
* SRM in LYON is still in bad shape. Experts are working on it.

=== 8th June ===
;14h20
* FDR2 AODs/DPDs/TAGs/ESDs are being exported to LYON (IN2P3-CC_DATADISK), but haven't been shipped to T2s.

;Number of datasets transfered by far
$ dq2-list-dataset-site IN2P3-CC_DATADISK|grep fdr08_run2.*|wc -l
68

;Throughput(T0-T1,snapshot)(MB/s)
[[Image:LYON.throughput.14400 200806081420.png]]

;Number of transfer errors
[[Image:LYON.num file xs error.14400 200806081420.png]]

;Throughput(T1-T2,snapshot)(MB/s)
[[Image:LYON.T2.throughput.14400 T1-T2 200806081420.png]]

=== 9th June ===
;0h20
* FDR datasets has begun to be shipped to T2s.
* Some T2s (e.g. BEIJING) subscribed dedicated datasets, which will be done later in today's morning.

;Shipped datasets

IN2P3-CC_DATADISK 315
BEIJING-LCG2_DATADISK 0
GRIF-LAL_DATADISK 79
GRIF-LPNHE_DATADISK 39
GRIF-SACLAY_DATADISK 38
IN2P3-CPPM_DATADISK 0
IN2P3-LAPP_DATADISK 12
IN2P3-LPC_DATADISK 12
IN2P3-LPSC_DATADISK 32
RO-02-NIPNE_DATADISK 29
RO-07-NIPNE_DATADISK 46
TOKYO-LCG2_DATADISK 134

;Free space

BEIJING-LCG2_DATADISK 6.366 6.366 6.343
GRIF-LAL_DATADISK 6.0 6.0 5.579
GRIF-LPNHE_DATADISK 3.906 3.906 3.817
GRIF-SACLAY_DATADISK 6.0 6.0 5.927
IN2P3-CPPM_DATADISK 1.0 1.0 0.977
IN2P3-LAPP_DATADISK 4.0 4.0 4.0
IN2P3-LPC_DATADISK 0 0 0
IN2P3-LPSC_DATADISK 0.488 0.488 0.439
RO-02-NIPNE_DATADISK 3.0 3.0 2.986
RO-07-NIPNE_DATADISK 2.0 2.0 1.896
TOKYO-LCG2_DATADISK 15.0 15.0 13.313

;Troughputs(snapshot)(MB/s)
[[Image:LYON.throughput.14400 200806090046.png]]

;Number of transfer erros(snapshot)
* Note: No errors found in FDR2 transfers. Errors showed in this snapshot are since other production data transfers.
[[Image:LYON.num file xs error.14400 200806090047.png]]

;09h24

;Troughputs(snapshot)(MB/s)
[[Image:LYON.throughput.14400 200806090922.png]]

;Number of transfer erros(snapshot)
* Note: No errors found in FDR2 transfers. Errors showed in this snapshot are since other production data transfers.
[[Image:LYON.num file xs error.14400 20806090922.png]]

;Shipped datasets

IN2P3-CC_DATADISK 319
BEIJING-LCG2_DATADISK 0
GRIF-LAL_DATADISK 79
GRIF-LPNHE_DATADISK 39
GRIF-SACLAY_DATADISK 38
IN2P3-CPPM_DATADISK 0
IN2P3-LAPP_DATADISK 12
IN2P3-LPC_DATADISK 12
IN2P3-LPSC_DATADISK 32
RO-02-NIPNE_DATADISK 59
RO-07-NIPNE_DATADISK 49
TOKYO-LCG2_DATADISK 134

;Free space
BEIJING-LCG2_DATADISK 6.366 6.366 6.343
GRIF-LAL_DATADISK 6.0 6.0 5.579
GRIF-LPNHE_DATADISK 3.906 3.906 3.817
GRIF-SACLAY_DATADISK 6.0 6.0 5.927
IN2P3-CPPM_DATADISK 1.0 1.0 0.977
IN2P3-LAPP_DATADISK 4.0 4.0 4.0
IN2P3-LPC_DATADISK 3.0 3.0 3.0
IN2P3-LPSC_DATADISK 0.488 0.488 0.439
RO-02-NIPNE_DATADISK 3.0 3.0 2.945
RO-07-NIPNE_DATADISK 2.0 2.0 1.885
TOKYO-LCG2_DATADISK 15.0 15.0 13.313

;14h00
Many failed with RO-02-NIPNE_DATADISK.

[FTS] FTS State [Failed] FTS Retries [1] Reason [TRANSFER error during TRANSFER phase: [PERMISSION] the server sent an error response: 550 550 rfio write failure: Permission denied.] Source Host [ccsrm.in2p3.fr] 16
[FTS] FTS State [Failed] FTS Retries [1] Reason [TRANSFER error during TRANSFER phase: [TRANSFER_TIMEOUT] gridftp_copy_wait: Connection timed out] Source Host [ccsrm.in2p3.fr] 9
[FTS] FTS State [Failed] FTS Retries [1] Reason [TRANSFER error during TRANSFER phase: [GRIDFTP] the server sent an error response: 426 426 Transfer aborted (Unexpected Exception : java.io.IOException: Broken pipe)] Source Host [ccsrm.in2p3.fr] 2
[FTS] FTS State [Failed] FTS Retries [1] Reason [The process serving the transfer (status = PREPARING) is no longer active (could not open file /proc/3775/cmdline)] Source Host [ccsrm.in2p3.fr] 1
[FTS] FTS State [Failed] FTS Retries [1] Reason [The process serving the transfer (status = PREPARING) is no longer active (could not open file /proc/30076/cmdline)] Source Host [ccsrm.in2p3.fr] 1

;20h10
* FDR datasets have finished transferred to all T2s except BEIJING-LCG2_DATADISK and RO-02-NIPNE_DATADISK.
* Many "Connection timed out" errors found at BEIJING-LCG2_DATADISK and RO-02-NIPNE_DATADISK.

[FTS] FTS State [Failed] FTS Retries [1] Reason [SOURCE error during PREPARATION phase: [CONNECTION] failed to contact on remote SRM [httpg://ccsrm.in2p3.fr:8443/srm/managerv2]. Givin' up after 3 tries] Source Host [ccsrm.in2p3.fr] 111
[FTS] FTS State [Failed] FTS Retries [1] Reason [TRANSFER error during TRANSFER phase: [TRANSFER_TIMEOUT] gridftp_copy_wait: Connection timed out] Source Host [ccsrm.in2p3.fr] 95
[FTS] FTS State [Failed] FTS Retries [1] Reason [SOURCE error during PREPARATION phase: [REQUEST_TIMEOUT] failed to prepare source file in 180 seconds] Source Host [ccsrm.in2p3.fr] 3

; Shipped datasets

IN2P3-CC_DATADISK 134
BEIJING-LCG2_DATADISK 79
GRIF-LAL_DATADISK 67
GRIF-LPNHE_DATADISK 32
GRIF-SACLAY_DATADISK 26
IN2P3-CPPM_DATADISK 52
IN2P3-LAPP_DATADISK 52
IN2P3-LPC_DATADISK 52
IN2P3-LPSC_DATADISK 32
RO-02-NIPNE_DATADISK 49
RO-07-NIPNE_DATADISK 49
TOKYO-LCG2_DATADISK 126

=== 10th June ===

;Summary

All T2s except BEIJING-LCG2_DATADISK and RO-02-NIPNE_DATADISK were in good status and received all the exported datasets when FDR transferring.
The problem of "Connection timed out" with transfers to BEIJING will be hopefully solved by tuned the transfer duration >1800s by Lionel.
RO-02-NIPNE is said being upgraded from SL3 to SL4.

=== 11st June ===

; Anouncement
All FDR2 datasets will be deleted from all sites since part of the detector was not included in the reconstruction. It was not detected by the Data Quality because it is run on the express stream which is correct.

Old datasets will be deleted centrally by Stephane and new datasets that would be named begin with "ccrc08-run.0024*" will be re-generated and re-exported soon in today.

T2s will receive the same share rate as requested for FDR (except that DPD are not produced).

=== 12nd June ===
;01h20
* Function Test datasets have been re-exported to T2s. For this moment, all T2 sites seems in good shape to accept datasets except RO-07-NIPNE and RO-02-NIPNE that are in downtime.

* Datasets shipped:

IN2P3-CC_DATADISK 387
BEIJING-LCG2_DATADISK 46
GRIF-LAL_DATADISK 141
GRIF-LPNHE_DATADISK 79
GRIF-SACLAY_DATADISK 90
IN2P3-CPPM_DATADISK 22
IN2P3-LAPP_DATADISK 22
IN2P3-LPC_DATADISK 36
IN2P3-LPSC_DATADISK 17
RO-02-NIPNE_DATADISK 20
RO-07-NIPNE_DATADISK 33
TOKYO-LCG2_DATADISK 310

;Troughput
[[Image:LYON.T2.throughput.14400 200806120120.png]]

;Transfer errors
* Note that the errors showed in this snapshot involves all transfers mixed with other transfers(e.g MC/Panda). So it may not reflect the real situation FT transfers.
[[Image:LYON.T2.num file xs error.14400 200806120120.png]]

Atlas:FDR2

2008-06-12T09:51:20Z

Erming: /* 12nd June */

= Monitoring pages/graphs =
=== FTS ===
http://cctoolsafs.in2p3.fr/fts/monitoring/prod/ftsmonitor.php?vo=atlas

=== T0-T1 (ALL) ===
; FDR monitoring page - T1s
http://atldq2pro.cern.ch:8000/ft/mon/fdrmon_tier1s.html

http://dashb-atlas-data-tier0.cern.ch/dashboard/request.py/site
*Throughput 
:http://dashb-atlas-data-tier0.cern.ch/dashboard/templates/plots/OVERVIEW.throughput.14400.png
*Errors 
:http://dashb-atlas-data-tier0.cern.ch/dashboard/templates/plots/OVERVIEW.num_file_xs_error.14400.png

=== T0-T1 (Lyon) ===
http://dashb-atlas-data-tier0.cern.ch/dashboard/request.py/site?statsInterval=4&name=LYON
*Throughput T0-LYON 
:http://dashb-atlas-data-tier0.cern.ch/dashboard/templates/plots/LYON.throughput.14400.png
*Errors T0-LYON 
:http://dashb-atlas-data-tier0.cern.ch/dashboard/templates/plots/LYON.num_file_xs_error.14400.png
*Throughput Others-LYON 
:http://dashb-atlas-data.cern.ch/dashboard/templates/plots/LYON.T1.throughput.14400.png
*Errors Others-LYON :http://dashb-atlas-data.cern.ch/dashboard/templates/plots/LYON.T1.num_file_xs_error.14400.png

=== T1-T1 ===

=== T1-T2 (Lyon) ===
; FDR monitoring page
http://atldq2pro.cern.ch:8000/ft/mon/fdrmon_TiersInfo.html

http://atldq2pro.cern.ch:8000/ft/mon/fdrmon_T1-T2_matrix_day.html

; Dashboard
http://dashb-atlas-data.cern.ch/dashboard/request.py/site?statsInterval=4&name=LYON
*Throughput 
:http://dashb-atlas-data.cern.ch/dashboard/templates/plots/LYON.T2.throughput.14400.png
*Errors
:http://dashb-atlas-data.cern.ch/dashboard/templates/plots/LYON.T2.num_file_xs_error.14400.png

=== Network Graphs ===
; IN2P3 Weathermap
* http://netstat.in2p3.fr/weathermap/
* http://netstat.in2p3.fr/weathermap/graphiques/

; cc-in2p3 : [http://netstat.in2p3.fr/weathermap/graphiques/lyo-cern.html lyo-cern] (lhcopn-in2p3.cern.ch) http://netstat.in2p3.fr/weathermap/graphiques/lyo-cern-daily.gif
; lyon-nord : [http://netstat.in2p3.fr/weathermap/graphiques/lyo-nrd.html lyo-nrd] http://netstat.in2p3.fr/weathermap/graphiques/lyo-nrd-daily.gif
; orsay : [http://netstat.in2p3.fr/weathermap/graphiques/orsay.html orsay] (in2p3-orsay.cssi.renater.fr http://netstat.in2p3.fr/weathermap/graphiques/orsay-daily.gif
; lal-cc : [http://netstat.in2p3.fr/weathermap/graphiques/lal-cc.html Liaison LAL - CC] http://netstat.in2p3.fr/weathermap/graphiques/lal-cc-daily.gif
; lal : [http://netstat.in2p3.fr/weathermap/graphiques/lal.html lal] http://netstat.in2p3.fr/weathermap/graphiques/lal-daily.gif
; lpnhe: [http://netstat.in2p3.fr/weathermap/graphiques/lpnhe-nrd.html lpnhe-nrd] (in2p3-jussieu.cssi.renater.fr) http://netstat.in2p3.fr/weathermap/graphiques/lpnhe-nrd-daily.gif [http://netstat.in2p3.fr/weathermap/graphiques/lpnhe.html lpnhe] (Paris-LPNHE.in2p3.fr) http://netstat.in2p3.fr/weathermap/graphiques/lpnhe-daily.gif
; lapp : [http://netstat.in2p3.fr/weathermap/graphiques/ann-nrd.html ann-nrd] http://netstat.in2p3.fr/weathermap/graphiques/ann-nrd-daily.gif
; lpc: [http://netstat.in2p3.fr/weathermap/graphiques/lpc-cf.html lpc-cf]
http://netstat.in2p3.fr/weathermap/graphiques/lpc-cf-daily.gif
; lpsc : [http://netstat.in2p3.fr/weathermap/graphiques/lpsc.html lpsc] http://netstat.in2p3.fr/weathermap/graphiques/lpsc-daily.gif
; cppm : [http://netstat.in2p3.fr/weathermap/graphiques/cppm.html cppm] 
http://netstat.in2p3.fr/weathermap/graphiques/cppm-daily.gif
; international (tokyo etc.) : [http://netstat.in2p3.fr/weathermap/graphiques/parisnrd.html parisnrd] 
http://netstat.in2p3.fr/weathermap/graphiques/parisnrd-daily.gif
; GEANT - NYC : [http://dc-snmp.wcc.grnoc.iu.edu/manlan/show-graph.cgi?title=sw.newy32aoa.manlan.internet2.edu--te10/1&rrdname=sw.newy32aoa.manlan.internet2.edu--te10_1.rrd MANLAN] 
http://dc-snmp.wcc.grnoc.iu.edu/manlan/img/sw.newy32aoa.manlan.internet2.edu--te10_1-std5.gif
; NYC - TOKYO : [http://dc-snmp.wcc.grnoc.iu.edu/manlan/show-graph.cgi?title=sw.newy32aoa.manlan.internet2.edu--te11/1&rrdname=sw.newy32aoa.manlan.internet2.edu--te11_1.rrd MANLAN] 
http://dc-snmp.wcc.grnoc.iu.edu/manlan/img/sw.newy32aoa.manlan.internet2.edu--te11_1-std5.gif
; tokyo : [http://www.icepp.jp/monitor/gridftp/monitor.html gridftp monitor] http://www.icepp.jp/monitor/gridftp/img/all.png

= T0-T1 =

= T1-T2 =
==== General informations ====
;FDR page:
https://twiki.cern.ch/twiki/bin/view/Atlas/FullDressRehearsal

;DDM_in_FRD2: https://twiki.cern.ch/twiki/bin/view/Atlas/DDMOperationsGroup#DDM_in_FRD2

;Replication status(checked ~ every 2h?)
http://pandamon.usatlas.bnl.gov:25880/server/pandamon/query?mode=listFDR

;Lyon FTS monitor: http://cctoolsafs.in2p3.fr/fts/monitoring/prod/ftsmonitor.php?vo=atlas

;Contacts:

* IN2P3-CPPM_DATADISK :
* IN2P3-LPSC_DATADISK :
* IN2P3-LPC_DATADISK :
* GRIF-LAL_DATADISK : (grid.admin a lal.in2p3.fr)
* GRIF-LPNHE_DATADISK :
* GRIF-SACLAY_DATADISK :
* BEIJING-LCG2_DATADISK : (yanxf a ihep.ac.cn, Erming.Pei a cern.ch)
* RO-07-NIPNE_DATADISK : (ciubancan a nipne.ro)
* RO-02-NIPNE_DATADISK : (tpreda a nipne.ro)
* TOKYO-LCG2_DATADISK : (lcg-admin a icepp.s.u-tokyo.ac.jp)

===== Shares =====
Only srmv2+ space tokens sites will collect data. The list of streams is : Egamma, Bphysics, MinBias, Muon and Jet. If no precise stream name is associated to the share, the site will get any stream. The site in parenthesis will collect the complementary datasets. The T2 ressources at T1 means that users without production role can access CPUs at the T1 without the Production role.
* US :
o AGLT2_DATADISK : 100 %
o MWT2_DATADISK: 100 %
o NET2_DATADISK: 100 %
o SLACXRD_DATADISK: 100 %
o SWT2_CPB_DATADISK: 100 %
o T2 CPU ressources in BNL to access AOD
* CA :
o SFU-LCG2_DATADISK : 40%
o VICTORIA-LCG2_DATADISK : 30%
o ALBERTA-LCG2_DATADISK : 30%
o TORONTO-LCG2_DATADISK : 50 %
* IT :
o INFN-MILANO_DATADISK : 25%
o INFN-FRASCATI_DATADISK : 25%
o INFN-ROMA1_DATADISK : 100%
o INFN-NAPOLI-ATLAS_DATADISK : 100 %
* TW :
o TW-FTT_DATADISK : 50 %
* NL
o JINR-LCG2_DATADISK : 50% Jet (RU-PROTVINO-IHEP_DATADISK)
o JINR-LCG2_DATADISK : 50% MinBias (RU-PROTVINO-IHEP_DATADISK)
o JINR-LCG2_DATADISK : 50% Muon (RU-PROTVINO-IHEP_DATADISK)
o JINR-LCG2_DATADISK : 50% Bphysics (RU-PNPI_DATADISK)
o RU-PROTVINO-IHEP_DATADISK : 50% Jet (JINR-LCG2_DATADISK)
o RU-PROTVINO-IHEP_DATADISK : 50% Muon (JINR-LCG2_DATADISK)
o RU-PROTVINO-IHEP_DATADISK : 50% MinBias (JINR-LCG2_DATADISK)
o RU-PROTVINO-IHEP_DATADISK : 50% Egamma (RU-PNPI_DATADISK)
o RU-PNPI_DATADISK : 50% Egamma (RU-PROTVINO-IHEP_DATADISK)
o RU-PNPI_DATADISK : 50% Bphysics (JINR-LCG2_DATADISK)
o RU-PNPI_DATADISK : 50% Jet (RRC-KI_DATADISK)
o RU-PNPI_DATADISK : 50% MinBias (RRC-KI_DATADISK)
o RRC-KI_DATADISK : 50% Jet (RU-PNPI_DATADISK)
o RRC-KI_DATADISK : 50% MinBias (RU-PNPI_DATADISK)
o RRC-KI_DATADISK : 50% Egamma
o RRC-KI_DATADISK : 50% Bphysics
o CSTCDIE_DATADISK : 20%
* FR :
o IN2P3-LAPP_DATADISK : 100 % Egamma
o IN2P3-CPPM_DATADISK : 100 % Egamma
o IN2P3-LPSC_DATADISK : 5 %
o IN2P3-LPC_DATADISK : 100% Muon, 50% Egamma
o BEIJING-LCG2_DATADISK : 100 % Egamma, 50% Muon, 35% MinBias
o RO-07-NIPNE_DATADISK : 10%
o RO-02-NIPNE_DATADISK : 10%
o GRIF-LAL_DATADISK : 45%
o GRIF-LPNHE_DATADISK : 25%
o GRIF-SACLAY_DATADISK : 30 %
o TOKYO-LCG2_DATADISK : 100%
o T2 CPU ressources in LYON to access AOD
* DE :
o DESY-HH_DATADISK : 100%
o CSCS-LCG2_DATADISK: 100%
o DESY-ZN_DATADISK : 50%
o GOEGRID_DATADISK : 50%
o WUPPERTALPROD_DATADISK: 25%
o CYFRONET-LCG2_DATADISK : 15%
o PRAGUELCG2_DATADISK : 25%
o LRZ-LMU_DATADISK : 35%
o UNI-FREIBURG_DATADISK : 100% Jet, 100% Muon
* UK :
o UKI-SOUTHGRID-CAM-HEP_DATADISK : 50 % Muon (UKI-NORTHGRID-LANCS-HEP_DATADISK
o UKI-NORTHGRID-LANCS-HEP_DATADISK : 50 % Muon (UKI-SOUTHGRID-CAM-HEP_DATADISK)
o UKI-NORTHGRID-SHEF-HEP_DATADISK : 50 % MinBias (UKI-SCOTGRID-GLASGOW_DATADISK)
o UKI-SCOTGRID-GLASGOW_DATADISK : 50 % MinBias (UKI-NORTHGRID-SHEF-HEP_DATADISK)
o UKI-LT2-RHUL_DATADISK : 50 % Egamma (UKI-NORTHGRID-LIV-HEP_DATADISK)
o UKI-NORTHGRID-LIV-HEP_DATADISK : 50 % Egamma (UKI-LT2-RHUL_DATADISK)
o UKI-SOUTHGRID-RALPP_DATADISK : 50 Egamma (UKI-LT2-RHUL_DATADISK)
o UKI-SOUTHGRID-BHAM-HEP_DATADISK : 50 Jet (UKI-SOUTHGRID-OX-HEP_DATADISK)
* ES :
o IFIC-LCG2_DATADISK: 50%
o UAM-LCG2_DATADISK: 25%
o IFAE_DATADISK: 25%
* NDGF :
o SIGNET_DATADISK : 50 %

== Logbook ==

=== 2nd June ===

* Since Lyon is in donwtime, so there's no transfers from LYON to T2s today.
http://cc.in2p3.fr/cc_accueil.php3?lang=fr

*Free Space

Token Total Guaranteed Unused
------------------------------------------
BEIJING-LCG2_DATADISK 6.366 6.366 6.343
GRIF-LAL_DATADISK 6.0 6.0 5.732
GRIF-LPNHE_DATADISK 3.906 3.906 3.89
GRIF-SACLAY_DATADISK 6.0 6.0 5.994
IN2P3-CPPM_DATADISK 1.0 1.0 0.977
IN2P3-LAPP_DATADISK 4.0 4.0 4.0
IN2P3-LPC_DATADISK 3.0 3.0 3.0
IN2P3-LPSC_DATADISK 0.488 0.488 0.482
RO-02-NIPNE_DATADISK 3.0 3.0 3.0
RO-07-NIPNE_DATADISK 2.0 2.0 2.0
TOKYO-LCG2_DATADISK 15.0 15.0 13.643

=== 3rd June ===

* Lyon is still in donwtime, no transfers from LYON to T2s in this morning.

; Throughput(snapshot) (MB/s) : [[Image:LYON.throughput.14400_20080603.png]]
; Number of transfer errors(snapshot) : [[Image:LYON.num_file_xs_error.14400_20080603.png]]

=== 4th June ===
;22h02
* Still no transfers, desipte that all the services in LYON shows OK.

;Throughput(snapshot)
[[Image:LYON.throughput.3600_20080604.png]]

;Services in LYON
[[Image:services_cc-in2p3_2008060422h15.PNG]]

;Free space

BEIJING-LCG2_DATADISK 6.366 6.366 6.343
BEIJING-LCG2_MCDISK 5.457 5.457 5.457
GRIF-LAL_DATADISK 6.0 6.0 5.732
GRIF-LAL_MCDISK 7.0 7.0 3.724
GRIF-LPNHE_DATADISK 3.906 3.906 3.89
GRIF-LPNHE_MCDISK 1.0 1.0 0.194
GRIF-SACLAY_DATADISK 6.0 6.0 5.994
GRIF-SACLAY_MCDISK 2.0 2.0 1.713
IN2P3-CPPM_DATADISK 1.0 1.0 0.977
IN2P3-CPPM_MCDISK 0.781 0.781 0.765
IN2P3-LAPP_DATADISK 4.0 4.0 4.0
IN2P3-LAPP_MCDISK 2.0 2.0 0.745
IN2P3-LPC_DATADISK 3.0 3.0 3.0
IN2P3-LPC_MCDISK 2.0 2.0 1.936
IN2P3-LPSC_DATADISK 0.488 0.488 0.482
IN2P3-LPSC_MCDISK 2.0 2.0 1.518
RO-02-NIPNE_DATADISK 3.0 3.0 3.0
RO-02-NIPNE_MCDISK 1.0 1.0 0.995
RO-07-NIPNE_DATADISK 2.0 2.0 2.0
RO-07-NIPNE_MCDISK 1.0 1.0 0.654
TOKYO-LCG2_DATADISK 15.0 15.0 13.643
TOKYO-LCG2_MCDISK 10.0 10.0 10.0

=== 5th June ===

* Data produced on Tuesday are supposed to be shipped in this night.

;14h50
* Now the RAW data has begun to shipped to LYON.

;Throughput(snapshot)(MB/s)
[[Image:LYON.throughput.14400_RAW_20080605_1816.png]]

;Number of transfer errors (snapshot)
[[Image:LYON.num_file_xs_error.14400_20080605_1823.png]]

;21h30
* IN2P3-CC_MCDISK has some problem of staging file. But it should have nothing to do with T0 export.

* 47 datasets have been exported to T1s. The total is 90.

;Throughput(snapshot)(MB/s)
[[Image:LYON.throughput.14400 20080605 2140.png]]

;Number of transfer errors (snapshot) -- It's the problem of staging file in T1 and nothing to do with FDR transfers.
[[Image:LYON.num_file_xs_error.14400_20080605_2140.png]]

=== 6th June ===
;09h50

;Throughput(snapshot) (MB/s)
[[Image:LYON.throughput.14400 200806060 0958.png]]

;Number of transfer errors (snapshot)
[[Image:LYON.num file xs error.14400 20080606 0959.png]]

;10h03
* Now all the RAW datasets have been exported to T1. The total number is 90.
http://atlas.web.cern.ch/Atlas/tzero/prod2/monitoring/tables/ddm_RAW_status.html

=== 7th June ===
* at CERN, will re-start the bulk processing with using the upcoming AtlasPoint1-14.1.0.12 release cache.

;11h30
* SRM in LYON is still in bad shape. Experts are working on it.

=== 8th June ===
;14h20
* FDR2 AODs/DPDs/TAGs/ESDs are being exported to LYON (IN2P3-CC_DATADISK), but haven't been shipped to T2s.

;Number of datasets transfered by far
$ dq2-list-dataset-site IN2P3-CC_DATADISK|grep fdr08_run2.*|wc -l
68

;Throughput(T0-T1,snapshot)(MB/s)
[[Image:LYON.throughput.14400 200806081420.png]]

;Number of transfer errors
[[Image:LYON.num file xs error.14400 200806081420.png]]

;Throughput(T1-T2,snapshot)(MB/s)
[[Image:LYON.T2.throughput.14400 T1-T2 200806081420.png]]

=== 9th June ===
;0h20
* FDR datasets has begun to be shipped to T2s.
* Some T2s (e.g. BEIJING) subscribed dedicated datasets, which will be done later in today's morning.

;Shipped datasets

IN2P3-CC_DATADISK 315
BEIJING-LCG2_DATADISK 0
GRIF-LAL_DATADISK 79
GRIF-LPNHE_DATADISK 39
GRIF-SACLAY_DATADISK 38
IN2P3-CPPM_DATADISK 0
IN2P3-LAPP_DATADISK 12
IN2P3-LPC_DATADISK 12
IN2P3-LPSC_DATADISK 32
RO-02-NIPNE_DATADISK 29
RO-07-NIPNE_DATADISK 46
TOKYO-LCG2_DATADISK 134

;Free space

BEIJING-LCG2_DATADISK 6.366 6.366 6.343
GRIF-LAL_DATADISK 6.0 6.0 5.579
GRIF-LPNHE_DATADISK 3.906 3.906 3.817
GRIF-SACLAY_DATADISK 6.0 6.0 5.927
IN2P3-CPPM_DATADISK 1.0 1.0 0.977
IN2P3-LAPP_DATADISK 4.0 4.0 4.0
IN2P3-LPC_DATADISK 0 0 0
IN2P3-LPSC_DATADISK 0.488 0.488 0.439
RO-02-NIPNE_DATADISK 3.0 3.0 2.986
RO-07-NIPNE_DATADISK 2.0 2.0 1.896
TOKYO-LCG2_DATADISK 15.0 15.0 13.313

;Troughputs(snapshot)(MB/s)
[[Image:LYON.throughput.14400 200806090046.png]]

;Number of transfer erros(snapshot)
* Note: No errors found in FDR2 transfers. Errors showed in this snapshot are since other production data transfers.
[[Image:LYON.num file xs error.14400 200806090047.png]]

;09h24

;Troughputs(snapshot)(MB/s)
[[Image:LYON.throughput.14400 200806090922.png]]

;Number of transfer erros(snapshot)
* Note: No errors found in FDR2 transfers. Errors showed in this snapshot are since other production data transfers.
[[Image:LYON.num file xs error.14400 20806090922.png]]

;Shipped datasets

IN2P3-CC_DATADISK 319
BEIJING-LCG2_DATADISK 0
GRIF-LAL_DATADISK 79
GRIF-LPNHE_DATADISK 39
GRIF-SACLAY_DATADISK 38
IN2P3-CPPM_DATADISK 0
IN2P3-LAPP_DATADISK 12
IN2P3-LPC_DATADISK 12
IN2P3-LPSC_DATADISK 32
RO-02-NIPNE_DATADISK 59
RO-07-NIPNE_DATADISK 49
TOKYO-LCG2_DATADISK 134

;Free space
BEIJING-LCG2_DATADISK 6.366 6.366 6.343
GRIF-LAL_DATADISK 6.0 6.0 5.579
GRIF-LPNHE_DATADISK 3.906 3.906 3.817
GRIF-SACLAY_DATADISK 6.0 6.0 5.927
IN2P3-CPPM_DATADISK 1.0 1.0 0.977
IN2P3-LAPP_DATADISK 4.0 4.0 4.0
IN2P3-LPC_DATADISK 3.0 3.0 3.0
IN2P3-LPSC_DATADISK 0.488 0.488 0.439
RO-02-NIPNE_DATADISK 3.0 3.0 2.945
RO-07-NIPNE_DATADISK 2.0 2.0 1.885
TOKYO-LCG2_DATADISK 15.0 15.0 13.313

;14h00
Many failed with RO-02-NIPNE_DATADISK.

[FTS] FTS State [Failed] FTS Retries [1] Reason [TRANSFER error during TRANSFER phase: [PERMISSION] the server sent an error response: 550 550 rfio write failure: Permission denied.] Source Host [ccsrm.in2p3.fr] 16
[FTS] FTS State [Failed] FTS Retries [1] Reason [TRANSFER error during TRANSFER phase: [TRANSFER_TIMEOUT] gridftp_copy_wait: Connection timed out] Source Host [ccsrm.in2p3.fr] 9
[FTS] FTS State [Failed] FTS Retries [1] Reason [TRANSFER error during TRANSFER phase: [GRIDFTP] the server sent an error response: 426 426 Transfer aborted (Unexpected Exception : java.io.IOException: Broken pipe)] Source Host [ccsrm.in2p3.fr] 2
[FTS] FTS State [Failed] FTS Retries [1] Reason [The process serving the transfer (status = PREPARING) is no longer active (could not open file /proc/3775/cmdline)] Source Host [ccsrm.in2p3.fr] 1
[FTS] FTS State [Failed] FTS Retries [1] Reason [The process serving the transfer (status = PREPARING) is no longer active (could not open file /proc/30076/cmdline)] Source Host [ccsrm.in2p3.fr] 1

;20h10
* FDR datasets have finished transferred to all T2s except BEIJING-LCG2_DATADISK and RO-02-NIPNE_DATADISK.
* Many "Connection timed out" errors found at BEIJING-LCG2_DATADISK and RO-02-NIPNE_DATADISK.

[FTS] FTS State [Failed] FTS Retries [1] Reason [SOURCE error during PREPARATION phase: [CONNECTION] failed to contact on remote SRM [httpg://ccsrm.in2p3.fr:8443/srm/managerv2]. Givin' up after 3 tries] Source Host [ccsrm.in2p3.fr] 111
[FTS] FTS State [Failed] FTS Retries [1] Reason [TRANSFER error during TRANSFER phase: [TRANSFER_TIMEOUT] gridftp_copy_wait: Connection timed out] Source Host [ccsrm.in2p3.fr] 95
[FTS] FTS State [Failed] FTS Retries [1] Reason [SOURCE error during PREPARATION phase: [REQUEST_TIMEOUT] failed to prepare source file in 180 seconds] Source Host [ccsrm.in2p3.fr] 3

; Shipped datasets

IN2P3-CC_DATADISK 134
BEIJING-LCG2_DATADISK 79
GRIF-LAL_DATADISK 67
GRIF-LPNHE_DATADISK 32
GRIF-SACLAY_DATADISK 26
IN2P3-CPPM_DATADISK 52
IN2P3-LAPP_DATADISK 52
IN2P3-LPC_DATADISK 52
IN2P3-LPSC_DATADISK 32
RO-02-NIPNE_DATADISK 49
RO-07-NIPNE_DATADISK 49
TOKYO-LCG2_DATADISK 126

=== 10th June ===

;Summary

All T2s except BEIJING-LCG2_DATADISK and RO-02-NIPNE_DATADISK were in good status and received all the exported datasets when FDR transferring.
The problem of "Connection timed out" with transfers to BEIJING will be hopefully solved by tuned the transfer duration >1800s by Lionel.
RO-02-NIPNE is said being upgraded from SL3 to SL4.

=== 11st June ===

; Anouncement
All FDR2 datasets will be deleted from all sites since part of the detector was not included in the reconstruction. It was not detected by the Data Quality because it is run on the express stream which is correct.

Old datasets will be deleted centrally by Stephane and new datasets that would be named begin with "ccrc08-run.0024*" will be re-generated and re-exported soon in today.

T2s will receive the same share rate as requested for FDR (except that DPD are not produced).

=== 12nd June ===
;01h20
* FDR2 datasets have been re-exported to T2s. For this moment, all T2 sites seems in good shape to accept datasets except RO-07-NIPNE and RO-02-NIPNE that are in downtime.

* Datasets shipped:

IN2P3-CC_DATADISK 387
BEIJING-LCG2_DATADISK 46
GRIF-LAL_DATADISK 141
GRIF-LPNHE_DATADISK 79
GRIF-SACLAY_DATADISK 90
IN2P3-CPPM_DATADISK 22
IN2P3-LAPP_DATADISK 22
IN2P3-LPC_DATADISK 36
IN2P3-LPSC_DATADISK 17
RO-02-NIPNE_DATADISK 20
RO-07-NIPNE_DATADISK 33
TOKYO-LCG2_DATADISK 310

;Troughput
[[Image:LYON.T2.throughput.14400 200806120120.png]]

;Transfer errors
* Note that the errors showed in this snapshot involves all transfers mixed with other transfers(e.g MC/Panda). So it may not reflect the real situation of FDR2 transfers.
[[Image:LYON.T2.num file xs error.14400 200806120120.png]]

Atlas:FDR2

2008-06-12T09:48:57Z

Erming: /* 12nd June */

= Monitoring pages/graphs =
=== FTS ===
http://cctoolsafs.in2p3.fr/fts/monitoring/prod/ftsmonitor.php?vo=atlas

=== T0-T1 (ALL) ===
; FDR monitoring page - T1s
http://atldq2pro.cern.ch:8000/ft/mon/fdrmon_tier1s.html

http://dashb-atlas-data-tier0.cern.ch/dashboard/request.py/site
*Throughput 
:http://dashb-atlas-data-tier0.cern.ch/dashboard/templates/plots/OVERVIEW.throughput.14400.png
*Errors 
:http://dashb-atlas-data-tier0.cern.ch/dashboard/templates/plots/OVERVIEW.num_file_xs_error.14400.png

=== T0-T1 (Lyon) ===
http://dashb-atlas-data-tier0.cern.ch/dashboard/request.py/site?statsInterval=4&name=LYON
*Throughput T0-LYON 
:http://dashb-atlas-data-tier0.cern.ch/dashboard/templates/plots/LYON.throughput.14400.png
*Errors T0-LYON 
:http://dashb-atlas-data-tier0.cern.ch/dashboard/templates/plots/LYON.num_file_xs_error.14400.png
*Throughput Others-LYON 
:http://dashb-atlas-data.cern.ch/dashboard/templates/plots/LYON.T1.throughput.14400.png
*Errors Others-LYON :http://dashb-atlas-data.cern.ch/dashboard/templates/plots/LYON.T1.num_file_xs_error.14400.png

=== T1-T1 ===

=== T1-T2 (Lyon) ===
; FDR monitoring page
http://atldq2pro.cern.ch:8000/ft/mon/fdrmon_TiersInfo.html

http://atldq2pro.cern.ch:8000/ft/mon/fdrmon_T1-T2_matrix_day.html

; Dashboard
http://dashb-atlas-data.cern.ch/dashboard/request.py/site?statsInterval=4&name=LYON
*Throughput 
:http://dashb-atlas-data.cern.ch/dashboard/templates/plots/LYON.T2.throughput.14400.png
*Errors
:http://dashb-atlas-data.cern.ch/dashboard/templates/plots/LYON.T2.num_file_xs_error.14400.png

=== Network Graphs ===
; IN2P3 Weathermap
* http://netstat.in2p3.fr/weathermap/
* http://netstat.in2p3.fr/weathermap/graphiques/

; cc-in2p3 : [http://netstat.in2p3.fr/weathermap/graphiques/lyo-cern.html lyo-cern] (lhcopn-in2p3.cern.ch) http://netstat.in2p3.fr/weathermap/graphiques/lyo-cern-daily.gif
; lyon-nord : [http://netstat.in2p3.fr/weathermap/graphiques/lyo-nrd.html lyo-nrd] http://netstat.in2p3.fr/weathermap/graphiques/lyo-nrd-daily.gif
; orsay : [http://netstat.in2p3.fr/weathermap/graphiques/orsay.html orsay] (in2p3-orsay.cssi.renater.fr http://netstat.in2p3.fr/weathermap/graphiques/orsay-daily.gif
; lal-cc : [http://netstat.in2p3.fr/weathermap/graphiques/lal-cc.html Liaison LAL - CC] http://netstat.in2p3.fr/weathermap/graphiques/lal-cc-daily.gif
; lal : [http://netstat.in2p3.fr/weathermap/graphiques/lal.html lal] http://netstat.in2p3.fr/weathermap/graphiques/lal-daily.gif
; lpnhe: [http://netstat.in2p3.fr/weathermap/graphiques/lpnhe-nrd.html lpnhe-nrd] (in2p3-jussieu.cssi.renater.fr) http://netstat.in2p3.fr/weathermap/graphiques/lpnhe-nrd-daily.gif [http://netstat.in2p3.fr/weathermap/graphiques/lpnhe.html lpnhe] (Paris-LPNHE.in2p3.fr) http://netstat.in2p3.fr/weathermap/graphiques/lpnhe-daily.gif
; lapp : [http://netstat.in2p3.fr/weathermap/graphiques/ann-nrd.html ann-nrd] http://netstat.in2p3.fr/weathermap/graphiques/ann-nrd-daily.gif
; lpc: [http://netstat.in2p3.fr/weathermap/graphiques/lpc-cf.html lpc-cf]
http://netstat.in2p3.fr/weathermap/graphiques/lpc-cf-daily.gif
; lpsc : [http://netstat.in2p3.fr/weathermap/graphiques/lpsc.html lpsc] http://netstat.in2p3.fr/weathermap/graphiques/lpsc-daily.gif
; cppm : [http://netstat.in2p3.fr/weathermap/graphiques/cppm.html cppm] 
http://netstat.in2p3.fr/weathermap/graphiques/cppm-daily.gif
; international (tokyo etc.) : [http://netstat.in2p3.fr/weathermap/graphiques/parisnrd.html parisnrd] 
http://netstat.in2p3.fr/weathermap/graphiques/parisnrd-daily.gif
; GEANT - NYC : [http://dc-snmp.wcc.grnoc.iu.edu/manlan/show-graph.cgi?title=sw.newy32aoa.manlan.internet2.edu--te10/1&rrdname=sw.newy32aoa.manlan.internet2.edu--te10_1.rrd MANLAN] 
http://dc-snmp.wcc.grnoc.iu.edu/manlan/img/sw.newy32aoa.manlan.internet2.edu--te10_1-std5.gif
; NYC - TOKYO : [http://dc-snmp.wcc.grnoc.iu.edu/manlan/show-graph.cgi?title=sw.newy32aoa.manlan.internet2.edu--te11/1&rrdname=sw.newy32aoa.manlan.internet2.edu--te11_1.rrd MANLAN] 
http://dc-snmp.wcc.grnoc.iu.edu/manlan/img/sw.newy32aoa.manlan.internet2.edu--te11_1-std5.gif
; tokyo : [http://www.icepp.jp/monitor/gridftp/monitor.html gridftp monitor] http://www.icepp.jp/monitor/gridftp/img/all.png

= T0-T1 =

= T1-T2 =
==== General informations ====
;FDR page:
https://twiki.cern.ch/twiki/bin/view/Atlas/FullDressRehearsal

;DDM_in_FRD2: https://twiki.cern.ch/twiki/bin/view/Atlas/DDMOperationsGroup#DDM_in_FRD2

;Replication status(checked ~ every 2h?)
http://pandamon.usatlas.bnl.gov:25880/server/pandamon/query?mode=listFDR

;Lyon FTS monitor: http://cctoolsafs.in2p3.fr/fts/monitoring/prod/ftsmonitor.php?vo=atlas

;Contacts:

* IN2P3-CPPM_DATADISK :
* IN2P3-LPSC_DATADISK :
* IN2P3-LPC_DATADISK :
* GRIF-LAL_DATADISK : (grid.admin a lal.in2p3.fr)
* GRIF-LPNHE_DATADISK :
* GRIF-SACLAY_DATADISK :
* BEIJING-LCG2_DATADISK : (yanxf a ihep.ac.cn, Erming.Pei a cern.ch)
* RO-07-NIPNE_DATADISK : (ciubancan a nipne.ro)
* RO-02-NIPNE_DATADISK : (tpreda a nipne.ro)
* TOKYO-LCG2_DATADISK : (lcg-admin a icepp.s.u-tokyo.ac.jp)

===== Shares =====
Only srmv2+ space tokens sites will collect data. The list of streams is : Egamma, Bphysics, MinBias, Muon and Jet. If no precise stream name is associated to the share, the site will get any stream. The site in parenthesis will collect the complementary datasets. The T2 ressources at T1 means that users without production role can access CPUs at the T1 without the Production role.
* US :
o AGLT2_DATADISK : 100 %
o MWT2_DATADISK: 100 %
o NET2_DATADISK: 100 %
o SLACXRD_DATADISK: 100 %
o SWT2_CPB_DATADISK: 100 %
o T2 CPU ressources in BNL to access AOD
* CA :
o SFU-LCG2_DATADISK : 40%
o VICTORIA-LCG2_DATADISK : 30%
o ALBERTA-LCG2_DATADISK : 30%
o TORONTO-LCG2_DATADISK : 50 %
* IT :
o INFN-MILANO_DATADISK : 25%
o INFN-FRASCATI_DATADISK : 25%
o INFN-ROMA1_DATADISK : 100%
o INFN-NAPOLI-ATLAS_DATADISK : 100 %
* TW :
o TW-FTT_DATADISK : 50 %
* NL
o JINR-LCG2_DATADISK : 50% Jet (RU-PROTVINO-IHEP_DATADISK)
o JINR-LCG2_DATADISK : 50% MinBias (RU-PROTVINO-IHEP_DATADISK)
o JINR-LCG2_DATADISK : 50% Muon (RU-PROTVINO-IHEP_DATADISK)
o JINR-LCG2_DATADISK : 50% Bphysics (RU-PNPI_DATADISK)
o RU-PROTVINO-IHEP_DATADISK : 50% Jet (JINR-LCG2_DATADISK)
o RU-PROTVINO-IHEP_DATADISK : 50% Muon (JINR-LCG2_DATADISK)
o RU-PROTVINO-IHEP_DATADISK : 50% MinBias (JINR-LCG2_DATADISK)
o RU-PROTVINO-IHEP_DATADISK : 50% Egamma (RU-PNPI_DATADISK)
o RU-PNPI_DATADISK : 50% Egamma (RU-PROTVINO-IHEP_DATADISK)
o RU-PNPI_DATADISK : 50% Bphysics (JINR-LCG2_DATADISK)
o RU-PNPI_DATADISK : 50% Jet (RRC-KI_DATADISK)
o RU-PNPI_DATADISK : 50% MinBias (RRC-KI_DATADISK)
o RRC-KI_DATADISK : 50% Jet (RU-PNPI_DATADISK)
o RRC-KI_DATADISK : 50% MinBias (RU-PNPI_DATADISK)
o RRC-KI_DATADISK : 50% Egamma
o RRC-KI_DATADISK : 50% Bphysics
o CSTCDIE_DATADISK : 20%
* FR :
o IN2P3-LAPP_DATADISK : 100 % Egamma
o IN2P3-CPPM_DATADISK : 100 % Egamma
o IN2P3-LPSC_DATADISK : 5 %
o IN2P3-LPC_DATADISK : 100% Muon, 50% Egamma
o BEIJING-LCG2_DATADISK : 100 % Egamma, 50% Muon, 35% MinBias
o RO-07-NIPNE_DATADISK : 10%
o RO-02-NIPNE_DATADISK : 10%
o GRIF-LAL_DATADISK : 45%
o GRIF-LPNHE_DATADISK : 25%
o GRIF-SACLAY_DATADISK : 30 %
o TOKYO-LCG2_DATADISK : 100%
o T2 CPU ressources in LYON to access AOD
* DE :
o DESY-HH_DATADISK : 100%
o CSCS-LCG2_DATADISK: 100%
o DESY-ZN_DATADISK : 50%
o GOEGRID_DATADISK : 50%
o WUPPERTALPROD_DATADISK: 25%
o CYFRONET-LCG2_DATADISK : 15%
o PRAGUELCG2_DATADISK : 25%
o LRZ-LMU_DATADISK : 35%
o UNI-FREIBURG_DATADISK : 100% Jet, 100% Muon
* UK :
o UKI-SOUTHGRID-CAM-HEP_DATADISK : 50 % Muon (UKI-NORTHGRID-LANCS-HEP_DATADISK
o UKI-NORTHGRID-LANCS-HEP_DATADISK : 50 % Muon (UKI-SOUTHGRID-CAM-HEP_DATADISK)
o UKI-NORTHGRID-SHEF-HEP_DATADISK : 50 % MinBias (UKI-SCOTGRID-GLASGOW_DATADISK)
o UKI-SCOTGRID-GLASGOW_DATADISK : 50 % MinBias (UKI-NORTHGRID-SHEF-HEP_DATADISK)
o UKI-LT2-RHUL_DATADISK : 50 % Egamma (UKI-NORTHGRID-LIV-HEP_DATADISK)
o UKI-NORTHGRID-LIV-HEP_DATADISK : 50 % Egamma (UKI-LT2-RHUL_DATADISK)
o UKI-SOUTHGRID-RALPP_DATADISK : 50 Egamma (UKI-LT2-RHUL_DATADISK)
o UKI-SOUTHGRID-BHAM-HEP_DATADISK : 50 Jet (UKI-SOUTHGRID-OX-HEP_DATADISK)
* ES :
o IFIC-LCG2_DATADISK: 50%
o UAM-LCG2_DATADISK: 25%
o IFAE_DATADISK: 25%
* NDGF :
o SIGNET_DATADISK : 50 %

== Logbook ==

=== 2nd June ===

* Since Lyon is in donwtime, so there's no transfers from LYON to T2s today.
http://cc.in2p3.fr/cc_accueil.php3?lang=fr

*Free Space

Token Total Guaranteed Unused
------------------------------------------
BEIJING-LCG2_DATADISK 6.366 6.366 6.343
GRIF-LAL_DATADISK 6.0 6.0 5.732
GRIF-LPNHE_DATADISK 3.906 3.906 3.89
GRIF-SACLAY_DATADISK 6.0 6.0 5.994
IN2P3-CPPM_DATADISK 1.0 1.0 0.977
IN2P3-LAPP_DATADISK 4.0 4.0 4.0
IN2P3-LPC_DATADISK 3.0 3.0 3.0
IN2P3-LPSC_DATADISK 0.488 0.488 0.482
RO-02-NIPNE_DATADISK 3.0 3.0 3.0
RO-07-NIPNE_DATADISK 2.0 2.0 2.0
TOKYO-LCG2_DATADISK 15.0 15.0 13.643

=== 3rd June ===

* Lyon is still in donwtime, no transfers from LYON to T2s in this morning.

; Throughput(snapshot) (MB/s) : [[Image:LYON.throughput.14400_20080603.png]]
; Number of transfer errors(snapshot) : [[Image:LYON.num_file_xs_error.14400_20080603.png]]

=== 4th June ===
;22h02
* Still no transfers, desipte that all the services in LYON shows OK.

;Throughput(snapshot)
[[Image:LYON.throughput.3600_20080604.png]]

;Services in LYON
[[Image:services_cc-in2p3_2008060422h15.PNG]]

;Free space

BEIJING-LCG2_DATADISK 6.366 6.366 6.343
BEIJING-LCG2_MCDISK 5.457 5.457 5.457
GRIF-LAL_DATADISK 6.0 6.0 5.732
GRIF-LAL_MCDISK 7.0 7.0 3.724
GRIF-LPNHE_DATADISK 3.906 3.906 3.89
GRIF-LPNHE_MCDISK 1.0 1.0 0.194
GRIF-SACLAY_DATADISK 6.0 6.0 5.994
GRIF-SACLAY_MCDISK 2.0 2.0 1.713
IN2P3-CPPM_DATADISK 1.0 1.0 0.977
IN2P3-CPPM_MCDISK 0.781 0.781 0.765
IN2P3-LAPP_DATADISK 4.0 4.0 4.0
IN2P3-LAPP_MCDISK 2.0 2.0 0.745
IN2P3-LPC_DATADISK 3.0 3.0 3.0
IN2P3-LPC_MCDISK 2.0 2.0 1.936
IN2P3-LPSC_DATADISK 0.488 0.488 0.482
IN2P3-LPSC_MCDISK 2.0 2.0 1.518
RO-02-NIPNE_DATADISK 3.0 3.0 3.0
RO-02-NIPNE_MCDISK 1.0 1.0 0.995
RO-07-NIPNE_DATADISK 2.0 2.0 2.0
RO-07-NIPNE_MCDISK 1.0 1.0 0.654
TOKYO-LCG2_DATADISK 15.0 15.0 13.643
TOKYO-LCG2_MCDISK 10.0 10.0 10.0

=== 5th June ===

* Data produced on Tuesday are supposed to be shipped in this night.

;14h50
* Now the RAW data has begun to shipped to LYON.

;Throughput(snapshot)(MB/s)
[[Image:LYON.throughput.14400_RAW_20080605_1816.png]]

;Number of transfer errors (snapshot)
[[Image:LYON.num_file_xs_error.14400_20080605_1823.png]]

;21h30
* IN2P3-CC_MCDISK has some problem of staging file. But it should have nothing to do with T0 export.

* 47 datasets have been exported to T1s. The total is 90.

;Throughput(snapshot)(MB/s)
[[Image:LYON.throughput.14400 20080605 2140.png]]

;Number of transfer errors (snapshot) -- It's the problem of staging file in T1 and nothing to do with FDR transfers.
[[Image:LYON.num_file_xs_error.14400_20080605_2140.png]]

=== 6th June ===
;09h50

;Throughput(snapshot) (MB/s)
[[Image:LYON.throughput.14400 200806060 0958.png]]

;Number of transfer errors (snapshot)
[[Image:LYON.num file xs error.14400 20080606 0959.png]]

;10h03
* Now all the RAW datasets have been exported to T1. The total number is 90.
http://atlas.web.cern.ch/Atlas/tzero/prod2/monitoring/tables/ddm_RAW_status.html

=== 7th June ===
* at CERN, will re-start the bulk processing with using the upcoming AtlasPoint1-14.1.0.12 release cache.

;11h30
* SRM in LYON is still in bad shape. Experts are working on it.

=== 8th June ===
;14h20
* FDR2 AODs/DPDs/TAGs/ESDs are being exported to LYON (IN2P3-CC_DATADISK), but haven't been shipped to T2s.

;Number of datasets transfered by far
$ dq2-list-dataset-site IN2P3-CC_DATADISK|grep fdr08_run2.*|wc -l
68

;Throughput(T0-T1,snapshot)(MB/s)
[[Image:LYON.throughput.14400 200806081420.png]]

;Number of transfer errors
[[Image:LYON.num file xs error.14400 200806081420.png]]

;Throughput(T1-T2,snapshot)(MB/s)
[[Image:LYON.T2.throughput.14400 T1-T2 200806081420.png]]

=== 9th June ===
;0h20
* FDR datasets has begun to be shipped to T2s.
* Some T2s (e.g. BEIJING) subscribed dedicated datasets, which will be done later in today's morning.

;Shipped datasets

IN2P3-CC_DATADISK 315
BEIJING-LCG2_DATADISK 0
GRIF-LAL_DATADISK 79
GRIF-LPNHE_DATADISK 39
GRIF-SACLAY_DATADISK 38
IN2P3-CPPM_DATADISK 0
IN2P3-LAPP_DATADISK 12
IN2P3-LPC_DATADISK 12
IN2P3-LPSC_DATADISK 32
RO-02-NIPNE_DATADISK 29
RO-07-NIPNE_DATADISK 46
TOKYO-LCG2_DATADISK 134

;Free space

BEIJING-LCG2_DATADISK 6.366 6.366 6.343
GRIF-LAL_DATADISK 6.0 6.0 5.579
GRIF-LPNHE_DATADISK 3.906 3.906 3.817
GRIF-SACLAY_DATADISK 6.0 6.0 5.927
IN2P3-CPPM_DATADISK 1.0 1.0 0.977
IN2P3-LAPP_DATADISK 4.0 4.0 4.0
IN2P3-LPC_DATADISK 0 0 0
IN2P3-LPSC_DATADISK 0.488 0.488 0.439
RO-02-NIPNE_DATADISK 3.0 3.0 2.986
RO-07-NIPNE_DATADISK 2.0 2.0 1.896
TOKYO-LCG2_DATADISK 15.0 15.0 13.313

;Troughputs(snapshot)(MB/s)
[[Image:LYON.throughput.14400 200806090046.png]]

;Number of transfer erros(snapshot)
* Note: No errors found in FDR2 transfers. Errors showed in this snapshot are since other production data transfers.
[[Image:LYON.num file xs error.14400 200806090047.png]]

;09h24

;Troughputs(snapshot)(MB/s)
[[Image:LYON.throughput.14400 200806090922.png]]

;Number of transfer erros(snapshot)
* Note: No errors found in FDR2 transfers. Errors showed in this snapshot are since other production data transfers.
[[Image:LYON.num file xs error.14400 20806090922.png]]

;Shipped datasets

IN2P3-CC_DATADISK 319
BEIJING-LCG2_DATADISK 0
GRIF-LAL_DATADISK 79
GRIF-LPNHE_DATADISK 39
GRIF-SACLAY_DATADISK 38
IN2P3-CPPM_DATADISK 0
IN2P3-LAPP_DATADISK 12
IN2P3-LPC_DATADISK 12
IN2P3-LPSC_DATADISK 32
RO-02-NIPNE_DATADISK 59
RO-07-NIPNE_DATADISK 49
TOKYO-LCG2_DATADISK 134

;Free space
BEIJING-LCG2_DATADISK 6.366 6.366 6.343
GRIF-LAL_DATADISK 6.0 6.0 5.579
GRIF-LPNHE_DATADISK 3.906 3.906 3.817
GRIF-SACLAY_DATADISK 6.0 6.0 5.927
IN2P3-CPPM_DATADISK 1.0 1.0 0.977
IN2P3-LAPP_DATADISK 4.0 4.0 4.0
IN2P3-LPC_DATADISK 3.0 3.0 3.0
IN2P3-LPSC_DATADISK 0.488 0.488 0.439
RO-02-NIPNE_DATADISK 3.0 3.0 2.945
RO-07-NIPNE_DATADISK 2.0 2.0 1.885
TOKYO-LCG2_DATADISK 15.0 15.0 13.313

;14h00
Many failed with RO-02-NIPNE_DATADISK.

[FTS] FTS State [Failed] FTS Retries [1] Reason [TRANSFER error during TRANSFER phase: [PERMISSION] the server sent an error response: 550 550 rfio write failure: Permission denied.] Source Host [ccsrm.in2p3.fr] 16
[FTS] FTS State [Failed] FTS Retries [1] Reason [TRANSFER error during TRANSFER phase: [TRANSFER_TIMEOUT] gridftp_copy_wait: Connection timed out] Source Host [ccsrm.in2p3.fr] 9
[FTS] FTS State [Failed] FTS Retries [1] Reason [TRANSFER error during TRANSFER phase: [GRIDFTP] the server sent an error response: 426 426 Transfer aborted (Unexpected Exception : java.io.IOException: Broken pipe)] Source Host [ccsrm.in2p3.fr] 2
[FTS] FTS State [Failed] FTS Retries [1] Reason [The process serving the transfer (status = PREPARING) is no longer active (could not open file /proc/3775/cmdline)] Source Host [ccsrm.in2p3.fr] 1
[FTS] FTS State [Failed] FTS Retries [1] Reason [The process serving the transfer (status = PREPARING) is no longer active (could not open file /proc/30076/cmdline)] Source Host [ccsrm.in2p3.fr] 1

;20h10
* FDR datasets have finished transferred to all T2s except BEIJING-LCG2_DATADISK and RO-02-NIPNE_DATADISK.
* Many "Connection timed out" errors found at BEIJING-LCG2_DATADISK and RO-02-NIPNE_DATADISK.

[FTS] FTS State [Failed] FTS Retries [1] Reason [SOURCE error during PREPARATION phase: [CONNECTION] failed to contact on remote SRM [httpg://ccsrm.in2p3.fr:8443/srm/managerv2]. Givin' up after 3 tries] Source Host [ccsrm.in2p3.fr] 111
[FTS] FTS State [Failed] FTS Retries [1] Reason [TRANSFER error during TRANSFER phase: [TRANSFER_TIMEOUT] gridftp_copy_wait: Connection timed out] Source Host [ccsrm.in2p3.fr] 95
[FTS] FTS State [Failed] FTS Retries [1] Reason [SOURCE error during PREPARATION phase: [REQUEST_TIMEOUT] failed to prepare source file in 180 seconds] Source Host [ccsrm.in2p3.fr] 3

; Shipped datasets

IN2P3-CC_DATADISK 134
BEIJING-LCG2_DATADISK 79
GRIF-LAL_DATADISK 67
GRIF-LPNHE_DATADISK 32
GRIF-SACLAY_DATADISK 26
IN2P3-CPPM_DATADISK 52
IN2P3-LAPP_DATADISK 52
IN2P3-LPC_DATADISK 52
IN2P3-LPSC_DATADISK 32
RO-02-NIPNE_DATADISK 49
RO-07-NIPNE_DATADISK 49
TOKYO-LCG2_DATADISK 126

=== 10th June ===

;Summary

All T2s except BEIJING-LCG2_DATADISK and RO-02-NIPNE_DATADISK were in good status and received all the exported datasets when FDR transferring.
The problem of "Connection timed out" with transfers to BEIJING will be hopefully solved by tuned the transfer duration >1800s by Lionel.
RO-02-NIPNE is said being upgraded from SL3 to SL4.

=== 11st June ===

; Anouncement
All FDR2 datasets will be deleted from all sites since part of the detector was not included in the reconstruction. It was not detected by the Data Quality because it is run on the express stream which is correct.

Old datasets will be deleted centrally by Stephane and new datasets that would be named begin with "ccrc08-run.0024*" will be re-generated and re-exported soon in today.

T2s will receive the same share rate as requested for FDR (except that DPD are not produced).

=== 12nd June ===
;01h20
* FDR2 datasets have been re-exported to T2s. For this moment, all T2 sites seems in good shape to accept datasets except RO-07-NIPNE and RO-02-NIPNE that are in downtime.

* Datasets shipped:

IN2P3-CC_DATADISK 387
BEIJING-LCG2_DATADISK 46
GRIF-LAL_DATADISK 141
GRIF-LPNHE_DATADISK 79
GRIF-SACLAY_DATADISK 90
IN2P3-CPPM_DATADISK 22
IN2P3-LAPP_DATADISK 22
IN2P3-LPC_DATADISK 36
IN2P3-LPSC_DATADISK 17
RO-02-NIPNE_DATADISK 20
RO-07-NIPNE_DATADISK 33
TOKYO-LCG2_DATADISK 310

;Troughput
[[Image:LYON.T2.throughput.14400 200806120120.png]]

;Transfer errors
* Note that the errors showed in this snapshot involves all transfers mixed with other transfers(e.g MC/Panda). So it may not reflect the real situation.
[[Image:LYON.T2.num file xs error.14400 200806120120.png]]

Atlas:FDR2

2008-06-11T23:38:06Z

Erming: /* 12nd June */

= Monitoring pages/graphs =
=== FTS ===
http://cctoolsafs.in2p3.fr/fts/monitoring/prod/ftsmonitor.php?vo=atlas

=== T0-T1 (ALL) ===
; FDR monitoring page - T1s
http://atldq2pro.cern.ch:8000/ft/mon/fdrmon_tier1s.html

http://dashb-atlas-data-tier0.cern.ch/dashboard/request.py/site
*Throughput 
:http://dashb-atlas-data-tier0.cern.ch/dashboard/templates/plots/OVERVIEW.throughput.14400.png
*Errors 
:http://dashb-atlas-data-tier0.cern.ch/dashboard/templates/plots/OVERVIEW.num_file_xs_error.14400.png

=== T0-T1 (Lyon) ===
http://dashb-atlas-data-tier0.cern.ch/dashboard/request.py/site?statsInterval=4&name=LYON
*Throughput T0-LYON 
:http://dashb-atlas-data-tier0.cern.ch/dashboard/templates/plots/LYON.throughput.14400.png
*Errors T0-LYON 
:http://dashb-atlas-data-tier0.cern.ch/dashboard/templates/plots/LYON.num_file_xs_error.14400.png
*Throughput Others-LYON 
:http://dashb-atlas-data.cern.ch/dashboard/templates/plots/LYON.T1.throughput.14400.png
*Errors Others-LYON :http://dashb-atlas-data.cern.ch/dashboard/templates/plots/LYON.T1.num_file_xs_error.14400.png

=== T1-T1 ===

=== T1-T2 (Lyon) ===
; FDR monitoring page
http://atldq2pro.cern.ch:8000/ft/mon/fdrmon_TiersInfo.html

http://atldq2pro.cern.ch:8000/ft/mon/fdrmon_T1-T2_matrix_day.html

; Dashboard
http://dashb-atlas-data.cern.ch/dashboard/request.py/site?statsInterval=4&name=LYON
*Throughput 
:http://dashb-atlas-data.cern.ch/dashboard/templates/plots/LYON.T2.throughput.14400.png
*Errors
:http://dashb-atlas-data.cern.ch/dashboard/templates/plots/LYON.T2.num_file_xs_error.14400.png

=== Network Graphs ===
; IN2P3 Weathermap
* http://netstat.in2p3.fr/weathermap/
* http://netstat.in2p3.fr/weathermap/graphiques/

; cc-in2p3 : [http://netstat.in2p3.fr/weathermap/graphiques/lyo-cern.html lyo-cern] (lhcopn-in2p3.cern.ch) http://netstat.in2p3.fr/weathermap/graphiques/lyo-cern-daily.gif
; lyon-nord : [http://netstat.in2p3.fr/weathermap/graphiques/lyo-nrd.html lyo-nrd] http://netstat.in2p3.fr/weathermap/graphiques/lyo-nrd-daily.gif
; orsay : [http://netstat.in2p3.fr/weathermap/graphiques/orsay.html orsay] (in2p3-orsay.cssi.renater.fr http://netstat.in2p3.fr/weathermap/graphiques/orsay-daily.gif
; lal-cc : [http://netstat.in2p3.fr/weathermap/graphiques/lal-cc.html Liaison LAL - CC] http://netstat.in2p3.fr/weathermap/graphiques/lal-cc-daily.gif
; lal : [http://netstat.in2p3.fr/weathermap/graphiques/lal.html lal] http://netstat.in2p3.fr/weathermap/graphiques/lal-daily.gif
; lpnhe: [http://netstat.in2p3.fr/weathermap/graphiques/lpnhe-nrd.html lpnhe-nrd] (in2p3-jussieu.cssi.renater.fr) http://netstat.in2p3.fr/weathermap/graphiques/lpnhe-nrd-daily.gif [http://netstat.in2p3.fr/weathermap/graphiques/lpnhe.html lpnhe] (Paris-LPNHE.in2p3.fr) http://netstat.in2p3.fr/weathermap/graphiques/lpnhe-daily.gif
; lapp : [http://netstat.in2p3.fr/weathermap/graphiques/ann-nrd.html ann-nrd] http://netstat.in2p3.fr/weathermap/graphiques/ann-nrd-daily.gif
; lpc: [http://netstat.in2p3.fr/weathermap/graphiques/lpc-cf.html lpc-cf]
http://netstat.in2p3.fr/weathermap/graphiques/lpc-cf-daily.gif
; lpsc : [http://netstat.in2p3.fr/weathermap/graphiques/lpsc.html lpsc] http://netstat.in2p3.fr/weathermap/graphiques/lpsc-daily.gif
; cppm : [http://netstat.in2p3.fr/weathermap/graphiques/cppm.html cppm] 
http://netstat.in2p3.fr/weathermap/graphiques/cppm-daily.gif
; international (tokyo etc.) : [http://netstat.in2p3.fr/weathermap/graphiques/parisnrd.html parisnrd] 
http://netstat.in2p3.fr/weathermap/graphiques/parisnrd-daily.gif
; GEANT - NYC : [http://dc-snmp.wcc.grnoc.iu.edu/manlan/show-graph.cgi?title=sw.newy32aoa.manlan.internet2.edu--te10/1&rrdname=sw.newy32aoa.manlan.internet2.edu--te10_1.rrd MANLAN] 
http://dc-snmp.wcc.grnoc.iu.edu/manlan/img/sw.newy32aoa.manlan.internet2.edu--te10_1-std5.gif
; NYC - TOKYO : [http://dc-snmp.wcc.grnoc.iu.edu/manlan/show-graph.cgi?title=sw.newy32aoa.manlan.internet2.edu--te11/1&rrdname=sw.newy32aoa.manlan.internet2.edu--te11_1.rrd MANLAN] 
http://dc-snmp.wcc.grnoc.iu.edu/manlan/img/sw.newy32aoa.manlan.internet2.edu--te11_1-std5.gif
; tokyo : [http://www.icepp.jp/monitor/gridftp/monitor.html gridftp monitor] http://www.icepp.jp/monitor/gridftp/img/all.png

= T0-T1 =

= T1-T2 =
==== General informations ====
;FDR page:
https://twiki.cern.ch/twiki/bin/view/Atlas/FullDressRehearsal

;DDM_in_FRD2: https://twiki.cern.ch/twiki/bin/view/Atlas/DDMOperationsGroup#DDM_in_FRD2

;Replication status(checked ~ every 2h?)
http://pandamon.usatlas.bnl.gov:25880/server/pandamon/query?mode=listFDR

;Lyon FTS monitor: http://cctoolsafs.in2p3.fr/fts/monitoring/prod/ftsmonitor.php?vo=atlas

;Contacts:

* IN2P3-CPPM_DATADISK :
* IN2P3-LPSC_DATADISK :
* IN2P3-LPC_DATADISK :
* GRIF-LAL_DATADISK : (grid.admin a lal.in2p3.fr)
* GRIF-LPNHE_DATADISK :
* GRIF-SACLAY_DATADISK :
* BEIJING-LCG2_DATADISK : (yanxf a ihep.ac.cn, Erming.Pei a cern.ch)
* RO-07-NIPNE_DATADISK : (ciubancan a nipne.ro)
* RO-02-NIPNE_DATADISK : (tpreda a nipne.ro)
* TOKYO-LCG2_DATADISK : (lcg-admin a icepp.s.u-tokyo.ac.jp)

===== Shares =====
Only srmv2+ space tokens sites will collect data. The list of streams is : Egamma, Bphysics, MinBias, Muon and Jet. If no precise stream name is associated to the share, the site will get any stream. The site in parenthesis will collect the complementary datasets. The T2 ressources at T1 means that users without production role can access CPUs at the T1 without the Production role.
* US :
o AGLT2_DATADISK : 100 %
o MWT2_DATADISK: 100 %
o NET2_DATADISK: 100 %
o SLACXRD_DATADISK: 100 %
o SWT2_CPB_DATADISK: 100 %
o T2 CPU ressources in BNL to access AOD
* CA :
o SFU-LCG2_DATADISK : 40%
o VICTORIA-LCG2_DATADISK : 30%
o ALBERTA-LCG2_DATADISK : 30%
o TORONTO-LCG2_DATADISK : 50 %
* IT :
o INFN-MILANO_DATADISK : 25%
o INFN-FRASCATI_DATADISK : 25%
o INFN-ROMA1_DATADISK : 100%
o INFN-NAPOLI-ATLAS_DATADISK : 100 %
* TW :
o TW-FTT_DATADISK : 50 %
* NL
o JINR-LCG2_DATADISK : 50% Jet (RU-PROTVINO-IHEP_DATADISK)
o JINR-LCG2_DATADISK : 50% MinBias (RU-PROTVINO-IHEP_DATADISK)
o JINR-LCG2_DATADISK : 50% Muon (RU-PROTVINO-IHEP_DATADISK)
o JINR-LCG2_DATADISK : 50% Bphysics (RU-PNPI_DATADISK)
o RU-PROTVINO-IHEP_DATADISK : 50% Jet (JINR-LCG2_DATADISK)
o RU-PROTVINO-IHEP_DATADISK : 50% Muon (JINR-LCG2_DATADISK)
o RU-PROTVINO-IHEP_DATADISK : 50% MinBias (JINR-LCG2_DATADISK)
o RU-PROTVINO-IHEP_DATADISK : 50% Egamma (RU-PNPI_DATADISK)
o RU-PNPI_DATADISK : 50% Egamma (RU-PROTVINO-IHEP_DATADISK)
o RU-PNPI_DATADISK : 50% Bphysics (JINR-LCG2_DATADISK)
o RU-PNPI_DATADISK : 50% Jet (RRC-KI_DATADISK)
o RU-PNPI_DATADISK : 50% MinBias (RRC-KI_DATADISK)
o RRC-KI_DATADISK : 50% Jet (RU-PNPI_DATADISK)
o RRC-KI_DATADISK : 50% MinBias (RU-PNPI_DATADISK)
o RRC-KI_DATADISK : 50% Egamma
o RRC-KI_DATADISK : 50% Bphysics
o CSTCDIE_DATADISK : 20%
* FR :
o IN2P3-LAPP_DATADISK : 100 % Egamma
o IN2P3-CPPM_DATADISK : 100 % Egamma
o IN2P3-LPSC_DATADISK : 5 %
o IN2P3-LPC_DATADISK : 100% Muon, 50% Egamma
o BEIJING-LCG2_DATADISK : 100 % Egamma, 50% Muon, 35% MinBias
o RO-07-NIPNE_DATADISK : 10%
o RO-02-NIPNE_DATADISK : 10%
o GRIF-LAL_DATADISK : 45%
o GRIF-LPNHE_DATADISK : 25%
o GRIF-SACLAY_DATADISK : 30 %
o TOKYO-LCG2_DATADISK : 100%
o T2 CPU ressources in LYON to access AOD
* DE :
o DESY-HH_DATADISK : 100%
o CSCS-LCG2_DATADISK: 100%
o DESY-ZN_DATADISK : 50%
o GOEGRID_DATADISK : 50%
o WUPPERTALPROD_DATADISK: 25%
o CYFRONET-LCG2_DATADISK : 15%
o PRAGUELCG2_DATADISK : 25%
o LRZ-LMU_DATADISK : 35%
o UNI-FREIBURG_DATADISK : 100% Jet, 100% Muon
* UK :
o UKI-SOUTHGRID-CAM-HEP_DATADISK : 50 % Muon (UKI-NORTHGRID-LANCS-HEP_DATADISK
o UKI-NORTHGRID-LANCS-HEP_DATADISK : 50 % Muon (UKI-SOUTHGRID-CAM-HEP_DATADISK)
o UKI-NORTHGRID-SHEF-HEP_DATADISK : 50 % MinBias (UKI-SCOTGRID-GLASGOW_DATADISK)
o UKI-SCOTGRID-GLASGOW_DATADISK : 50 % MinBias (UKI-NORTHGRID-SHEF-HEP_DATADISK)
o UKI-LT2-RHUL_DATADISK : 50 % Egamma (UKI-NORTHGRID-LIV-HEP_DATADISK)
o UKI-NORTHGRID-LIV-HEP_DATADISK : 50 % Egamma (UKI-LT2-RHUL_DATADISK)
o UKI-SOUTHGRID-RALPP_DATADISK : 50 Egamma (UKI-LT2-RHUL_DATADISK)
o UKI-SOUTHGRID-BHAM-HEP_DATADISK : 50 Jet (UKI-SOUTHGRID-OX-HEP_DATADISK)
* ES :
o IFIC-LCG2_DATADISK: 50%
o UAM-LCG2_DATADISK: 25%
o IFAE_DATADISK: 25%
* NDGF :
o SIGNET_DATADISK : 50 %

== Logbook ==

=== 2nd June ===

* Since Lyon is in donwtime, so there's no transfers from LYON to T2s today.
http://cc.in2p3.fr/cc_accueil.php3?lang=fr

*Free Space

Token Total Guaranteed Unused
------------------------------------------
BEIJING-LCG2_DATADISK 6.366 6.366 6.343
GRIF-LAL_DATADISK 6.0 6.0 5.732
GRIF-LPNHE_DATADISK 3.906 3.906 3.89
GRIF-SACLAY_DATADISK 6.0 6.0 5.994
IN2P3-CPPM_DATADISK 1.0 1.0 0.977
IN2P3-LAPP_DATADISK 4.0 4.0 4.0
IN2P3-LPC_DATADISK 3.0 3.0 3.0
IN2P3-LPSC_DATADISK 0.488 0.488 0.482
RO-02-NIPNE_DATADISK 3.0 3.0 3.0
RO-07-NIPNE_DATADISK 2.0 2.0 2.0
TOKYO-LCG2_DATADISK 15.0 15.0 13.643

=== 3rd June ===

* Lyon is still in donwtime, no transfers from LYON to T2s in this morning.

; Throughput(snapshot) (MB/s) : [[Image:LYON.throughput.14400_20080603.png]]
; Number of transfer errors(snapshot) : [[Image:LYON.num_file_xs_error.14400_20080603.png]]

=== 4th June ===
;22h02
* Still no transfers, desipte that all the services in LYON shows OK.

;Throughput(snapshot)
[[Image:LYON.throughput.3600_20080604.png]]

;Services in LYON
[[Image:services_cc-in2p3_2008060422h15.PNG]]

;Free space

BEIJING-LCG2_DATADISK 6.366 6.366 6.343
BEIJING-LCG2_MCDISK 5.457 5.457 5.457
GRIF-LAL_DATADISK 6.0 6.0 5.732
GRIF-LAL_MCDISK 7.0 7.0 3.724
GRIF-LPNHE_DATADISK 3.906 3.906 3.89
GRIF-LPNHE_MCDISK 1.0 1.0 0.194
GRIF-SACLAY_DATADISK 6.0 6.0 5.994
GRIF-SACLAY_MCDISK 2.0 2.0 1.713
IN2P3-CPPM_DATADISK 1.0 1.0 0.977
IN2P3-CPPM_MCDISK 0.781 0.781 0.765
IN2P3-LAPP_DATADISK 4.0 4.0 4.0
IN2P3-LAPP_MCDISK 2.0 2.0 0.745
IN2P3-LPC_DATADISK 3.0 3.0 3.0
IN2P3-LPC_MCDISK 2.0 2.0 1.936
IN2P3-LPSC_DATADISK 0.488 0.488 0.482
IN2P3-LPSC_MCDISK 2.0 2.0 1.518
RO-02-NIPNE_DATADISK 3.0 3.0 3.0
RO-02-NIPNE_MCDISK 1.0 1.0 0.995
RO-07-NIPNE_DATADISK 2.0 2.0 2.0
RO-07-NIPNE_MCDISK 1.0 1.0 0.654
TOKYO-LCG2_DATADISK 15.0 15.0 13.643
TOKYO-LCG2_MCDISK 10.0 10.0 10.0

=== 5th June ===

* Data produced on Tuesday are supposed to be shipped in this night.

;14h50
* Now the RAW data has begun to shipped to LYON.

;Throughput(snapshot)(MB/s)
[[Image:LYON.throughput.14400_RAW_20080605_1816.png]]

;Number of transfer errors (snapshot)
[[Image:LYON.num_file_xs_error.14400_20080605_1823.png]]

;21h30
* IN2P3-CC_MCDISK has some problem of staging file. But it should have nothing to do with T0 export.

* 47 datasets have been exported to T1s. The total is 90.

;Throughput(snapshot)(MB/s)
[[Image:LYON.throughput.14400 20080605 2140.png]]

;Number of transfer errors (snapshot) -- It's the problem of staging file in T1 and nothing to do with FDR transfers.
[[Image:LYON.num_file_xs_error.14400_20080605_2140.png]]

=== 6th June ===
;09h50

;Throughput(snapshot) (MB/s)
[[Image:LYON.throughput.14400 200806060 0958.png]]

;Number of transfer errors (snapshot)
[[Image:LYON.num file xs error.14400 20080606 0959.png]]

;10h03
* Now all the RAW datasets have been exported to T1. The total number is 90.
http://atlas.web.cern.ch/Atlas/tzero/prod2/monitoring/tables/ddm_RAW_status.html

=== 7th June ===
* at CERN, will re-start the bulk processing with using the upcoming AtlasPoint1-14.1.0.12 release cache.

;11h30
* SRM in LYON is still in bad shape. Experts are working on it.

=== 8th June ===
;14h20
* FDR2 AODs/DPDs/TAGs/ESDs are being exported to LYON (IN2P3-CC_DATADISK), but haven't been shipped to T2s.

;Number of datasets transfered by far
$ dq2-list-dataset-site IN2P3-CC_DATADISK|grep fdr08_run2.*|wc -l
68

;Throughput(T0-T1,snapshot)(MB/s)
[[Image:LYON.throughput.14400 200806081420.png]]

;Number of transfer errors
[[Image:LYON.num file xs error.14400 200806081420.png]]

;Throughput(T1-T2,snapshot)(MB/s)
[[Image:LYON.T2.throughput.14400 T1-T2 200806081420.png]]

=== 9th June ===
;0h20
* FDR datasets has begun to be shipped to T2s.
* Some T2s (e.g. BEIJING) subscribed dedicated datasets, which will be done later in today's morning.

;Shipped datasets

IN2P3-CC_DATADISK 315
BEIJING-LCG2_DATADISK 0
GRIF-LAL_DATADISK 79
GRIF-LPNHE_DATADISK 39
GRIF-SACLAY_DATADISK 38
IN2P3-CPPM_DATADISK 0
IN2P3-LAPP_DATADISK 12
IN2P3-LPC_DATADISK 12
IN2P3-LPSC_DATADISK 32
RO-02-NIPNE_DATADISK 29
RO-07-NIPNE_DATADISK 46
TOKYO-LCG2_DATADISK 134

;Free space

BEIJING-LCG2_DATADISK 6.366 6.366 6.343
GRIF-LAL_DATADISK 6.0 6.0 5.579
GRIF-LPNHE_DATADISK 3.906 3.906 3.817
GRIF-SACLAY_DATADISK 6.0 6.0 5.927
IN2P3-CPPM_DATADISK 1.0 1.0 0.977
IN2P3-LAPP_DATADISK 4.0 4.0 4.0
IN2P3-LPC_DATADISK 0 0 0
IN2P3-LPSC_DATADISK 0.488 0.488 0.439
RO-02-NIPNE_DATADISK 3.0 3.0 2.986
RO-07-NIPNE_DATADISK 2.0 2.0 1.896
TOKYO-LCG2_DATADISK 15.0 15.0 13.313

;Troughputs(snapshot)(MB/s)
[[Image:LYON.throughput.14400 200806090046.png]]

;Number of transfer erros(snapshot)
* Note: No errors found in FDR2 transfers. Errors showed in this snapshot are since other production data transfers.
[[Image:LYON.num file xs error.14400 200806090047.png]]

;09h24

;Troughputs(snapshot)(MB/s)
[[Image:LYON.throughput.14400 200806090922.png]]

;Number of transfer erros(snapshot)
* Note: No errors found in FDR2 transfers. Errors showed in this snapshot are since other production data transfers.
[[Image:LYON.num file xs error.14400 20806090922.png]]

;Shipped datasets

IN2P3-CC_DATADISK 319
BEIJING-LCG2_DATADISK 0
GRIF-LAL_DATADISK 79
GRIF-LPNHE_DATADISK 39
GRIF-SACLAY_DATADISK 38
IN2P3-CPPM_DATADISK 0
IN2P3-LAPP_DATADISK 12
IN2P3-LPC_DATADISK 12
IN2P3-LPSC_DATADISK 32
RO-02-NIPNE_DATADISK 59
RO-07-NIPNE_DATADISK 49
TOKYO-LCG2_DATADISK 134

;Free space
BEIJING-LCG2_DATADISK 6.366 6.366 6.343
GRIF-LAL_DATADISK 6.0 6.0 5.579
GRIF-LPNHE_DATADISK 3.906 3.906 3.817
GRIF-SACLAY_DATADISK 6.0 6.0 5.927
IN2P3-CPPM_DATADISK 1.0 1.0 0.977
IN2P3-LAPP_DATADISK 4.0 4.0 4.0
IN2P3-LPC_DATADISK 3.0 3.0 3.0
IN2P3-LPSC_DATADISK 0.488 0.488 0.439
RO-02-NIPNE_DATADISK 3.0 3.0 2.945
RO-07-NIPNE_DATADISK 2.0 2.0 1.885
TOKYO-LCG2_DATADISK 15.0 15.0 13.313

;14h00
Many failed with RO-02-NIPNE_DATADISK.

[FTS] FTS State [Failed] FTS Retries [1] Reason [TRANSFER error during TRANSFER phase: [PERMISSION] the server sent an error response: 550 550 rfio write failure: Permission denied.] Source Host [ccsrm.in2p3.fr] 16
[FTS] FTS State [Failed] FTS Retries [1] Reason [TRANSFER error during TRANSFER phase: [TRANSFER_TIMEOUT] gridftp_copy_wait: Connection timed out] Source Host [ccsrm.in2p3.fr] 9
[FTS] FTS State [Failed] FTS Retries [1] Reason [TRANSFER error during TRANSFER phase: [GRIDFTP] the server sent an error response: 426 426 Transfer aborted (Unexpected Exception : java.io.IOException: Broken pipe)] Source Host [ccsrm.in2p3.fr] 2
[FTS] FTS State [Failed] FTS Retries [1] Reason [The process serving the transfer (status = PREPARING) is no longer active (could not open file /proc/3775/cmdline)] Source Host [ccsrm.in2p3.fr] 1
[FTS] FTS State [Failed] FTS Retries [1] Reason [The process serving the transfer (status = PREPARING) is no longer active (could not open file /proc/30076/cmdline)] Source Host [ccsrm.in2p3.fr] 1

;20h10
* FDR datasets have finished transferred to all T2s except BEIJING-LCG2_DATADISK and RO-02-NIPNE_DATADISK.
* Many "Connection timed out" errors found at BEIJING-LCG2_DATADISK and RO-02-NIPNE_DATADISK.

[FTS] FTS State [Failed] FTS Retries [1] Reason [SOURCE error during PREPARATION phase: [CONNECTION] failed to contact on remote SRM [httpg://ccsrm.in2p3.fr:8443/srm/managerv2]. Givin' up after 3 tries] Source Host [ccsrm.in2p3.fr] 111
[FTS] FTS State [Failed] FTS Retries [1] Reason [TRANSFER error during TRANSFER phase: [TRANSFER_TIMEOUT] gridftp_copy_wait: Connection timed out] Source Host [ccsrm.in2p3.fr] 95
[FTS] FTS State [Failed] FTS Retries [1] Reason [SOURCE error during PREPARATION phase: [REQUEST_TIMEOUT] failed to prepare source file in 180 seconds] Source Host [ccsrm.in2p3.fr] 3

; Shipped datasets

IN2P3-CC_DATADISK 134
BEIJING-LCG2_DATADISK 79
GRIF-LAL_DATADISK 67
GRIF-LPNHE_DATADISK 32
GRIF-SACLAY_DATADISK 26
IN2P3-CPPM_DATADISK 52
IN2P3-LAPP_DATADISK 52
IN2P3-LPC_DATADISK 52
IN2P3-LPSC_DATADISK 32
RO-02-NIPNE_DATADISK 49
RO-07-NIPNE_DATADISK 49
TOKYO-LCG2_DATADISK 126

=== 10th June ===

;Summary

All T2s except BEIJING-LCG2_DATADISK and RO-02-NIPNE_DATADISK were in good status and received all the exported datasets when FDR transferring.
The problem of "Connection timed out" with transfers to BEIJING will be hopefully solved by tuned the transfer duration >1800s by Lionel.
RO-02-NIPNE is said being upgraded from SL3 to SL4.

=== 11st June ===

; Anouncement
All FDR2 datasets will be deleted from all sites since part of the detector was not included in the reconstruction. It was not detected by the Data Quality because it is run on the express stream which is correct.

Old datasets will be deleted centrally by Stephane and new datasets that would be named begin with "ccrc08-run.0024*" will be re-generated and re-exported soon in today.

T2s will receive the same share rate as requested for FDR (except that DPD are not produced).

=== 12nd June ===
;01h20
* FDR2 datasets have been re-exported to T2s. For this moment, all T2 sites seems in good shape to accept datasets except RO-07-NIPNE and RO-02-NIPNE which have a same slight error:
"LFC exception [Cannot connect to LFC [lfc://lfc-prod.in2p3.fr:/grid/atlas]]"

* Datasets shipped:

IN2P3-CC_DATADISK 387
BEIJING-LCG2_DATADISK 46
GRIF-LAL_DATADISK 141
GRIF-LPNHE_DATADISK 79
GRIF-SACLAY_DATADISK 90
IN2P3-CPPM_DATADISK 22
IN2P3-LAPP_DATADISK 22
IN2P3-LPC_DATADISK 36
IN2P3-LPSC_DATADISK 17
RO-02-NIPNE_DATADISK 20
RO-07-NIPNE_DATADISK 33
TOKYO-LCG2_DATADISK 310

;Troughput
[[Image:LYON.T2.throughput.14400 200806120120.png]]

;Transfer errors
[[Image:LYON.T2.num file xs error.14400 200806120120.png]]

File:LYON.T2.num file xs error.14400 200806120120.png

2008-06-11T23:36:49Z

Erming:

File:LYON.T2.throughput.14400 200806120120.png

2008-06-11T23:35:43Z

Erming:

Atlas:FDR2

2008-06-11T09:41:47Z

Erming: /* 10th June */

Atlas:FDR2

2008-06-11T09:39:33Z

Erming: /* 10th June */

= Monitoring pages/graphs =
=== FTS ===
http://cctoolsafs.in2p3.fr/fts/monitoring/prod/ftsmonitor.php?vo=atlas

=== T0-T1 (ALL) ===
; FDR monitoring page - T1s
http://atldq2pro.cern.ch:8000/ft/mon/fdrmon_tier1s.html

http://dashb-atlas-data-tier0.cern.ch/dashboard/request.py/site
*Throughput 
:http://dashb-atlas-data-tier0.cern.ch/dashboard/templates/plots/OVERVIEW.throughput.14400.png
*Errors 
:http://dashb-atlas-data-tier0.cern.ch/dashboard/templates/plots/OVERVIEW.num_file_xs_error.14400.png

=== T0-T1 (Lyon) ===
http://dashb-atlas-data-tier0.cern.ch/dashboard/request.py/site?statsInterval=4&name=LYON
*Throughput T0-LYON 
:http://dashb-atlas-data-tier0.cern.ch/dashboard/templates/plots/LYON.throughput.14400.png
*Errors T0-LYON 
:http://dashb-atlas-data-tier0.cern.ch/dashboard/templates/plots/LYON.num_file_xs_error.14400.png
*Throughput Others-LYON 
:http://dashb-atlas-data.cern.ch/dashboard/templates/plots/LYON.T1.throughput.14400.png
*Errors Others-LYON :http://dashb-atlas-data.cern.ch/dashboard/templates/plots/LYON.T1.num_file_xs_error.14400.png

=== T1-T1 ===

=== T1-T2 (Lyon) ===
; FDR monitoring page
http://atldq2pro.cern.ch:8000/ft/mon/fdrmon_TiersInfo.html

http://atldq2pro.cern.ch:8000/ft/mon/fdrmon_T1-T2_matrix_day.html

; Dashboard
http://dashb-atlas-data.cern.ch/dashboard/request.py/site?statsInterval=4&name=LYON
*Throughput 
:http://dashb-atlas-data.cern.ch/dashboard/templates/plots/LYON.T2.throughput.14400.png
*Errors
:http://dashb-atlas-data.cern.ch/dashboard/templates/plots/LYON.T2.num_file_xs_error.14400.png

=== Network Graphs ===
; IN2P3 Weathermap
* http://netstat.in2p3.fr/weathermap/
* http://netstat.in2p3.fr/weathermap/graphiques/

; cc-in2p3 : [http://netstat.in2p3.fr/weathermap/graphiques/lyo-cern.html lyo-cern] (lhcopn-in2p3.cern.ch) http://netstat.in2p3.fr/weathermap/graphiques/lyo-cern-daily.gif
; lyon-nord : [http://netstat.in2p3.fr/weathermap/graphiques/lyo-nrd.html lyo-nrd] http://netstat.in2p3.fr/weathermap/graphiques/lyo-nrd-daily.gif
; orsay : [http://netstat.in2p3.fr/weathermap/graphiques/orsay.html orsay] (in2p3-orsay.cssi.renater.fr http://netstat.in2p3.fr/weathermap/graphiques/orsay-daily.gif
; lal-cc : [http://netstat.in2p3.fr/weathermap/graphiques/lal-cc.html Liaison LAL - CC] http://netstat.in2p3.fr/weathermap/graphiques/lal-cc-daily.gif
; lal : [http://netstat.in2p3.fr/weathermap/graphiques/lal.html lal] http://netstat.in2p3.fr/weathermap/graphiques/lal-daily.gif
; lpnhe: [http://netstat.in2p3.fr/weathermap/graphiques/lpnhe-nrd.html lpnhe-nrd] (in2p3-jussieu.cssi.renater.fr) http://netstat.in2p3.fr/weathermap/graphiques/lpnhe-nrd-daily.gif [http://netstat.in2p3.fr/weathermap/graphiques/lpnhe.html lpnhe] (Paris-LPNHE.in2p3.fr) http://netstat.in2p3.fr/weathermap/graphiques/lpnhe-daily.gif
; lapp : [http://netstat.in2p3.fr/weathermap/graphiques/ann-nrd.html ann-nrd] http://netstat.in2p3.fr/weathermap/graphiques/ann-nrd-daily.gif
; lpc: [http://netstat.in2p3.fr/weathermap/graphiques/lpc-cf.html lpc-cf]
http://netstat.in2p3.fr/weathermap/graphiques/lpc-cf-daily.gif
; lpsc : [http://netstat.in2p3.fr/weathermap/graphiques/lpsc.html lpsc] http://netstat.in2p3.fr/weathermap/graphiques/lpsc-daily.gif
; cppm : [http://netstat.in2p3.fr/weathermap/graphiques/cppm.html cppm] 
http://netstat.in2p3.fr/weathermap/graphiques/cppm-daily.gif
; international (tokyo etc.) : [http://netstat.in2p3.fr/weathermap/graphiques/parisnrd.html parisnrd] 
http://netstat.in2p3.fr/weathermap/graphiques/parisnrd-daily.gif
; GEANT - NYC : [http://dc-snmp.wcc.grnoc.iu.edu/manlan/show-graph.cgi?title=sw.newy32aoa.manlan.internet2.edu--te10/1&rrdname=sw.newy32aoa.manlan.internet2.edu--te10_1.rrd MANLAN] 
http://dc-snmp.wcc.grnoc.iu.edu/manlan/img/sw.newy32aoa.manlan.internet2.edu--te10_1-std5.gif
; NYC - TOKYO : [http://dc-snmp.wcc.grnoc.iu.edu/manlan/show-graph.cgi?title=sw.newy32aoa.manlan.internet2.edu--te11/1&rrdname=sw.newy32aoa.manlan.internet2.edu--te11_1.rrd MANLAN] 
http://dc-snmp.wcc.grnoc.iu.edu/manlan/img/sw.newy32aoa.manlan.internet2.edu--te11_1-std5.gif
; tokyo : [http://www.icepp.jp/monitor/gridftp/monitor.html gridftp monitor] http://www.icepp.jp/monitor/gridftp/img/all.png

= T0-T1 =

= T1-T2 =
==== General informations ====
;FDR page:
https://twiki.cern.ch/twiki/bin/view/Atlas/FullDressRehearsal

;DDM_in_FRD2: https://twiki.cern.ch/twiki/bin/view/Atlas/DDMOperationsGroup#DDM_in_FRD2

;Replication status(checked ~ every 2h?)
http://pandamon.usatlas.bnl.gov:25880/server/pandamon/query?mode=listFDR

;Lyon FTS monitor: http://cctoolsafs.in2p3.fr/fts/monitoring/prod/ftsmonitor.php?vo=atlas

;Contacts:

* IN2P3-CPPM_DATADISK :
* IN2P3-LPSC_DATADISK :
* IN2P3-LPC_DATADISK :
* GRIF-LAL_DATADISK : (grid.admin a lal.in2p3.fr)
* GRIF-LPNHE_DATADISK :
* GRIF-SACLAY_DATADISK :
* BEIJING-LCG2_DATADISK : (yanxf a ihep.ac.cn, Erming.Pei a cern.ch)
* RO-07-NIPNE_DATADISK : (ciubancan a nipne.ro)
* RO-02-NIPNE_DATADISK : (tpreda a nipne.ro)
* TOKYO-LCG2_DATADISK : (lcg-admin a icepp.s.u-tokyo.ac.jp)

===== Shares =====
Only srmv2+ space tokens sites will collect data. The list of streams is : Egamma, Bphysics, MinBias, Muon and Jet. If no precise stream name is associated to the share, the site will get any stream. The site in parenthesis will collect the complementary datasets. The T2 ressources at T1 means that users without production role can access CPUs at the T1 without the Production role.
* US :
o AGLT2_DATADISK : 100 %
o MWT2_DATADISK: 100 %
o NET2_DATADISK: 100 %
o SLACXRD_DATADISK: 100 %
o SWT2_CPB_DATADISK: 100 %
o T2 CPU ressources in BNL to access AOD
* CA :
o SFU-LCG2_DATADISK : 40%
o VICTORIA-LCG2_DATADISK : 30%
o ALBERTA-LCG2_DATADISK : 30%
o TORONTO-LCG2_DATADISK : 50 %
* IT :
o INFN-MILANO_DATADISK : 25%
o INFN-FRASCATI_DATADISK : 25%
o INFN-ROMA1_DATADISK : 100%
o INFN-NAPOLI-ATLAS_DATADISK : 100 %
* TW :
o TW-FTT_DATADISK : 50 %
* NL
o JINR-LCG2_DATADISK : 50% Jet (RU-PROTVINO-IHEP_DATADISK)
o JINR-LCG2_DATADISK : 50% MinBias (RU-PROTVINO-IHEP_DATADISK)
o JINR-LCG2_DATADISK : 50% Muon (RU-PROTVINO-IHEP_DATADISK)
o JINR-LCG2_DATADISK : 50% Bphysics (RU-PNPI_DATADISK)
o RU-PROTVINO-IHEP_DATADISK : 50% Jet (JINR-LCG2_DATADISK)
o RU-PROTVINO-IHEP_DATADISK : 50% Muon (JINR-LCG2_DATADISK)
o RU-PROTVINO-IHEP_DATADISK : 50% MinBias (JINR-LCG2_DATADISK)
o RU-PROTVINO-IHEP_DATADISK : 50% Egamma (RU-PNPI_DATADISK)
o RU-PNPI_DATADISK : 50% Egamma (RU-PROTVINO-IHEP_DATADISK)
o RU-PNPI_DATADISK : 50% Bphysics (JINR-LCG2_DATADISK)
o RU-PNPI_DATADISK : 50% Jet (RRC-KI_DATADISK)
o RU-PNPI_DATADISK : 50% MinBias (RRC-KI_DATADISK)
o RRC-KI_DATADISK : 50% Jet (RU-PNPI_DATADISK)
o RRC-KI_DATADISK : 50% MinBias (RU-PNPI_DATADISK)
o RRC-KI_DATADISK : 50% Egamma
o RRC-KI_DATADISK : 50% Bphysics
o CSTCDIE_DATADISK : 20%
* FR :
o IN2P3-LAPP_DATADISK : 100 % Egamma
o IN2P3-CPPM_DATADISK : 100 % Egamma
o IN2P3-LPSC_DATADISK : 5 %
o IN2P3-LPC_DATADISK : 100% Muon, 50% Egamma
o BEIJING-LCG2_DATADISK : 100 % Egamma, 50% Muon, 35% MinBias
o RO-07-NIPNE_DATADISK : 10%
o RO-02-NIPNE_DATADISK : 10%
o GRIF-LAL_DATADISK : 45%
o GRIF-LPNHE_DATADISK : 25%
o GRIF-SACLAY_DATADISK : 30 %
o TOKYO-LCG2_DATADISK : 100%
o T2 CPU ressources in LYON to access AOD
* DE :
o DESY-HH_DATADISK : 100%
o CSCS-LCG2_DATADISK: 100%
o DESY-ZN_DATADISK : 50%
o GOEGRID_DATADISK : 50%
o WUPPERTALPROD_DATADISK: 25%
o CYFRONET-LCG2_DATADISK : 15%
o PRAGUELCG2_DATADISK : 25%
o LRZ-LMU_DATADISK : 35%
o UNI-FREIBURG_DATADISK : 100% Jet, 100% Muon
* UK :
o UKI-SOUTHGRID-CAM-HEP_DATADISK : 50 % Muon (UKI-NORTHGRID-LANCS-HEP_DATADISK
o UKI-NORTHGRID-LANCS-HEP_DATADISK : 50 % Muon (UKI-SOUTHGRID-CAM-HEP_DATADISK)
o UKI-NORTHGRID-SHEF-HEP_DATADISK : 50 % MinBias (UKI-SCOTGRID-GLASGOW_DATADISK)
o UKI-SCOTGRID-GLASGOW_DATADISK : 50 % MinBias (UKI-NORTHGRID-SHEF-HEP_DATADISK)
o UKI-LT2-RHUL_DATADISK : 50 % Egamma (UKI-NORTHGRID-LIV-HEP_DATADISK)
o UKI-NORTHGRID-LIV-HEP_DATADISK : 50 % Egamma (UKI-LT2-RHUL_DATADISK)
o UKI-SOUTHGRID-RALPP_DATADISK : 50 Egamma (UKI-LT2-RHUL_DATADISK)
o UKI-SOUTHGRID-BHAM-HEP_DATADISK : 50 Jet (UKI-SOUTHGRID-OX-HEP_DATADISK)
* ES :
o IFIC-LCG2_DATADISK: 50%
o UAM-LCG2_DATADISK: 25%
o IFAE_DATADISK: 25%
* NDGF :
o SIGNET_DATADISK : 50 %

== Logbook ==

=== 2nd June ===

* Since Lyon is in donwtime, so there's no transfers from LYON to T2s today.
http://cc.in2p3.fr/cc_accueil.php3?lang=fr

*Free Space

Token Total Guaranteed Unused
------------------------------------------
BEIJING-LCG2_DATADISK 6.366 6.366 6.343
GRIF-LAL_DATADISK 6.0 6.0 5.732
GRIF-LPNHE_DATADISK 3.906 3.906 3.89
GRIF-SACLAY_DATADISK 6.0 6.0 5.994
IN2P3-CPPM_DATADISK 1.0 1.0 0.977
IN2P3-LAPP_DATADISK 4.0 4.0 4.0
IN2P3-LPC_DATADISK 3.0 3.0 3.0
IN2P3-LPSC_DATADISK 0.488 0.488 0.482
RO-02-NIPNE_DATADISK 3.0 3.0 3.0
RO-07-NIPNE_DATADISK 2.0 2.0 2.0
TOKYO-LCG2_DATADISK 15.0 15.0 13.643

=== 3rd June ===

* Lyon is still in donwtime, no transfers from LYON to T2s in this morning.

; Throughput(snapshot) (MB/s) : [[Image:LYON.throughput.14400_20080603.png]]
; Number of transfer errors(snapshot) : [[Image:LYON.num_file_xs_error.14400_20080603.png]]

=== 4th June ===
;22h02
* Still no transfers, desipte that all the services in LYON shows OK.

;Throughput(snapshot)
[[Image:LYON.throughput.3600_20080604.png]]

;Services in LYON
[[Image:services_cc-in2p3_2008060422h15.PNG]]

;Free space

BEIJING-LCG2_DATADISK 6.366 6.366 6.343
BEIJING-LCG2_MCDISK 5.457 5.457 5.457
GRIF-LAL_DATADISK 6.0 6.0 5.732
GRIF-LAL_MCDISK 7.0 7.0 3.724
GRIF-LPNHE_DATADISK 3.906 3.906 3.89
GRIF-LPNHE_MCDISK 1.0 1.0 0.194
GRIF-SACLAY_DATADISK 6.0 6.0 5.994
GRIF-SACLAY_MCDISK 2.0 2.0 1.713
IN2P3-CPPM_DATADISK 1.0 1.0 0.977
IN2P3-CPPM_MCDISK 0.781 0.781 0.765
IN2P3-LAPP_DATADISK 4.0 4.0 4.0
IN2P3-LAPP_MCDISK 2.0 2.0 0.745
IN2P3-LPC_DATADISK 3.0 3.0 3.0
IN2P3-LPC_MCDISK 2.0 2.0 1.936
IN2P3-LPSC_DATADISK 0.488 0.488 0.482
IN2P3-LPSC_MCDISK 2.0 2.0 1.518
RO-02-NIPNE_DATADISK 3.0 3.0 3.0
RO-02-NIPNE_MCDISK 1.0 1.0 0.995
RO-07-NIPNE_DATADISK 2.0 2.0 2.0
RO-07-NIPNE_MCDISK 1.0 1.0 0.654
TOKYO-LCG2_DATADISK 15.0 15.0 13.643
TOKYO-LCG2_MCDISK 10.0 10.0 10.0

=== 5th June ===

* Data produced on Tuesday are supposed to be shipped in this night.

;14h50
* Now the RAW data has begun to shipped to LYON.

;Throughput(snapshot)(MB/s)
[[Image:LYON.throughput.14400_RAW_20080605_1816.png]]

;Number of transfer errors (snapshot)
[[Image:LYON.num_file_xs_error.14400_20080605_1823.png]]

;21h30
* IN2P3-CC_MCDISK has some problem of staging file. But it should have nothing to do with T0 export.

* 47 datasets have been exported to T1s. The total is 90.

;Throughput(snapshot)(MB/s)
[[Image:LYON.throughput.14400 20080605 2140.png]]

;Number of transfer errors (snapshot) -- It's the problem of staging file in T1 and nothing to do with FDR transfers.
[[Image:LYON.num_file_xs_error.14400_20080605_2140.png]]

=== 6th June ===
;09h50

;Throughput(snapshot) (MB/s)
[[Image:LYON.throughput.14400 200806060 0958.png]]

;Number of transfer errors (snapshot)
[[Image:LYON.num file xs error.14400 20080606 0959.png]]

;10h03
* Now all the RAW datasets have been exported to T1. The total number is 90.
http://atlas.web.cern.ch/Atlas/tzero/prod2/monitoring/tables/ddm_RAW_status.html

=== 7th June ===
* at CERN, will re-start the bulk processing with using the upcoming AtlasPoint1-14.1.0.12 release cache.

;11h30
* SRM in LYON is still in bad shape. Experts are working on it.

=== 8th June ===
;14h20
* FDR2 AODs/DPDs/TAGs/ESDs are being exported to LYON (IN2P3-CC_DATADISK), but haven't been shipped to T2s.

;Number of datasets transfered by far
$ dq2-list-dataset-site IN2P3-CC_DATADISK|grep fdr08_run2.*|wc -l
68

;Throughput(T0-T1,snapshot)(MB/s)
[[Image:LYON.throughput.14400 200806081420.png]]

;Number of transfer errors
[[Image:LYON.num file xs error.14400 200806081420.png]]

;Throughput(T1-T2,snapshot)(MB/s)
[[Image:LYON.T2.throughput.14400 T1-T2 200806081420.png]]

=== 9th June ===
;0h20
* FDR datasets has begun to be shipped to T2s.
* Some T2s (e.g. BEIJING) subscribed dedicated datasets, which will be done later in today's morning.

;Shipped datasets

IN2P3-CC_DATADISK 315
BEIJING-LCG2_DATADISK 0
GRIF-LAL_DATADISK 79
GRIF-LPNHE_DATADISK 39
GRIF-SACLAY_DATADISK 38
IN2P3-CPPM_DATADISK 0
IN2P3-LAPP_DATADISK 12
IN2P3-LPC_DATADISK 12
IN2P3-LPSC_DATADISK 32
RO-02-NIPNE_DATADISK 29
RO-07-NIPNE_DATADISK 46
TOKYO-LCG2_DATADISK 134

;Free space

BEIJING-LCG2_DATADISK 6.366 6.366 6.343
GRIF-LAL_DATADISK 6.0 6.0 5.579
GRIF-LPNHE_DATADISK 3.906 3.906 3.817
GRIF-SACLAY_DATADISK 6.0 6.0 5.927
IN2P3-CPPM_DATADISK 1.0 1.0 0.977
IN2P3-LAPP_DATADISK 4.0 4.0 4.0
IN2P3-LPC_DATADISK 0 0 0
IN2P3-LPSC_DATADISK 0.488 0.488 0.439
RO-02-NIPNE_DATADISK 3.0 3.0 2.986
RO-07-NIPNE_DATADISK 2.0 2.0 1.896
TOKYO-LCG2_DATADISK 15.0 15.0 13.313

;Troughputs(snapshot)(MB/s)
[[Image:LYON.throughput.14400 200806090046.png]]

;Number of transfer erros(snapshot)
* Note: No errors found in FDR2 transfers. Errors showed in this snapshot are since other production data transfers.
[[Image:LYON.num file xs error.14400 200806090047.png]]

;09h24

;Troughputs(snapshot)(MB/s)
[[Image:LYON.throughput.14400 200806090922.png]]

;Number of transfer erros(snapshot)
* Note: No errors found in FDR2 transfers. Errors showed in this snapshot are since other production data transfers.
[[Image:LYON.num file xs error.14400 20806090922.png]]

;Shipped datasets

IN2P3-CC_DATADISK 319
BEIJING-LCG2_DATADISK 0
GRIF-LAL_DATADISK 79
GRIF-LPNHE_DATADISK 39
GRIF-SACLAY_DATADISK 38
IN2P3-CPPM_DATADISK 0
IN2P3-LAPP_DATADISK 12
IN2P3-LPC_DATADISK 12
IN2P3-LPSC_DATADISK 32
RO-02-NIPNE_DATADISK 59
RO-07-NIPNE_DATADISK 49
TOKYO-LCG2_DATADISK 134

;Free space
BEIJING-LCG2_DATADISK 6.366 6.366 6.343
GRIF-LAL_DATADISK 6.0 6.0 5.579
GRIF-LPNHE_DATADISK 3.906 3.906 3.817
GRIF-SACLAY_DATADISK 6.0 6.0 5.927
IN2P3-CPPM_DATADISK 1.0 1.0 0.977
IN2P3-LAPP_DATADISK 4.0 4.0 4.0
IN2P3-LPC_DATADISK 3.0 3.0 3.0
IN2P3-LPSC_DATADISK 0.488 0.488 0.439
RO-02-NIPNE_DATADISK 3.0 3.0 2.945
RO-07-NIPNE_DATADISK 2.0 2.0 1.885
TOKYO-LCG2_DATADISK 15.0 15.0 13.313

;14h00
Many failed with RO-02-NIPNE_DATADISK.

[FTS] FTS State [Failed] FTS Retries [1] Reason [TRANSFER error during TRANSFER phase: [PERMISSION] the server sent an error response: 550 550 rfio write failure: Permission denied.] Source Host [ccsrm.in2p3.fr] 16
[FTS] FTS State [Failed] FTS Retries [1] Reason [TRANSFER error during TRANSFER phase: [TRANSFER_TIMEOUT] gridftp_copy_wait: Connection timed out] Source Host [ccsrm.in2p3.fr] 9
[FTS] FTS State [Failed] FTS Retries [1] Reason [TRANSFER error during TRANSFER phase: [GRIDFTP] the server sent an error response: 426 426 Transfer aborted (Unexpected Exception : java.io.IOException: Broken pipe)] Source Host [ccsrm.in2p3.fr] 2
[FTS] FTS State [Failed] FTS Retries [1] Reason [The process serving the transfer (status = PREPARING) is no longer active (could not open file /proc/3775/cmdline)] Source Host [ccsrm.in2p3.fr] 1
[FTS] FTS State [Failed] FTS Retries [1] Reason [The process serving the transfer (status = PREPARING) is no longer active (could not open file /proc/30076/cmdline)] Source Host [ccsrm.in2p3.fr] 1

;20h10
* FDR datasets have finished transferred to all T2s except BEIJING-LCG2_DATADISK and RO-02-NIPNE_DATADISK.
* Many "Connection timed out" errors found at BEIJING-LCG2_DATADISK and RO-02-NIPNE_DATADISK.

[FTS] FTS State [Failed] FTS Retries [1] Reason [SOURCE error during PREPARATION phase: [CONNECTION] failed to contact on remote SRM [httpg://ccsrm.in2p3.fr:8443/srm/managerv2]. Givin' up after 3 tries] Source Host [ccsrm.in2p3.fr] 111
[FTS] FTS State [Failed] FTS Retries [1] Reason [TRANSFER error during TRANSFER phase: [TRANSFER_TIMEOUT] gridftp_copy_wait: Connection timed out] Source Host [ccsrm.in2p3.fr] 95
[FTS] FTS State [Failed] FTS Retries [1] Reason [SOURCE error during PREPARATION phase: [REQUEST_TIMEOUT] failed to prepare source file in 180 seconds] Source Host [ccsrm.in2p3.fr] 3

; Shipped datasets

IN2P3-CC_DATADISK 134
BEIJING-LCG2_DATADISK 79
GRIF-LAL_DATADISK 67
GRIF-LPNHE_DATADISK 32
GRIF-SACLAY_DATADISK 26
IN2P3-CPPM_DATADISK 52
IN2P3-LAPP_DATADISK 52
IN2P3-LPC_DATADISK 52
IN2P3-LPSC_DATADISK 32
RO-02-NIPNE_DATADISK 49
RO-07-NIPNE_DATADISK 49
TOKYO-LCG2_DATADISK 126

=== 10th June ===

;Summary

All T2s except BEIJING-LCG2_DATADISK and RO-02-NIPNE_DATADISK were in good status when FDR transferring.
The problem of "Connection timed out" with transfers to BEIJING will be hopefully solved by tuned the transfer duration >1800s by Lionel.
RO-02-NIPNE is said being upgraded from SL3 to SL4.

=== 11st June ===

; Anouncement
All FDR2 datasets will be deleted from all sites since part of the detector was not included in the reconstruction. It was not detected by the Data Quality because it is run on the express stream which is correct.

Old datasets will be deleted centrally by Stephane and new datasets that would be named begin with "ccrc08-run.0024*" will be re-generated and re-exported soon in today.

T2s will receive the same share rate as requested for FDR (except that DPD are not produced).

=== 12nd June ===

Atlas:FDR2

2008-06-11T09:39:01Z

Erming: /* 11st June */

= Monitoring pages/graphs =
=== FTS ===
http://cctoolsafs.in2p3.fr/fts/monitoring/prod/ftsmonitor.php?vo=atlas

=== T0-T1 (ALL) ===
; FDR monitoring page - T1s
http://atldq2pro.cern.ch:8000/ft/mon/fdrmon_tier1s.html

http://dashb-atlas-data-tier0.cern.ch/dashboard/request.py/site
*Throughput 
:http://dashb-atlas-data-tier0.cern.ch/dashboard/templates/plots/OVERVIEW.throughput.14400.png
*Errors 
:http://dashb-atlas-data-tier0.cern.ch/dashboard/templates/plots/OVERVIEW.num_file_xs_error.14400.png

=== T0-T1 (Lyon) ===
http://dashb-atlas-data-tier0.cern.ch/dashboard/request.py/site?statsInterval=4&name=LYON
*Throughput T0-LYON 
:http://dashb-atlas-data-tier0.cern.ch/dashboard/templates/plots/LYON.throughput.14400.png
*Errors T0-LYON 
:http://dashb-atlas-data-tier0.cern.ch/dashboard/templates/plots/LYON.num_file_xs_error.14400.png
*Throughput Others-LYON 
:http://dashb-atlas-data.cern.ch/dashboard/templates/plots/LYON.T1.throughput.14400.png
*Errors Others-LYON :http://dashb-atlas-data.cern.ch/dashboard/templates/plots/LYON.T1.num_file_xs_error.14400.png

=== T1-T1 ===

=== T1-T2 (Lyon) ===
; FDR monitoring page
http://atldq2pro.cern.ch:8000/ft/mon/fdrmon_TiersInfo.html

http://atldq2pro.cern.ch:8000/ft/mon/fdrmon_T1-T2_matrix_day.html

; Dashboard
http://dashb-atlas-data.cern.ch/dashboard/request.py/site?statsInterval=4&name=LYON
*Throughput 
:http://dashb-atlas-data.cern.ch/dashboard/templates/plots/LYON.T2.throughput.14400.png
*Errors
:http://dashb-atlas-data.cern.ch/dashboard/templates/plots/LYON.T2.num_file_xs_error.14400.png

=== Network Graphs ===
; IN2P3 Weathermap
* http://netstat.in2p3.fr/weathermap/
* http://netstat.in2p3.fr/weathermap/graphiques/

; cc-in2p3 : [http://netstat.in2p3.fr/weathermap/graphiques/lyo-cern.html lyo-cern] (lhcopn-in2p3.cern.ch) http://netstat.in2p3.fr/weathermap/graphiques/lyo-cern-daily.gif
; lyon-nord : [http://netstat.in2p3.fr/weathermap/graphiques/lyo-nrd.html lyo-nrd] http://netstat.in2p3.fr/weathermap/graphiques/lyo-nrd-daily.gif
; orsay : [http://netstat.in2p3.fr/weathermap/graphiques/orsay.html orsay] (in2p3-orsay.cssi.renater.fr http://netstat.in2p3.fr/weathermap/graphiques/orsay-daily.gif
; lal-cc : [http://netstat.in2p3.fr/weathermap/graphiques/lal-cc.html Liaison LAL - CC] http://netstat.in2p3.fr/weathermap/graphiques/lal-cc-daily.gif
; lal : [http://netstat.in2p3.fr/weathermap/graphiques/lal.html lal] http://netstat.in2p3.fr/weathermap/graphiques/lal-daily.gif
; lpnhe: [http://netstat.in2p3.fr/weathermap/graphiques/lpnhe-nrd.html lpnhe-nrd] (in2p3-jussieu.cssi.renater.fr) http://netstat.in2p3.fr/weathermap/graphiques/lpnhe-nrd-daily.gif [http://netstat.in2p3.fr/weathermap/graphiques/lpnhe.html lpnhe] (Paris-LPNHE.in2p3.fr) http://netstat.in2p3.fr/weathermap/graphiques/lpnhe-daily.gif
; lapp : [http://netstat.in2p3.fr/weathermap/graphiques/ann-nrd.html ann-nrd] http://netstat.in2p3.fr/weathermap/graphiques/ann-nrd-daily.gif
; lpc: [http://netstat.in2p3.fr/weathermap/graphiques/lpc-cf.html lpc-cf]
http://netstat.in2p3.fr/weathermap/graphiques/lpc-cf-daily.gif
; lpsc : [http://netstat.in2p3.fr/weathermap/graphiques/lpsc.html lpsc] http://netstat.in2p3.fr/weathermap/graphiques/lpsc-daily.gif
; cppm : [http://netstat.in2p3.fr/weathermap/graphiques/cppm.html cppm] 
http://netstat.in2p3.fr/weathermap/graphiques/cppm-daily.gif
; international (tokyo etc.) : [http://netstat.in2p3.fr/weathermap/graphiques/parisnrd.html parisnrd] 
http://netstat.in2p3.fr/weathermap/graphiques/parisnrd-daily.gif
; GEANT - NYC : [http://dc-snmp.wcc.grnoc.iu.edu/manlan/show-graph.cgi?title=sw.newy32aoa.manlan.internet2.edu--te10/1&rrdname=sw.newy32aoa.manlan.internet2.edu--te10_1.rrd MANLAN] 
http://dc-snmp.wcc.grnoc.iu.edu/manlan/img/sw.newy32aoa.manlan.internet2.edu--te10_1-std5.gif
; NYC - TOKYO : [http://dc-snmp.wcc.grnoc.iu.edu/manlan/show-graph.cgi?title=sw.newy32aoa.manlan.internet2.edu--te11/1&rrdname=sw.newy32aoa.manlan.internet2.edu--te11_1.rrd MANLAN] 
http://dc-snmp.wcc.grnoc.iu.edu/manlan/img/sw.newy32aoa.manlan.internet2.edu--te11_1-std5.gif
; tokyo : [http://www.icepp.jp/monitor/gridftp/monitor.html gridftp monitor] http://www.icepp.jp/monitor/gridftp/img/all.png

= T0-T1 =

= T1-T2 =
==== General informations ====
;FDR page:
https://twiki.cern.ch/twiki/bin/view/Atlas/FullDressRehearsal

;DDM_in_FRD2: https://twiki.cern.ch/twiki/bin/view/Atlas/DDMOperationsGroup#DDM_in_FRD2

;Replication status(checked ~ every 2h?)
http://pandamon.usatlas.bnl.gov:25880/server/pandamon/query?mode=listFDR

;Lyon FTS monitor: http://cctoolsafs.in2p3.fr/fts/monitoring/prod/ftsmonitor.php?vo=atlas

;Contacts:

* IN2P3-CPPM_DATADISK :
* IN2P3-LPSC_DATADISK :
* IN2P3-LPC_DATADISK :
* GRIF-LAL_DATADISK : (grid.admin a lal.in2p3.fr)
* GRIF-LPNHE_DATADISK :
* GRIF-SACLAY_DATADISK :
* BEIJING-LCG2_DATADISK : (yanxf a ihep.ac.cn, Erming.Pei a cern.ch)
* RO-07-NIPNE_DATADISK : (ciubancan a nipne.ro)
* RO-02-NIPNE_DATADISK : (tpreda a nipne.ro)
* TOKYO-LCG2_DATADISK : (lcg-admin a icepp.s.u-tokyo.ac.jp)

===== Shares =====
Only srmv2+ space tokens sites will collect data. The list of streams is : Egamma, Bphysics, MinBias, Muon and Jet. If no precise stream name is associated to the share, the site will get any stream. The site in parenthesis will collect the complementary datasets. The T2 ressources at T1 means that users without production role can access CPUs at the T1 without the Production role.
* US :
o AGLT2_DATADISK : 100 %
o MWT2_DATADISK: 100 %
o NET2_DATADISK: 100 %
o SLACXRD_DATADISK: 100 %
o SWT2_CPB_DATADISK: 100 %
o T2 CPU ressources in BNL to access AOD
* CA :
o SFU-LCG2_DATADISK : 40%
o VICTORIA-LCG2_DATADISK : 30%
o ALBERTA-LCG2_DATADISK : 30%
o TORONTO-LCG2_DATADISK : 50 %
* IT :
o INFN-MILANO_DATADISK : 25%
o INFN-FRASCATI_DATADISK : 25%
o INFN-ROMA1_DATADISK : 100%
o INFN-NAPOLI-ATLAS_DATADISK : 100 %
* TW :
o TW-FTT_DATADISK : 50 %
* NL
o JINR-LCG2_DATADISK : 50% Jet (RU-PROTVINO-IHEP_DATADISK)
o JINR-LCG2_DATADISK : 50% MinBias (RU-PROTVINO-IHEP_DATADISK)
o JINR-LCG2_DATADISK : 50% Muon (RU-PROTVINO-IHEP_DATADISK)
o JINR-LCG2_DATADISK : 50% Bphysics (RU-PNPI_DATADISK)
o RU-PROTVINO-IHEP_DATADISK : 50% Jet (JINR-LCG2_DATADISK)
o RU-PROTVINO-IHEP_DATADISK : 50% Muon (JINR-LCG2_DATADISK)
o RU-PROTVINO-IHEP_DATADISK : 50% MinBias (JINR-LCG2_DATADISK)
o RU-PROTVINO-IHEP_DATADISK : 50% Egamma (RU-PNPI_DATADISK)
o RU-PNPI_DATADISK : 50% Egamma (RU-PROTVINO-IHEP_DATADISK)
o RU-PNPI_DATADISK : 50% Bphysics (JINR-LCG2_DATADISK)
o RU-PNPI_DATADISK : 50% Jet (RRC-KI_DATADISK)
o RU-PNPI_DATADISK : 50% MinBias (RRC-KI_DATADISK)
o RRC-KI_DATADISK : 50% Jet (RU-PNPI_DATADISK)
o RRC-KI_DATADISK : 50% MinBias (RU-PNPI_DATADISK)
o RRC-KI_DATADISK : 50% Egamma
o RRC-KI_DATADISK : 50% Bphysics
o CSTCDIE_DATADISK : 20%
* FR :
o IN2P3-LAPP_DATADISK : 100 % Egamma
o IN2P3-CPPM_DATADISK : 100 % Egamma
o IN2P3-LPSC_DATADISK : 5 %
o IN2P3-LPC_DATADISK : 100% Muon, 50% Egamma
o BEIJING-LCG2_DATADISK : 100 % Egamma, 50% Muon, 35% MinBias
o RO-07-NIPNE_DATADISK : 10%
o RO-02-NIPNE_DATADISK : 10%
o GRIF-LAL_DATADISK : 45%
o GRIF-LPNHE_DATADISK : 25%
o GRIF-SACLAY_DATADISK : 30 %
o TOKYO-LCG2_DATADISK : 100%
o T2 CPU ressources in LYON to access AOD
* DE :
o DESY-HH_DATADISK : 100%
o CSCS-LCG2_DATADISK: 100%
o DESY-ZN_DATADISK : 50%
o GOEGRID_DATADISK : 50%
o WUPPERTALPROD_DATADISK: 25%
o CYFRONET-LCG2_DATADISK : 15%
o PRAGUELCG2_DATADISK : 25%
o LRZ-LMU_DATADISK : 35%
o UNI-FREIBURG_DATADISK : 100% Jet, 100% Muon
* UK :
o UKI-SOUTHGRID-CAM-HEP_DATADISK : 50 % Muon (UKI-NORTHGRID-LANCS-HEP_DATADISK
o UKI-NORTHGRID-LANCS-HEP_DATADISK : 50 % Muon (UKI-SOUTHGRID-CAM-HEP_DATADISK)
o UKI-NORTHGRID-SHEF-HEP_DATADISK : 50 % MinBias (UKI-SCOTGRID-GLASGOW_DATADISK)
o UKI-SCOTGRID-GLASGOW_DATADISK : 50 % MinBias (UKI-NORTHGRID-SHEF-HEP_DATADISK)
o UKI-LT2-RHUL_DATADISK : 50 % Egamma (UKI-NORTHGRID-LIV-HEP_DATADISK)
o UKI-NORTHGRID-LIV-HEP_DATADISK : 50 % Egamma (UKI-LT2-RHUL_DATADISK)
o UKI-SOUTHGRID-RALPP_DATADISK : 50 Egamma (UKI-LT2-RHUL_DATADISK)
o UKI-SOUTHGRID-BHAM-HEP_DATADISK : 50 Jet (UKI-SOUTHGRID-OX-HEP_DATADISK)
* ES :
o IFIC-LCG2_DATADISK: 50%
o UAM-LCG2_DATADISK: 25%
o IFAE_DATADISK: 25%
* NDGF :
o SIGNET_DATADISK : 50 %

== Logbook ==

=== 2nd June ===

* Since Lyon is in donwtime, so there's no transfers from LYON to T2s today.
http://cc.in2p3.fr/cc_accueil.php3?lang=fr

*Free Space

Token Total Guaranteed Unused
------------------------------------------
BEIJING-LCG2_DATADISK 6.366 6.366 6.343
GRIF-LAL_DATADISK 6.0 6.0 5.732
GRIF-LPNHE_DATADISK 3.906 3.906 3.89
GRIF-SACLAY_DATADISK 6.0 6.0 5.994
IN2P3-CPPM_DATADISK 1.0 1.0 0.977
IN2P3-LAPP_DATADISK 4.0 4.0 4.0
IN2P3-LPC_DATADISK 3.0 3.0 3.0
IN2P3-LPSC_DATADISK 0.488 0.488 0.482
RO-02-NIPNE_DATADISK 3.0 3.0 3.0
RO-07-NIPNE_DATADISK 2.0 2.0 2.0
TOKYO-LCG2_DATADISK 15.0 15.0 13.643

=== 3rd June ===

* Lyon is still in donwtime, no transfers from LYON to T2s in this morning.

; Throughput(snapshot) (MB/s) : [[Image:LYON.throughput.14400_20080603.png]]
; Number of transfer errors(snapshot) : [[Image:LYON.num_file_xs_error.14400_20080603.png]]

=== 4th June ===
;22h02
* Still no transfers, desipte that all the services in LYON shows OK.

;Throughput(snapshot)
[[Image:LYON.throughput.3600_20080604.png]]

;Services in LYON
[[Image:services_cc-in2p3_2008060422h15.PNG]]

;Free space

BEIJING-LCG2_DATADISK 6.366 6.366 6.343
BEIJING-LCG2_MCDISK 5.457 5.457 5.457
GRIF-LAL_DATADISK 6.0 6.0 5.732
GRIF-LAL_MCDISK 7.0 7.0 3.724
GRIF-LPNHE_DATADISK 3.906 3.906 3.89
GRIF-LPNHE_MCDISK 1.0 1.0 0.194
GRIF-SACLAY_DATADISK 6.0 6.0 5.994
GRIF-SACLAY_MCDISK 2.0 2.0 1.713
IN2P3-CPPM_DATADISK 1.0 1.0 0.977
IN2P3-CPPM_MCDISK 0.781 0.781 0.765
IN2P3-LAPP_DATADISK 4.0 4.0 4.0
IN2P3-LAPP_MCDISK 2.0 2.0 0.745
IN2P3-LPC_DATADISK 3.0 3.0 3.0
IN2P3-LPC_MCDISK 2.0 2.0 1.936
IN2P3-LPSC_DATADISK 0.488 0.488 0.482
IN2P3-LPSC_MCDISK 2.0 2.0 1.518
RO-02-NIPNE_DATADISK 3.0 3.0 3.0
RO-02-NIPNE_MCDISK 1.0 1.0 0.995
RO-07-NIPNE_DATADISK 2.0 2.0 2.0
RO-07-NIPNE_MCDISK 1.0 1.0 0.654
TOKYO-LCG2_DATADISK 15.0 15.0 13.643
TOKYO-LCG2_MCDISK 10.0 10.0 10.0

=== 5th June ===

* Data produced on Tuesday are supposed to be shipped in this night.

;14h50
* Now the RAW data has begun to shipped to LYON.

;Throughput(snapshot)(MB/s)
[[Image:LYON.throughput.14400_RAW_20080605_1816.png]]

;Number of transfer errors (snapshot)
[[Image:LYON.num_file_xs_error.14400_20080605_1823.png]]

;21h30
* IN2P3-CC_MCDISK has some problem of staging file. But it should have nothing to do with T0 export.

* 47 datasets have been exported to T1s. The total is 90.

;Throughput(snapshot)(MB/s)
[[Image:LYON.throughput.14400 20080605 2140.png]]

;Number of transfer errors (snapshot) -- It's the problem of staging file in T1 and nothing to do with FDR transfers.
[[Image:LYON.num_file_xs_error.14400_20080605_2140.png]]

=== 6th June ===
;09h50

;Throughput(snapshot) (MB/s)
[[Image:LYON.throughput.14400 200806060 0958.png]]

;Number of transfer errors (snapshot)
[[Image:LYON.num file xs error.14400 20080606 0959.png]]

;10h03
* Now all the RAW datasets have been exported to T1. The total number is 90.
http://atlas.web.cern.ch/Atlas/tzero/prod2/monitoring/tables/ddm_RAW_status.html

=== 7th June ===
* at CERN, will re-start the bulk processing with using the upcoming AtlasPoint1-14.1.0.12 release cache.

;11h30
* SRM in LYON is still in bad shape. Experts are working on it.

=== 8th June ===
;14h20
* FDR2 AODs/DPDs/TAGs/ESDs are being exported to LYON (IN2P3-CC_DATADISK), but haven't been shipped to T2s.

;Number of datasets transfered by far
$ dq2-list-dataset-site IN2P3-CC_DATADISK|grep fdr08_run2.*|wc -l
68

;Throughput(T0-T1,snapshot)(MB/s)
[[Image:LYON.throughput.14400 200806081420.png]]

;Number of transfer errors
[[Image:LYON.num file xs error.14400 200806081420.png]]

;Throughput(T1-T2,snapshot)(MB/s)
[[Image:LYON.T2.throughput.14400 T1-T2 200806081420.png]]

=== 9th June ===
;0h20
* FDR datasets has begun to be shipped to T2s.
* Some T2s (e.g. BEIJING) subscribed dedicated datasets, which will be done later in today's morning.

;Shipped datasets

IN2P3-CC_DATADISK 315
BEIJING-LCG2_DATADISK 0
GRIF-LAL_DATADISK 79
GRIF-LPNHE_DATADISK 39
GRIF-SACLAY_DATADISK 38
IN2P3-CPPM_DATADISK 0
IN2P3-LAPP_DATADISK 12
IN2P3-LPC_DATADISK 12
IN2P3-LPSC_DATADISK 32
RO-02-NIPNE_DATADISK 29
RO-07-NIPNE_DATADISK 46
TOKYO-LCG2_DATADISK 134

;Free space

BEIJING-LCG2_DATADISK 6.366 6.366 6.343
GRIF-LAL_DATADISK 6.0 6.0 5.579
GRIF-LPNHE_DATADISK 3.906 3.906 3.817
GRIF-SACLAY_DATADISK 6.0 6.0 5.927
IN2P3-CPPM_DATADISK 1.0 1.0 0.977
IN2P3-LAPP_DATADISK 4.0 4.0 4.0
IN2P3-LPC_DATADISK 0 0 0
IN2P3-LPSC_DATADISK 0.488 0.488 0.439
RO-02-NIPNE_DATADISK 3.0 3.0 2.986
RO-07-NIPNE_DATADISK 2.0 2.0 1.896
TOKYO-LCG2_DATADISK 15.0 15.0 13.313

;Troughputs(snapshot)(MB/s)
[[Image:LYON.throughput.14400 200806090046.png]]

;Number of transfer erros(snapshot)
* Note: No errors found in FDR2 transfers. Errors showed in this snapshot are since other production data transfers.
[[Image:LYON.num file xs error.14400 200806090047.png]]

;09h24

;Troughputs(snapshot)(MB/s)
[[Image:LYON.throughput.14400 200806090922.png]]

;Number of transfer erros(snapshot)
* Note: No errors found in FDR2 transfers. Errors showed in this snapshot are since other production data transfers.
[[Image:LYON.num file xs error.14400 20806090922.png]]

;Shipped datasets

IN2P3-CC_DATADISK 319
BEIJING-LCG2_DATADISK 0
GRIF-LAL_DATADISK 79
GRIF-LPNHE_DATADISK 39
GRIF-SACLAY_DATADISK 38
IN2P3-CPPM_DATADISK 0
IN2P3-LAPP_DATADISK 12
IN2P3-LPC_DATADISK 12
IN2P3-LPSC_DATADISK 32
RO-02-NIPNE_DATADISK 59
RO-07-NIPNE_DATADISK 49
TOKYO-LCG2_DATADISK 134

;Free space
BEIJING-LCG2_DATADISK 6.366 6.366 6.343
GRIF-LAL_DATADISK 6.0 6.0 5.579
GRIF-LPNHE_DATADISK 3.906 3.906 3.817
GRIF-SACLAY_DATADISK 6.0 6.0 5.927
IN2P3-CPPM_DATADISK 1.0 1.0 0.977
IN2P3-LAPP_DATADISK 4.0 4.0 4.0
IN2P3-LPC_DATADISK 3.0 3.0 3.0
IN2P3-LPSC_DATADISK 0.488 0.488 0.439
RO-02-NIPNE_DATADISK 3.0 3.0 2.945
RO-07-NIPNE_DATADISK 2.0 2.0 1.885
TOKYO-LCG2_DATADISK 15.0 15.0 13.313

;14h00
Many failed with RO-02-NIPNE_DATADISK.

[FTS] FTS State [Failed] FTS Retries [1] Reason [TRANSFER error during TRANSFER phase: [PERMISSION] the server sent an error response: 550 550 rfio write failure: Permission denied.] Source Host [ccsrm.in2p3.fr] 16
[FTS] FTS State [Failed] FTS Retries [1] Reason [TRANSFER error during TRANSFER phase: [TRANSFER_TIMEOUT] gridftp_copy_wait: Connection timed out] Source Host [ccsrm.in2p3.fr] 9
[FTS] FTS State [Failed] FTS Retries [1] Reason [TRANSFER error during TRANSFER phase: [GRIDFTP] the server sent an error response: 426 426 Transfer aborted (Unexpected Exception : java.io.IOException: Broken pipe)] Source Host [ccsrm.in2p3.fr] 2
[FTS] FTS State [Failed] FTS Retries [1] Reason [The process serving the transfer (status = PREPARING) is no longer active (could not open file /proc/3775/cmdline)] Source Host [ccsrm.in2p3.fr] 1
[FTS] FTS State [Failed] FTS Retries [1] Reason [The process serving the transfer (status = PREPARING) is no longer active (could not open file /proc/30076/cmdline)] Source Host [ccsrm.in2p3.fr] 1

;20h10
* FDR datasets have finished transferred to all T2s except BEIJING-LCG2_DATADISK and RO-02-NIPNE_DATADISK.
* Many "Connection timed out" errors found at BEIJING-LCG2_DATADISK and RO-02-NIPNE_DATADISK.

[FTS] FTS State [Failed] FTS Retries [1] Reason [SOURCE error during PREPARATION phase: [CONNECTION] failed to contact on remote SRM [httpg://ccsrm.in2p3.fr:8443/srm/managerv2]. Givin' up after 3 tries] Source Host [ccsrm.in2p3.fr] 111
[FTS] FTS State [Failed] FTS Retries [1] Reason [TRANSFER error during TRANSFER phase: [TRANSFER_TIMEOUT] gridftp_copy_wait: Connection timed out] Source Host [ccsrm.in2p3.fr] 95
[FTS] FTS State [Failed] FTS Retries [1] Reason [SOURCE error during PREPARATION phase: [REQUEST_TIMEOUT] failed to prepare source file in 180 seconds] Source Host [ccsrm.in2p3.fr] 3

; Shipped datasets

IN2P3-CC_DATADISK 134
BEIJING-LCG2_DATADISK 79
GRIF-LAL_DATADISK 67
GRIF-LPNHE_DATADISK 32
GRIF-SACLAY_DATADISK 26
IN2P3-CPPM_DATADISK 52
IN2P3-LAPP_DATADISK 52
IN2P3-LPC_DATADISK 52
IN2P3-LPSC_DATADISK 32
RO-02-NIPNE_DATADISK 49
RO-07-NIPNE_DATADISK 49
TOKYO-LCG2_DATADISK 126

=== 10th June ===

;Summary

All T2s except BEIJING-LCG2_DATADISK and RO-02-NIPNE_DATADISK were in good status when we made FDR transfers.
The problem of "Connection timed out" with transfers to BEIJING will be hopefully solved by tuned the transfer duration >1800s by Lionel.
RO-02-NIPNE is said being upgraded from SL3 to SL4.

=== 11st June ===

; Anouncement
All FDR2 datasets will be deleted from all sites since part of the detector was not included in the reconstruction. It was not detected by the Data Quality because it is run on the express stream which is correct.

Old datasets will be deleted centrally by Stephane and new datasets that would be named begin with "ccrc08-run.0024*" will be re-generated and re-exported soon in today.

T2s will receive the same share rate as requested for FDR (except that DPD are not produced).

=== 12nd June ===

Atlas:FDR2

2008-06-11T09:38:48Z

Erming: /* 10th June */

= Monitoring pages/graphs =
=== FTS ===
http://cctoolsafs.in2p3.fr/fts/monitoring/prod/ftsmonitor.php?vo=atlas

=== T0-T1 (ALL) ===
; FDR monitoring page - T1s
http://atldq2pro.cern.ch:8000/ft/mon/fdrmon_tier1s.html

http://dashb-atlas-data-tier0.cern.ch/dashboard/request.py/site
*Throughput 
:http://dashb-atlas-data-tier0.cern.ch/dashboard/templates/plots/OVERVIEW.throughput.14400.png
*Errors 
:http://dashb-atlas-data-tier0.cern.ch/dashboard/templates/plots/OVERVIEW.num_file_xs_error.14400.png

=== T0-T1 (Lyon) ===
http://dashb-atlas-data-tier0.cern.ch/dashboard/request.py/site?statsInterval=4&name=LYON
*Throughput T0-LYON 
:http://dashb-atlas-data-tier0.cern.ch/dashboard/templates/plots/LYON.throughput.14400.png
*Errors T0-LYON 
:http://dashb-atlas-data-tier0.cern.ch/dashboard/templates/plots/LYON.num_file_xs_error.14400.png
*Throughput Others-LYON 
:http://dashb-atlas-data.cern.ch/dashboard/templates/plots/LYON.T1.throughput.14400.png
*Errors Others-LYON :http://dashb-atlas-data.cern.ch/dashboard/templates/plots/LYON.T1.num_file_xs_error.14400.png

=== T1-T1 ===

=== T1-T2 (Lyon) ===
; FDR monitoring page
http://atldq2pro.cern.ch:8000/ft/mon/fdrmon_TiersInfo.html

http://atldq2pro.cern.ch:8000/ft/mon/fdrmon_T1-T2_matrix_day.html

; Dashboard
http://dashb-atlas-data.cern.ch/dashboard/request.py/site?statsInterval=4&name=LYON
*Throughput 
:http://dashb-atlas-data.cern.ch/dashboard/templates/plots/LYON.T2.throughput.14400.png
*Errors
:http://dashb-atlas-data.cern.ch/dashboard/templates/plots/LYON.T2.num_file_xs_error.14400.png

=== Network Graphs ===
; IN2P3 Weathermap
* http://netstat.in2p3.fr/weathermap/
* http://netstat.in2p3.fr/weathermap/graphiques/

; cc-in2p3 : [http://netstat.in2p3.fr/weathermap/graphiques/lyo-cern.html lyo-cern] (lhcopn-in2p3.cern.ch) http://netstat.in2p3.fr/weathermap/graphiques/lyo-cern-daily.gif
; lyon-nord : [http://netstat.in2p3.fr/weathermap/graphiques/lyo-nrd.html lyo-nrd] http://netstat.in2p3.fr/weathermap/graphiques/lyo-nrd-daily.gif
; orsay : [http://netstat.in2p3.fr/weathermap/graphiques/orsay.html orsay] (in2p3-orsay.cssi.renater.fr http://netstat.in2p3.fr/weathermap/graphiques/orsay-daily.gif
; lal-cc : [http://netstat.in2p3.fr/weathermap/graphiques/lal-cc.html Liaison LAL - CC] http://netstat.in2p3.fr/weathermap/graphiques/lal-cc-daily.gif
; lal : [http://netstat.in2p3.fr/weathermap/graphiques/lal.html lal] http://netstat.in2p3.fr/weathermap/graphiques/lal-daily.gif
; lpnhe: [http://netstat.in2p3.fr/weathermap/graphiques/lpnhe-nrd.html lpnhe-nrd] (in2p3-jussieu.cssi.renater.fr) http://netstat.in2p3.fr/weathermap/graphiques/lpnhe-nrd-daily.gif [http://netstat.in2p3.fr/weathermap/graphiques/lpnhe.html lpnhe] (Paris-LPNHE.in2p3.fr) http://netstat.in2p3.fr/weathermap/graphiques/lpnhe-daily.gif
; lapp : [http://netstat.in2p3.fr/weathermap/graphiques/ann-nrd.html ann-nrd] http://netstat.in2p3.fr/weathermap/graphiques/ann-nrd-daily.gif
; lpc: [http://netstat.in2p3.fr/weathermap/graphiques/lpc-cf.html lpc-cf]
http://netstat.in2p3.fr/weathermap/graphiques/lpc-cf-daily.gif
; lpsc : [http://netstat.in2p3.fr/weathermap/graphiques/lpsc.html lpsc] http://netstat.in2p3.fr/weathermap/graphiques/lpsc-daily.gif
; cppm : [http://netstat.in2p3.fr/weathermap/graphiques/cppm.html cppm] 
http://netstat.in2p3.fr/weathermap/graphiques/cppm-daily.gif
; international (tokyo etc.) : [http://netstat.in2p3.fr/weathermap/graphiques/parisnrd.html parisnrd] 
http://netstat.in2p3.fr/weathermap/graphiques/parisnrd-daily.gif
; GEANT - NYC : [http://dc-snmp.wcc.grnoc.iu.edu/manlan/show-graph.cgi?title=sw.newy32aoa.manlan.internet2.edu--te10/1&rrdname=sw.newy32aoa.manlan.internet2.edu--te10_1.rrd MANLAN] 
http://dc-snmp.wcc.grnoc.iu.edu/manlan/img/sw.newy32aoa.manlan.internet2.edu--te10_1-std5.gif
; NYC - TOKYO : [http://dc-snmp.wcc.grnoc.iu.edu/manlan/show-graph.cgi?title=sw.newy32aoa.manlan.internet2.edu--te11/1&rrdname=sw.newy32aoa.manlan.internet2.edu--te11_1.rrd MANLAN] 
http://dc-snmp.wcc.grnoc.iu.edu/manlan/img/sw.newy32aoa.manlan.internet2.edu--te11_1-std5.gif
; tokyo : [http://www.icepp.jp/monitor/gridftp/monitor.html gridftp monitor] http://www.icepp.jp/monitor/gridftp/img/all.png

= T0-T1 =

= T1-T2 =
==== General informations ====
;FDR page:
https://twiki.cern.ch/twiki/bin/view/Atlas/FullDressRehearsal

;DDM_in_FRD2: https://twiki.cern.ch/twiki/bin/view/Atlas/DDMOperationsGroup#DDM_in_FRD2

;Replication status(checked ~ every 2h?)
http://pandamon.usatlas.bnl.gov:25880/server/pandamon/query?mode=listFDR

;Lyon FTS monitor: http://cctoolsafs.in2p3.fr/fts/monitoring/prod/ftsmonitor.php?vo=atlas

;Contacts:

* IN2P3-CPPM_DATADISK :
* IN2P3-LPSC_DATADISK :
* IN2P3-LPC_DATADISK :
* GRIF-LAL_DATADISK : (grid.admin a lal.in2p3.fr)
* GRIF-LPNHE_DATADISK :
* GRIF-SACLAY_DATADISK :
* BEIJING-LCG2_DATADISK : (yanxf a ihep.ac.cn, Erming.Pei a cern.ch)
* RO-07-NIPNE_DATADISK : (ciubancan a nipne.ro)
* RO-02-NIPNE_DATADISK : (tpreda a nipne.ro)
* TOKYO-LCG2_DATADISK : (lcg-admin a icepp.s.u-tokyo.ac.jp)

===== Shares =====
Only srmv2+ space tokens sites will collect data. The list of streams is : Egamma, Bphysics, MinBias, Muon and Jet. If no precise stream name is associated to the share, the site will get any stream. The site in parenthesis will collect the complementary datasets. The T2 ressources at T1 means that users without production role can access CPUs at the T1 without the Production role.
* US :
o AGLT2_DATADISK : 100 %
o MWT2_DATADISK: 100 %
o NET2_DATADISK: 100 %
o SLACXRD_DATADISK: 100 %
o SWT2_CPB_DATADISK: 100 %
o T2 CPU ressources in BNL to access AOD
* CA :
o SFU-LCG2_DATADISK : 40%
o VICTORIA-LCG2_DATADISK : 30%
o ALBERTA-LCG2_DATADISK : 30%
o TORONTO-LCG2_DATADISK : 50 %
* IT :
o INFN-MILANO_DATADISK : 25%
o INFN-FRASCATI_DATADISK : 25%
o INFN-ROMA1_DATADISK : 100%
o INFN-NAPOLI-ATLAS_DATADISK : 100 %
* TW :
o TW-FTT_DATADISK : 50 %
* NL
o JINR-LCG2_DATADISK : 50% Jet (RU-PROTVINO-IHEP_DATADISK)
o JINR-LCG2_DATADISK : 50% MinBias (RU-PROTVINO-IHEP_DATADISK)
o JINR-LCG2_DATADISK : 50% Muon (RU-PROTVINO-IHEP_DATADISK)
o JINR-LCG2_DATADISK : 50% Bphysics (RU-PNPI_DATADISK)
o RU-PROTVINO-IHEP_DATADISK : 50% Jet (JINR-LCG2_DATADISK)
o RU-PROTVINO-IHEP_DATADISK : 50% Muon (JINR-LCG2_DATADISK)
o RU-PROTVINO-IHEP_DATADISK : 50% MinBias (JINR-LCG2_DATADISK)
o RU-PROTVINO-IHEP_DATADISK : 50% Egamma (RU-PNPI_DATADISK)
o RU-PNPI_DATADISK : 50% Egamma (RU-PROTVINO-IHEP_DATADISK)
o RU-PNPI_DATADISK : 50% Bphysics (JINR-LCG2_DATADISK)
o RU-PNPI_DATADISK : 50% Jet (RRC-KI_DATADISK)
o RU-PNPI_DATADISK : 50% MinBias (RRC-KI_DATADISK)
o RRC-KI_DATADISK : 50% Jet (RU-PNPI_DATADISK)
o RRC-KI_DATADISK : 50% MinBias (RU-PNPI_DATADISK)
o RRC-KI_DATADISK : 50% Egamma
o RRC-KI_DATADISK : 50% Bphysics
o CSTCDIE_DATADISK : 20%
* FR :
o IN2P3-LAPP_DATADISK : 100 % Egamma
o IN2P3-CPPM_DATADISK : 100 % Egamma
o IN2P3-LPSC_DATADISK : 5 %
o IN2P3-LPC_DATADISK : 100% Muon, 50% Egamma
o BEIJING-LCG2_DATADISK : 100 % Egamma, 50% Muon, 35% MinBias
o RO-07-NIPNE_DATADISK : 10%
o RO-02-NIPNE_DATADISK : 10%
o GRIF-LAL_DATADISK : 45%
o GRIF-LPNHE_DATADISK : 25%
o GRIF-SACLAY_DATADISK : 30 %
o TOKYO-LCG2_DATADISK : 100%
o T2 CPU ressources in LYON to access AOD
* DE :
o DESY-HH_DATADISK : 100%
o CSCS-LCG2_DATADISK: 100%
o DESY-ZN_DATADISK : 50%
o GOEGRID_DATADISK : 50%
o WUPPERTALPROD_DATADISK: 25%
o CYFRONET-LCG2_DATADISK : 15%
o PRAGUELCG2_DATADISK : 25%
o LRZ-LMU_DATADISK : 35%
o UNI-FREIBURG_DATADISK : 100% Jet, 100% Muon
* UK :
o UKI-SOUTHGRID-CAM-HEP_DATADISK : 50 % Muon (UKI-NORTHGRID-LANCS-HEP_DATADISK
o UKI-NORTHGRID-LANCS-HEP_DATADISK : 50 % Muon (UKI-SOUTHGRID-CAM-HEP_DATADISK)
o UKI-NORTHGRID-SHEF-HEP_DATADISK : 50 % MinBias (UKI-SCOTGRID-GLASGOW_DATADISK)
o UKI-SCOTGRID-GLASGOW_DATADISK : 50 % MinBias (UKI-NORTHGRID-SHEF-HEP_DATADISK)
o UKI-LT2-RHUL_DATADISK : 50 % Egamma (UKI-NORTHGRID-LIV-HEP_DATADISK)
o UKI-NORTHGRID-LIV-HEP_DATADISK : 50 % Egamma (UKI-LT2-RHUL_DATADISK)
o UKI-SOUTHGRID-RALPP_DATADISK : 50 Egamma (UKI-LT2-RHUL_DATADISK)
o UKI-SOUTHGRID-BHAM-HEP_DATADISK : 50 Jet (UKI-SOUTHGRID-OX-HEP_DATADISK)
* ES :
o IFIC-LCG2_DATADISK: 50%
o UAM-LCG2_DATADISK: 25%
o IFAE_DATADISK: 25%
* NDGF :
o SIGNET_DATADISK : 50 %

== Logbook ==

=== 2nd June ===

* Since Lyon is in donwtime, so there's no transfers from LYON to T2s today.
http://cc.in2p3.fr/cc_accueil.php3?lang=fr

*Free Space

Token Total Guaranteed Unused
------------------------------------------
BEIJING-LCG2_DATADISK 6.366 6.366 6.343
GRIF-LAL_DATADISK 6.0 6.0 5.732
GRIF-LPNHE_DATADISK 3.906 3.906 3.89
GRIF-SACLAY_DATADISK 6.0 6.0 5.994
IN2P3-CPPM_DATADISK 1.0 1.0 0.977
IN2P3-LAPP_DATADISK 4.0 4.0 4.0
IN2P3-LPC_DATADISK 3.0 3.0 3.0
IN2P3-LPSC_DATADISK 0.488 0.488 0.482
RO-02-NIPNE_DATADISK 3.0 3.0 3.0
RO-07-NIPNE_DATADISK 2.0 2.0 2.0
TOKYO-LCG2_DATADISK 15.0 15.0 13.643

=== 3rd June ===

* Lyon is still in donwtime, no transfers from LYON to T2s in this morning.

; Throughput(snapshot) (MB/s) : [[Image:LYON.throughput.14400_20080603.png]]
; Number of transfer errors(snapshot) : [[Image:LYON.num_file_xs_error.14400_20080603.png]]

=== 4th June ===
;22h02
* Still no transfers, desipte that all the services in LYON shows OK.

;Throughput(snapshot)
[[Image:LYON.throughput.3600_20080604.png]]

;Services in LYON
[[Image:services_cc-in2p3_2008060422h15.PNG]]

;Free space

BEIJING-LCG2_DATADISK 6.366 6.366 6.343
BEIJING-LCG2_MCDISK 5.457 5.457 5.457
GRIF-LAL_DATADISK 6.0 6.0 5.732
GRIF-LAL_MCDISK 7.0 7.0 3.724
GRIF-LPNHE_DATADISK 3.906 3.906 3.89
GRIF-LPNHE_MCDISK 1.0 1.0 0.194
GRIF-SACLAY_DATADISK 6.0 6.0 5.994
GRIF-SACLAY_MCDISK 2.0 2.0 1.713
IN2P3-CPPM_DATADISK 1.0 1.0 0.977
IN2P3-CPPM_MCDISK 0.781 0.781 0.765
IN2P3-LAPP_DATADISK 4.0 4.0 4.0
IN2P3-LAPP_MCDISK 2.0 2.0 0.745
IN2P3-LPC_DATADISK 3.0 3.0 3.0
IN2P3-LPC_MCDISK 2.0 2.0 1.936
IN2P3-LPSC_DATADISK 0.488 0.488 0.482
IN2P3-LPSC_MCDISK 2.0 2.0 1.518
RO-02-NIPNE_DATADISK 3.0 3.0 3.0
RO-02-NIPNE_MCDISK 1.0 1.0 0.995
RO-07-NIPNE_DATADISK 2.0 2.0 2.0
RO-07-NIPNE_MCDISK 1.0 1.0 0.654
TOKYO-LCG2_DATADISK 15.0 15.0 13.643
TOKYO-LCG2_MCDISK 10.0 10.0 10.0

=== 5th June ===

* Data produced on Tuesday are supposed to be shipped in this night.

;14h50
* Now the RAW data has begun to shipped to LYON.

;Throughput(snapshot)(MB/s)
[[Image:LYON.throughput.14400_RAW_20080605_1816.png]]

;Number of transfer errors (snapshot)
[[Image:LYON.num_file_xs_error.14400_20080605_1823.png]]

;21h30
* IN2P3-CC_MCDISK has some problem of staging file. But it should have nothing to do with T0 export.

* 47 datasets have been exported to T1s. The total is 90.

;Throughput(snapshot)(MB/s)
[[Image:LYON.throughput.14400 20080605 2140.png]]

;Number of transfer errors (snapshot) -- It's the problem of staging file in T1 and nothing to do with FDR transfers.
[[Image:LYON.num_file_xs_error.14400_20080605_2140.png]]

=== 6th June ===
;09h50

;Throughput(snapshot) (MB/s)
[[Image:LYON.throughput.14400 200806060 0958.png]]

;Number of transfer errors (snapshot)
[[Image:LYON.num file xs error.14400 20080606 0959.png]]

;10h03
* Now all the RAW datasets have been exported to T1. The total number is 90.
http://atlas.web.cern.ch/Atlas/tzero/prod2/monitoring/tables/ddm_RAW_status.html

=== 7th June ===
* at CERN, will re-start the bulk processing with using the upcoming AtlasPoint1-14.1.0.12 release cache.

;11h30
* SRM in LYON is still in bad shape. Experts are working on it.

=== 8th June ===
;14h20
* FDR2 AODs/DPDs/TAGs/ESDs are being exported to LYON (IN2P3-CC_DATADISK), but haven't been shipped to T2s.

;Number of datasets transfered by far
$ dq2-list-dataset-site IN2P3-CC_DATADISK|grep fdr08_run2.*|wc -l
68

;Throughput(T0-T1,snapshot)(MB/s)
[[Image:LYON.throughput.14400 200806081420.png]]

;Number of transfer errors
[[Image:LYON.num file xs error.14400 200806081420.png]]

;Throughput(T1-T2,snapshot)(MB/s)
[[Image:LYON.T2.throughput.14400 T1-T2 200806081420.png]]

=== 9th June ===
;0h20
* FDR datasets has begun to be shipped to T2s.
* Some T2s (e.g. BEIJING) subscribed dedicated datasets, which will be done later in today's morning.

;Shipped datasets

IN2P3-CC_DATADISK 315
BEIJING-LCG2_DATADISK 0
GRIF-LAL_DATADISK 79
GRIF-LPNHE_DATADISK 39
GRIF-SACLAY_DATADISK 38
IN2P3-CPPM_DATADISK 0
IN2P3-LAPP_DATADISK 12
IN2P3-LPC_DATADISK 12
IN2P3-LPSC_DATADISK 32
RO-02-NIPNE_DATADISK 29
RO-07-NIPNE_DATADISK 46
TOKYO-LCG2_DATADISK 134

;Free space

BEIJING-LCG2_DATADISK 6.366 6.366 6.343
GRIF-LAL_DATADISK 6.0 6.0 5.579
GRIF-LPNHE_DATADISK 3.906 3.906 3.817
GRIF-SACLAY_DATADISK 6.0 6.0 5.927
IN2P3-CPPM_DATADISK 1.0 1.0 0.977
IN2P3-LAPP_DATADISK 4.0 4.0 4.0
IN2P3-LPC_DATADISK 0 0 0
IN2P3-LPSC_DATADISK 0.488 0.488 0.439
RO-02-NIPNE_DATADISK 3.0 3.0 2.986
RO-07-NIPNE_DATADISK 2.0 2.0 1.896
TOKYO-LCG2_DATADISK 15.0 15.0 13.313

;Troughputs(snapshot)(MB/s)
[[Image:LYON.throughput.14400 200806090046.png]]

;Number of transfer erros(snapshot)
* Note: No errors found in FDR2 transfers. Errors showed in this snapshot are since other production data transfers.
[[Image:LYON.num file xs error.14400 200806090047.png]]

;09h24

;Troughputs(snapshot)(MB/s)
[[Image:LYON.throughput.14400 200806090922.png]]

;Number of transfer erros(snapshot)
* Note: No errors found in FDR2 transfers. Errors showed in this snapshot are since other production data transfers.
[[Image:LYON.num file xs error.14400 20806090922.png]]

;Shipped datasets

IN2P3-CC_DATADISK 319
BEIJING-LCG2_DATADISK 0
GRIF-LAL_DATADISK 79
GRIF-LPNHE_DATADISK 39
GRIF-SACLAY_DATADISK 38
IN2P3-CPPM_DATADISK 0
IN2P3-LAPP_DATADISK 12
IN2P3-LPC_DATADISK 12
IN2P3-LPSC_DATADISK 32
RO-02-NIPNE_DATADISK 59
RO-07-NIPNE_DATADISK 49
TOKYO-LCG2_DATADISK 134

;Free space
BEIJING-LCG2_DATADISK 6.366 6.366 6.343
GRIF-LAL_DATADISK 6.0 6.0 5.579
GRIF-LPNHE_DATADISK 3.906 3.906 3.817
GRIF-SACLAY_DATADISK 6.0 6.0 5.927
IN2P3-CPPM_DATADISK 1.0 1.0 0.977
IN2P3-LAPP_DATADISK 4.0 4.0 4.0
IN2P3-LPC_DATADISK 3.0 3.0 3.0
IN2P3-LPSC_DATADISK 0.488 0.488 0.439
RO-02-NIPNE_DATADISK 3.0 3.0 2.945
RO-07-NIPNE_DATADISK 2.0 2.0 1.885
TOKYO-LCG2_DATADISK 15.0 15.0 13.313

;14h00
Many failed with RO-02-NIPNE_DATADISK.

[FTS] FTS State [Failed] FTS Retries [1] Reason [TRANSFER error during TRANSFER phase: [PERMISSION] the server sent an error response: 550 550 rfio write failure: Permission denied.] Source Host [ccsrm.in2p3.fr] 16
[FTS] FTS State [Failed] FTS Retries [1] Reason [TRANSFER error during TRANSFER phase: [TRANSFER_TIMEOUT] gridftp_copy_wait: Connection timed out] Source Host [ccsrm.in2p3.fr] 9
[FTS] FTS State [Failed] FTS Retries [1] Reason [TRANSFER error during TRANSFER phase: [GRIDFTP] the server sent an error response: 426 426 Transfer aborted (Unexpected Exception : java.io.IOException: Broken pipe)] Source Host [ccsrm.in2p3.fr] 2
[FTS] FTS State [Failed] FTS Retries [1] Reason [The process serving the transfer (status = PREPARING) is no longer active (could not open file /proc/3775/cmdline)] Source Host [ccsrm.in2p3.fr] 1
[FTS] FTS State [Failed] FTS Retries [1] Reason [The process serving the transfer (status = PREPARING) is no longer active (could not open file /proc/30076/cmdline)] Source Host [ccsrm.in2p3.fr] 1

;20h10
* FDR datasets have finished transferred to all T2s except BEIJING-LCG2_DATADISK and RO-02-NIPNE_DATADISK.
* Many "Connection timed out" errors found at BEIJING-LCG2_DATADISK and RO-02-NIPNE_DATADISK.

[FTS] FTS State [Failed] FTS Retries [1] Reason [SOURCE error during PREPARATION phase: [CONNECTION] failed to contact on remote SRM [httpg://ccsrm.in2p3.fr:8443/srm/managerv2]. Givin' up after 3 tries] Source Host [ccsrm.in2p3.fr] 111
[FTS] FTS State [Failed] FTS Retries [1] Reason [TRANSFER error during TRANSFER phase: [TRANSFER_TIMEOUT] gridftp_copy_wait: Connection timed out] Source Host [ccsrm.in2p3.fr] 95
[FTS] FTS State [Failed] FTS Retries [1] Reason [SOURCE error during PREPARATION phase: [REQUEST_TIMEOUT] failed to prepare source file in 180 seconds] Source Host [ccsrm.in2p3.fr] 3

; Shipped datasets

IN2P3-CC_DATADISK 134
BEIJING-LCG2_DATADISK 79
GRIF-LAL_DATADISK 67
GRIF-LPNHE_DATADISK 32
GRIF-SACLAY_DATADISK 26
IN2P3-CPPM_DATADISK 52
IN2P3-LAPP_DATADISK 52
IN2P3-LPC_DATADISK 52
IN2P3-LPSC_DATADISK 32
RO-02-NIPNE_DATADISK 49
RO-07-NIPNE_DATADISK 49
TOKYO-LCG2_DATADISK 126

=== 10th June ===

;Summary

All T2s except BEIJING-LCG2_DATADISK and RO-02-NIPNE_DATADISK were in good status when we made FDR transfers.
The problem of "Connection timed out" with transfers to BEIJING will be hopefully solved by tuned the transfer duration >1800s by Lionel.
RO-02-NIPNE is said being upgraded from SL3 to SL4.

=== 11st June ===

=== 12nd June ===

Atlas:FDR2

2008-06-09T18:18:52Z

Erming: /* 9th June */

Atlas:FDR2

2008-06-09T12:19:02Z

Erming: /* 9th June */

= Monitoring pages/graphs =
=== FTS ===
http://cctoolsafs.in2p3.fr/fts/monitoring/prod/ftsmonitor.php?vo=atlas

=== T0-T1 (ALL) ===
; FDR monitoring page - T1s
http://atldq2pro.cern.ch:8000/ft/mon/fdrmon_tier1s.html

http://dashb-atlas-data-tier0.cern.ch/dashboard/request.py/site
*Throughput 
:http://dashb-atlas-data-tier0.cern.ch/dashboard/templates/plots/OVERVIEW.throughput.14400.png
*Errors 
:http://dashb-atlas-data-tier0.cern.ch/dashboard/templates/plots/OVERVIEW.num_file_xs_error.14400.png

=== T0-T1 (Lyon) ===
http://dashb-atlas-data-tier0.cern.ch/dashboard/request.py/site?statsInterval=4&name=LYON
*Throughput T0-LYON 
:http://dashb-atlas-data-tier0.cern.ch/dashboard/templates/plots/LYON.throughput.14400.png
*Errors T0-LYON 
:http://dashb-atlas-data-tier0.cern.ch/dashboard/templates/plots/LYON.num_file_xs_error.14400.png
*Throughput Others-LYON 
:http://dashb-atlas-data.cern.ch/dashboard/templates/plots/LYON.T1.throughput.14400.png
*Errors Others-LYON :http://dashb-atlas-data.cern.ch/dashboard/templates/plots/LYON.T1.num_file_xs_error.14400.png

=== T1-T1 ===

=== T1-T2 (Lyon) ===
; FDR monitoring page
http://atldq2pro.cern.ch:8000/ft/mon/fdrmon_TiersInfo.html

http://atldq2pro.cern.ch:8000/ft/mon/fdrmon_T1-T2_matrix_day.html

; Dashboard
http://dashb-atlas-data.cern.ch/dashboard/request.py/site?statsInterval=4&name=LYON
*Throughput 
:http://dashb-atlas-data.cern.ch/dashboard/templates/plots/LYON.T2.throughput.14400.png
*Errors
:http://dashb-atlas-data.cern.ch/dashboard/templates/plots/LYON.T2.num_file_xs_error.14400.png

=== Network Graphs ===
; IN2P3 Weathermap
* http://netstat.in2p3.fr/weathermap/
* http://netstat.in2p3.fr/weathermap/graphiques/

; cc-in2p3 : [http://netstat.in2p3.fr/weathermap/graphiques/lyo-cern.html lyo-cern] (lhcopn-in2p3.cern.ch) http://netstat.in2p3.fr/weathermap/graphiques/lyo-cern-daily.gif
; lyon-nord : [http://netstat.in2p3.fr/weathermap/graphiques/lyo-nrd.html lyo-nrd] http://netstat.in2p3.fr/weathermap/graphiques/lyo-nrd-daily.gif
; orsay : [http://netstat.in2p3.fr/weathermap/graphiques/orsay.html orsay] (in2p3-orsay.cssi.renater.fr http://netstat.in2p3.fr/weathermap/graphiques/orsay-daily.gif
; lal-cc : [http://netstat.in2p3.fr/weathermap/graphiques/lal-cc.html Liaison LAL - CC] http://netstat.in2p3.fr/weathermap/graphiques/lal-cc-daily.gif
; lal : [http://netstat.in2p3.fr/weathermap/graphiques/lal.html lal] http://netstat.in2p3.fr/weathermap/graphiques/lal-daily.gif
; lpnhe: [http://netstat.in2p3.fr/weathermap/graphiques/lpnhe-nrd.html lpnhe-nrd] (in2p3-jussieu.cssi.renater.fr) http://netstat.in2p3.fr/weathermap/graphiques/lpnhe-nrd-daily.gif [http://netstat.in2p3.fr/weathermap/graphiques/lpnhe.html lpnhe] (Paris-LPNHE.in2p3.fr) http://netstat.in2p3.fr/weathermap/graphiques/lpnhe-daily.gif
; lapp : [http://netstat.in2p3.fr/weathermap/graphiques/ann-nrd.html ann-nrd] http://netstat.in2p3.fr/weathermap/graphiques/ann-nrd-daily.gif
; lpc: [http://netstat.in2p3.fr/weathermap/graphiques/lpc-cf.html lpc-cf]
http://netstat.in2p3.fr/weathermap/graphiques/lpc-cf-daily.gif
; lpsc : [http://netstat.in2p3.fr/weathermap/graphiques/lpsc.html lpsc] http://netstat.in2p3.fr/weathermap/graphiques/lpsc-daily.gif
; cppm : [http://netstat.in2p3.fr/weathermap/graphiques/cppm.html cppm] 
http://netstat.in2p3.fr/weathermap/graphiques/cppm-daily.gif
; international (tokyo etc.) : [http://netstat.in2p3.fr/weathermap/graphiques/parisnrd.html parisnrd] 
http://netstat.in2p3.fr/weathermap/graphiques/parisnrd-daily.gif
; GEANT - NYC : [http://dc-snmp.wcc.grnoc.iu.edu/manlan/show-graph.cgi?title=sw.newy32aoa.manlan.internet2.edu--te10/1&rrdname=sw.newy32aoa.manlan.internet2.edu--te10_1.rrd MANLAN] 
http://dc-snmp.wcc.grnoc.iu.edu/manlan/img/sw.newy32aoa.manlan.internet2.edu--te10_1-std5.gif
; NYC - TOKYO : [http://dc-snmp.wcc.grnoc.iu.edu/manlan/show-graph.cgi?title=sw.newy32aoa.manlan.internet2.edu--te11/1&rrdname=sw.newy32aoa.manlan.internet2.edu--te11_1.rrd MANLAN] 
http://dc-snmp.wcc.grnoc.iu.edu/manlan/img/sw.newy32aoa.manlan.internet2.edu--te11_1-std5.gif
; tokyo : [http://www.icepp.jp/monitor/gridftp/monitor.html gridftp monitor] http://www.icepp.jp/monitor/gridftp/img/all.png

= T0-T1 =

= T1-T2 =
==== General informations ====
;FDR page:
https://twiki.cern.ch/twiki/bin/view/Atlas/FullDressRehearsal

;DDM_in_FRD2: https://twiki.cern.ch/twiki/bin/view/Atlas/DDMOperationsGroup#DDM_in_FRD2

;Replication status(checked ~ every 2h?)
http://pandamon.usatlas.bnl.gov:25880/server/pandamon/query?mode=listFDR

;Lyon FTS monitor: http://cctoolsafs.in2p3.fr/fts/monitoring/prod/ftsmonitor.php?vo=atlas

;Contacts:

* IN2P3-CPPM_DATADISK :
* IN2P3-LPSC_DATADISK :
* IN2P3-LPC_DATADISK :
* GRIF-LAL_DATADISK : (grid.admin a lal.in2p3.fr)
* GRIF-LPNHE_DATADISK :
* GRIF-SACLAY_DATADISK :
* BEIJING-LCG2_DATADISK : (yanxf a ihep.ac.cn, Erming.Pei a cern.ch)
* RO-07-NIPNE_DATADISK : (ciubancan a nipne.ro)
* RO-02-NIPNE_DATADISK : (tpreda a nipne.ro)
* TOKYO-LCG2_DATADISK : (lcg-admin a icepp.s.u-tokyo.ac.jp)

===== Shares =====
Only srmv2+ space tokens sites will collect data. The list of streams is : Egamma, Bphysics, MinBias, Muon and Jet. If no precise stream name is associated to the share, the site will get any stream. The site in parenthesis will collect the complementary datasets. The T2 ressources at T1 means that users without production role can access CPUs at the T1 without the Production role.
* US :
o AGLT2_DATADISK : 100 %
o MWT2_DATADISK: 100 %
o NET2_DATADISK: 100 %
o SLACXRD_DATADISK: 100 %
o SWT2_CPB_DATADISK: 100 %
o T2 CPU ressources in BNL to access AOD
* CA :
o SFU-LCG2_DATADISK : 40%
o VICTORIA-LCG2_DATADISK : 30%
o ALBERTA-LCG2_DATADISK : 30%
o TORONTO-LCG2_DATADISK : 50 %
* IT :
o INFN-MILANO_DATADISK : 25%
o INFN-FRASCATI_DATADISK : 25%
o INFN-ROMA1_DATADISK : 100%
o INFN-NAPOLI-ATLAS_DATADISK : 100 %
* TW :
o TW-FTT_DATADISK : 50 %
* NL
o JINR-LCG2_DATADISK : 50% Jet (RU-PROTVINO-IHEP_DATADISK)
o JINR-LCG2_DATADISK : 50% MinBias (RU-PROTVINO-IHEP_DATADISK)
o JINR-LCG2_DATADISK : 50% Muon (RU-PROTVINO-IHEP_DATADISK)
o JINR-LCG2_DATADISK : 50% Bphysics (RU-PNPI_DATADISK)
o RU-PROTVINO-IHEP_DATADISK : 50% Jet (JINR-LCG2_DATADISK)
o RU-PROTVINO-IHEP_DATADISK : 50% Muon (JINR-LCG2_DATADISK)
o RU-PROTVINO-IHEP_DATADISK : 50% MinBias (JINR-LCG2_DATADISK)
o RU-PROTVINO-IHEP_DATADISK : 50% Egamma (RU-PNPI_DATADISK)
o RU-PNPI_DATADISK : 50% Egamma (RU-PROTVINO-IHEP_DATADISK)
o RU-PNPI_DATADISK : 50% Bphysics (JINR-LCG2_DATADISK)
o RU-PNPI_DATADISK : 50% Jet (RRC-KI_DATADISK)
o RU-PNPI_DATADISK : 50% MinBias (RRC-KI_DATADISK)
o RRC-KI_DATADISK : 50% Jet (RU-PNPI_DATADISK)
o RRC-KI_DATADISK : 50% MinBias (RU-PNPI_DATADISK)
o RRC-KI_DATADISK : 50% Egamma
o RRC-KI_DATADISK : 50% Bphysics
o CSTCDIE_DATADISK : 20%
* FR :
o IN2P3-LAPP_DATADISK : 100 % Egamma
o IN2P3-CPPM_DATADISK : 100 % Egamma
o IN2P3-LPSC_DATADISK : 5 %
o IN2P3-LPC_DATADISK : 100% Muon, 50% Egamma
o BEIJING-LCG2_DATADISK : 100 % Egamma, 50% Muon, 35% MinBias
o RO-07-NIPNE_DATADISK : 10%
o RO-02-NIPNE_DATADISK : 10%
o GRIF-LAL_DATADISK : 45%
o GRIF-LPNHE_DATADISK : 25%
o GRIF-SACLAY_DATADISK : 30 %
o TOKYO-LCG2_DATADISK : 100%
o T2 CPU ressources in LYON to access AOD
* DE :
o DESY-HH_DATADISK : 100%
o CSCS-LCG2_DATADISK: 100%
o DESY-ZN_DATADISK : 50%
o GOEGRID_DATADISK : 50%
o WUPPERTALPROD_DATADISK: 25%
o CYFRONET-LCG2_DATADISK : 15%
o PRAGUELCG2_DATADISK : 25%
o LRZ-LMU_DATADISK : 35%
o UNI-FREIBURG_DATADISK : 100% Jet, 100% Muon
* UK :
o UKI-SOUTHGRID-CAM-HEP_DATADISK : 50 % Muon (UKI-NORTHGRID-LANCS-HEP_DATADISK
o UKI-NORTHGRID-LANCS-HEP_DATADISK : 50 % Muon (UKI-SOUTHGRID-CAM-HEP_DATADISK)
o UKI-NORTHGRID-SHEF-HEP_DATADISK : 50 % MinBias (UKI-SCOTGRID-GLASGOW_DATADISK)
o UKI-SCOTGRID-GLASGOW_DATADISK : 50 % MinBias (UKI-NORTHGRID-SHEF-HEP_DATADISK)
o UKI-LT2-RHUL_DATADISK : 50 % Egamma (UKI-NORTHGRID-LIV-HEP_DATADISK)
o UKI-NORTHGRID-LIV-HEP_DATADISK : 50 % Egamma (UKI-LT2-RHUL_DATADISK)
o UKI-SOUTHGRID-RALPP_DATADISK : 50 Egamma (UKI-LT2-RHUL_DATADISK)
o UKI-SOUTHGRID-BHAM-HEP_DATADISK : 50 Jet (UKI-SOUTHGRID-OX-HEP_DATADISK)
* ES :
o IFIC-LCG2_DATADISK: 50%
o UAM-LCG2_DATADISK: 25%
o IFAE_DATADISK: 25%
* NDGF :
o SIGNET_DATADISK : 50 %

== Logbook ==

=== 2nd June ===

* Since Lyon is in donwtime, so there's no transfers from LYON to T2s today.
http://cc.in2p3.fr/cc_accueil.php3?lang=fr

*Free Space

Token Total Guaranteed Unused
------------------------------------------
BEIJING-LCG2_DATADISK 6.366 6.366 6.343
GRIF-LAL_DATADISK 6.0 6.0 5.732
GRIF-LPNHE_DATADISK 3.906 3.906 3.89
GRIF-SACLAY_DATADISK 6.0 6.0 5.994
IN2P3-CPPM_DATADISK 1.0 1.0 0.977
IN2P3-LAPP_DATADISK 4.0 4.0 4.0
IN2P3-LPC_DATADISK 3.0 3.0 3.0
IN2P3-LPSC_DATADISK 0.488 0.488 0.482
RO-02-NIPNE_DATADISK 3.0 3.0 3.0
RO-07-NIPNE_DATADISK 2.0 2.0 2.0
TOKYO-LCG2_DATADISK 15.0 15.0 13.643

=== 3rd June ===

* Lyon is still in donwtime, no transfers from LYON to T2s in this morning.

; Throughput(snapshot) (MB/s) : [[Image:LYON.throughput.14400_20080603.png]]
; Number of transfer errors(snapshot) : [[Image:LYON.num_file_xs_error.14400_20080603.png]]

=== 4th June ===
;22h02
* Still no transfers, desipte that all the services in LYON shows OK.

;Throughput(snapshot)
[[Image:LYON.throughput.3600_20080604.png]]

;Services in LYON
[[Image:services_cc-in2p3_2008060422h15.PNG]]

;Free space

BEIJING-LCG2_DATADISK 6.366 6.366 6.343
BEIJING-LCG2_MCDISK 5.457 5.457 5.457
GRIF-LAL_DATADISK 6.0 6.0 5.732
GRIF-LAL_MCDISK 7.0 7.0 3.724
GRIF-LPNHE_DATADISK 3.906 3.906 3.89
GRIF-LPNHE_MCDISK 1.0 1.0 0.194
GRIF-SACLAY_DATADISK 6.0 6.0 5.994
GRIF-SACLAY_MCDISK 2.0 2.0 1.713
IN2P3-CPPM_DATADISK 1.0 1.0 0.977
IN2P3-CPPM_MCDISK 0.781 0.781 0.765
IN2P3-LAPP_DATADISK 4.0 4.0 4.0
IN2P3-LAPP_MCDISK 2.0 2.0 0.745
IN2P3-LPC_DATADISK 3.0 3.0 3.0
IN2P3-LPC_MCDISK 2.0 2.0 1.936
IN2P3-LPSC_DATADISK 0.488 0.488 0.482
IN2P3-LPSC_MCDISK 2.0 2.0 1.518
RO-02-NIPNE_DATADISK 3.0 3.0 3.0
RO-02-NIPNE_MCDISK 1.0 1.0 0.995
RO-07-NIPNE_DATADISK 2.0 2.0 2.0
RO-07-NIPNE_MCDISK 1.0 1.0 0.654
TOKYO-LCG2_DATADISK 15.0 15.0 13.643
TOKYO-LCG2_MCDISK 10.0 10.0 10.0

=== 5th June ===

* Data produced on Tuesday are supposed to be shipped in this night.

;14h50
* Now the RAW data has begun to shipped to LYON.

;Throughput(snapshot)(MB/s)
[[Image:LYON.throughput.14400_RAW_20080605_1816.png]]

;Number of transfer errors (snapshot)
[[Image:LYON.num_file_xs_error.14400_20080605_1823.png]]

;21h30
* IN2P3-CC_MCDISK has some problem of staging file. But it should have nothing to do with T0 export.

* 47 datasets have been exported to T1s. The total is 90.

;Throughput(snapshot)(MB/s)
[[Image:LYON.throughput.14400 20080605 2140.png]]

;Number of transfer errors (snapshot) -- It's the problem of staging file in T1 and nothing to do with FDR transfers.
[[Image:LYON.num_file_xs_error.14400_20080605_2140.png]]

=== 6th June ===
;09h50

;Throughput(snapshot) (MB/s)
[[Image:LYON.throughput.14400 200806060 0958.png]]

;Number of transfer errors (snapshot)
[[Image:LYON.num file xs error.14400 20080606 0959.png]]

;10h03
* Now all the RAW datasets have been exported to T1. The total number is 90.
http://atlas.web.cern.ch/Atlas/tzero/prod2/monitoring/tables/ddm_RAW_status.html

=== 7th June ===
* at CERN, will re-start the bulk processing with using the upcoming AtlasPoint1-14.1.0.12 release cache.

;11h30
* SRM in LYON is still in bad shape. Experts are working on it.

=== 8th June ===
;14h20
* FDR2 AODs/DPDs/TAGs/ESDs are being exported to LYON (IN2P3-CC_DATADISK), but haven't been shipped to T2s.

;Number of datasets transfered by far
$ dq2-list-dataset-site IN2P3-CC_DATADISK|grep fdr08_run2.*|wc -l
68

;Throughput(T0-T1,snapshot)(MB/s)
[[Image:LYON.throughput.14400 200806081420.png]]

;Number of transfer errors
[[Image:LYON.num file xs error.14400 200806081420.png]]

;Throughput(T1-T2,snapshot)(MB/s)
[[Image:LYON.T2.throughput.14400 T1-T2 200806081420.png]]

=== 9th June ===
;0h20
* FDR datasets has begun to be shipped to T2s.
* Some T2s (e.g. BEIJING) subscribed dedicated datasets, which will be done later in today's morning.

;Shipped datasets

IN2P3-CC_DATADISK 315
BEIJING-LCG2_DATADISK 0
GRIF-LAL_DATADISK 79
GRIF-LPNHE_DATADISK 39
GRIF-SACLAY_DATADISK 38
IN2P3-CPPM_DATADISK 0
IN2P3-LAPP_DATADISK 12
IN2P3-LPC_DATADISK 12
IN2P3-LPSC_DATADISK 32
RO-02-NIPNE_DATADISK 29
RO-07-NIPNE_DATADISK 46
TOKYO-LCG2_DATADISK 134

;Free space

BEIJING-LCG2_DATADISK 6.366 6.366 6.343
GRIF-LAL_DATADISK 6.0 6.0 5.579
GRIF-LPNHE_DATADISK 3.906 3.906 3.817
GRIF-SACLAY_DATADISK 6.0 6.0 5.927
IN2P3-CPPM_DATADISK 1.0 1.0 0.977
IN2P3-LAPP_DATADISK 4.0 4.0 4.0
IN2P3-LPC_DATADISK 0 0 0
IN2P3-LPSC_DATADISK 0.488 0.488 0.439
RO-02-NIPNE_DATADISK 3.0 3.0 2.986
RO-07-NIPNE_DATADISK 2.0 2.0 1.896
TOKYO-LCG2_DATADISK 15.0 15.0 13.313

;Troughputs(snapshot)(MB/s)
[[Image:LYON.throughput.14400 200806090046.png]]

;Number of transfer erros(snapshot)
* Note: No errors found in FDR2 transfers. Errors showed in this snapshot are since other production data transfers.
[[Image:LYON.num file xs error.14400 200806090047.png]]

;09h24

;Troughputs(snapshot)(MB/s)
[[Image:LYON.throughput.14400 200806090922.png]]

;Number of transfer erros(snapshot)
* Note: No errors found in FDR2 transfers. Errors showed in this snapshot are since other production data transfers.
[[Image:LYON.num file xs error.14400 20806090922.png]]

;Shipped datasets

IN2P3-CC_DATADISK 319
BEIJING-LCG2_DATADISK 0
GRIF-LAL_DATADISK 79
GRIF-LPNHE_DATADISK 39
GRIF-SACLAY_DATADISK 38
IN2P3-CPPM_DATADISK 0
IN2P3-LAPP_DATADISK 12
IN2P3-LPC_DATADISK 12
IN2P3-LPSC_DATADISK 32
RO-02-NIPNE_DATADISK 59
RO-07-NIPNE_DATADISK 49
TOKYO-LCG2_DATADISK 134

;Free space
BEIJING-LCG2_DATADISK 6.366 6.366 6.343
GRIF-LAL_DATADISK 6.0 6.0 5.579
GRIF-LPNHE_DATADISK 3.906 3.906 3.817
GRIF-SACLAY_DATADISK 6.0 6.0 5.927
IN2P3-CPPM_DATADISK 1.0 1.0 0.977
IN2P3-LAPP_DATADISK 4.0 4.0 4.0
IN2P3-LPC_DATADISK 3.0 3.0 3.0
IN2P3-LPSC_DATADISK 0.488 0.488 0.439
RO-02-NIPNE_DATADISK 3.0 3.0 2.945
RO-07-NIPNE_DATADISK 2.0 2.0 1.885
TOKYO-LCG2_DATADISK 15.0 15.0 13.313

;14h00
Many failed with RO-02-NIPNE_DATADISK.

[FTS] FTS State [Failed] FTS Retries [1] Reason [TRANSFER error during TRANSFER phase: [PERMISSION] the server sent an error response: 550 550 rfio write failure: Permission denied.] Source Host [ccsrm.in2p3.fr] 16
[FTS] FTS State [Failed] FTS Retries [1] Reason [TRANSFER error during TRANSFER phase: [TRANSFER_TIMEOUT] gridftp_copy_wait: Connection timed out] Source Host [ccsrm.in2p3.fr] 9
[FTS] FTS State [Failed] FTS Retries [1] Reason [TRANSFER error during TRANSFER phase: [GRIDFTP] the server sent an error response: 426 426 Transfer aborted (Unexpected Exception : java.io.IOException: Broken pipe)] Source Host [ccsrm.in2p3.fr] 2
[FTS] FTS State [Failed] FTS Retries [1] Reason [The process serving the transfer (status = PREPARING) is no longer active (could not open file /proc/3775/cmdline)] Source Host [ccsrm.in2p3.fr] 1
[FTS] FTS State [Failed] FTS Retries [1] Reason [The process serving the transfer (status = PREPARING) is no longer active (could not open file /proc/30076/cmdline)] Source Host [ccsrm.in2p3.fr] 1

=== 10th June ===

=== 11st June ===

=== 12nd June ===

Atlas:FDR2

2008-06-09T08:35:04Z

Erming: /* 9th June */

Atlas:FDR2

2008-06-09T08:15:49Z

Erming: /* 9th June */

Atlas:FDR2

2008-06-09T07:43:59Z

Erming: /* 9th June */

Atlas:FDR2

2008-06-09T07:42:33Z

Erming: /* 9th June */

Atlas:FDR2

2008-06-09T07:30:01Z

Erming: /* 9th June */

File:LYON.num file xs error.14400 20806090922.png

2008-06-09T07:25:24Z

Erming:

File:LYON.throughput.14400 200806090922.png

2008-06-09T07:24:08Z

Erming:

Atlas:FDR2

2008-06-08T22:51:14Z

Erming: /* 9th June */

Atlas:FDR2

2008-06-08T22:50:11Z

Erming: /* 8th June */

Atlas:FDR2

2008-06-08T22:48:46Z

Erming: /* 8th June */

Atlas:FDR2

2008-06-08T22:48:15Z

Erming: /* 9th June */

Atlas:FDR2

2008-06-08T22:47:58Z

Erming: /* 8th June */

File:LYON.num file xs error.14400 200806090047.png

2008-06-08T22:47:46Z

Erming:

File:LYON.throughput.14400 200806090046.png

2008-06-08T22:46:31Z

Erming:

Atlas:FDR2

2008-06-08T22:44:57Z

Erming: /* 8th June */

Atlas:FDR2

2008-06-08T12:42:46Z

Erming: /* 8th June */

Atlas:FDR2

2008-06-08T12:42:31Z

Erming: /* 8th June */

Atlas:FDR2

2008-06-08T12:41:19Z

Erming: /* 8th June */

File:LYON.T2.throughput.14400 T1-T2 200806081420.png

2008-06-08T12:40:29Z

Erming:

Atlas:FDR2

2008-06-08T12:38:09Z

Erming: /* 8th June */

File:LYON.num file xs error.14400 200806081420.png

2008-06-08T12:37:58Z

Erming:

File:LYON.throughput.14400 200806081420.png

2008-06-08T12:36:52Z

Erming:

Atlas:FDR2

2008-06-07T19:52:00Z

Erming: /* 7th June */

Atlas:FDR2

2008-06-07T19:41:55Z

Erming: /* 7th June */

Atlas:FDR2

2008-06-07T19:40:33Z

Erming: /* 7th June */

Atlas:FDR2

2008-06-07T19:40:13Z

Erming: /* 7th June */

Atlas:FDR2

2008-06-06T08:05:56Z

Erming: /* 6th June */

File:LYON.num file xs error.14400 20080606 0959.png

2008-06-06T08:00:03Z

Erming: