Difference between revisions of "WLCG Data transfers"

Un article de lcgwiki.
Jump to: navigation, search
(→‎XRootD Monitoring : Site resolution in France (affichage "n/a"))
(→‎XRootD Monitoring : Site resolution in France)
Ligne 73: Ligne 73:
  
 
== XRootD Monitoring : Site resolution in France  ==
 
== XRootD Monitoring : Site resolution in France  ==
−
'''Affichage "n/a"'''
+
'''Affichage "n/a"''' <br>
 
Par ex. Trafic XRootD ATLAS vu au niveau du SE du LAPP au cours des 4 dernières heures :
 
Par ex. Trafic XRootD ATLAS vu au niveau du SE du LAPP au cours des 4 dernières heures :
 
* http://dashb-wlcg-transfers.cern.ch/ui/#grouping.dst=(site)&p.grouping=dst&src.site=(LAPP)&technology=(xrootd)&vo=(atlas)   
 
* http://dashb-wlcg-transfers.cern.ch/ui/#grouping.dst=(site)&p.grouping=dst&src.site=(LAPP)&technology=(xrootd)&vo=(atlas)   
Ligne 82: Ligne 82:
  
 
n/a is an indication that the topology resolution (aka the mapping between xrootd servers and clients to GRID site names) encountered problems. The topology resolution is done at the dashboard level, and France required some special configuration too, so let us go through and check if there is some mapping not properly specified or missing.
 
n/a is an indication that the topology resolution (aka the mapping between xrootd servers and clients to GRID site names) encountered problems. The topology resolution is done at the dashboard level, and France required some special configuration too, so let us go through and check if there is some mapping not properly specified or missing.
−
* Contact : Luca Magnoni via Julia Andreeva.
+
 
−
* Suivi via ticket GGUS  : https://ggus.eu/index.php?mode=ticket_info&ticket_id=121143
+
==== TABLE A VERIFIER/CORRIGER"
−
* Voici la table '''A VERIFIER/CORRIGER''' assurant le mapping entre site client XRootD et site de grille :
+
* Ci-dessous la table assurant le mapping entre site client XRootD et site de grille :
 +
* via ticket GGUS  : https://ggus.eu/index.php?mode=ticket_info&ticket_id=121143 ; Contact : Luca Magnoni (CERN)
 +
 
''for the xrootd dashboard the mapping is done with a static lookup of the client domain to a known list of grid sites (e.g."client_domain":"lal.in2p3.fr" is mapped to GRIF). If the mapping fails, the statistics is reported to the n/a category, as a fallback. For France in particular, if a client log arrives with the "in2p3" domain without a known sub-domain, it goes into a "IN2P3failover" category. Below the current list (actually a dictionary) for ATLAS and CMS, please let me know if you see anything incorrect or missing, of if you need any other info.''  
 
''for the xrootd dashboard the mapping is done with a static lookup of the client domain to a known list of grid sites (e.g."client_domain":"lal.in2p3.fr" is mapped to GRIF). If the mapping fails, the statistics is reported to the n/a category, as a fallback. For France in particular, if a client log arrives with the "in2p3" domain without a known sub-domain, it goes into a "IN2P3failover" category. Below the current list (actually a dictionary) for ATLAS and CMS, please let me know if you see anything incorrect or missing, of if you need any other info.''  
  

Version du 10:52, 13 mai 2016

Mars 2016 Petit Tour des Dashboards de monitoring des transferts de données :


Outils communs de monitoring des transferts de données

PRELIMINAIRE

Dashboard WLCG

ALICE, ATLAS, CMS, LHCb - Transferts FTS & XRootD Dashboard : http://dashb-wlcg-transfers.cern.ch/ui
Voir certains extraits du Dashboard WLCG

Dashboard FTS

ATLAS, CMS, LHCb - uniquement transferts FTS : http://dashb-fts-transfers.cern.ch/ui
Extraits (FC) du Dashboard FTS

  • Possibilité de spécifier un intervalle de dates plus important, intérêt peut être pour le monitoring d’un site multi-VO sinon préférer l’outil spécifique à la VO par ex. monitoring rucio dans le cas ATLAS http://dashb-atlas-ddm.cern.ch/ddm2/
  • Il y a + de détails : efficacité, erreurs… qu'avec le dashboard général de WLCG

Dashboard XRooTD

'ATLAS, CMS - uniquement trafics XRootD ': http://dashb-wdt-xrootd.cern.ch/ui

Monitoring XRootD ATLAS / CMS (configuration du)

Dans certains sites FR, il n'y a clairement pas de monitoring des trafics XrootD ATLAS / CMS

ATLAS : les trafics fts & XRootd, en accès local & remote sont correctement monitorés au LPSC, CPPM, LAPP mais ce n'est pas le cas partout :

CMS : les trafics fts & XRootd, en accès local & remote sont correctement monitorés au GRIF (IRFU, LLR) et à l'IPHC :

Il y avait 2 petites erreurs (aujourd'hui corrigées) dans les valeurs par défaut proposées par Quattor  :

  1. au niveau du numéro de port (9330) du collecteur EU atlas-fax-eu-collector.cern.ch:9330 (pour ATLAS)
  2. au niveau du host EU: CMS-AAA-EU-COLLECTOR.cern.ch:9330 (pour CMS)

Liens vers les pages qui font foi :

Il reste à vérifier/modifier la configuration du monitoring XRootD DPM/dCache au CC pour la remontée des infos ATLAS et CMS dans les quelques sites concernés.

XRootD Monitoring : Site resolution in France

Affichage "n/a"
Par ex. Trafic XRootD ATLAS vu au niveau du SE du LAPP au cours des 4 dernières heures :

idem pour CMS et le LLR :

n/a is an indication that the topology resolution (aka the mapping between xrootd servers and clients to GRID site names) encountered problems. The topology resolution is done at the dashboard level, and France required some special configuration too, so let us go through and check if there is some mapping not properly specified or missing.

==== TABLE A VERIFIER/CORRIGER"

for the xrootd dashboard the mapping is done with a static lookup of the client domain to a known list of grid sites (e.g."client_domain":"lal.in2p3.fr" is mapped to GRIF). If the mapping fails, the statistics is reported to the n/a category, as a fallback. For France in particular, if a client log arrives with the "in2p3" domain without a known sub-domain, it goes into a "IN2P3failover" category. Below the current list (actually a dictionary) for ATLAS and CMS, please let me know if you see anything incorrect or missing, of if you need any other info. 

  {
   "ATLAS": {
       "in2p3.fr": {
           "host_dict": {
               "lpn": {
                   "VOname": "GRIF-LPNHE",
                   "SiteName": "GRIF",
                   "Country": "France"
               },
               "lal": {
                   "VOname": "GRIF-LAL",
                   "SiteName": "GRIF",
                   "Country": "France"
               },
               "ccsrm" A CHANGER pour "ccw": {
                   "VOname": "IN2P3-CC",
                   "SiteName": "IN2P3-CC",
                   "Country": "France"
               },
               "mar": {
                   "VOname": "IN2P3-CPPM",
                   "SiteName": "IN2P3-CPPM",
                   "Country": "France"
               },
               "lapp": {
                   "VOname": "IN2P3-LAPP",
                   "SiteName": "IN2P3-LAPP",
                   "Country": "France"
               },
               "clrlcg": {
                   "VOname": "IN2P3-LPC",
                   "SiteName": "IN2P3-LPC",
                   "Country": "France"
               },
               "lpsc": {
                   "VOname": "IN2P3-LPSC",
                   "SiteName": "IN2P3-LPSC",
                   "Country": "France"
               }
           },
           "SiteName": "IN2P3",
           "VOname": "IN2P3failover",
           "Country": "France"
       }
     }
   }



   {
   "CMS": {
       "in2p3.fr": {
           "host_dict": {
               "lapp": {
                   "VOname": "T2_FR_GRIF_LLR",
                   "SiteName": "IN2P3-LAPP",
                   "Country": "France"
               },  A SUPPRIMER ! 
               "lpn": {
                   "VOname": "T2_FR_GRIF_IRFU",
                   "SiteName": "GRIF-LPNHE",
                   "Country": "France"
               },
               "lal": {
                   "VOname": "T2_FR_GRIF_LLR",
                   "SiteName": "GRIF-LAL",
                   "Country": "France"
               },
               "lpsc": {
                   "VOname": "IN2P3-LPSC",
                   "SiteName": "IN2P3-LPSC",
                   "Country": "France"
               },
               "mar": {
                   "VOname": "IN2P3-CPPM",
                   "SiteName": "CPPM",
                   "Country": "France"
               },A SUPPRIMER ! 
               "polgrid": {
                   "VOname": "T2_FR_GRIF_LLR",
                   "SiteName": "GRIF",
                   "Country": "France"
               },
               "sbgwn": {
                   "VOname": "T2_FR_IPHC",
                   "SiteName": "IN2P3-IRES",
                   "Country": "France"
               },
               "llrgrwn": {
                   "VOname": "T2_FR_GRIF_LLR",
                   "SiteName": "GRIFS",
                   "Country": "France"
               }
           },
           "SiteName": "IN2P3",
           "VOname": "IN2P3failover",
           "Country": "France"
       }
   }

Outils spécifiques aux VOs

Liens

Récupérée de « http://lcg.in2p3.fr/wiki/index.php?title=WLCG_Data_transfers&oldid=7950 »