Difference between revisions of "Atlas:StorageRequirements"

Un article de lcgwiki.
Jump to: navigation, search
(Deployment)
(ATLASUSERDISK)
Ligne 246: Ligne 246:
 
** <code>dpns-setacl -m g:atlas/Role=production:rwx,m:rwx,d:g:atlas/Role=production:rwx,d:m:rwx $DPNS_HOME/atlas/atlasuserdisk</code>
 
** <code>dpns-setacl -m g:atlas/Role=production:rwx,m:rwx,d:g:atlas/Role=production:rwx,d:m:rwx $DPNS_HOME/atlas/atlasuserdisk</code>
 
** <code>dpns-getacl $DPNS_HOME/atlas/atlasuserdisk</code>
 
** <code>dpns-getacl $DPNS_HOME/atlas/atlasuserdisk</code>
 +
<ul><pre>
 +
# group: atlas
 +
user::rwx
 +
group::rwx
 +
group:atlas:rwx
 +
group:atlas/Role=lcgadmin:rwx
 +
group:atlas/Role=production:rwx
 +
mask::rwx
 +
other::r-x
 +
default:user::rwx
 +
default:group::rwx
 +
default:group:atlas:rwx
 +
default:group:atlas/Role=lcgadmin:rwx
 +
default:group:atlas/Role=production:rwx
 +
default:mask::rwx
 +
default:other::r-x
 +
</pre></ul>
 +
 +
== ATLASSCRATCHDISK ==
 +
<span style="color:#FF0000;">This space is to be deployed to replace ATLASUSERDISK</span>.
 +
During the migraiton period, the both spaces should be available. The size of the space
 +
is preferably the same as ATLASUSERDISK, but it is also possible to start with a small size
 +
and increase it while monitoring the usage and decreasing ATLASUSERDISK.
 +
 +
This is the scratch space for users to put the output of their analysis jobs run at the site before copying them to their final destination. The files are to be deleted by central operation after a certain period of time.
 +
* 5TB for a typical T2 with ~500 CPUs and ~100 TB disk
 +
* VOMS group associated with the space: /atlas
 +
* Namespace directory to be created: .../atlas/atlasscratchdisk/
 +
* Namespace ACL: writable by all ATLAS users (group /atlas) and /atlas/Role=production (for central deletion). <span style="color:#FF0000;">all the subdirectories should follow this.</span>
 +
* Normally, sites have already this namespace created by user analysis jobs so far.
 +
* example commands
 +
** <code>dpns-entergrpmap --group "atlas"</code>
 +
** <code>dpns-entergrpmap --group "atlas/Role=production"</code>
 +
** <code>dpns-mkdir $DPNS_HOME/atlas/atlasscratchdisk</code>
 +
** <code>dpns-mkdir $DPNS_HOME/atlas/atlasscratchdisk</code>
 +
** <code>dpns-setacl -m g:atlas:rwx,m:rwx,d:g:atlas:r-x,d:m:rwx $DPNS_HOME/atlas/atlasscratchdisk</code>
 +
** <code>dpns-setacl -m g:atlas/Role=production:rwx,m:rwx,d:g:atlas/Role=production:rwx,d:m:rwx $DPNS_HOME/atlas/atlasscratchdisk</code>
 +
** <code>dpns-getacl $DPNS_HOME/atlas/atlasscratchdisk</code>
 
<ul><pre>
 
<ul><pre>
 
# group: atlas
 
# group: atlas

Version du 16:21, 5 mars 2009

Introduction

Sources of information:


Some comments about the contents

  • The requirements for ATLASGROUPDISK is temporary due to a technical limitation in DPM. Atlas has contacted DPM developpers. The proper implementation is expected to be available in September.


  • Site admin's are requested to create at least the space tokens. The namespace directories can be left to atlas production if the top directory is properly set up so that /atlas/Role=production is allowed to execute dpns-mkdir and dpns-setacl remotely. (Of course they are welcome to create them by themselves).


  • In general read permission should be given to all ATLAS users everywhere.
  • In general write permission should be given to /atlas/Role=production everywhere, so that the ATLAS central deletion tool can work.
  • It is suggested at LAPP to give write permission to atlas/Role=lcgadmin everywhere, so that the ATLAS admin can control the storage. (less people in Role=lcgadmin than in than in /atlas/Role=production). Stephane will discuss this issue with ATLAS.

Space Requirements

ATLAS top directory

  • On this page the top directory for ATLAS is represented by .../atlas. Each site should replace it according to their SE configuration.
    • eg. lapp-se01.in2p3.fr:/dpm/in2p3.fr/home/atlas for lapp.
  • Although it is up to sites policies, but it is recommended
    • to have this top directory configured so that only /atlas/Role=production and /atlas/Role=lcgadmin can create files/directories underneath and ordinary users cannot.


Summary table for Space token and Name space

  • The size in < > is for a typical T2 with ~500 CPUs and ~100 TB disk
space token voms group to be associated space to be reserved namespace directory(-ies) to be created
ATLASDATADISK /atlas/Role=production not known yet
see below*
.../atlas/atlasdatadisk

write permission only to /atlas/Role=production

ATLASMCDISK /atlas/Role=production a share of 60TB (plus possible extra)
see below*
.../atlas/atlasmcdisk
write permission only to atlas/Role=production
ATLASPRODDISK /atlas/Role=production <2TB>, (or 6GB/kSI2K)
(see below*)
.../atlas/atlasproddisk
write permission only to /atlas/Role=production
ATLASGROUPDISK /atlas (temporary solution. see below*) a share of 30TB
(see below*)
.../atlas/atlasgroupdisk
write permission to /atlas/Role=production


.../atlas/atlasgroupdisk/$GROUP
write permission to /atlas/Role=production and /atlas/$GROUP/Role=production

($GROUP: see below*)

ATLASUSERDISK
(Atlas StorageSetUp)
/atlas <5TB>
(see below*)
.../atlas/atlasuserdisk
write permission to all ATLAS users
write permission to /atlas/Role=production for central deletion
the same for all the subdirectories
ATLASLOCALGROUPDISK
(Atlas StorageSetUp)
/atlas/<locality>, <locality>=fr,ro,cn,jp,...
/atlas
sites to decide .../atlas/atlaslocalgroupdisk
write permission to local ATLAS users (/atlas/<locality>)
write permission to /atlas/Role=production for central deletion


  • A T3 will need ATLASDATADISK if it would like to receive real data, ATLASMCDISK to received simulated data, ATLASGROUPDISK to receive group analysis data, and ATLASPRODDISK to contribute official production. ATLASUSERDISK is not necessary, but may be needed if the site contributes non-local user analysis.
  • The spacetokens ATLASGRP<GROUP> (eg. ATLASGRPTOP), which were requested for ccrc08/fdr, are not required any longer, being repleaced with a single spacetoken ATLASGROUPDISK. There should not be a file using this spacetoken. They can be removced.
  • The spacetoken ATLASENDUSER, which was requested for ccrc08/fdr, is not required any longer, being replaced with ATLASUSERDISK. It can be removed.

ATLASDATADISK

  • Estimation of size comes later
  • VOMS group associated with the space: /atlas/Role=production
  • Namespace directory to be created: $DPNS_HOME/atlas/atlasdatadisk
    • Normally, sites have already this namespace created.
  • Namespace ACL: writable by only atlas/Role=production, readable by all ATLAS users
# group: atlas/Role=production
user::rwx
group::rwx              #effective:rwx
group:atlas/Role=production:rwx         #effective:rwx
mask::rwx
other::r-x
default:user::rwx
default:group::rwx
default:group:atlas/Role=production:rwx
default:mask::rwx
default:other::r-x

ATLASMCDISK

This is the space where simulated data produced centrally go in.

  • 15TB for a typical T2 with ~500 CPU?fs and ~100 TB disk
    • 60TB for a T2 requesting for 100% AOD.
    • The size will increase in case D1PD are requested by the site. (100% D1PD will be 30TB)
    • The required space is 1/3 of the above for the period until Sep 2008, 2/3 for Sep - Dec 2008, and full for Jan - Mar 2009.
    • using the share at ccrc08, the required size for each FR sites are;
                           Required Space at
       Site         Share  Jul08  Sep08  Dec08  Possibly with D1PD
       IN2P3-LAPP     25%   5 TB  10 TB  15 TB  22.5 TB
       IN2P3-CPPM      5%   1 TB   2 TB   3 TB   4.5 TB
       IN2P3-LPSC      5%   1 TB   2 TB   3 TB   4.5 TB
       IN2P3-LPC      25%   5 TB  10 TB  15 TB  22.5 TB
       BEIJING-LCG2   20%   4 TB   8 TB  12 TB  18 TB
       RO-07-NIPNE    10%   2 TB   4 TB   6 TB   9 TB
       RO-02-NIPNE    10%   2 TB   4 TB   6 TB   9 TB
       GRIF-LAL       45%   9 TB  18 TB  27 TB  40.5 TB 
       GRIF-LPNHE     25%   5 TB  10 TB  15 TB  22.5 TB
       GRIF-SACLAY    30%   6 TB  12 TB  18 TB  27 TB
       TOKYO-LCG2    100%  20 TB  40 TB  60 TB  90 TB
      
  • VOMS group associated with the space: /atlas/Role=production
  • Namespace directory to be created: $DPNS_HOME/atlas/atlasmcdisk
    • Normally, sites have already this namespace created.
  • Namespace ACL: writable by only atlas/Role=production, readable by all ATLAS users
    # group: atlas/Role=production
    user::rwx
    group::rwx              #effective:r-x
    group:atlas/Role=production:rwx         #effective:rwx
    mask::rwx
    other::r-x
    default:user::rwx
    default:group::rwx
    default:group:atlas/Role=production:rwx
    default:mask::rwx
    default:other::r-x
    

ATLASPRODDISK

This is the space used by the ATLAS production system (panda) for putting input files for the jobs to run on the site, downloading them from T1, and output files from the jobs which are to be replicated to the T1. Both input and output files are to be deleted centrally after a certain period.

  • 2TB for a typical T2 with ~500 CPUs (the size is to be re-visited)
    • scales with the CPU capacity of the site.
    • will be larger if the reconstruction jobs run on the site.
    • My private estimation is 6GB/kSI2K (400kSI2K/event for simulation, 4MB/HITS, 7days to keep).
    • Using the MoU value, the required space for each site is;
      Site    kSI2K  Space
      BEIJING  200    1.2 TB
      GRIF     800    4.8 TB (federation)
      LAPP     440    2.6 TB
      LPC      400    2.4 TB
      TOKYO   1000    6 TB
      RO       400    2.4 TB (federation)
      
  • VOMS group associated with the space: /atlas/Role=production
  • Namespace directory to be created: $DPNS_HOME/atlas/atlasproddisk
  • Namespace ACL: writable by only atlas/Role=production, readable by all ATLAS users
    # group: atlas/Role=production
    user::rwx
    group::rwx              #effective:rwx
    group:atlas/Role=production:rwx         #effective:rwx
    mask::rwx
    other::r-x
    default:user::rwx
    default:group::rwx
    default:group:atlas/Role=production:rwx
    default:mask::rwx
    default:other::r-x
    

ATLASGROUPDISK

This is the space to put group DPD files, which are not created by the central production system, but by group analyses with the group production roles.

  • 6TB for a typical T2 with ~500 CPUs and ~100 TB disk
    • 100% D2PD will be 30TB
    • The required space is 1/3 of the above for the period until Sep 2008, 2/3 for Sep - Dec 2008, and full for Jan - Mar 2009.
    • using the share at ccrc08, the required size for each FR sites are;
                           Required Space by
       Site         Share  Jul08    Sep08   Dec08
       IN2P3-LAPP     25%   2.5 TB   5 TB    7.5 TB  
       IN2P3-CPPM      5%   0.5 TB   1 TB    1.5 TB 
       IN2P3-LPSC      5%   0.5 TB   1 TB    1.5 TB 
       IN2P3-LPC      25%   2.5 TB   5 TB    7.5 TB 
       BEIJING-LCG2   20%   2 TB     4 TB    6 TB 
       RO-07-NIPNE    10%   1 TB     2 TB    3 TB 
       RO-02-NIPNE    10%   1 TB     2 TB    3 TB  
       GRIF-LAL       45%   4.5 TB   9 TB   13.5 TB 
       GRIF-LPNHE     25%   2.5 TB   5 TB    7.5 TB 
       GRIF-SACLAY    30%   3 TB     6 TB    9 TB 
       TOKYO-LCG2    100%  10 TB    20 TB   30 TB 
      
  • One single space to be reserved for all the group activities ($GROUP = phys-beauty, phys-exotics, phys-gener, phys-hi, phys-higgs, phys-lumin, phys-sm, phys-susy, phys-top, perf-egamma, perf-flavtag, perf-jets, perf-muons, perf-tau, etc.)
  • VOMS group associated with the space: /atlas
    • A temporary solution until multiple group support to the spaces is available.
    • Once it is available, the groups will be /atlas/Role=production and /atlas/$GROUP/Role=production for all $GROUP
  • Namespaces directory to be created and their ACLs:
    • $DPNS_HOME/atlas/atlasgroupdisk: writable by only atlas/Role=production, readable by all ATLAS users
    • $DPNS_HOME/atlas/atlasgroupdisk/$GROUP: writable by atlas/Role=production and /atlas/$GROUP/Role=production, readable by all ATLAS users
    • eg. for phys-higgs
    # group: atlas/Role=production
    user::rwx
    group::rwx              #effective:rwx
    group:atlas/Role=production:rwx         #effective:rwx
    group:atlas/phys-higgs/Role=production:rwx         #effective:rwx
    mask::rwx
    other::r-x
    default:user::rwx
    default:group::rwx
    default:group:atlas/Role=production:rwx
    default:group:atlas/phys-higgs/Role=production:rwx
    default:mask::rwx
    default:other::r-x
    

ATLASUSERDISK

This is the scratch space for users to put the output of their analysis jobs run at the site before copying them to their final destination. The files are to be deleted by central operation after a certain period of time.

  • 5TB for a typical T2 with ~500 CPUs and ~100 TB disk
  • VOMS group associated with the space: /atlas
  • Namespace directory to be created: .../atlas/atlasuserdisk/ (the old .../atlas/user is to be used for files without spacetoken)
  • Namespace ACL: writable by all ATLAS users (group /atlas) and /atlas/Role=production (for central deletion). The user directories underneath should be writable only by the owner and /atlas/Role=production (for central deletion). all the subdirectories should follow this.
  • Normally, sites have already this namespace created by user analysis jobs so far.
  • example commands
    • dpns-entergrpmap --group "atlas"
    • dpns-entergrpmap --group "atlas/Role=production"
    • dpns-mkdir $DPNS_HOME/atlas/atlasuserdisk
    • dpns-mkdir $DPNS_HOME/atlas/atlasuserdisk
    • dpns-setacl -m g:atlas:rwx,m:rwx,d:g:atlas:r-x,d:m:rwx $DPNS_HOME/atlas/atlasuserdisk
    • dpns-setacl -m g:atlas/Role=production:rwx,m:rwx,d:g:atlas/Role=production:rwx,d:m:rwx $DPNS_HOME/atlas/atlasuserdisk
    • dpns-getacl $DPNS_HOME/atlas/atlasuserdisk
    # group: atlas
    user::rwx
    group::rwx
    group:atlas:rwx
    group:atlas/Role=lcgadmin:rwx
    group:atlas/Role=production:rwx
    mask::rwx
    other::r-x
    default:user::rwx
    default:group::rwx
    default:group:atlas:rwx
    default:group:atlas/Role=lcgadmin:rwx
    default:group:atlas/Role=production:rwx
    default:mask::rwx
    default:other::r-x
    

ATLASSCRATCHDISK

This space is to be deployed to replace ATLASUSERDISK. During the migraiton period, the both spaces should be available. The size of the space is preferably the same as ATLASUSERDISK, but it is also possible to start with a small size and increase it while monitoring the usage and decreasing ATLASUSERDISK.

This is the scratch space for users to put the output of their analysis jobs run at the site before copying them to their final destination. The files are to be deleted by central operation after a certain period of time.

  • 5TB for a typical T2 with ~500 CPUs and ~100 TB disk
  • VOMS group associated with the space: /atlas
  • Namespace directory to be created: .../atlas/atlasscratchdisk/
  • Namespace ACL: writable by all ATLAS users (group /atlas) and /atlas/Role=production (for central deletion). all the subdirectories should follow this.
  • Normally, sites have already this namespace created by user analysis jobs so far.
  • example commands
    • dpns-entergrpmap --group "atlas"
    • dpns-entergrpmap --group "atlas/Role=production"
    • dpns-mkdir $DPNS_HOME/atlas/atlasscratchdisk
    • dpns-mkdir $DPNS_HOME/atlas/atlasscratchdisk
    • dpns-setacl -m g:atlas:rwx,m:rwx,d:g:atlas:r-x,d:m:rwx $DPNS_HOME/atlas/atlasscratchdisk
    • dpns-setacl -m g:atlas/Role=production:rwx,m:rwx,d:g:atlas/Role=production:rwx,d:m:rwx $DPNS_HOME/atlas/atlasscratchdisk
    • dpns-getacl $DPNS_HOME/atlas/atlasscratchdisk
    # group: atlas
    user::rwx
    group::rwx
    group:atlas:rwx
    group:atlas/Role=lcgadmin:rwx
    group:atlas/Role=production:rwx
    mask::rwx
    other::r-x
    default:user::rwx
    default:group::rwx
    default:group:atlas:rwx
    default:group:atlas/Role=lcgadmin:rwx
    default:group:atlas/Role=production:rwx
    default:mask::rwx
    default:other::r-x
    

ATLASLOCALGROUPDISK

This is a space for "local" users of the site, and is not included in the pledged resources to ATLAS.

  • See the Atlas StorageSetUp wiki page.
  • size to be decided by sites.
    • the resources not included in the pledge.
  • VOMS group associated with the space: /atlas/fr (or /atlas/ro, /atlas/cn, /atlas/jp correspondingly) /atlas
  • name space: $DPNS_HOME/atlas/atlaslocalgroupdisk (the path previously required .../atlas/fr/user (or .../atlas/ro/user, .../atlas/cn/user, .../atlas/jp/user, etc.) is to be used for files without space token)
  • ACL: write permission only to /atlas/fr group (or /atlas/ro, /atlas/cn, /atlas/jp correspondingly) and /atlas/Role=production
  • example ACL:
    # group: atlas/fr
    user::rwx
    group::rwx
    group:atlas/Role=lcgadmin:rwx
    group:atlas/Role=production:rwx
    group:atlas/fr:rwx              #effective:rwx
    mask::rwx
    other::r-x
    default:user::rwx
    default:group::r-x
    default:group:atlas:r-x
    default:group:atlas/Role=lcgadmin:rwx
    default:group:atlas/Role=production:rwx
    default:group:atlas/fr:rwx
    default:mask::rwx
    default:other::r-x
    

Deployment

Space Reservation 
  • The table shows spaces reserved/required.
    • The reserved spaces are obtained using srm commands.
    • The required spaces are obtained using the shares at ccrc08, thus they might change.
 Site         Share  DATADISK   MCDISK     PRODDISK  GROUPDISK    USERDISK  LOCAL
 IN2P3-LAPP     25%  5/ TB      5/15 TB    2/2.6     0.5/7.5 TB   0.5/      0.5 TB
 IN2P3-LPC      25%  5/ TB      5/15 TB    2/2.4     1/7.5 TB     2/
 IN2P3-CPPM      5%  1/ TB      1.5/3 TB   1/        0.5/1.5 TB   0.5/ TB   (0.5) TB
 IN2P3-LPSC      5%  1/ TB      2/3 TB     0.5/      0.5/1.5 TB   0.5/
 BEIJING-LCG2   20%  4.5/ TB    5.5/12 TB  0.9/1.2   0.9/6 TB     0.9/
 RO-07-NIPNE    10%  3/ TB      9/6 TB     2/TB      3/3 TB       5/TB
 RO-02-NIPNE    10%  5/ TB      9/6 TB     2/        3/3 TB       3/
 GRIF-LAL       45%  6/ TB      15/27 TB   6/        6/13.5 TB    2/
 GRIF-SACLAY    30%  4/         6/18 TB    2/         /9 TB       2/
 GRIF-LPNHE     25%  4/ TB      10/15 TB   1/        4/7.5 TB     3/        1.0 TB
 TOKYO-LCG2    100%  15/ TB     15/60 TB   2/6 TB     /30 TB      5/        5.0 TB
ATLASUSERDISK
SE owner group g: g:atlas g:atlas/ Role= production g:atlas/ Role= lcgadmin d:g: d:g:atlas d:g:atlas/ Role= production d:g:atlas/ Role= lcgadmin subdirectories
clrlcgse01.in2p3.fr root atlas rwx rwx rwx rwx rwx rwx rwx rwx SAM:ok, user08:ok, user08.EricLancon:?, user08.JohannesElmsheuser:?
grid05.lal.in2p3.fr Stephane Jezequel atlas/Role=production rwx rwx rwx rwx rwx rwx rwx rwx user:ok, user08:ok, user08.EricLancon:?, user08.FredericDerue:?, user08.JohannesElmsheuser:?
node12.datagrid.cea.fr root root rwx rwx rwx none rwx rwx rwx none SAM:ok, user:ok, user08:ok, user08.EricLancon:?, user08.JohannesElmsheuser:ok
lpnse1.in2p3.fr I Ueda atlas/Role=production rwx rwx rwx rwx rwx rwx rwx rwx SAM:ok, user08:ok, user08.EricLancon:ok, user08.FredericDerue:ok, user08.JohannesElmsheuser:
lapp-se01.in2p3.fr I Ueda atlas/Role=production rwx rwx rwx rwx rwx rwx rwx rwx SAM:ok, users:ok, user08:ok, user08.EricLancon:?, user08.JohannesElmsheuser:?
marsedpm.in2p3.fr I Ueda atlas/Role=production rwx rwx rwx rwx rwx rwx rwx rwx no subdirectory
lpsc-se-dpm-server.in2p3.fr Alessandro Di Girolamo atlas/Role=production rwx rwx rwx rwx rwx rwx rwx rwx SAM:ok
lcg-se01.icepp.jp I Ueda atlas/Role=production rwx rwx rwx rwx rwx rwx rwx rwx not checked yet
tbat05.nipne.ro root atlas/Role=production rwx rwx rwx rwx rwx rwx rwx rwx SAM:ok, user08.EricLancon:?
tbit00.nipne.ro Eric Lancon atlas/Role=pilot rwx rwx rwx rwx rwx rwx rwx rwx user08.EricLancon:?


ATLASLOCALGROUPDISK
SE owner group g: g:atlas g:atlas/ <locality> g:atlas/ Role= production g:atlas/ Role= lcgadmin d:g d:g:atlas d:g:atlas/
<locality>
d:g:atlas/
Role= production
d:g:atlas/
Role= lcgadmin
subdirectories
clrlcgse01.in2p3.fr root root rwx none rwx rwx rwx rwx none rwx rwx rwx none
grid05.lal.in2p3.fr Stephane Jezequel atlas/fr rwx r-x none rwx rwx rwx r-x rwx rwx rwx no subdirectory, 1 file (test1 by Stephane)
node12.datagrid.cea.fr root root rwx none rwx rwx rwx rwx none rwx rwx rwx jerome:NG
lpnse1.in2p3.fr root atlas/ Role=production rwx none rwx rwx rwx rwx none rwx rwx rwx none
lapp-se01.in2p3.fr root atlas/ Role=production r-x none rwx rwx rwx rwx none rwx rwx rwx SAM:ok
marsedpm.in2p3.fr
lpsc-se-dpm-server.in2p3.fr
lcg-se01.icepp.jp root atlas/jp rwx none rwx rwx rwx r-x none rwx rwx rwx SAM:ok, data08_cos:?, fdr08_run2:?
tbat05.nipne.ro
tbit00.nipne.ro