LCG-FR / SA1-FR Monitoring WG CACTI

Installation de CACTI

http://www.cacti.net/

Installation

Suivre le manuel très bien fait: http://www.cacti.net/downloads/docs/pdf/manual.pdf

Installation des RPMs

Un certian nombre de RPMs utiles se trouvent sur le repositary du LAL:

wget http://quattorsrv.lal.in2p3.fr/packages/cacti/cacti-0.8.7b-2.el4.rf.noarch.rpm
wget http://quattorsrv.lal.in2p3.fr/packages/os/sl460-x86_64/base/SL/RPMS/php-snmp-4.3.9-3.22.9.x86_64.rpm
wget http://quattorsrv.lal.in2p3.fr/packages/os/sl460-x86_64/base/SL/RPMS/\
 mysql-server-4.1.20-3.RHEL4.1.el4_6.1.x86_64.rpm
wget http://quattorsrv.lal.in2p3.fr/packages/os/sl460-x86_64/base/SL/RPMS/perl-DBD-MySQL-2.9004-3.1.x86_64.rpm

rpm -ivh http://quattorsrv.lal.in2p3.fr/packages/os/sl460-x86_64/base/SL/RPMS/perl-DBD-MySQL-2.9004-3.1.x86_64.rpm mysql-server-4.1.20-3.RHEL4.1.el4_6.1.x86_64.rpm
 

Config Mysql

Se référer a la Doc: http://www.cacti.net/downloads/docs/pdf/manual.pdf

Ceci n'est qu'un copier/coller des actions faites sur un site:

/etc/init.d/mysqld start
mysqladmin --user=root password *yourpasswd*
mysqladmin --user=root create cacti -p
mysql cacti -p  < /var/www/cacti/cacti.sql

Modifier 
/var/www/cacti/include/config.php

Config http

Modifier le fichier:

/etc/httpd/conf.d/cacti.conf

Config php

Modifier le fichier /etc/php.ini:

memory_limit = 128M

Vérifier que snmp V1 répond

snmpwalk -v 1 -c public 'your switch'

Configuration

Ajout des nouveaux devices en utilisant les commandes en ligne

Cela permet d'automatiser la configuration de plusieurs machines:

http://www.cacti.net/downloads/docs/html/scripts.html

Ajouter un device (attention utiliser template 1 pour pouvoir recuperer les stats des interfaces)

for i in $(cat /root/CACTI/wnlistplat); do php -q add_device.php --avail=ping --ping_method=udp --ping_retries=1 \
 --description=$i --ip=$i --template=1 --community="public" ; done

Ajouter les devices dans un tree:

for i in $(seq  83 147); do php -q add_tree.php --type=node --node-type=host --tree-id=4 --host-id=$i; done

Ajouter les graphes (type 'ds') des interfaces pour tous les devices:

for i in $(seq  83 147); do php -q add_graphs.php --host-id=$i --graph-type=ds --snmp-query-id=1 \
--snmp-query- type-id=13 --snmp-field=ifOperStatus  --snmp-value=Up --graph-template-id=2 ; done

Ajouter les graphes cg linux

for i in $(seq  83 147); do  for j in  7 8 9 10 12 ; do  php -q add_graphs.php --host-id=$i  \
--snmp-query-id=6 --graph- template-id=$j --graph-type=cg; done; done

Configuration de snmpd

exemple de fichier /etc/snmp/snmpd.conf:

com2sec roUser default public
group roGroup v1 roUser
view firstView included .1
access roGroup "" any noauth exact firstView none none
syslocation YourSite
syscontact <grid.support@YourSite>

Dépannage

/var/www/cacti/log/cacti.log

• Time < Poller Interval
• Hosts
• DataSources
• RRDsProcessed

rrdtool fetch

rrdtool fetch /var/www/cacti/rra/<rrdfile>.rrd AVERAGE

Optimisation

Migrer vers InnoDB

Pour pouvoir utiliser des locks par ligne

Créer des indexes

http://bugs.cacti.net/view.php?id=1333

create index `data_template_data_id` ON `data_input_data` (`data_template_data_id`);
create index `host_id_snmp_query_id_snmp_index` ON data_local (`host_id`,`snmp_query_id`,`snmp_index`);
create index `local_data_id_data_source_name` ON data_template_rrd (`local_data_id`,`data_source_name`);
create index `graph_template_id_local_graph_id` ON graph_templates_item (`graph_template_id`,`local_graph_id`);
create index `local_graph_template_item_id` ON graph_templates_item (`local_graph_template_item_id`);
create index `host_id_snmp_query_id_snmp_index` ON host_snmp_cache (`host_id`,`snmp_query_id`,`snmp_index`);
create index `local_data_id_rrd_path` ON poller_item (`local_data_id`,`rrd_path`);
create index `host_id_rrd_next_step` ON poller_item (`host_id`,`rrd_next_step`);
create index order_key3 on graph_tree_items (order_key(3));
create index order_key6 on graph_tree_items (order_key(6));
create index order_key9 on graph_tree_items (order_key(9));
create index host_id_snmp_query_id ON host_snmp_cache (host_id,snmp_query_id);
create index host_id_snmp_port ON poller_item (host_id,snmp_port);

Spine

Poller alternatif optimisé.

• Site: http://www.cacti.net/spine_info.php
• Téléchargement: http://www.cacti.net/spine_download.php
• Installation: http://www.cacti.net/spine_install.php

Pré-requis

• net-snmp-devel
• mysql
• mysql-devel
• openssl-devel

Installation

wget http://www.cacti.net/downloads/spine/cacti-spine-0.8.7a.tar.gz
tar xvzf cacti-spine-0.8.7a.tar.gz
cd spine-0.8.7a
./configure
make
make install

• Editer /usr/local/spine/spine.conf
• Cacti -> Console -> Settings -> Paths -> Spine Poller File Path

/usr/local/spine/spine

• Cacti -> Console -> Settings -> Poller -> Poller Type

spine

Extension

Plugin Architecture

Pour permettre l'ajout de plugins:

• http://cactiusers.org/wiki/PluginArchitectureInstall

Aggregate

Pour grouper plusieurs graphiques:

• http://docs.cacti.net/?q=node/392

Disk I/O

Templates pour les entrées / sorties des disques:

• http://www.goldfisch.at/knowwiki/howtos/cacti#disk_io

removespikes

Pour supprimer les "spikes" dans les rrds:

• http://oss.oetiker.ch/rrdtool/pub/contrib/removespikes-20080226-mkn.txt
• http://oss.oetiker.ch/rrdtool/pub/contrib/removespikes-20080226-mkn.tar.gz

Grille

Exemple d'utilisation des data queries:

• http://docs.cacti.net/?q=node/284

• En utilisant les plugins et des scripts on peut obtenir des graphiques personnalisées:

/var/www/cacti/scripts/query_CE_queues.pl

Script (brouillon) pour obtenir l'information a propos de les queues d'une Computing Element gLite

#!/usr/bin/perl
use strict;
use warnings;
use Switch;
my $target = $ARGV[0];
my $action = $ARGV[1];
my $base = 'Mds-Vo-name=resource,o=grid';
my $filter = "GlueCEUniqueID=$target:*";
my $attributes = 'GlueCEStateRunningJobs GlueCEStateWaitingJobs GlueCEInfoTotalCPUs GlueCEStateFreeCPUs GlueCEStateEstimatedResponseTime GlueCEName';
my @ldif = qx(ldapsearch -x -h $target -p2170 -LLL -b $base $filter $attributes 2>/dev/null) or die;
my @data;
my $i = 0;
my $j;
foreach my $line (@ldif) {
  chomp($line);
  if ($line =~ /^dn/) {
    $i++;
  } elsif ( $line =~ /^$/) {
    # skip
  } elsif ( $line =~ /^ /) {
    # skip
  } else {
    my ($key, $value)=split(/: /,$line);
    $data[$i]{$key}=$value;
  }
}
my $n = scalar(@data);
switch ($action) {
  case "index" {
    for($i=1;$i<$n;$i++) {
      $j = $data[$i];
      print "$j->{'GlueCEName'}\n";
    }
  }
  case "query" {
    for($i=1;$i<$n;$i++) {
      $j = $data[$i];
      print "$j->{'GlueCEName'}:$j->{$ARGV[2]}\n";
    }
  }
  case "get" {
    for($i=1;$i<$n;$i++) {
      $j = $data[$i];
      if ( $j->{'GlueCEName'} eq $ARGV[3]){
        print "$j->{$ARGV[2]}";
      }
    }
  }
  else {
    die;
  }
}

XML d'intégration

/var/www/cacti/resource/script_queries/CE_queues.xml

<interface>
 <name>Get CE queue information</name>
 <script_path>perl |path_cacti|/scripts/query_CE_queues.pl</script_path>
 <arg_prepend>|host_hostname|</arg_prepend>
 <arg_index>index</arg_index>
 <arg_query>query</arg_query>
 <arg_get>get</arg_get>
 <output_delimeter>:</output_delimeter>
 <index_order>GlueCEName</index_order>
 <index_order_type>alphanumeric</index_order_type>
 <index_title_format>|chosen_order_field|</index_title_format>
 <fields>
  <GlueCEName>
   <name>Queue</name>
   <direction>input</direction>
   <query_name>GlueCEName</query_name>
  </GlueCEName>
  <GlueCEStateFreeCPUs>
   <name>Free</name>
   <direction>output</direction>
   <query_name>GlueCEStateFreeCPUs</query_name>
  </GlueCEStateFreeCPUs>
  <GlueCEInfoTotalCPUs>
   <name>Total</name>
   <direction>output</direction>
   <query_name>GlueCEInfoTotalCPUs</query_name>
  </GlueCEInfoTotalCPUs>
  <GlueCEStateEstimatedResponseTime>
   <name>ERT</name>
   <direction>output</direction>
   <query_name>GlueCEStateEstimatedResponseTime</query_name>
  </GlueCEStateEstimatedResponseTime>
  <GlueCEStateWaitingJobs>
   <name>Waiting</name>
   <direction>output</direction>
   <query_name>GlueCEStateWaitingJobs</query_name>
  </GlueCEStateWaitingJobs>
  <GlueCEStateRunningJobs>
   <name>Running</name>
   <direction>output</direction>
   <query_name>GlueCEStateRunningJobs</query_name>
  </GlueCEStateRunningJobs>
 <fields>
</interface>

Data Query

Data Query XML

<cacti> 
 <hash_0400168d4b0cca81d487d54968ca0d3aa894e8>
  <name>Computing Element queues</name>
  <description>Computing Element queues</description>
  <xml_path><path_cacti>/resource/script_queries/CE_queues.xml</xml_path>
  <data_input_id>hash_03001680e9e4c4191a5da189ae26d0e237f015</data_input_id>
  <graphs>
   <hash_1100164f193eb62b5e9cf98c55daaa1cceccfd>
    <name>Computing Element Queue</name>
    <graph_template_id>hash_000016a634e798f897368bece7eeaac73081f6</graph_template_id>
    <rrd>
     <item_000>
      <snmp_field_name>GlueCEInfoTotalCPUs</snmp_field_name>
      <data_template_id>hash_0100160c1cc1d1178b8cf8cf5876bc8de87cf4</data_template_id>
      <data_template_rrd_id>hash_080016ef38e0fcb85d21995f3eda4984cb5c04</data_template_rrd_id>
     </item_000>
     <item_001>
      <snmp_field_name>GlueCEStateEstimatedResponseTime</snmp_field_name>
      <data_template_id>hash_0100160c1cc1d1178b8cf8cf5876bc8de87cf4</data_template_id>
      <data_template_rrd_id>hash_08001610e49122694f56ff5d99799888a8a4d7</data_template_rrd_id>
     </item_001>
     <item_002>
      <snmp_field_name>GlueCEStateWaitingJobs</snmp_field_name>
      <data_template_id>hash_0100160c1cc1d1178b8cf8cf5876bc8de87cf4</data_template_id>
      <data_template_rrd_id>hash_0800164d9f3e87d54e4b2fed9dd793ebf23505</data_template_rrd_id>
     </item_002>
     <item_003>
      <snmp_field_name>GlueCEStateFreeCPUs</snmp_field_name>
      <data_template_id>hash_0100160c1cc1d1178b8cf8cf5876bc8de87cf4</data_template_id>
      <data_template_rrd_id>hash_080016bbcb3088736558f6689336d05fb1aff7</data_template_rrd_id>
     </item_003>
     <item_004>
      <snmp_field_name>GlueCEStateRunningJobs</snmp_field_name>
      <data_template_id>hash_0100160c1cc1d1178b8cf8cf5876bc8de87cf4</data_template_id>
      <data_template_rrd_id>hash_08001629c804736fcbccf55f085a1c32049c4d</data_template_rrd_id>
     </item_004>
    </rrd>
    <sv_graph>
    </sv_graph>
    <sv_data_source>
    </sv_data_source>
   </hash_1100164f193eb62b5e9cf98c55daaa1cceccfd>
  </graphs>
 </hash_0400168d4b0cca81d487d54968ca0d3aa894e8>
</cacti>

Data Template

Data Template XML

<cacti> 
 <hash_0100160c1cc1d1178b8cf8cf5876bc8de87cf4>
  <name>Computer Element Queue</name>
  <ds>
   <t_name></t_name>
   <name>|host_description| - Queue - |query_GlueCEName|</name>
   <data_input_id>hash_03001680e9e4c4191a5da189ae26d0e237f015</data_input_id>
   <t_rra_id></t_rra_id>
   <t_rrd_step></t_rrd_step>
   <rrd_step>300</rrd_step>
   <t_active></t_active>
   <active>on</active>
   <rra_items>hash_150016c21df5178e5c955013591239eb0afd46|hash_1500160d9c0af8b8acdc7807943937b3208e29|hash_1500166fc2d038fb42950138b0ce3e9874cc60|hash_150016e36f3adb9f152adfa5dc50fd2b23337e|hash_150016283ea2bf1634d92ce081ec82a634f513|hash_150016f044e9143dda57b227728577faf08e7c</rra_items>
  </ds>
  <items>
   <hash_080016ef38e0fcb85d21995f3eda4984cb5c04>
    <t_data_source_name></t_data_source_name>
    <data_source_name>Total</data_source_name>
    <t_rrd_minimum></t_rrd_minimum>
    <rrd_minimum>0</rrd_minimum>
    <t_rrd_maximum></t_rrd_maximum>
    <rrd_maximum>U</rrd_maximum>
    <t_data_source_type_id></t_data_source_type_id>
    <data_source_type_id>1</data_source_type_id>
    <t_rrd_heartbeat></t_rrd_heartbeat>
    <rrd_heartbeat>600</rrd_heartbeat>
    <t_data_input_field_id></t_data_input_field_id>
    <data_input_field_id>0</data_input_field_id>
   </hash_080016ef38e0fcb85d21995f3eda4984cb5c04>
   <hash_08001610e49122694f56ff5d99799888a8a4d7>
    <t_data_source_name></t_data_source_name>
    <data_source_name>ERT</data_source_name>
    <t_rrd_minimum></t_rrd_minimum>
    <rrd_minimum>0</rrd_minimum>
    <t_rrd_maximum></t_rrd_maximum>
    <rrd_maximum>U</rrd_maximum>
    <t_data_source_type_id></t_data_source_type_id>
    <data_source_type_id>1</data_source_type_id>
    <t_rrd_heartbeat></t_rrd_heartbeat>
    <rrd_heartbeat>600</rrd_heartbeat>
    <t_data_input_field_id></t_data_input_field_id>
    <data_input_field_id>0</data_input_field_id>
   </hash_08001610e49122694f56ff5d99799888a8a4d7>
   <hash_0800164d9f3e87d54e4b2fed9dd793ebf23505>
    <t_data_source_name></t_data_source_name>
    <data_source_name>Waiting</data_source_name>
    <t_rrd_minimum></t_rrd_minimum>
    <rrd_minimum>0</rrd_minimum>
    <t_rrd_maximum></t_rrd_maximum>
    <rrd_maximum>U</rrd_maximum>
    <t_data_source_type_id></t_data_source_type_id>
    <data_source_type_id>1</data_source_type_id>
    <t_rrd_heartbeat></t_rrd_heartbeat>
    <rrd_heartbeat>600</rrd_heartbeat>
    <t_data_input_field_id></t_data_input_field_id>
    <data_input_field_id>0</data_input_field_id>
   </hash_0800164d9f3e87d54e4b2fed9dd793ebf23505>
   <hash_080016bbcb3088736558f6689336d05fb1aff7>
    <t_data_source_name></t_data_source_name>
    <data_source_name>Free</data_source_name>
    <t_rrd_minimum></t_rrd_minimum>
    <rrd_minimum>0</rrd_minimum>
    <t_rrd_maximum></t_rrd_maximum>
    <rrd_maximum>U</rrd_maximum>
    <t_data_source_type_id></t_data_source_type_id>
    <data_source_type_id>1</data_source_type_id>
    <t_rrd_heartbeat></t_rrd_heartbeat>
    <rrd_heartbeat>600</rrd_heartbeat>
    <t_data_input_field_id></t_data_input_field_id>
    <data_input_field_id>0</data_input_field_id>
   </hash_080016bbcb3088736558f6689336d05fb1aff7>
   <hash_08001629c804736fcbccf55f085a1c32049c4d>
    <t_data_source_name></t_data_source_name>
    <data_source_name>Running</data_source_name>
    <t_rrd_minimum></t_rrd_minimum>
    <rrd_minimum>0</rrd_minimum>
    <t_rrd_maximum></t_rrd_maximum>
    <rrd_maximum>U</rrd_maximum>
    <t_data_source_type_id></t_data_source_type_id>
    <data_source_type_id>1</data_source_type_id>
    <t_rrd_heartbeat></t_rrd_heartbeat>
    <rrd_heartbeat>600</rrd_heartbeat>
    <t_data_input_field_id></t_data_input_field_id>
    <data_input_field_id>0</data_input_field_id>
   </hash_08001629c804736fcbccf55f085a1c32049c4d>
  </items>
  
   <item_000>
    <data_input_field_id>hash_070016d39556ecad6166701bfb0e28c5a11108</data_input_field_id>
    <t_value>on</t_value>
    <value></value>
   </item_000>
   <item_001>
    <data_input_field_id>hash_0700163b7caa46eb809fc238de6ef18b6e10d5</data_input_field_id>
    <t_value>on</t_value>
    <value></value>
   </item_001>
   <item_002>
    <data_input_field_id>hash_07001674af2e42dc12956c4817c2ef5d9983f9</data_input_field_id>
    <t_value>on</t_value>
    <value></value>
   </item_002>
  
 </hash_0100160c1cc1d1178b8cf8cf5876bc8de87cf4>
</cacti>

Graph Template

Graph Template XML

<cacti> 
 <hash_000016a634e798f897368bece7eeaac73081f6>
  <name>Computing Element Queue</name>
  <graph>
   <t_title></t_title>
   <title>|host_description| - Queue - |query_GlueCEName|</title>
   <t_image_format_id></t_image_format_id>
   <image_format_id>1</image_format_id>
   <t_height></t_height>
   <height>120</height>
   <t_width></t_width>
   <width>720</width>
   <t_slope_mode></t_slope_mode>
   <slope_mode></slope_mode>
   <t_auto_scale></t_auto_scale>
   <auto_scale>on</auto_scale>
   <t_auto_scale_opts></t_auto_scale_opts>
   <auto_scale_opts>4</auto_scale_opts>
   <t_auto_scale_log></t_auto_scale_log>
   <auto_scale_log></auto_scale_log>
   <t_scale_log_units></t_scale_log_units>
   <scale_log_units></scale_log_units>
   <t_auto_scale_rigid></t_auto_scale_rigid>
   <auto_scale_rigid></auto_scale_rigid>
   <t_auto_padding></t_auto_padding>
   <auto_padding>on</auto_padding>
   <t_export></t_export>
   <export>on</export>
   <t_upper_limit></t_upper_limit>
   <upper_limit>1</upper_limit>
   <t_lower_limit></t_lower_limit>
   <lower_limit>0</lower_limit>
   <t_base_value></t_base_value>
   <base_value>1000</base_value>
   <t_unit_value></t_unit_value>
   <unit_value></unit_value>
   <t_unit_exponent_value></t_unit_exponent_value>
   <unit_exponent_value></unit_exponent_value>
   <t_vertical_label></t_vertical_label>
   <vertical_label></vertical_label>
  </graph>
  <items>
   <hash_10001660c385532179d47b371dd26af21136f7>
    <task_item_id>hash_08001629c804736fcbccf55f085a1c32049c4d</task_item_id>
    <color_id>00FF00</color_id>
    <alpha>FF</alpha>
    <graph_type_id>4</graph_type_id>
    <consolidation_function_id>1</consolidation_function_id>
    <cdef_id>0</cdef_id>
    <value></value>
    <gprint_id>hash_06001619414480d6897c8731c7dc6c5310653e</gprint_id>
    <text_format>Running</text_format>
    <hard_return></hard_return>
    <sequence>1</sequence>
   </hash_10001660c385532179d47b371dd26af21136f7>
   <hash_100016452ea7619f44f9240bd05e282ea820d5>
    <task_item_id>hash_080016bbcb3088736558f6689336d05fb1aff7</task_item_id>
    <color_id>0</color_id>
    <alpha>FF</alpha>
    <graph_type_id>9</graph_type_id>
    <consolidation_function_id>4</consolidation_function_id>
    <cdef_id>0</cdef_id>
    <value></value>
    <gprint_id>hash_06001619414480d6897c8731c7dc6c5310653e</gprint_id>
    <text_format>Free:</text_format>
    <hard_return></hard_return>
    <sequence>2</sequence>
   </hash_100016452ea7619f44f9240bd05e282ea820d5>
   <hash_10001625c74d47ed5f241cd5cb1fd21dea93c0>
    <task_item_id>hash_08001629c804736fcbccf55f085a1c32049c4d</task_item_id>
    <color_id>0</color_id>
    <alpha>FF</alpha>
    <graph_type_id>9</graph_type_id>
    <consolidation_function_id>4</consolidation_function_id>
    <cdef_id>0</cdef_id>
    <value></value>
    <gprint_id>hash_06001619414480d6897c8731c7dc6c5310653e</gprint_id>
    <text_format>Current:</text_format>
    <hard_return></hard_return>
    <sequence>3</sequence>
   </hash_10001625c74d47ed5f241cd5cb1fd21dea93c0>
   <hash_100016c3fea60fba0b5de3cf9f496a3e14f38a>
    <task_item_id>hash_08001629c804736fcbccf55f085a1c32049c4d</task_item_id>
    <color_id>0</color_id>
    <alpha>FF</alpha>
    <graph_type_id>9</graph_type_id>
    <consolidation_function_id>1</consolidation_function_id>
    <cdef_id>0</cdef_id>
    <value></value>
    <gprint_id>hash_06001619414480d6897c8731c7dc6c5310653e</gprint_id>
    <text_format>Average:</text_format>
    <hard_return></hard_return>
    <sequence>4</sequence>
   </hash_100016c3fea60fba0b5de3cf9f496a3e14f38a>
   <hash_100016fa1e55813d5021546ca2f0c19a3d0611>
    <task_item_id>hash_08001629c804736fcbccf55f085a1c32049c4d</task_item_id>
    <color_id>0</color_id>
    <alpha>FF</alpha>
    <graph_type_id>9</graph_type_id>
    <consolidation_function_id>3</consolidation_function_id>
    <cdef_id>0</cdef_id>
    <value></value>
    <gprint_id>hash_06001619414480d6897c8731c7dc6c5310653e</gprint_id>
    <text_format>Max:</text_format>
    <hard_return></hard_return>
    <sequence>5</sequence>
   </hash_100016fa1e55813d5021546ca2f0c19a3d0611>
   <hash_100016dfd0d0c949e51b170dbcdcf13bbf8acb>
    <task_item_id>hash_08001629c804736fcbccf55f085a1c32049c4d</task_item_id>
    <color_id>0</color_id>
    <alpha>FF</alpha>
    <graph_type_id>9</graph_type_id>
    <consolidation_function_id>2</consolidation_function_id>
    <cdef_id>0</cdef_id>
    <value></value>
    <gprint_id>hash_06001619414480d6897c8731c7dc6c5310653e</gprint_id>
    <text_format>Min:</text_format>
    <hard_return>on</hard_return>
    <sequence>6</sequence>
   </hash_100016dfd0d0c949e51b170dbcdcf13bbf8acb>
   <hash_1000160d9e710a70c26579ad7f5ba9074a0d07>
    <task_item_id>hash_0800164d9f3e87d54e4b2fed9dd793ebf23505</task_item_id>
    <color_id>FF0000</color_id>
    <alpha>FF</alpha>
    <graph_type_id>4</graph_type_id>
    <consolidation_function_id>1</consolidation_function_id>
    <cdef_id>0</cdef_id>
    <value></value>
    <gprint_id>hash_06001619414480d6897c8731c7dc6c5310653e</gprint_id>
    <text_format>Waiting</text_format>
    <hard_return></hard_return>
    <sequence>7</sequence>
   </hash_1000160d9e710a70c26579ad7f5ba9074a0d07>
   <hash_1000166b65aea95d4fe79da13d8f81be8f8a15>
    <task_item_id>hash_08001610e49122694f56ff5d99799888a8a4d7</task_item_id>
    <color_id>0</color_id>
    <alpha>FF</alpha>
    <graph_type_id>9</graph_type_id>
    <consolidation_function_id>4</consolidation_function_id>
    <cdef_id>0</cdef_id>
    <value></value>
    <gprint_id>hash_06001619414480d6897c8731c7dc6c5310653e</gprint_id>
    <text_format> ERT:</text_format>
    <hard_return></hard_return>
    <sequence>8</sequence>
   </hash_1000166b65aea95d4fe79da13d8f81be8f8a15>
   <hash_100016146031fe57480308160a76c2b3d692d0>
    <task_item_id>hash_0800164d9f3e87d54e4b2fed9dd793ebf23505</task_item_id>
    <color_id>0</color_id>
    <alpha>FF</alpha>
    <graph_type_id>9</graph_type_id>
    <consolidation_function_id>4</consolidation_function_id>
    <cdef_id>0</cdef_id>
    <value></value>
    <gprint_id>hash_06001619414480d6897c8731c7dc6c5310653e</gprint_id>
    <text_format>Current:</text_format>
    <hard_return></hard_return>
    <sequence>9</sequence>
   </hash_100016146031fe57480308160a76c2b3d692d0>
   <hash_100016d0f7d613f0427ac1f149adfd8b912584>
    <task_item_id>hash_0800164d9f3e87d54e4b2fed9dd793ebf23505</task_item_id>
    <color_id>0</color_id>
    <alpha>FF</alpha>
    <graph_type_id>9</graph_type_id>
    <consolidation_function_id>1</consolidation_function_id>
    <cdef_id>0</cdef_id>
    <value></value>
    <gprint_id>hash_06001619414480d6897c8731c7dc6c5310653e</gprint_id>
    <text_format>Average:</text_format>
    <hard_return></hard_return>
    <sequence>10</sequence>
   </hash_100016d0f7d613f0427ac1f149adfd8b912584>
   <hash_100016d56925ac40dd6abfc099b74370874e01>
    <task_item_id>hash_0800164d9f3e87d54e4b2fed9dd793ebf23505</task_item_id>
    <color_id>0</color_id>
    <alpha>FF</alpha>
    <graph_type_id>9</graph_type_id>
    <consolidation_function_id>3</consolidation_function_id>
    <cdef_id>0</cdef_id>
    <value></value>
    <gprint_id>hash_06001619414480d6897c8731c7dc6c5310653e</gprint_id>
    <text_format>Max:</text_format>
    <hard_return></hard_return>
    <sequence>11</sequence>
   </hash_100016d56925ac40dd6abfc099b74370874e01>
   <hash_100016e3c1745f1ee0ceea123c83a82f6a9cd5>
    <task_item_id>hash_0800164d9f3e87d54e4b2fed9dd793ebf23505</task_item_id>
    <color_id>0</color_id>
    <alpha>FF</alpha>
    <graph_type_id>9</graph_type_id>
    <consolidation_function_id>2</consolidation_function_id>
    <cdef_id>0</cdef_id>
    <value></value>
    <gprint_id>hash_06001619414480d6897c8731c7dc6c5310653e</gprint_id>
    <text_format>Min:</text_format>
    <hard_return>on</hard_return>
    <sequence>12</sequence>
   </hash_100016e3c1745f1ee0ceea123c83a82f6a9cd5>
  </items>
  <inputs>
   <hash_090016e5c6d1e07fc28d04f13a70d2a52ed82a>
    <name>Data Source [Waiting]</name>
    <description></description>
    <column_name>task_item_id</column_name>
    <items>hash_0000160d9e710a70c26579ad7f5ba9074a0d07|hash_000016146031fe57480308160a76c2b3d692d0|hash_000016d0f7d613f0427ac1f149adfd8b912584|hash_000016d56925ac40dd6abfc099b74370874e01|hash_000016e3c1745f1ee0ceea123c83a82f6a9cd5</items>
   </hash_090016e5c6d1e07fc28d04f13a70d2a52ed82a>
   <hash_090016a15ef2b79dfc4ab1765c10b1c52f7bfc>
    <name>Data Source [ERT]</name>
    <description></description>
    <column_name>task_item_id</column_name>
    <items>hash_0000166b65aea95d4fe79da13d8f81be8f8a15</items>
   </hash_090016a15ef2b79dfc4ab1765c10b1c52f7bfc>
   <hash_090016a00d6f19c821f807210f1c427cd17122>
    <name>Data Source [Running]</name>
    <description></description>
    <column_name>task_item_id</column_name>
    <items>hash_00001660c385532179d47b371dd26af21136f7|hash_00001625c74d47ed5f241cd5cb1fd21dea93c0|hash_000016c3fea60fba0b5de3cf9f496a3e14f38a|hash_000016fa1e55813d5021546ca2f0c19a3d0611|hash_000016dfd0d0c949e51b170dbcdcf13bbf8acb</items>
   </hash_090016a00d6f19c821f807210f1c427cd17122>
   <hash_09001691371f8e47f9fb6b9a8fc8e3611a2a82>
    <name>Data Source [Free]</name>
    <description></description>
    <column_name>task_item_id</column_name>
    <items>hash_000016452ea7619f44f9240bd05e282ea820d5</items>
   </hash_09001691371f8e47f9fb6b9a8fc8e3611a2a82>
  </inputs>
 </hash_000016a634e798f897368bece7eeaac73081f6>
</cacti>