LCG-FR / SA1-FR Monitoring WG CACTI: Difference between revisions

Un article de lcgwiki.
Jump to navigation Jump to search
LEROY (talk | contribs)
LEROY (talk | contribs)
 
Ligne 73: Ligne 73:
  view firstView included .1
  view firstView included .1
  access roGroup "" any noauth exact firstView none none
  access roGroup "" any noauth exact firstView none none
  syslocation LPNHE
  syslocation YourSite
  syscontact <grid.support@lpnhe.in2p3.fr>
  syscontact <grid.support@YourSite>


= Dépannage =
= Dépannage =

Latest revision as of 17:42, 17 décembre 2008

Installation de CACTI

http://www.cacti.net/

Installation

Suivre le manuel très bien fait: http://www.cacti.net/downloads/docs/pdf/manual.pdf

Installation des RPMs

Un certian nombre de RPMs utiles se trouvent sur le repositary du LAL:

wget http://quattorsrv.lal.in2p3.fr/packages/cacti/cacti-0.8.7b-2.el4.rf.noarch.rpm
wget http://quattorsrv.lal.in2p3.fr/packages/os/sl460-x86_64/base/SL/RPMS/php-snmp-4.3.9-3.22.9.x86_64.rpm
wget http://quattorsrv.lal.in2p3.fr/packages/os/sl460-x86_64/base/SL/RPMS/\
 mysql-server-4.1.20-3.RHEL4.1.el4_6.1.x86_64.rpm
wget http://quattorsrv.lal.in2p3.fr/packages/os/sl460-x86_64/base/SL/RPMS/perl-DBD-MySQL-2.9004-3.1.x86_64.rpm
rpm -ivh http://quattorsrv.lal.in2p3.fr/packages/os/sl460-x86_64/base/SL/RPMS/perl-DBD-MySQL-2.9004-3.1.x86_64.rpm mysql-server-4.1.20-3.RHEL4.1.el4_6.1.x86_64.rpm
 

Config Mysql

Se référer a la Doc: http://www.cacti.net/downloads/docs/pdf/manual.pdf

Ceci n'est qu'un copier/coller des actions faites sur un site:

/etc/init.d/mysqld start
mysqladmin --user=root password *yourpasswd*
mysqladmin --user=root create cacti -p
mysql cacti -p  < /var/www/cacti/cacti.sql
Modifier 
/var/www/cacti/include/config.php

Config http

Modifier le fichier:

/etc/httpd/conf.d/cacti.conf

Config php

Modifier le fichier /etc/php.ini:

memory_limit = 128M

Vérifier que snmp V1 répond

snmpwalk -v 1 -c public 'your switch'

Configuration

Ajout des nouveaux devices en utilisant les commandes en ligne

Cela permet d'automatiser la configuration de plusieurs machines:

http://www.cacti.net/downloads/docs/html/scripts.html

Ajouter un device (attention utiliser template 1 pour pouvoir recuperer les stats des interfaces)

for i in $(cat /root/CACTI/wnlistplat); do php -q add_device.php --avail=ping --ping_method=udp --ping_retries=1 \
 --description=$i --ip=$i --template=1 --community="public" ; done

Ajouter les devices dans un tree:

for i in $(seq  83 147); do php -q add_tree.php --type=node --node-type=host --tree-id=4 --host-id=$i; done

Ajouter les graphes (type 'ds') des interfaces pour tous les devices:

for i in $(seq  83 147); do php -q add_graphs.php --host-id=$i --graph-type=ds --snmp-query-id=1 \
--snmp-query- type-id=13 --snmp-field=ifOperStatus  --snmp-value=Up --graph-template-id=2 ; done

Ajouter les graphes cg linux

for i in $(seq  83 147); do  for j in  7 8 9 10 12 ; do  php -q add_graphs.php --host-id=$i  \
--snmp-query-id=6 --graph- template-id=$j --graph-type=cg; done; done

Configuration de snmpd

exemple de fichier /etc/snmp/snmpd.conf:

com2sec roUser default public
group roGroup v1 roUser
view firstView included .1
access roGroup "" any noauth exact firstView none none
syslocation YourSite
syscontact <grid.support@YourSite>

Dépannage

/var/www/cacti/log/cacti.log

  • Time < Poller Interval
  • Hosts
  • DataSources
  • RRDsProcessed

rrdtool fetch

rrdtool fetch /var/www/cacti/rra/<rrdfile>.rrd AVERAGE

Optimisation

Migrer vers InnoDB

Pour pouvoir utiliser des locks par ligne

Créer des indexes

http://bugs.cacti.net/view.php?id=1333

create index `data_template_data_id` ON `data_input_data` (`data_template_data_id`);
create index `host_id_snmp_query_id_snmp_index` ON data_local (`host_id`,`snmp_query_id`,`snmp_index`);
create index `local_data_id_data_source_name` ON data_template_rrd (`local_data_id`,`data_source_name`);
create index `graph_template_id_local_graph_id` ON graph_templates_item (`graph_template_id`,`local_graph_id`);
create index `local_graph_template_item_id` ON graph_templates_item (`local_graph_template_item_id`);
create index `host_id_snmp_query_id_snmp_index` ON host_snmp_cache (`host_id`,`snmp_query_id`,`snmp_index`);
create index `local_data_id_rrd_path` ON poller_item (`local_data_id`,`rrd_path`);
create index `host_id_rrd_next_step` ON poller_item (`host_id`,`rrd_next_step`);
create index order_key3 on graph_tree_items (order_key(3));
create index order_key6 on graph_tree_items (order_key(6));
create index order_key9 on graph_tree_items (order_key(9));
create index host_id_snmp_query_id ON host_snmp_cache (host_id,snmp_query_id);
create index host_id_snmp_port ON poller_item (host_id,snmp_port);

Spine

Poller alternatif optimisé.

Pré-requis

  • net-snmp-devel
  • mysql
  • mysql-devel
  • openssl-devel

Installation

wget http://www.cacti.net/downloads/spine/cacti-spine-0.8.7a.tar.gz
tar xvzf cacti-spine-0.8.7a.tar.gz
cd spine-0.8.7a
./configure
make
make install
  • Editer /usr/local/spine/spine.conf
  • Cacti -> Console -> Settings -> Paths -> Spine Poller File Path
/usr/local/spine/spine
  • Cacti -> Console -> Settings -> Poller -> Poller Type
spine

Extension

Plugin Architecture

Pour permettre l'ajout de plugins:

Aggregate

Pour grouper plusieurs graphiques:

Disk I/O

Templates pour les entrées / sorties des disques:

removespikes

Pour supprimer les "spikes" dans les rrds:

Grille

Exemple d'utilisation des data queries:

  • En utilisant les plugins et des scripts on peut obtenir des graphiques personnalisées:

/var/www/cacti/scripts/query_CE_queues.pl

Script (brouillon) pour obtenir l'information a propos de les queues d'une Computing Element gLite

#!/usr/bin/perl
use strict;
use warnings;
use Switch;
my $target = $ARGV[0];
my $action = $ARGV[1];
my $base = 'Mds-Vo-name=resource,o=grid';
my $filter = "GlueCEUniqueID=$target:*";
my $attributes = 'GlueCEStateRunningJobs GlueCEStateWaitingJobs GlueCEInfoTotalCPUs GlueCEStateFreeCPUs GlueCEStateEstimatedResponseTime GlueCEName';
my @ldif = qx(ldapsearch -x -h $target -p2170 -LLL -b $base $filter $attributes 2>/dev/null) or die;
my @data;
my $i = 0;
my $j;
foreach my $line (@ldif) {
  chomp($line);
  if ($line =~ /^dn/) {
    $i++;
  } elsif ( $line =~ /^$/) {
    # skip
  } elsif ( $line =~ /^ /) {
    # skip
  } else {
    my ($key, $value)=split(/: /,$line);
    $data[$i]{$key}=$value;
  }
}
my $n = scalar(@data);
switch ($action) {
  case "index" {
    for($i=1;$i<$n;$i++) {
      $j = $data[$i];
      print "$j->{'GlueCEName'}\n";
    }
  }
  case "query" {
    for($i=1;$i<$n;$i++) {
      $j = $data[$i];
      print "$j->{'GlueCEName'}:$j->{$ARGV[2]}\n";
    }
  }
  case "get" {
    for($i=1;$i<$n;$i++) {
      $j = $data[$i];
      if ( $j->{'GlueCEName'} eq $ARGV[3]){
        print "$j->{$ARGV[2]}";
      }
    }
  }
  else {
    die;
  }
}

XML d'intégration

/var/www/cacti/resource/script_queries/CE_queues.xml

<interface>
 <name>Get CE queue information</name>
 <script_path>perl |path_cacti|/scripts/query_CE_queues.pl</script_path>
 <arg_prepend>|host_hostname|</arg_prepend>
 <arg_index>index</arg_index>
 <arg_query>query</arg_query>
 <arg_get>get</arg_get>
 <output_delimeter>:</output_delimeter>
 <index_order>GlueCEName</index_order>
 <index_order_type>alphanumeric</index_order_type>
 <index_title_format>|chosen_order_field|</index_title_format>
 <fields>
  <GlueCEName>
   <name>Queue</name>
   <direction>input</direction>
   <query_name>GlueCEName</query_name>
  </GlueCEName>
  <GlueCEStateFreeCPUs>
   <name>Free</name>
   <direction>output</direction>
   <query_name>GlueCEStateFreeCPUs</query_name>
  </GlueCEStateFreeCPUs>
  <GlueCEInfoTotalCPUs>
   <name>Total</name>
   <direction>output</direction>
   <query_name>GlueCEInfoTotalCPUs</query_name>
  </GlueCEInfoTotalCPUs>
  <GlueCEStateEstimatedResponseTime>
   <name>ERT</name>
   <direction>output</direction>
   <query_name>GlueCEStateEstimatedResponseTime</query_name>
  </GlueCEStateEstimatedResponseTime>
  <GlueCEStateWaitingJobs>
   <name>Waiting</name>
   <direction>output</direction>
   <query_name>GlueCEStateWaitingJobs</query_name>
  </GlueCEStateWaitingJobs>
  <GlueCEStateRunningJobs>
   <name>Running</name>
   <direction>output</direction>
   <query_name>GlueCEStateRunningJobs</query_name>
  </GlueCEStateRunningJobs>
 <fields>
</interface>

Data Query

Data Query XML

<cacti> 
 <hash_0400168d4b0cca81d487d54968ca0d3aa894e8>
  <name>Computing Element queues</name>
  <description>Computing Element queues</description>
  <xml_path><path_cacti>/resource/script_queries/CE_queues.xml</xml_path>
  <data_input_id>hash_03001680e9e4c4191a5da189ae26d0e237f015</data_input_id>
  <graphs>
   <hash_1100164f193eb62b5e9cf98c55daaa1cceccfd>
    <name>Computing Element Queue</name>
    <graph_template_id>hash_000016a634e798f897368bece7eeaac73081f6</graph_template_id>
    <rrd>
     <item_000>
      <snmp_field_name>GlueCEInfoTotalCPUs</snmp_field_name>
      <data_template_id>hash_0100160c1cc1d1178b8cf8cf5876bc8de87cf4</data_template_id>
      <data_template_rrd_id>hash_080016ef38e0fcb85d21995f3eda4984cb5c04</data_template_rrd_id>
     </item_000>
     <item_001>
      <snmp_field_name>GlueCEStateEstimatedResponseTime</snmp_field_name>
      <data_template_id>hash_0100160c1cc1d1178b8cf8cf5876bc8de87cf4</data_template_id>
      <data_template_rrd_id>hash_08001610e49122694f56ff5d99799888a8a4d7</data_template_rrd_id>
     </item_001>
     <item_002>
      <snmp_field_name>GlueCEStateWaitingJobs</snmp_field_name>
      <data_template_id>hash_0100160c1cc1d1178b8cf8cf5876bc8de87cf4</data_template_id>
      <data_template_rrd_id>hash_0800164d9f3e87d54e4b2fed9dd793ebf23505</data_template_rrd_id>
     </item_002>
     <item_003>
      <snmp_field_name>GlueCEStateFreeCPUs</snmp_field_name>
      <data_template_id>hash_0100160c1cc1d1178b8cf8cf5876bc8de87cf4</data_template_id>
      <data_template_rrd_id>hash_080016bbcb3088736558f6689336d05fb1aff7</data_template_rrd_id>
     </item_003>
     <item_004>
      <snmp_field_name>GlueCEStateRunningJobs</snmp_field_name>
      <data_template_id>hash_0100160c1cc1d1178b8cf8cf5876bc8de87cf4</data_template_id>
      <data_template_rrd_id>hash_08001629c804736fcbccf55f085a1c32049c4d</data_template_rrd_id>
     </item_004>
    </rrd>
    <sv_graph>
    </sv_graph>
    <sv_data_source>
    </sv_data_source>
   </hash_1100164f193eb62b5e9cf98c55daaa1cceccfd>
  </graphs>
 </hash_0400168d4b0cca81d487d54968ca0d3aa894e8>
</cacti>

Data Template

Data Template XML

<cacti> 
 <hash_0100160c1cc1d1178b8cf8cf5876bc8de87cf4>
  <name>Computer Element Queue</name>
  <ds>
   <t_name></t_name>
   <name>|host_description| - Queue - |query_GlueCEName|</name>
   <data_input_id>hash_03001680e9e4c4191a5da189ae26d0e237f015</data_input_id>
   <t_rra_id></t_rra_id>
   <t_rrd_step></t_rrd_step>
   <rrd_step>300</rrd_step>
   <t_active></t_active>
   <active>on</active>
   <rra_items>hash_150016c21df5178e5c955013591239eb0afd46|hash_1500160d9c0af8b8acdc7807943937b3208e29|hash_1500166fc2d038fb42950138b0ce3e9874cc60|hash_150016e36f3adb9f152adfa5dc50fd2b23337e|hash_150016283ea2bf1634d92ce081ec82a634f513|hash_150016f044e9143dda57b227728577faf08e7c</rra_items>
  </ds>
  <items>
   <hash_080016ef38e0fcb85d21995f3eda4984cb5c04>
    <t_data_source_name></t_data_source_name>
    <data_source_name>Total</data_source_name>
    <t_rrd_minimum></t_rrd_minimum>
    <rrd_minimum>0</rrd_minimum>
    <t_rrd_maximum></t_rrd_maximum>
    <rrd_maximum>U</rrd_maximum>
    <t_data_source_type_id></t_data_source_type_id>
    <data_source_type_id>1</data_source_type_id>
    <t_rrd_heartbeat></t_rrd_heartbeat>
    <rrd_heartbeat>600</rrd_heartbeat>
    <t_data_input_field_id></t_data_input_field_id>
    <data_input_field_id>0</data_input_field_id>
   </hash_080016ef38e0fcb85d21995f3eda4984cb5c04>
   <hash_08001610e49122694f56ff5d99799888a8a4d7>
    <t_data_source_name></t_data_source_name>
    <data_source_name>ERT</data_source_name>
    <t_rrd_minimum></t_rrd_minimum>
    <rrd_minimum>0</rrd_minimum>
    <t_rrd_maximum></t_rrd_maximum>
    <rrd_maximum>U</rrd_maximum>
    <t_data_source_type_id></t_data_source_type_id>
    <data_source_type_id>1</data_source_type_id>
    <t_rrd_heartbeat></t_rrd_heartbeat>
    <rrd_heartbeat>600</rrd_heartbeat>
    <t_data_input_field_id></t_data_input_field_id>
    <data_input_field_id>0</data_input_field_id>
   </hash_08001610e49122694f56ff5d99799888a8a4d7>
   <hash_0800164d9f3e87d54e4b2fed9dd793ebf23505>
    <t_data_source_name></t_data_source_name>
    <data_source_name>Waiting</data_source_name>
    <t_rrd_minimum></t_rrd_minimum>
    <rrd_minimum>0</rrd_minimum>
    <t_rrd_maximum></t_rrd_maximum>
    <rrd_maximum>U</rrd_maximum>
    <t_data_source_type_id></t_data_source_type_id>
    <data_source_type_id>1</data_source_type_id>
    <t_rrd_heartbeat></t_rrd_heartbeat>
    <rrd_heartbeat>600</rrd_heartbeat>
    <t_data_input_field_id></t_data_input_field_id>
    <data_input_field_id>0</data_input_field_id>
   </hash_0800164d9f3e87d54e4b2fed9dd793ebf23505>
   <hash_080016bbcb3088736558f6689336d05fb1aff7>
    <t_data_source_name></t_data_source_name>
    <data_source_name>Free</data_source_name>
    <t_rrd_minimum></t_rrd_minimum>
    <rrd_minimum>0</rrd_minimum>
    <t_rrd_maximum></t_rrd_maximum>
    <rrd_maximum>U</rrd_maximum>
    <t_data_source_type_id></t_data_source_type_id>
    <data_source_type_id>1</data_source_type_id>
    <t_rrd_heartbeat></t_rrd_heartbeat>
    <rrd_heartbeat>600</rrd_heartbeat>
    <t_data_input_field_id></t_data_input_field_id>
    <data_input_field_id>0</data_input_field_id>
   </hash_080016bbcb3088736558f6689336d05fb1aff7>
   <hash_08001629c804736fcbccf55f085a1c32049c4d>
    <t_data_source_name></t_data_source_name>
    <data_source_name>Running</data_source_name>
    <t_rrd_minimum></t_rrd_minimum>
    <rrd_minimum>0</rrd_minimum>
    <t_rrd_maximum></t_rrd_maximum>
    <rrd_maximum>U</rrd_maximum>
    <t_data_source_type_id></t_data_source_type_id>
    <data_source_type_id>1</data_source_type_id>
    <t_rrd_heartbeat></t_rrd_heartbeat>
    <rrd_heartbeat>600</rrd_heartbeat>
    <t_data_input_field_id></t_data_input_field_id>
    <data_input_field_id>0</data_input_field_id>
   </hash_08001629c804736fcbccf55f085a1c32049c4d>
  </items>
  
   <item_000>
    <data_input_field_id>hash_070016d39556ecad6166701bfb0e28c5a11108</data_input_field_id>
    <t_value>on</t_value>
    <value></value>
   </item_000>
   <item_001>
    <data_input_field_id>hash_0700163b7caa46eb809fc238de6ef18b6e10d5</data_input_field_id>
    <t_value>on</t_value>
    <value></value>
   </item_001>
   <item_002>
    <data_input_field_id>hash_07001674af2e42dc12956c4817c2ef5d9983f9</data_input_field_id>
    <t_value>on</t_value>
    <value></value>
   </item_002>
  
 </hash_0100160c1cc1d1178b8cf8cf5876bc8de87cf4>
</cacti>

Graph Template

Graph Template XML

<cacti> 
 <hash_000016a634e798f897368bece7eeaac73081f6>
  <name>Computing Element Queue</name>
  <graph>
   <t_title></t_title>
   <title>|host_description| - Queue - |query_GlueCEName|</title>
   <t_image_format_id></t_image_format_id>
   <image_format_id>1</image_format_id>
   <t_height></t_height>
   <height>120</height>
   <t_width></t_width>
   <width>720</width>
   <t_slope_mode></t_slope_mode>
   <slope_mode></slope_mode>
   <t_auto_scale></t_auto_scale>
   <auto_scale>on</auto_scale>
   <t_auto_scale_opts></t_auto_scale_opts>
   <auto_scale_opts>4</auto_scale_opts>
   <t_auto_scale_log></t_auto_scale_log>
   <auto_scale_log></auto_scale_log>
   <t_scale_log_units></t_scale_log_units>
   <scale_log_units></scale_log_units>
   <t_auto_scale_rigid></t_auto_scale_rigid>
   <auto_scale_rigid></auto_scale_rigid>
   <t_auto_padding></t_auto_padding>
   <auto_padding>on</auto_padding>
   <t_export></t_export>
   <export>on</export>
   <t_upper_limit></t_upper_limit>
   <upper_limit>1</upper_limit>
   <t_lower_limit></t_lower_limit>
   <lower_limit>0</lower_limit>
   <t_base_value></t_base_value>
   <base_value>1000</base_value>
   <t_unit_value></t_unit_value>
   <unit_value></unit_value>
   <t_unit_exponent_value></t_unit_exponent_value>
   <unit_exponent_value></unit_exponent_value>
   <t_vertical_label></t_vertical_label>
   <vertical_label></vertical_label>
  </graph>
  <items>
   <hash_10001660c385532179d47b371dd26af21136f7>
    <task_item_id>hash_08001629c804736fcbccf55f085a1c32049c4d</task_item_id>
    <color_id>00FF00</color_id>
    <alpha>FF</alpha>
    <graph_type_id>4</graph_type_id>
    <consolidation_function_id>1</consolidation_function_id>
    <cdef_id>0</cdef_id>
    <value></value>
    <gprint_id>hash_06001619414480d6897c8731c7dc6c5310653e</gprint_id>
    <text_format>Running</text_format>
    <hard_return></hard_return>
    <sequence>1</sequence>
   </hash_10001660c385532179d47b371dd26af21136f7>
   <hash_100016452ea7619f44f9240bd05e282ea820d5>
    <task_item_id>hash_080016bbcb3088736558f6689336d05fb1aff7</task_item_id>
    <color_id>0</color_id>
    <alpha>FF</alpha>
    <graph_type_id>9</graph_type_id>
    <consolidation_function_id>4</consolidation_function_id>
    <cdef_id>0</cdef_id>
    <value></value>
    <gprint_id>hash_06001619414480d6897c8731c7dc6c5310653e</gprint_id>
    <text_format>Free:</text_format>
    <hard_return></hard_return>
    <sequence>2</sequence>
   </hash_100016452ea7619f44f9240bd05e282ea820d5>
   <hash_10001625c74d47ed5f241cd5cb1fd21dea93c0>
    <task_item_id>hash_08001629c804736fcbccf55f085a1c32049c4d</task_item_id>
    <color_id>0</color_id>
    <alpha>FF</alpha>
    <graph_type_id>9</graph_type_id>
    <consolidation_function_id>4</consolidation_function_id>
    <cdef_id>0</cdef_id>
    <value></value>
    <gprint_id>hash_06001619414480d6897c8731c7dc6c5310653e</gprint_id>
    <text_format>Current:</text_format>
    <hard_return></hard_return>
    <sequence>3</sequence>
   </hash_10001625c74d47ed5f241cd5cb1fd21dea93c0>
   <hash_100016c3fea60fba0b5de3cf9f496a3e14f38a>
    <task_item_id>hash_08001629c804736fcbccf55f085a1c32049c4d</task_item_id>
    <color_id>0</color_id>
    <alpha>FF</alpha>
    <graph_type_id>9</graph_type_id>
    <consolidation_function_id>1</consolidation_function_id>
    <cdef_id>0</cdef_id>
    <value></value>
    <gprint_id>hash_06001619414480d6897c8731c7dc6c5310653e</gprint_id>
    <text_format>Average:</text_format>
    <hard_return></hard_return>
    <sequence>4</sequence>
   </hash_100016c3fea60fba0b5de3cf9f496a3e14f38a>
   <hash_100016fa1e55813d5021546ca2f0c19a3d0611>
    <task_item_id>hash_08001629c804736fcbccf55f085a1c32049c4d</task_item_id>
    <color_id>0</color_id>
    <alpha>FF</alpha>
    <graph_type_id>9</graph_type_id>
    <consolidation_function_id>3</consolidation_function_id>
    <cdef_id>0</cdef_id>
    <value></value>
    <gprint_id>hash_06001619414480d6897c8731c7dc6c5310653e</gprint_id>
    <text_format>Max:</text_format>
    <hard_return></hard_return>
    <sequence>5</sequence>
   </hash_100016fa1e55813d5021546ca2f0c19a3d0611>
   <hash_100016dfd0d0c949e51b170dbcdcf13bbf8acb>
    <task_item_id>hash_08001629c804736fcbccf55f085a1c32049c4d</task_item_id>
    <color_id>0</color_id>
    <alpha>FF</alpha>
    <graph_type_id>9</graph_type_id>
    <consolidation_function_id>2</consolidation_function_id>
    <cdef_id>0</cdef_id>
    <value></value>
    <gprint_id>hash_06001619414480d6897c8731c7dc6c5310653e</gprint_id>
    <text_format>Min:</text_format>
    <hard_return>on</hard_return>
    <sequence>6</sequence>
   </hash_100016dfd0d0c949e51b170dbcdcf13bbf8acb>
   <hash_1000160d9e710a70c26579ad7f5ba9074a0d07>
    <task_item_id>hash_0800164d9f3e87d54e4b2fed9dd793ebf23505</task_item_id>
    <color_id>FF0000</color_id>
    <alpha>FF</alpha>
    <graph_type_id>4</graph_type_id>
    <consolidation_function_id>1</consolidation_function_id>
    <cdef_id>0</cdef_id>
    <value></value>
    <gprint_id>hash_06001619414480d6897c8731c7dc6c5310653e</gprint_id>
    <text_format>Waiting</text_format>
    <hard_return></hard_return>
    <sequence>7</sequence>
   </hash_1000160d9e710a70c26579ad7f5ba9074a0d07>
   <hash_1000166b65aea95d4fe79da13d8f81be8f8a15>
    <task_item_id>hash_08001610e49122694f56ff5d99799888a8a4d7</task_item_id>
    <color_id>0</color_id>
    <alpha>FF</alpha>
    <graph_type_id>9</graph_type_id>
    <consolidation_function_id>4</consolidation_function_id>
    <cdef_id>0</cdef_id>
    <value></value>
    <gprint_id>hash_06001619414480d6897c8731c7dc6c5310653e</gprint_id>
    <text_format> ERT:</text_format>
    <hard_return></hard_return>
    <sequence>8</sequence>
   </hash_1000166b65aea95d4fe79da13d8f81be8f8a15>
   <hash_100016146031fe57480308160a76c2b3d692d0>
    <task_item_id>hash_0800164d9f3e87d54e4b2fed9dd793ebf23505</task_item_id>
    <color_id>0</color_id>
    <alpha>FF</alpha>
    <graph_type_id>9</graph_type_id>
    <consolidation_function_id>4</consolidation_function_id>
    <cdef_id>0</cdef_id>
    <value></value>
    <gprint_id>hash_06001619414480d6897c8731c7dc6c5310653e</gprint_id>
    <text_format>Current:</text_format>
    <hard_return></hard_return>
    <sequence>9</sequence>
   </hash_100016146031fe57480308160a76c2b3d692d0>
   <hash_100016d0f7d613f0427ac1f149adfd8b912584>
    <task_item_id>hash_0800164d9f3e87d54e4b2fed9dd793ebf23505</task_item_id>
    <color_id>0</color_id>
    <alpha>FF</alpha>
    <graph_type_id>9</graph_type_id>
    <consolidation_function_id>1</consolidation_function_id>
    <cdef_id>0</cdef_id>
    <value></value>
    <gprint_id>hash_06001619414480d6897c8731c7dc6c5310653e</gprint_id>
    <text_format>Average:</text_format>
    <hard_return></hard_return>
    <sequence>10</sequence>
   </hash_100016d0f7d613f0427ac1f149adfd8b912584>
   <hash_100016d56925ac40dd6abfc099b74370874e01>
    <task_item_id>hash_0800164d9f3e87d54e4b2fed9dd793ebf23505</task_item_id>
    <color_id>0</color_id>
    <alpha>FF</alpha>
    <graph_type_id>9</graph_type_id>
    <consolidation_function_id>3</consolidation_function_id>
    <cdef_id>0</cdef_id>
    <value></value>
    <gprint_id>hash_06001619414480d6897c8731c7dc6c5310653e</gprint_id>
    <text_format>Max:</text_format>
    <hard_return></hard_return>
    <sequence>11</sequence>
   </hash_100016d56925ac40dd6abfc099b74370874e01>
   <hash_100016e3c1745f1ee0ceea123c83a82f6a9cd5>
    <task_item_id>hash_0800164d9f3e87d54e4b2fed9dd793ebf23505</task_item_id>
    <color_id>0</color_id>
    <alpha>FF</alpha>
    <graph_type_id>9</graph_type_id>
    <consolidation_function_id>2</consolidation_function_id>
    <cdef_id>0</cdef_id>
    <value></value>
    <gprint_id>hash_06001619414480d6897c8731c7dc6c5310653e</gprint_id>
    <text_format>Min:</text_format>
    <hard_return>on</hard_return>
    <sequence>12</sequence>
   </hash_100016e3c1745f1ee0ceea123c83a82f6a9cd5>
  </items>
  <inputs>
   <hash_090016e5c6d1e07fc28d04f13a70d2a52ed82a>
    <name>Data Source [Waiting]</name>
    <description></description>
    <column_name>task_item_id</column_name>
    <items>hash_0000160d9e710a70c26579ad7f5ba9074a0d07|hash_000016146031fe57480308160a76c2b3d692d0|hash_000016d0f7d613f0427ac1f149adfd8b912584|hash_000016d56925ac40dd6abfc099b74370874e01|hash_000016e3c1745f1ee0ceea123c83a82f6a9cd5</items>
   </hash_090016e5c6d1e07fc28d04f13a70d2a52ed82a>
   <hash_090016a15ef2b79dfc4ab1765c10b1c52f7bfc>
    <name>Data Source [ERT]</name>
    <description></description>
    <column_name>task_item_id</column_name>
    <items>hash_0000166b65aea95d4fe79da13d8f81be8f8a15</items>
   </hash_090016a15ef2b79dfc4ab1765c10b1c52f7bfc>
   <hash_090016a00d6f19c821f807210f1c427cd17122>
    <name>Data Source [Running]</name>
    <description></description>
    <column_name>task_item_id</column_name>
    <items>hash_00001660c385532179d47b371dd26af21136f7|hash_00001625c74d47ed5f241cd5cb1fd21dea93c0|hash_000016c3fea60fba0b5de3cf9f496a3e14f38a|hash_000016fa1e55813d5021546ca2f0c19a3d0611|hash_000016dfd0d0c949e51b170dbcdcf13bbf8acb</items>
   </hash_090016a00d6f19c821f807210f1c427cd17122>
   <hash_09001691371f8e47f9fb6b9a8fc8e3611a2a82>
    <name>Data Source [Free]</name>
    <description></description>
    <column_name>task_item_id</column_name>
    <items>hash_000016452ea7619f44f9240bd05e282ea820d5</items>
   </hash_09001691371f8e47f9fb6b9a8fc8e3611a2a82>
  </inputs>
 </hash_000016a634e798f897368bece7eeaac73081f6>
</cacti>