lcgwiki - Contributions [en]

LCG-FR / SA1-FR Monitoring NagiosWithQuattor

2009-03-13T15:31:37Z

Fschaer: /* 2 conditions */

= Installing Nagios with quattor =

Nagios configuration requires both a set of client templates for commands to be run on clients by the Nagios Remote Plug-in Executor (NRPE) and a set of server templates configuring contacts for alarms, hosts to be monitored, services (AKA sensors) and so on.

You can contact '''frederic.schaer__arobase char__cea.fr''' in case of problem.

== Configuring the Nagios server ==

The configuration of a Nagios server is done in a set of standard templates, in the 'monitoring/nagios3' namespace.

=== Repository Used ===

*Sensors are provided for some of the grid plugins from the SA1 repository: http://www.sysadmin.hep.ac.uk/rpms/grid-services/RPMS.monitoring/

*RPMs for nagios and nagios-plugins (+dépendances) are compiled for each supported OS, and are put in the repository « updates » on quattorsrv.lal.in2p3.fr.
Ex. : http://quattor.web.lal.in2p3.fr/packages/os/sl440-i386/updates/

*Plugins « nagios-grid-plugins » are in noarch RPM in the repository « nagios »
Ex. : http://quattor.web.lal.in2p3.fr/packages/nagios/

see template : '''nagios3/plugins/config.tpl'''

=== Server Template ===

An example Nagios server template is here :

https://trac.lal.in2p3.fr/LCGQWG/browser/templates/trunk/clusters/example-3.1/profiles/nagios3-server.example.org.tpl

This machine should be a UI to monitor grid services.

=== Who is monitored, Client configuration ===

==== 2 conditions ====

*Hosts from site ('''variable SITES''') and present in '''config/’sitename’_nodes_properties.tpl''' will be monitored

Template example for hosts declaration are in LCGQWG:
https://trac.lal.in2p3.fr/LCGQWG/browser/templates/trunk/sites/example/site/config

*AND host that has NAGIOS_CLIENT_ENABLED set to "true":

variable NAGIOS_CLIENT_ENABLED ?= true;
include { if(NAGIOS_CLIENT_ENABLED) 'monitoring/nagios3/client/config'};

If you want to set this variable to "true" for all your hosts you can put it in the following template:

'''template site/pro_site_common_config'''

note : Please consider disabling client installation (by setting NAGIOS_CLIENT_ENABLED=false in the profiles) on nodes with unsupported OSes... otherwise you'll encounter quattor compilation errors.

==== You can tune this with ====

'''NAGIOS_IGNORED_NODES''' , '''NAGIOS_MONITORED_HOSTGROUPS'''

see the profile: https://trac.lal.in2p3.fr/LCGQWG/browser/templates/trunk/clusters/example-3.1/profiles/nagios3-server.example.org.tpl

==== Autorisation to contact NRPE ====

You need to declare your nagios server in the variable '''NAGIOS_SERVER' so it is able to contact NRPE daemon on the clients

=== What is monitored ===

==== Services ====
Services are added in the template '''« server/cfgfiles/services.tpl »''' ,
adding a service can be done like this :

variable TMP_SERVICE=nlist(
"use"," generic-service",
"host_name"," node07.org.fr",
"service_description"," Workers ssh_known_hosts",
"contact_groups"," admins",
"check_command"," check_nrpe_long!check_ssh_known_hosts!60",
"normal_check_interval"," 60 ; check every hour",
"max_check_attempts"," 1",
);
variable NAGIOS_SERVICES=nagios_add_service(TMP_SERVICE);

If the second parameter of the function '''nagios_add_service''' is « true » , a dependency will be added on the NRPE daemon for all the nodes « "*,!NOQUATTOR » for the service defined .... need to improve on this...

It's possible to add dependency on a services for a host, with a service from another host well defined:

variable NAGIOS_USER_DEFINED_HOST_DEPENDENCIES = nagios_add_host_service_dependency(
"node07.datagrid.cea.fr","nrpe daemon", "node07.datagrid.cea.fr","Workers ssh_known_hosts"
);

It's not possible to add dependency between hostgroups (for the moment ?)

==== commands and NRPE ====

Some Nagios configuration files don't need complex quattor structure template and so are created with '''filecopy''' :

*adding commands is done in:
'''monitoring/nagios3/server/cfgfiles/commands'''
*adding NRPE commands is done in:
'''monitoring/nagios3/client/cfgfiles/nrpe_commands'''

It's possible to add dependency on the NRPE daemon for services wich are not defined on all the hosts(template '''services.tpl''')

variable NRPE_HOSTGROUPS_SPECIFIC_DEPENDENT_SERVICES=nlist(
escape("CE,CE-MPI,!NOQUATTOR") ,"black hole workers",
escape("CE,CE-MPI,LFC,SE_DPM,SE_DISK,MON,VOBOX,WMS,!NOQUATTOR"),"host certificate",
escape("MON") ,"apel publisher",
escape("CE,CE-MPI") ,"apel parser",
escape("WN,CE-MPI,UI,VOBOX") ,"home partition freespace",
escape("WN") ,"pbs_mom transfers",
);

=== Proxy management ===
Need to have a valid certificate for local grid probe.
2 mechanisms are possible: Renewal and Retrieval.
In cfg/standard/monitoring/nagios3/server/config.tpl
include { if(NAGIOS_NCG_CONFIG && NAGIOS_MODE_PROXY_RENEW) 'monitoring/nagios3/server/vobox'};
include { if(NAGIOS_NCG_CONFIG && NAGIOS_MODE_PROXY_RETRIEVE) 'monitoring/nagios3/server/proxy_retrieval'};

Les variables associées:
{| border="1" width="100%"
| width="50%" |
NAGIOS_MODE_PROXY_RENEW
| width="50%" |
boolean. true if you want to use renewal mechanism
|----
|
NAGIOS_MODE_PROXY_RETRIEVE
| width="50%" |
boolean. true if you want to use retrieval mechanism
|----
|
NAGIOS_RENEW_PROXY
| width="50%" |
string. file where the proxy is renewed by the vobox mechanism renewal
|----
|
NAGIOS_OUTPUT_PROXY
| width="50%" |
string. file where the proxy should be retrieved by the retrieval cron proxy
|----
|
NAGIOS_MYPROXY_NAME
| width="50%" |
string. name of your proxy for later retrieval
|----
|
MYPROXY_SERVER
| width="50%" |
string. name of your myproxy server host
|----
|
NAGIOS_VONAME_PROXY
| width="50%" |
string. VO Used for voms authentication
|----
|}

== Les variables ==

{| border="1" width="100%"
| width="50%" |
NAGIOS_RPM_VERSION ?= "3.0.5-1"
| width="50%" |
string. Nagios Server RPM version
|----
|
variable NAGIOS_NODES_PROPERTIES ?= nlist()
| width="50%" |
nlist. The site nodes properties. For instance : nlist( escape("mynode.mydomain"), nlist("type","NFS", "monitoring","yes", "os","undef", "ip","192.68.0.1" , "hardware","undef"))
|----
|
variable NAGIOS_HTPASSWD_LOGIN ?= "nagios"
| width="50%" |
string. Defines the user name allowed to access the nagios interface.
|----
|
variable NAGIOS_HTPASSWD_PASS ?= "i88QUu1o8Jwq."
| width="50%" |
string. Defines the password associated with the user name. This is a md5 hash generated using the "openssl passwd" command. Default password is "nagios".
|----
|
variable NAGIOS_MONITORED_HOSTGROUPS ?= NAGIOS_KNOWN_HOSTGROUPS
| width="50%" |
list. Defines the list of hostgroups you want to define in Nagios. If different from NAGIOS_KNOWN_HOSTGROUPS, each node that is of a "type" not listed will be added to the hostgroup NAGIOS_DEFAULT_NODE_GROUP. Please not that KNOWN hostgrouops will still be defined, for compatibility reasons (for now).
|----
|
variable NAGIOS_DEFAULT_NODE_GROUP?="Others"
| width="50%" |
string. Nodes which type is not known will be put in this hostgroup
|----
|
variable NAGIOS_SERVER ?= FULL_HOSTNAME
| width="50%" |
string. Defines the NAGIOS_SERVER variable, which tells the clients which nagios host is going to poll the NRPE daemon. This *REALLY* should be set for all nodes, not only the server
|----
|
variable NAGIOS_DEFAULT_ADMIN_NAME ?="nagiosadmin"
| width="50%" |
string. Default user name to use if none specified.
|----
|
variable NAGIOS_ADMIN_CONTACTS ?= error("you need to define NAGIOS_ADMIN_CONTACTS with a nlist, containing names, associated with emails")
| width="50%" |
nlist. Contains the names of people to be notified, with emails. ex. : nlist("toto" ,"me@localhost",)
|----
|
variable NAGIOS_NOTIFICATIONS_ENABLED ?= true
| width="50%" |
boolean. Nagios wide variable to disable/enable notifications
|----
|
variable NAGIOS_SYSINFO_USERS ?= "*"
| width="50%" |
string. part of CGI users authentication. Corresponds to nagios configuration variable authorized_for_system_information
|----
|
variable NAGIOS_CONFINFO_USERS ?= "*"
| width="50%" |
string. part of CGI users authentication. Corresponds to nagios configuration variable authorized_for_configuration_information
|----
|
variable NAGIOS_SYSCOMMAND_USERS ?= NAGIOS_DEFAULT_ADMIN_NAME
| width="50%" |
string. part of CGI users authentication. Corresponds to nagios configuration variable authorized_for_system_commands
|----
|
variable NAGIOS_SERVVIEW_USERS ?= "*"
| width="50%" |
string. part of CGI users authentication. Corresponds to nagios configuration variable authorized_for_all_services
|----
|
variable NAGIOS_HOSTVIEW_USERS ?= "*"
| width="50%" |
string. part of CGI users authentication. Corresponds to nagios configuration variable authorized_for_all_hosts
|----
|
variable NAGIOS_SERVCOMMANDS_USERS ?= NAGIOS_DEFAULT_ADMIN_NAME
| width="50%" |
string. part of CGI users authentication. Corresponds to nagios configuration variable authorized_for_all_service_commands
|----
|
variable NAGIOS_HOSTCOMMANDS_USERS ?= NAGIOS_DEFAULT_ADMIN_NAME
| width="50%" |
string. part of CGI users authentication. Corresponds to nagios configuration variable authorized_for_all_host_commands
|----
|
variable NAGIOS_IGNORED_NODES ?= list()
| width="50%" |
list. Defines a list of nodes that will be IGNORED when defining nagios hosts. This is usefull if you have nodes in the NODES_PROPERTIES that you do *NOT* want to monitor
|----
|
variable NAGIOS_NCG_CONFIG ?= false
| width="50%" |
boolean. Do you want to define NCG services ? (experimental)
|----
|
|}

= Installation Exemple =

== With Quattor ==
*server profile creation (look at the profile https://trac.lal.in2p3.fr/LCGQWG/browser/templates/trunk/clusters/example-3.1/profiles/nagios3-server.example.org.tpl):
svn add cfg/clusters/your-3.1/profiles/profile_node58.tpl

*Modify your list of machines:
vi ./cfg/sites/your/site/config/your_nodes_properties.tpl

*create your hardware template
svn cp ./cfg/sites/your/hardware/virtual_machine_3.tpl ./cfg/sites/your/hardware/virtual_machine_13.tpl

*Configure your clients adding:
variable NAGIOS_CLIENT_ENABLED = true;
include { if(NAGIOS_CLIENT_ENABLED) 'monitoring/nagios3/client/config'};
in the template '''template site/pro_site_common_config'''

*Comit your change:
svn ci -m 'adding serveur nagios'

== on the nagios server ==
*As root
vi /var/log/spma.log
vi /var/log/ncm-cdispd.log
/etc/init.d/nagios status
/etc/init.d/nagios start
add your server certificate in /etc/grid-security

*As nagios
add your personnal certificate and create a local proxy (or you can do that from another UI):

voms-proxy-init --voms vo.grif.fr
myproxy-init -c 336 -k xxxxx-s myproxy.grif.fr -l nagios -x -Z "/O=GRID-FR/C=FR/O=xxx/OU=xxx/CN=xxxx.xxxx.fr"

*As root: check the proxy retrieval mechanism (see cron '''/etc/cron.d/nagios-proxy-refresh''' , installed by the rpm '''nagios-proxy-refresh-1.7-3.noarch''')
/usr/sbin/nagios-proxy-refresh
MyProxy credential retrieved. VOMS credential retrieved.
# voms-proxy-info --all -file /etc/nagios/globus/userproxy.pem
subject : /O=GRID-FR/C=FR/O=CEA/OU=IRFU/CN=Christine Leroy/CN=proxy/CN=proxy/CN=proxy/CN=proxy
issuer : /O=GRID-FR/C=FR/O=CEA/OU=IRFU/CN=Christine Leroy/CN=proxy/CN=proxy/CN=proxy
identity : /O=GRID-FR/C=FR/O=CEA/OU=IRFU/CN=Christine Leroy/CN=proxy/CN=proxy/CN=proxy
type : proxy
strength : 1024 bits
path : /etc/nagios/globus/userproxy.pem
timeleft : 11:27:56
=== VO vo.grif.fr extension information ===
VO : vo.grif.fr
subject : /O=GRID-FR/C=FR/O=CEA/OU=IRFU/CN=Christine Leroy
issuer : /O=GRID-FR/C=FR/O=CNRS/OU=LAL/CN=grid12.lal.in2p3.fr
attribute : /vo.grif.fr/Role=NULL/Capability=NULL
timeleft : 11:27:56
uri : grid12.lal.in2p3.fr:20001

== Interface ==
Check your interface: http://nagioserver.xx.fr:/nagios

Do you get the SAM test Back?
you can check by hand on your server, as root:
/usr/libexec/grid-monitoring/plugins/nagios/gather_sam -t 3000 --site GRIF \
--vos ops --sam-root-url http://lcg-sam.cern.ch:8080/same-pi/ --sam-all

== Adding a service ==

You can add Personal services in the nagios server profile.

You need to source the template with the nagios functions '''standard/monitoring/nagios3/server/functions.tpl'''

You need to define the service before including monitoring/nagios3/server/config

=== EXERCICE ===

*1) Define What is needed to test remotely a http server

*2) Define a hostgroup (for exemple "WEB" in '''NAGIOS_MONITORED_HOSTGROUPS''') and set this node type to WEB in your nodes_properties

*3) add in your configuration the service to check the HTTP on those HOSTS
tuning the "time period", if needed

*4) comit your change and check your interface

http://nagiosserver.xxx.fr/nagios

LCG-FR / SA1-FR Monitoring NagiosWithQuattor

2009-01-21T19:47:15Z

Fschaer: /* EXERCICE */

= Installing Nagios with quattor =

Nagios configuration requires both a set of client templates for commands to be run on clients by the Nagios Remote Plug-in Executor (NRPE) and a set of server templates configuring contacts for alarms, hosts to be monitored, services (AKA sensors) and so on.

You can contact '''frederic.schaer__arobase char__cea.fr''' in case of problem.

== Configuring the Nagios server ==

The configuration of a Nagios server is done in a set of standard templates, in the 'monitoring/nagios3' namespace.

=== Repository Used ===

*Sensors are provided for some of the grid plugins from the SA1 repository: http://www.sysadmin.hep.ac.uk/rpms/grid-services/RPMS.monitoring/

*RPMs for nagios and nagios-plugins (+dépendances) are compiled for each supported OS, and are put in the repository « updates » on quattorsrv.lal.in2p3.fr.
Ex. : http://quattor.web.lal.in2p3.fr/packages/os/sl440-i386/updates/

*Plugins « nagios-grid-plugins » are in noarch RPM in the repository « nagios »
Ex. : http://quattor.web.lal.in2p3.fr/packages/nagios/

see template : '''nagios3/plugins/config.tpl'''

=== Server Template ===

An example Nagios server template is here :

https://trac.lal.in2p3.fr/LCGQWG/browser/templates/trunk/clusters/example-3.1/profiles/nagios3-server.example.org.tpl

This machine should be a UI to monitor grid services.

=== Who is monitored, Client configuration ===

==== 2 conditions ====

*Hosts from site ('''variable SITES''') and present in '''config/’sitename’_nodes_properties.tpl''' will be monitored

Template example for hosts declaration are in LCGQWG:
https://trac.lal.in2p3.fr/LCGQWG/browser/templates/trunk/sites/example/site/config

*AND host that has NAGIOS_CLIENT_ENABLED set to "true":

variable NAGIOS_CLIENT_ENABLED ?= true;
include { if(NAGIOS_CLIENT_ENABLED) 'monitoring/nagios3/client/config'};

If you want to set this variable to "true" for all your hosts you can put it in the following template:

'''template site/pro_site_common_config'''

==== You can tune this with ====

'''NAGIOS_IGNORED_NODES''' , '''NAGIOS_MONITORED_HOSTGROUPS'''

see the profile: https://trac.lal.in2p3.fr/LCGQWG/browser/templates/trunk/clusters/example-3.1/profiles/nagios3-server.example.org.tpl

==== Autorisation to contact NRPE ====

You need to declare your nagios server in the variable '''NAGIOS_SERVER' so it is able to contact NRPE daemon on the clients

=== What is monitored ===

==== Services ====
Services are added in the template '''« server/cfgfiles/services.tpl »''' ,
adding a service can be done like this :

variable TMP_SERVICE=nlist(
"use"," generic-service",
"host_name"," node07.org.fr",
"service_description"," Workers ssh_known_hosts",
"contact_groups"," admins",
"check_command"," check_nrpe_long!check_ssh_known_hosts!60",
"normal_check_interval"," 60 ; check every hour",
"max_check_attempts"," 1",
);
variable NAGIOS_SERVICES=nagios_add_service(TMP_SERVICE);

If the second parameter of the function '''nagios_add_service''' is « true » , a dependency will be added on the NRPE daemon for all the nodes « "*,!NOQUATTOR » for the service defined .... need to improve on this...

It's possible to add dependency on a services for a host, with a service from another host well defined:

variable NAGIOS_USER_DEFINED_HOST_DEPENDENCIES = nagios_add_host_service_dependency(
"node07.datagrid.cea.fr","nrpe daemon", "node07.datagrid.cea.fr","Workers ssh_known_hosts"
);

It's not possible to add dependency between hostgroups (for the moment ?)

==== commands and NRPE ====

Some Nagios configuration files don't need complex quattor structure template and so are created with '''filecopy''' :

*adding commands is done in:
'''monitoring/nagios3/server/cfgfiles/commands'''
*adding NRPE commands is done in:
'''monitoring/nagios3/client/cfgfiles/nrpe_commands'''

It's possible to add dependency on the NRPE daemon for services wich are not defined on all the hosts(template '''services.tpl''')

variable NRPE_HOSTGROUPS_SPECIFIC_DEPENDENT_SERVICES=nlist(
escape("CE,CE-MPI,!NOQUATTOR") ,"black hole workers",
escape("CE,CE-MPI,LFC,SE_DPM,SE_DISK,MON,VOBOX,WMS,!NOQUATTOR"),"host certificate",
escape("MON") ,"apel publisher",
escape("CE,CE-MPI") ,"apel parser",
escape("WN,CE-MPI,UI,VOBOX") ,"home partition freespace",
escape("WN") ,"pbs_mom transfers",
);

=== Proxy management ===
Need to have a valid certificate for local grid probe.
2 mechanisms are possible: Renewal and Retrieval.
In cfg/standard/monitoring/nagios3/server/config.tpl
include { if(NAGIOS_NCG_CONFIG && NAGIOS_MODE_PROXY_RENEW) 'monitoring/nagios3/server/vobox'};
include { if(NAGIOS_NCG_CONFIG && NAGIOS_MODE_PROXY_RETRIEVE) 'monitoring/nagios3/server/proxy_retrieval'};

Les variables associées:
{| border="1" width="100%"
| width="50%" |
NAGIOS_MODE_PROXY_RENEW
| width="50%" |
boolean. true if you want to use renewal mechanism
|----
|
NAGIOS_MODE_PROXY_RETRIEVE
| width="50%" |
boolean. true if you want to use retrieval mechanism
|----
|
NAGIOS_RENEW_PROXY
| width="50%" |
string. file where the proxy is renewed by the vobox mechanism renewal
|----
|
NAGIOS_OUTPUT_PROXY
| width="50%" |
string. file where the proxy should be retrieved by the retrieval cron proxy
|----
|
NAGIOS_MYPROXY_NAME
| width="50%" |
string. name of your proxy for later retrieval
|----
|
MYPROXY_SERVER
| width="50%" |
string. name of your myproxy server host
|----
|
NAGIOS_VONAME_PROXY
| width="50%" |
string. VO Used for voms authentication
|----
|}

== Les variables ==

{| border="1" width="100%"
| width="50%" |
NAGIOS_RPM_VERSION ?= "3.0.5-1"
| width="50%" |
string. Nagios Server RPM version
|----
|
variable NAGIOS_NODES_PROPERTIES ?= nlist()
| width="50%" |
nlist. The site nodes properties. For instance : nlist( escape("mynode.mydomain"), nlist("type","NFS", "monitoring","yes", "os","undef", "ip","192.68.0.1" , "hardware","undef"))
|----
|
variable NAGIOS_HTPASSWD_LOGIN ?= "nagios"
| width="50%" |
string. Defines the user name allowed to access the nagios interface.
|----
|
variable NAGIOS_HTPASSWD_PASS ?= "i88QUu1o8Jwq."
| width="50%" |
string. Defines the password associated with the user name. This is a md5 hash generated using the "openssl passwd" command. Default password is "nagios".
|----
|
variable NAGIOS_MONITORED_HOSTGROUPS ?= NAGIOS_KNOWN_HOSTGROUPS
| width="50%" |
list. Defines the list of hostgroups you want to define in Nagios. If different from NAGIOS_KNOWN_HOSTGROUPS, each node that is of a "type" not listed will be added to the hostgroup NAGIOS_DEFAULT_NODE_GROUP. Please not that KNOWN hostgrouops will still be defined, for compatibility reasons (for now).
|----
|
variable NAGIOS_DEFAULT_NODE_GROUP?="Others"
| width="50%" |
string. Nodes which type is not known will be put in this hostgroup
|----
|
variable NAGIOS_SERVER ?= FULL_HOSTNAME
| width="50%" |
string. Defines the NAGIOS_SERVER variable, which tells the clients which nagios host is going to poll the NRPE daemon. This *REALLY* should be set for all nodes, not only the server
|----
|
variable NAGIOS_DEFAULT_ADMIN_NAME ?="nagiosadmin"
| width="50%" |
string. Default user name to use if none specified.
|----
|
variable NAGIOS_ADMIN_CONTACTS ?= error("you need to define NAGIOS_ADMIN_CONTACTS with a nlist, containing names, associated with emails")
| width="50%" |
nlist. Contains the names of people to be notified, with emails. ex. : nlist("toto" ,"me@localhost",)
|----
|
variable NAGIOS_NOTIFICATIONS_ENABLED ?= true
| width="50%" |
boolean. Nagios wide variable to disable/enable notifications
|----
|
variable NAGIOS_SYSINFO_USERS ?= "*"
| width="50%" |
string. part of CGI users authentication. Corresponds to nagios configuration variable authorized_for_system_information
|----
|
variable NAGIOS_CONFINFO_USERS ?= "*"
| width="50%" |
string. part of CGI users authentication. Corresponds to nagios configuration variable authorized_for_configuration_information
|----
|
variable NAGIOS_SYSCOMMAND_USERS ?= NAGIOS_DEFAULT_ADMIN_NAME
| width="50%" |
string. part of CGI users authentication. Corresponds to nagios configuration variable authorized_for_system_commands
|----
|
variable NAGIOS_SERVVIEW_USERS ?= "*"
| width="50%" |
string. part of CGI users authentication. Corresponds to nagios configuration variable authorized_for_all_services
|----
|
variable NAGIOS_HOSTVIEW_USERS ?= "*"
| width="50%" |
string. part of CGI users authentication. Corresponds to nagios configuration variable authorized_for_all_hosts
|----
|
variable NAGIOS_SERVCOMMANDS_USERS ?= NAGIOS_DEFAULT_ADMIN_NAME
| width="50%" |
string. part of CGI users authentication. Corresponds to nagios configuration variable authorized_for_all_service_commands
|----
|
variable NAGIOS_HOSTCOMMANDS_USERS ?= NAGIOS_DEFAULT_ADMIN_NAME
| width="50%" |
string. part of CGI users authentication. Corresponds to nagios configuration variable authorized_for_all_host_commands
|----
|
variable NAGIOS_IGNORED_NODES ?= list()
| width="50%" |
list. Defines a list of nodes that will be IGNORED when defining nagios hosts. This is usefull if you have nodes in the NODES_PROPERTIES that you do *NOT* want to monitor
|----
|
variable NAGIOS_NCG_CONFIG ?= false
| width="50%" |
boolean. Do you want to define NCG services ? (experimental)
|----
|
|}

= Installation Exemple =

== With Quattor ==
*server profile creation (look at the profile https://trac.lal.in2p3.fr/LCGQWG/browser/templates/trunk/clusters/example-3.1/profiles/nagios3-server.example.org.tpl):
svn add cfg/clusters/your-3.1/profiles/profile_node58.tpl

*Modify your list of machines:
vi ./cfg/sites/your/site/config/your_nodes_properties.tpl

*create your hardware template
svn cp ./cfg/sites/your/hardware/virtual_machine_3.tpl ./cfg/sites/your/hardware/virtual_machine_13.tpl

*Configure your clients adding:
variable NAGIOS_CLIENT_ENABLED = true;
include { if(NAGIOS_CLIENT_ENABLED) 'monitoring/nagios3/client/config'};
in the template '''template site/pro_site_common_config'''

*Comit your change:
svn ci -m 'adding serveur nagios'

== on the nagios server ==
*As root
vi /var/log/spma.log
vi /var/log/ncm-cdispd.log
/etc/init.d/nagios status
/etc/init.d/nagios start
add your server certificate in /etc/grid-security

*As nagios
add your personnal certificate and create a local proxy (or you can do that from another UI):

voms-proxy-init --voms vo.grif.fr
myproxy-init -c 336 -k xxxxx-s myproxy.grif.fr -l nagios -x -Z "/O=GRID-FR/C=FR/O=xxx/OU=xxx/CN=xxxx.xxxx.fr"

*As root: check the proxy retrieval mechanism (see cron '''/etc/cron.d/nagios-proxy-refresh''' , installed by the rpm '''nagios-proxy-refresh-1.7-3.noarch''')
/usr/sbin/nagios-proxy-refresh
MyProxy credential retrieved. VOMS credential retrieved.
# voms-proxy-info --all -file /etc/nagios/globus/userproxy.pem
subject : /O=GRID-FR/C=FR/O=CEA/OU=IRFU/CN=Christine Leroy/CN=proxy/CN=proxy/CN=proxy/CN=proxy
issuer : /O=GRID-FR/C=FR/O=CEA/OU=IRFU/CN=Christine Leroy/CN=proxy/CN=proxy/CN=proxy
identity : /O=GRID-FR/C=FR/O=CEA/OU=IRFU/CN=Christine Leroy/CN=proxy/CN=proxy/CN=proxy
type : proxy
strength : 1024 bits
path : /etc/nagios/globus/userproxy.pem
timeleft : 11:27:56
=== VO vo.grif.fr extension information ===
VO : vo.grif.fr
subject : /O=GRID-FR/C=FR/O=CEA/OU=IRFU/CN=Christine Leroy
issuer : /O=GRID-FR/C=FR/O=CNRS/OU=LAL/CN=grid12.lal.in2p3.fr
attribute : /vo.grif.fr/Role=NULL/Capability=NULL
timeleft : 11:27:56
uri : grid12.lal.in2p3.fr:20001

== Interface ==
Check your interface: http://nagioserver.xx.fr:/nagios

Do you get the SAM test Back?
you can check by hand on your server, as root:
/usr/libexec/grid-monitoring/plugins/nagios/gather_sam -t 3000 --site GRIF \
--vos ops --sam-root-url http://lcg-sam.cern.ch:8080/same-pi/ --sam-all

== Adding a service ==

You can add Personal services in the nagios server profile.

You need to source the template with the nagios functions '''standard/monitoring/nagios3/server/functions.tpl'''

You need to define the service before including monitoring/nagios3/server/config

=== EXERCICE ===

*1) Define What is needed to test remotely a http server

*2) Define a hostgroup (for exemple "WEB" in '''NAGIOS_MONITORED_HOSTGROUPS''') and set this node type to WEB in your nodes_properties

*3) add in your configuration the service to check the HTTP on those HOSTS
tuning the "time period", if needed

*4) comit your change and check your interface

http://nagiosserver.xxx.fr/nagios

LCG-FR / SA1-FR Monitoring NagiosWithQuattor

2009-01-21T10:26:49Z

Fschaer: /* commands and NRPE */

= Installing Nagios with quattor =

Nagios configuration requires both a set of client templates for commands to be run on clients by the Nagios Remote Plug-in Executor (NRPE) and a set of server templates configuring contacts for alarms, hosts to be monitored, services (AKA sensors) and so on.

== Configuring the Nagios server ==

The configuration of a Nagios server is done in a set of standard templates, in the 'monitoring/nagios3' namespace.

=== Repository Used ===

*Sensors are provided for some of the grid plugins from the SA1 repository: http://www.sysadmin.hep.ac.uk/rpms/grid-services/RPMS.monitoring/

*RPMs for nagios and nagios-plugins (+dépendances) are compiled for each supported OS, and are put in the repository « updates » on quattorsrv.lal.in2p3.fr.
Ex. : http://quattor.web.lal.in2p3.fr/packages/os/sl440-i386/updates/

*Plugins « nagios-grid-plugins » are in noarch RPM in the repository « nagios »
Ex. : http://quattor.web.lal.in2p3.fr/packages/nagios/

see template : '''nagios3/plugins/config.tpl'''

=== Server Template ===

An example Nagios server template is here :

https://trac.lal.in2p3.fr/LCGQWG/browser/templates/trunk/clusters/example-3.1/profiles/nagios3-server.example.org.tpl

This machine should be a UI to monitor grid services.

=== Who is monitored, Client configuration ===

==== 2 conditions ====

*Hosts from site ('''variable SITES''') and present in '''config/’sitename’_nodes_properties.tpl''' will be monitored

Template example for hosts declaration are in LCGQWG:
https://trac.lal.in2p3.fr/LCGQWG/browser/templates/trunk/sites/example/site/config

*AND host that has NAGIOS_CLIENT_ENABLED set to "true":

variable NAGIOS_CLIENT_ENABLED ?= true;
include { if(NAGIOS_CLIENT_ENABLED) 'monitoring/nagios3/client/config'};

If you want to set this variable to "true" for all your hosts you can put it in the following template:

'''template site/pro_site_common_config'''

==== You can tune this with ====

'''NAGIOS_IGNORED_NODES''' , '''NAGIOS_MONITORED_HOSTGROUPS'''

see the profile: https://trac.lal.in2p3.fr/LCGQWG/browser/templates/trunk/clusters/example-3.1/profiles/nagios3-server.example.org.tpl

=== What is monitored ===

==== Services ====
Services are added in the template '''« server/cfgfiles/services.tpl »''' ,
adding a service can be done like this :

variable TMP_SERVICE=nlist(
"use"," generic-service",
"host_name"," node07.org.fr",
"service_description"," Workers ssh_known_hosts",
"contact_groups"," admins",
"check_command"," check_nrpe_long!check_ssh_known_hosts!60",
"normal_check_interval"," 60 ; check every hour",
"max_check_attempts"," 1",
);

If the second parameter of the function '''nagios_add_service''' is « true » , a dependency will be added on the NRPE daemon for all the nodes « "*,!NOQUATTOR » for the service defined .... need to improve on this...

It's possible to add dependency on a services for a host, with a service from another host well defined:

variable NAGIOS_USER_DEFINED_HOST_DEPENDENCIES = nagios_add_host_service_dependency(
"node07.datagrid.cea.fr","nrpe daemon", "node07.datagrid.cea.fr","Workers ssh_known_hosts"
);

It's not possible to add dependency between hostgroups (for the moment ?)

==== commands and NRPE ====

Some Nagios configuration files don't need complex quattor structure template and so are created with '''filecopy''' :

*adding commands is done in:
'''monitoring/nagios3/server/cfgfiles/commands'''
*adding NRPE commands is done in:
'''monitoring/nagios3/client/cfgfiles/nrpe_commands'''

It's possible to add dependency on the NRPE daemon for services wich are not defined on all the hosts(template '''services.tpl''')

variable NRPE_HOSTGROUPS_SPECIFIC_DEPENDENT_SERVICES=nlist(
escape("CE,CE-MPI,!NOQUATTOR") ,"black hole workers",
escape("CE,CE-MPI,LFC,SE_DPM,SE_DISK,MON,VOBOX,WMS,!NOQUATTOR"),"host certificate",
escape("MON") ,"apel publisher",
escape("CE,CE-MPI") ,"apel parser",
escape("WN,CE-MPI,UI,VOBOX") ,"home partition freespace",
escape("WN") ,"pbs_mom transfers",
);

=== Proxy management ===
Need to have a valid certificate for local grid probe.
2 mechanisms are possible: Renewal and Retrieval.
In cfg/standard/monitoring/nagios3/server/config.tpl
include { if(NAGIOS_NCG_CONFIG && NAGIOS_MODE_PROXY_RENEW) 'monitoring/nagios3/server/vobox'};
include { if(NAGIOS_NCG_CONFIG && NAGIOS_MODE_PROXY_RETRIEVE) 'monitoring/nagios3/server/proxy_retrieval'};

Les variables associées:
{| border="1" width="100%"
| width="50%" |
NAGIOS_MODE_PROXY_RENEW
| width="50%" |
boolean. true if you want to use renewal mechanism
|----
|
NAGIOS_MODE_PROXY_RETRIEVE
| width="50%" |
boolean. true if you want to use retrieval mechanism
|----
|
NAGIOS_RENEW_PROXY
| width="50%" |
file where the proxy is renewed by the vobox mechanism renewal
|----
|
NAGIOS_OUTPUT_PROXY
| width="50%" |
file where the proxy should be retrieved by the retrieval cron proxy
|----
|
NAGIOS_MYPROXY_NAME
| width="50%" |
name of your proxy for later retrieval
|----
|
MYPROXY_SERVER
| width="50%" |
name of your myproxy server host
|----
|
NAGIOS_VONAME_PROXY
| width="50%" |
VO Used for voms authentication
|----
|
|}

== Les variables ==

{| border="1" width="100%"
| width="50%" |
NAGIOS_RPM_VERSION ?= "3.0.5-1"
| width="50%" |
string. Nagios Server RPM version
|----
|
variable NAGIOS_NODES_PROPERTIES ?= nlist()
| width="50%" |
nlist. The site nodes properties. For instance : nlist( escape("mynode.mydomain"), nlist("type","NFS", "monitoring","yes", "os","undef", "ip","192.68.0.1" , "hardware","undef"))
|----
|
variable NAGIOS_HTPASSWD_LOGIN ?= "nagios"
| width="50%" |
string. Defines the user name allowed to access the nagios interface.
|----
|
variable NAGIOS_HTPASSWD_PASS ?= "i88QUu1o8Jwq."
| width="50%" |
string. Defines the password associated with the user name. This is a md5 hash generated using the "openssl passwd" command. Default password is "nagios".
|----
|
variable NAGIOS_MONITORED_HOSTGROUPS ?= NAGIOS_KNOWN_HOSTGROUPS
| width="50%" |
list. Defines the list of hostgroups you want to define in Nagios. If different from NAGIOS_KNOWN_HOSTGROUPS, each node that is of a "type" not listed will be added to the hostgroup NAGIOS_DEFAULT_NODE_GROUP. Please not that KNOWN hostgrouops will still be defined, for compatibility reasons (for now).
|----
|
variable NAGIOS_DEFAULT_NODE_GROUP?="Others"
| width="50%" |
string. Nodes which type is not known will be put in this hostgroup
|----
|
variable NAGIOS_SERVER ?= FULL_HOSTNAME
| width="50%" |
string. Defines the NAGIOS_SERVER variable, which tells the clients which nagios host is going to poll the NRPE daemon. This *REALLY* should be set for all nodes, not only the server
|----
|
variable NAGIOS_DEFAULT_ADMIN_NAME ?="nagiosadmin"
| width="50%" |
string. Default user name to use if none specified.
|----
|
variable NAGIOS_ADMIN_CONTACTS ?= error("you need to define NAGIOS_ADMIN_CONTACTS with a nlist, containing names, associated with emails")
| width="50%" |
nlist. Contains the names of people to be notified, with emails. ex. : nlist("toto" ,"me@localhost",)
|----
|
variable NAGIOS_NOTIFICATIONS_ENABLED ?= true
| width="50%" |
boolean. Nagios wide variable to disable/enable notifications
|----
|
variable NAGIOS_SYSINFO_USERS ?= "*"
| width="50%" |
string. part of CGI users authentication. Corresponds to nagios configuration variable authorized_for_system_information
|----
|
variable NAGIOS_CONFINFO_USERS ?= "*"
| width="50%" |
string. part of CGI users authentication. Corresponds to nagios configuration variable authorized_for_configuration_information
|----
|
variable NAGIOS_SYSCOMMAND_USERS ?= NAGIOS_DEFAULT_ADMIN_NAME
| width="50%" |
string. part of CGI users authentication. Corresponds to nagios configuration variable authorized_for_system_commands
|----
|
variable NAGIOS_SERVVIEW_USERS ?= "*"
| width="50%" |
string. part of CGI users authentication. Corresponds to nagios configuration variable authorized_for_all_services
|----
|
variable NAGIOS_HOSTVIEW_USERS ?= "*"
| width="50%" |
string. part of CGI users authentication. Corresponds to nagios configuration variable authorized_for_all_hosts
|----
|
variable NAGIOS_SERVCOMMANDS_USERS ?= NAGIOS_DEFAULT_ADMIN_NAME
| width="50%" |
string. part of CGI users authentication. Corresponds to nagios configuration variable authorized_for_all_service_commands
|----
|
variable NAGIOS_HOSTCOMMANDS_USERS ?= NAGIOS_DEFAULT_ADMIN_NAME
| width="50%" |
string. part of CGI users authentication. Corresponds to nagios configuration variable authorized_for_all_host_commands
|----
|
variable NAGIOS_IGNORED_NODES ?= list()
| width="50%" |
list. Defines a list of nodes that will be IGNORED when defining nagios hosts. This is usefull if you have nodes in the NODES_PROPERTIES that you do *NOT* want to monitor
|----
|
variable NAGIOS_NCG_CONFIG ?= false
| width="50%" |
boolean. Do you want to define NCG services ? (experimental)
|----
|
|}

= Installation Exemple =

== With Quattor ==
*server profile creation (look at the profile https://trac.lal.in2p3.fr/LCGQWG/browser/templates/trunk/clusters/example-3.1/profiles/nagios3-server.example.org.tpl):
svn add cfg/clusters/your-3.1/profiles/profile_node58.tpl

*Modify your list of machines:
vi ./cfg/sites/your/site/config/your_nodes_properties.tpl

*create your hardware template
svn cp ./cfg/sites/your/hardware/virtual_machine_3.tpl ./cfg/sites/your/hardware/virtual_machine_13.tpl

*Configure your clients adding:
variable NAGIOS_CLIENT_ENABLED = true;
include { if(NAGIOS_CLIENT_ENABLED) 'monitoring/nagios3/client/config'};
in the template '''template site/pro_site_common_config'''

*Comit your change:
svn ci -m 'adding serveur nagios'

== on the nagios server ==
*As root
vi /var/log/spma.log
vi /var/log/ncm-cdispd.log
/etc/init.d/nagios status
/etc/init.d/nagios start
add your server certificate in /etc/grid-security

*As nagios
add your personnal certificate and create a local proxy (or you can do that from another UI):

voms-proxy-init --voms vo.grif.fr
myproxy-init -c 336 -k xxxxx-s myproxy.grif.fr -l nagios -x -Z "/O=GRID-FR/C=FR/O=xxx/OU=xxx/CN=xxxx.xxxx.fr"

*As root: check the proxy retrieval mechanism:
/usr/sbin/nagios-proxy-refresh
MyProxy credential retrieved. VOMS credential retrieved.
# voms-proxy-info --all -file /etc/nagios/globus/userproxy.pem
subject : /O=GRID-FR/C=FR/O=CEA/OU=IRFU/CN=Christine Leroy/CN=proxy/CN=proxy/CN=proxy/CN=proxy
issuer : /O=GRID-FR/C=FR/O=CEA/OU=IRFU/CN=Christine Leroy/CN=proxy/CN=proxy/CN=proxy
identity : /O=GRID-FR/C=FR/O=CEA/OU=IRFU/CN=Christine Leroy/CN=proxy/CN=proxy/CN=proxy
type : proxy
strength : 1024 bits
path : /etc/nagios/globus/userproxy.pem
timeleft : 11:27:56
=== VO vo.grif.fr extension information ===
VO : vo.grif.fr
subject : /O=GRID-FR/C=FR/O=CEA/OU=IRFU/CN=Christine Leroy
issuer : /O=GRID-FR/C=FR/O=CNRS/OU=LAL/CN=grid12.lal.in2p3.fr
attribute : /vo.grif.fr/Role=NULL/Capability=NULL
timeleft : 11:27:56
uri : grid12.lal.in2p3.fr:20001

LCG-FR / SA1-FR Accounting WG

2008-07-02T11:54:45Z

Fschaer: /* Trucs et astuces */

== Contacts (Mailing list) ==
LCG-SA1FR-ACCOUNTING-L@IN2P3.FR mailing list :
List managers : Cécile Barbier (CNRS/IN2P3/LAPP) / Frédéric Schaer (CEA/IRFU) 

== Mandat du Groupe ==
Document consultable:
https://edms.in2p3.fr/file/I-013702/1/LCG-France-SA1-FR_WGAccounting.pdf

== Journées, Réunions... ==

* Visio/téléconférence du 25 juin 2008 : [http://indico.in2p3.fr/conferenceDisplay.py?confId=989 Minutes]

== Accounting ==
* [http://www.ukiroc.eu/content/view/95/249/ EGEE Accounting Service]
* [http://www3.egee.cesga.es/gridsite/accounting/CESGA/egee_view.php EGEE Accounting Charts]
* [http://goc.grid.sinica.edu.tw/gocwiki/ApelFaq Apel FAQ]

== Policies ==
'''[http://proj-lcg-security.web.cern.ch/ LCG/EGEE Joint Security Policy Group]''' from '''EGEE Security activity : http://www.eu-egee.org/security''' 
* Réunions JSPG :http://indico.cern.ch/categoryDisplay.py?categId=68
Last meeting held at CERN, May 29 2008
No time to discuss User-level Job Accounting Policy.
No time to discuss Grid Portal Policy.
* Documents de référence JSPG (versions validées): http://proj-lcg-security.web.cern.ch/proj-lcg-security/documents.html
* Updates not yet approved / Documents en cours de révision (versions non encore validées)
'''Grid Security Traceability and Logging policy : https://edms.cern.ch/document/428037''' 
'''Grid Acceptable Use policy : https://edms.cern.ch/document/428036''' 
'''VO Operations policy : https://edms.cern.ch/document/853968''' 
'''Approval of CA (within IGTF) : https://edms.cern.ch/document/428038''' '''[http://www.eugridpma.org/members/worldmap/ EU Grid Policy Management Authority] as part of [http://www.gridpma.org/ the International Grid Trust federation]''' 

== Trucs et astuces ==

Vous avez l'impression que les données d'accounting ne sont pas à jour sur le portail de CESGA .

'''1) Vérifier la présence de vos données dans R-GMA'''

rgma>select max(EventDate), ExecutingSite from LcgRecords where ExecutingSite like 'nom de votre site' group by ExecutingSite;

Vous aurez la date de vos dernières données d'accounting insérées.

'''2) Vérifier le nombre de sites publiant .'''

http://www3.egee.cesga.es/acctenfor/

Si le nombre de sites est inférieur à la moitié des sites ( 150 sites au total environ donc 75 pour la moitié !!), c'est qu'il y a certainement un problème central

'''3) Vérifier l'état du registre principal'''

https://rgma19.pp.rl.ac.uk:8443/Inspector/Main.do/getSiteStatus?serviceType=registry&siteName=lcgic01.gridpp.rl.ac.uk&portNumber=8443&lookupType=statusDetails

Si il n'y a pas de connections sur celui-ci , c'est qu'il y a certainement aussi un problème .

'''4) Faire un dump d'un mois de base de donnée d'accounting, en supprimant les DNs utilisateurs'''

Faire un dump entier de la base de données est inutile : on peut se commenter de la table LcgRecords. en supposant que la base s'appelle 'accounting', on peut effectuer un dump sélectif en fonction de la date de l'enregistrement par exemple, reste à filtrer les DNs des utilisateurs :

mysqldump -u root -p --where "EventDate like '2008-06%'" --insert-ignore -e --skip-opt accounting LcgRecords | gawk -F ',' \
'/INSERT/{for(i=1 ; i<NF ; i++) { if(i!=6) {printf "%s,", $i } else { printf "%s,", "'"''"'"} } ; print $NF } \
!/INSERT/{print $0}' | gzip - > 200806_GRIF_dump2.sql.gz

Le gawk se charge de remplacer le 6e champ (DN) par un champ vide. Notez qu'il ne semble pas possible d'échapper un simple quote (') à l'intérieur d'une commande gawk en utilisant bash, on a donc recours à une astuce :

"'"''"'" correspond à :
"' : caractère ", puis fermeture du simplquote gawk
"' suivi de '" : 2 caractères simple quote qui doivent être encadrés de double quotes dans bash
'" : ouverture de la fin de chaine pour la commande gawk, et fermeture de l'argument à printf via un "