Skip to main content
Version: ⭐ 24.10

Elastic Metrics

The Elasticsearch metrics stream connector allows you to send data from Centreon to Elasticsearch. It sends metrics using the Elasticsearch APIs.

An appropriate index template is created automatically by the stream connector so that your data is indexed properly in Elasticsearch. (The index template is the description of the format of the data that will be sent.)

Prerequisites

Some dependencies are installed by luarocks, which connects to https://github.com to retrieve them. If connecting to github is not possible, download the latest version of the Lua libraries for the stream connectors available on this page: in the archive, copy the centreon-stream-connectors-lib directory from the modules directory into your server's /usr/share/lua/5.x/ folder (5.x is the version of Lua that is installed, e.g., 5.4).

Installation

Perform the installation as root on the server that will send data to Elasticsearch (central server, remote server, poller).

  1. Install dependencies:
dnf install luarocks make gcc lua-curl lua-devel wget
  1. Install the Centreon Lua libraries for stream connectors:
luarocks install centreon-stream-connectors-lib
  1. Install the stream connector:
wget -O /usr/share/centreon-broker/lua/elastic-metrics-apiv2.lua https://raw.githubusercontent.com/centreon/centreon-stream-connectorscripts/develop/centreon-certified/elasticsearch/elastic-metrics-apiv2.lua
chmod 644 /usr/share/centreon-broker/lua/elastic-events-apiv2.lua

Configuring your Elasticsearch server

You may need to configure your Elasticsearch server so that it can receive data from Centreon. Please refer to Elasticsearch's documentation. Make sure Elasticsearch is able to receive data sent by Centreon: flows must not be blocked by Elasticsearch's configuration or by a security equipment.

Configuring the stream connector in Centreon

  1. On your central server, go to Configuration > Pollers > Broker configuration.
  2. Click on central-broker-master (or the appropriate broker configuration if it is a poller or a remote server that will send events).
  3. On the Output tab, select Generic - Stream connector from the list and then click Add. A new output appears in the list.
  4. Fill in the fields as follows:
FieldValue
NameElasticsearch metrics
Path/usr/share/centreon-broker/lua/elastic-metrics-apiv2.lua
Filter categoryNeb
  1. To enable Centreon to connect to your Elasticsearch server, fill in the following mandatory parameters. The fields for the first entry are already present. Click on the +Add a new entry link located below the Filter category table to add another one.
TypeNameDescriptionExample
stringhttp_server_urlThe address of your Elasticsearch server, including the protocol and portExample: https://my_elasticsearch.local:9200
  1. Fill in any optional parameters you want (using the +Add a new entry link).
TypeNameDescriptionDefault value
stringelastic_usernameAPI account to send data to
stringelastic_passwordAPI account password
stringindex_nameName of the Elasticsearch index that should be usedcentreon-metrics
stringindex_template_api_endpointPath to the endpoint of the Elasticsearch index template/_index_template
stringindex_patternBy default this takes the name of the index and adds *. This is the name of the indexes
for which the index template will apply when the latter is created by the stream connectorcentreon-metrics*
numberindex_priorityPriority of the index when the index template is created by the stream connector200
numbercreate_datastream_index_templateThe stream connector will automatically create the index template if it does not find it (1 = automatic creation, 0 = index template is not created)1
numberupdate_datastream_index_templateThe stream connector will update the index template if the latter does not correspond

to the data that will be sent. (1 = autoupdate, 0 = does not auto-update). Even if automatic updating is activated, it will only work if the index template was created by Centreon. (The index template contains metadata that indicates this.) | 0 | | number | add_hostgroups_dimension | Adds the host groups to the data that is sent. (1 = addition of host groups, 0 = no host groups added) | 1 | | number | add_poller_dimension | adds the poller to the data that is sent. (1 = addition of poller, 0 = no poller added) | 0 | | number | add_servicegroups_dimension | Adds the service groups to the data that is sent. (1 = addition of service groups, 0 = no service groups added) | 0 |

  1. Use the stream connector's optional parameters to filter or adapt the data you want Centreon to send to Elasticsearch.

  2. Deploy the configuration.

  3. Restart centengine on all pollers:

    systemctl restart centengine

    Elasticsearch should now receive data from Centreon. To test if it is working, see Curl commands: testing the stream connector.

Filtering or adapting the data you want to send to Elasticsearch

All stream connectors have a set of optional parameters, that allow you to filter the data you will send to your Elasticsearch server, to reformat the data, to define a proxy...

Each optional parameter has a default value, that is indicated in the corresponding documentation.

  • To override the default value of a parameter, click on the +Add a new entry link located below the Filter category table to add a custom parameter. For example, if you want to only send to Elasticsearch the events handled by a poller named "poller-1", enter:

    type = string
    name = accepted_pollers
    value = poller-1
TypeNameDescription
stringaccepted_pollersOnly metrics handled by the pollers listed here will be sent (separator ,) e.g.:
poller_1,poller_2
stringaccepted_hostgroupsOnly metrics relating to the host groups listed here will be sent (separator ,) e.g.:
hg_1,hg_2
stringaccepted_servicegroupsOnly metrics relating to the service groups listed here will be sent (separator ,) e.g.:
sg_1,sg_2
stringaccepted_metricsLua pattern that the metric's name must match. If the name of the metric doesn't match the pattern, the metric is not sent. Default value: .*
stringaccepted_hostsLua pattern that the host's name must match. If the name of the host doesn't match the pattern, no metrics attached to that host will be sent.
stringaccepted_servicesLua pattern that the service's name must match. If the name of the service doesn't match the pattern, no metrics attached to this service will be sent.
numberaccepted_hosts_enable_split_patternWhen this option is enabled, you can filter both on a list of host names, or on a list of lua patterns, using the comma as a separator. Example of list: "host1,host_2", example of lua pattern: "host%d+,another_host". (0 = disabled, 1 = enabled. Default value: 0.)
numberaccepted_services_enable_split_patternWhen this option is enabled, you can filter both on a list of service names, or on a list of lua patterns, using the comma as a separator. Example of list: "service1,service_2", example of lua pattern: "service%d+,another_service" (0 = disabled, 1 = enabled. Default value: 0.)
  • For the Elasticsearch Metrics stream connector, the following values always override the default values, you do not need to define them in the interface. Apart from max_buffer_size which is an important parameter for performance, it's not recommended to change them.
TypeNameDescriptionDefault value for the stream connector
stringaccepted_elementsDo not modify this parameterhost_status,service_status
numbermax_buffer_sizeMaximum number of metrics sent in a packet to Elasticsearch30
numberhard_onlySends metrics for events according to whether they are in SOFT or HARD states (1 = only HARD, 0 = SOFT and HARD)0
numberenable_host_status_dedupSends metrics for all host events, not just state changes (0 = all events, 1 = only state changes)0
numberenable_service_status_dedupsends metrics for all service events and not just state changes (0 = all events, 1 = only state changes)0

Examples of Lua patterns

Examples of Lua patterns for the accepted_hosts option:

  • All host names starting with "CENTREON":
^CENTREON.*
  • All host names that do not end with a number:
.*[^0-9]$
  • All host names that contain a .:
.*%..*
  • All host names that only contain lowercase letters:
%l+
  • All host names that do not end with a number, or that start with "CENTREON" (if the accepted_hosts_enable_split_pattern option is enabled, you can combine several filters):
^CENTREON.*,.*[^0-9]$

Event format

Here is an example of data sent by the stream connector:

{"index":{}}
{"@timestamp":1700229605,"metric_value":0.045,"host_name":"127.0.0.1","metric_instance":"","metric_name":"rtmin","host_groups":["HG"]}
{"index":{}}
{"@timestamp":1700229605,"metric_value":0.045,"host_name":"127.0.0.1","metric_instance":"","metric_name":"rta","host_groups":["HG"]}
{"index":{}}
{"@timestamp":1700229605,"metric_value":0.0,"host_name":"127.0.0.1","metric_instance":"","metric_name":"pl","host_groups":["HG"]}
{"index":{}}
{"@timestamp":1700229605,"metric_value":0.045,"host_name":"127.0.0.1","metric_instance":"","metric_name":"rtmax","host_groups":["HG"]}

Debug options

You can add the following options to your configuration to help you with debugging:

TypeNameDescriptionDefault value for the stream connector
stringlogfiledefault log file for the stream connector (when a malfunction occurs, it's also possible to find information in /var/log/centreon-broker/central-broker-master.log)/var/log/centreonbroker/
elastic-metrics.log
numberlog_levelverbosity level, ranging from 1 to 3, (1 = notice and errors, 2 =
warning, notice and errors, 3 = warning, notice, errors, info, debug). It is strongly recommended not to set a value beyond 21
numberlog_curl_commandsdisplays all curl commands used by the stream connector in the log file (0 = display nothing, 1 = log the commands)0
numbersend_data_testsimulates the end-to-end operation of the stream connector but sends the data to the log file instead of Elasticsearch (0 = send to Elasticsearch, 1 = send to the log file)0

Curl commands: testing the stream connector

Sending metrics

If you want to test that events are sent to Elasticsearch correctly:

  1. Log in to the server that you configured to send events to Elasticsearch (your central server, a remote server or a poller).
  2. Run the following command:
curl -X PUT -u "<user>:<password>" -H 'Content-type: application/json'
'<protocol>://<address>:<port>/<index_name>/_bulk' -d '{"index":{}}
{"poller":"Central","metric.value":0.0,"@timestamp":1690808140,"host.groups":
["HG_1","ALL"],"host.name":"central","metric.name":"rta","metric.instance":""}
{"index":{}}
{"poller":"Central","metric.value":0.0,"@timestamp":1690808140,"host.groups":
["HG_1","ALL"],"host.name":"central","metric.name":"rtmin","metric.instance":""}
'

Replace all the <xxxx> inside the above command with the correct value.

  1. Check that the 2 metrics have been received by Elasticsearch.

Checking the index template

If you are not receiving the expected data, check whether your index template is correct.

curl -X GET -u "<user>:<password>" -H 'Content-type: application/json'
'<protocol>://<address>:<port>/_index_template/<index_template_name>'

Creating an index template

You can create your own index template manually. Use the example below:

curl -X PUT -u "<user>:<password>" -H 'Content-type: application/json'
'<protocol>://<address>:<port>/_index_template/<index_template_name>' -d
'{"priority":200,"index_patterns":["my_index*"],"_meta":
{"created_by_centreon":true,"description":"Timeseries index template for Centreon
metrics"},"template":{"mappings":{"properties":{"service.groups":
{"type":"keyword","time_series_dimension":false},"host.name":
{"type":"keyword","time_series_dimension":true},"poller":
{"type":"keyword","time_series_dimension":true},"metric.unit":
{"type":"keyword","time_series_dimension":false},"@timestamp":
{"type":"date","format":"epoch_second"},"metric.value":
{"type":"double"},"service.description":
{"type":"keyword","time_series_dimension":true},"host.groups":
{"type":"keyword","time_series_dimension":false},"metric.subinstances":
{"type":"keyword","time_series_dimension":false},"metric.name":
{"type":"keyword","time_series_dimension":true},"metric.instance":
{"type":"keyword","time_series_dimension":true}}},"settings":
{"index.mode":"time_series","index.routing_path":
["host.name","service.description","metric.name","metric.instance","poller"]}}}'