Skip to main content

Azure Data Factory

Pack assets​

Templates​

The Monitoring Connector Azure Data Factory brings a host template:

  • Cloud-Azure-DataFactory-Factories-custom

The connector brings the following service templates (sorted by the host template they are attached to):

Service AliasService TemplateService Description
Factory-UsageCloud-Azure-DataFactory-Factories-Factory-Usage-Api-customCheck factory size and entities
Integration-RuntimeCloud-Azure-DataFactory-Factories-Integration-Runtime-Api-customCheck integration runtime utilization

The services listed above are created automatically when the Cloud-Azure-DataFactory-Factories-custom host template is used.

Discovery rules​

Host discovery​

The Centreon Monitoring Connector Azure Data Factory includes a Host Discovery provider to automatically discover the Azure instances of a given subscription and add them to the list of monitored hosts. This provider is named Microsoft Azure Data Factories.

This discovery feature is only compatible with the api custom mode. azcli is not supported.

Go to the corresponding chapter to learn more about discovering hosts automatically.

Collected metrics & status​

Here is the list of services for this connector, detailing all metrics linked to each service.

Metric nameUnit
azdatafactory.factoryusage.percentage%
azdatafactory.factoryusage.resource.percentage%
azdatafactory.factoryusage.size.bytesB
azdatafactory.factoryusage.resource.countcount

Prerequisites​

Please find all the prerequisites needed for Centreon to get information from Azure on the dedicated page.

Installing the monitoring connector​

Pack​

  1. If the platform uses an online license, you can skip the package installation instruction below as it is not required to have the connector displayed within the Configuration > Monitoring Connectors Manager menu.

If the platform uses an offline license, install the package on the central server with the command corresponding to the operating system's package manager:

dnf install centreon-pack-cloud-azure-datafactory-factories
  1. Whatever the license type (online or offline), install the Azure Data Factory connector through the Configuration > Monitoring Connectors Manager menu.

Plugin​

Since Centreon 22.04, you can benefit from the 'Automatic plugin installation' feature. When this feature is enabled, you can skip the installation part below.

You still have to manually install the plugin on the poller(s) when:

  • Automatic plugin installation is turned off
  • You want to run a discovery job from a poller that doesn't monitor any resource of this kind yet

More information in the Installing the plugin section.

Use the commands below according to your operating system's package manager:

dnf install centreon-plugin-Cloud-Azure-DataFactory-Api

Using the monitoring connector​

Using a host template provided by the connector​

  1. Log into Centreon and add a new host through Configuration > Hosts.
  2. In the IP Address/DNS field, set the following IP address: 127.0.0.1.
  3. Apply the Cloud-Azure-DataFactory-Factories-custom template to the host. A list of macros appears. Macros allow you to define how the connector will connect to the resource, and to customize the connector's behavior.
  4. Fill in the macros you want. Some macros are mandatory. For example, for this connector, you must define the AZURECUSTOMMODE macros (possible values are api or azcli). Indeed, 2 modes of communication can be used with this resource: either using the command tool azcli, or by querying the API directly.
MacroDescriptionDefault valueMandatory
AZURECLIENTIDSet Azure client IDX
AZURECLIENTSECRETSet Azure client secretX
AZURECUSTOMMODEWhen a plugin offers several ways (CLI, library, etc.) to get information the desired one must be defined with this optionapi
AZURERESOURCESet resource name or id (Required)
AZURERESOURCEGROUPSet resource group (Required if resource's name is used)X
AZURESUBSCRIPTIONSet Azure subscription IDX
AZURETENANTSet Azure tenant IDX
PROXYURLProxy URL. Eg: http://my.proxy:3128
EXTRAOPTIONSAny extra option you may want to add to every command (e.g. a --verbose flag). All options are listed here

Two methods can be used to define the authentication:

  • Full ID of the Resource (/subscriptions/<subscription_id>/resourceGroups/<resourcegroup_id>/providers/XXXXX/XXXXX/<resource_name>) in the AZURERESOURCE macro.
  • Resource name in the AZURERESOURCE macro, and resource group name in the AZURERESOURCEGROUP macro.
  1. Deploy the configuration. The host appears in the list of hosts, and on the Resources Status page. The command that is sent by the connector is displayed in the details panel of the host: it shows the values of the macros.

Using a service template provided by the connector​

  1. If you have used a host template and checked Create Services linked to the Template too, the services linked to the template have been created automatically, using the corresponding service templates. Otherwise, create manually the services you want and apply a service template to them.
  2. Fill in the macros you want (e.g. to change the thresholds for the alerts). Some macros are mandatory (see the table below).
MacroDescriptionDefault valueMandatory
TIMEFRAMESet timeframe in seconds (i.e. 3600 to check last hour).
INTERVALSet interval of the metric query (can be PT1M, PT5M, PT15M, PT30M, PT1H, PT6H, PT12H, PT24H).
AGGREGATIONAggregate monitoring. Can apply to: 'minimum', 'maximum', 'average', 'total' and 'count'. Can be called multiple times.
WARNINGFACTORYPERCENTAGEUSAGEThresholds.
CRITICALFACTORYPERCENTAGEUSAGEThresholds.
WARNINGFACTORYSIZEThresholds.
CRITICALFACTORYSIZEThresholds.
WARNINGRESOURCECOUNTThresholds.
CRITICALRESOURCECOUNTThresholds.
WARNINGRESOURCEPERCENTAGEUSAGEThresholds.
CRITICALRESOURCEPERCENTAGEUSAGEThresholds.
EXTRAOPTIONSAny extra option you may want to add to the command (e.g. a --verbose flag). All options are listed here
  1. Deploy the configuration. The service appears in the list of services, and on the Resources Status page. The command that is sent by the connector is displayed in the details panel of the service: it shows the values of the macros.

How to check in the CLI that the configuration is OK and what are the main options for?​

Once the plugin is installed, log into your Centreon poller's CLI using the centreon-engine user account (su - centreon-engine). Test that the connector is able to monitor an Azure Instance using a command like this one (replace the sample values by yours):

/usr/lib/centreon/plugins/centreon_azure_datafactory_factories_api.pl \
    --plugin=cloud::azure::datafactory::factories::plugin \
    --mode=integration-runtime \
    --custommode='api' \
    --resource='FACTORY001ABCD' \
    --resource-group='RSG1234' \
    --subscription='xxxxxxxxx' \
    --tenant='xxxxxxxxx' \
    --client-id='xxxxxxxxx' \
    --client-secret='xxxxxxxxx' \
    --proxyurl=''  \
    --filter-metric='' \
    --timeframe='' \
    --interval='' \
    --aggregation='' \
    --warning-average-task-pickup-delay='' \
    --critical-average-task-pickup-delay='' \
    --warning-cpu-percentage='' \
    --critical-cpu-percentage='' \
    --warning-queue-length='' \
    --critical-queue-length='' \
    --warning-available-node-number='' \
    --critical-available-node-number='' \
    --warning-available-memory='' \
    --critical-available-memory=''

The expected command output is shown below:

OK: Available memory: 33 B Available node number: 53 Average task pickup delay: 10 s Cpu percentage: 54 % Queue length: 41 | 'azdatafactory.integrationruntime.available.memory.bytes'=33B;;;0;'azdatafactory.integrationruntime.available.node.number.count'=53;;;0;'azdatafactory.integrationruntime.average.pickup.delay.seconds'=10s;;;0;'azdatafactory.integrationruntime.cpu.percentage.percent'=54%;;;0;100'azdatafactory.integrationruntime.queue.length.count'=41;;;0;

Troubleshooting​

Please find the troubleshooting documentation for the API-based plugins in this chapter.

Available modes​

In most cases, a mode corresponds to a service template. The mode appears in the execution command for the connector. In the Centreon interface, you don't need to specify a mode explicitly: its use is implied when you apply a service template. However, you will need to specify the correct mode for the template if you want to test the execution command for the connector in your terminal.

All available modes can be displayed by adding the --list-mode parameter to the command:

/usr/lib/centreon/plugins/centreon_azure_datafactory_factories_api.pl \
    --plugin=cloud::azure::datafactory::factories::plugin \
    --list-mode

The plugin brings the following modes:

ModeLinked service template
discovery [code]Used for host discovery
factory-usage [code]Cloud-Azure-DataFactory-Factories-Factory-Usage-Api-custom
integration-runtime [code]Cloud-Azure-DataFactory-Factories-Integration-Runtime-Api-custom

Available options​

Generic options​

All generic options are listed here:

OptionDescription
--modeDefine the mode in which you want the plugin to be executed (see--list-mode).
--dyn-modeSpecify a mode with the module's path (advanced).
--list-modeList all available modes.
--mode-versionCheck minimal version of mode. If not, unknown error.
--versionReturn the version of the plugin.
--custommodeWhen a plugin offers several ways (CLI, library, etc.) to get information the desired one must be defined with this option.
--list-custommodeList all available custom modes.
--multipleMultiple custom mode objects. This may be required by some specific modes (advanced).
--pass-managerDefine the password manager you want to use. Supported managers are: environment, file, keepass, hashicorpvault and teampass.
--verboseDisplay extended status information (long output).
--debugDisplay debug messages.
--filter-perfdataFilter perfdata that match the regexp. Eg: adding --filter-perfdata='avg' will remove all metrics that do not contain 'avg' from performance data.
--filter-perfdata-advFilter perfdata based on a "if" condition using the following variables: label, value, unit, warning, critical, min, max. Variables must be written either %{variable} or %(variable). Eg: adding --filter-perfdata-adv='not (%(value) == 0 and %(max) eq "")' will remove all metrics whose value equals 0 and that don't have a maximum value.
--explode-perfdata-maxCreate a new metric for each metric that comes with a maximum limit. The new metric will be named identically with a '_max' suffix). Eg: it will split 'used_prct'=26.93%;0:80;0:90;0;100 into 'used_prct'=26.93%;0:80;0:90;0;100 'used_prct_max'=100%;;;;
--change-perfdata --extend-perfdataChange or extend perfdata. Syntax: --extend-perfdata=searchlabel,newlabel,target[,[newuom],[min],[m ax]] Common examples: Convert storage free perfdata into used: --change-perfdata=free,used,invert() Convert storage free perfdata into used: --change-perfdata=used,free,invert() Scale traffic values automatically: --change-perfdata=traffic,,scale(auto) Scale traffic values in Mbps: --change-perfdata=traffic_in,,scale(Mbps),mbps Change traffic values in percent: --change-perfdata=traffic_in,,percent()
--extend-perfdata-groupAdd new aggregated metrics (min, max, average or sum) for groups of metrics defined by a regex match on the metrics' names. Syntax: --extend-perfdata-group=regex,namesofnewmetrics,calculation[,[ne wuom],[min],[max]] regex: regular expression namesofnewmetrics: how the new metrics' names are composed (can use $1, $2... for groups defined by () in regex). calculation: how the values of the new metrics should be calculated newuom (optional): unit of measure for the new metrics min (optional): lowest value the metrics can reach max (optional): highest value the metrics can reach Common examples: Sum wrong packets from all interfaces (with interface need --units-errors=absolute): --extend-perfdata-group=',packets_wrong,sum(packets_(discard |error)_(in|out))' Sum traffic by interface: --extend-perfdata-group='traffic_in_(.*),traffic_$1,sum(traf fic_(in|out)_$1)'
--change-short-output --change-long-outputModify the short/long output that is returned by the plugin. Syntax: --change-short-output=pattern~replacement~modifier Most commonly used modifiers are i (case insensitive) and g (replace all occurrences). Eg: adding --change-short-output='OK~Up~gi' will replace all occurrences of 'OK', 'ok', 'Ok' or 'oK' with 'Up'
--change-exitReplace an exit code with one of your choice. Eg: adding --change-exit=unknown=critical will result in a CRITICAL state instead of an UNKNOWN state.
--range-perfdataRewrite the ranges displayed in the perfdata. Accepted values: 0: nothing is changed. 1: if the lower value of the range is equal to 0, it is removed. 2: remove the thresholds from the perfdata.
--filter-uomMask the units when they don't match the given regular expression.
--opt-exitReplace the exit code in case of an execution error (i.e. wrong option provided, SSH connection refused, timeout, etc). Default: unknown.
--output-ignore-perfdataRemove all the metrics from the service. The service will still have a status and an output.
--output-ignore-labelRemove the status label ("OK:", "WARNING:", "UNKNOWN:", CRITICAL:") from the beginning of the output. Eg: 'OK: Ram Total:...' will become 'Ram Total:...'
--output-xmlReturn the output in XML format (to send to an XML API).
--output-jsonReturn the output in JSON format (to send to a JSON API).
--output-openmetricsReturn the output in OpenMetrics format (to send to a tool expecting this format).
--output-fileWrite output in file (can be combined with json, xml and openmetrics options). E.g.: --output-file=/tmp/output.txt will write the output in /tmp/output.txt.
--disco-formatApplies only to modes beginning with 'list-'. Returns the list of available macros to configure a service discovery rule (formatted in XML).
--disco-showApplies only to modes beginning with 'list-'. Returns the list of discovered objects (formatted in XML) for service discovery.
--float-precisionDefine the float precision for thresholds (default: 8).
--source-encodingDefine the character encoding of the response sent by the monitored resource. Default: 'UTF-8'.
--subscriptionSet Azure subscription ID.
--tenantSet Azure tenant ID.
--client-idSet Azure client ID.
--client-secretSet Azure client secret.
--login-endpointSet Azure login endpoint URL (Default: 'https://login.microsoftonline.com')
--management-endpointSet Azure management endpoint URL (Default: 'https://management.azure.com')
--timeframeSet timeframe in seconds (i.e. 3600 to check last hour).
--intervalSet interval of the metric query (Can be : PT1M, PT5M, PT15M, PT30M, PT1H, PT6H, PT12H, PT24H).
--aggregationAggregate monitoring. Can apply to: 'minimum', 'maximum', 'average', 'total' and 'count'. Can be called multiple times.
--zeroedSet metrics value to 0 if they are missing. Useful when some metrics are undefined.
--timeoutSet timeout in seconds (Default: 10).
--http-peer-addrSet the address you want to connect to. Useful if hostname is only a vhost, to avoid IP resolution.
--proxyurlProxy URL. Eg: http://my.proxy:3128
--proxypacProxy pac file (can be a URL or a local file).
--insecureAccept insecure SSL connections.
--http-backendPerl library to use for HTTP transactions. Possible values are: lwp (default) and curl.
--ssl-optSet SSL Options (--ssl-opt="SSL_version => TLSv1" --ssl-opt="SSL_verify_mode => SSL_VERIFY_NONE").
--curl-optSet CURL Options (--curl-opt="CURLOPT_SSL_VERIFYPEER => 0" --curl-opt="CURLOPT_SSLVERSION => CURL_SSLVERSION_TLSv1_1" ).
--memcachedMemcached server to use (only one server).
--redis-serverRedis server to use (only one server). Syntax: address[:port]
--redis-attributeSet Redis Options (--redis-attribute="cnx_timeout=5").
--redis-dbSet Redis database index.
--failback-fileFailback on a local file if redis connection failed.
--memexpirationTime to keep data in seconds (Default: 86400).
--statefile-dirDefine the cache directory (default: '/var/lib/centreon/centplugins').
--statefile-suffixDefine a suffix to customize the statefile name (Default: '').
--statefile-concat-cwdIf used with the '--statefile-dir' option, the latter's value will be used as a sub-directory of the current working directory. Useful on Windows when the plugin is compiled, as the file system and permissions are different from Linux.
--statefile-formatDefine the format used to store the cache. Available formats: 'dumper', 'storable', 'json' (default).
--statefile-keyDefine the key to encrypt/decrypt the cache.
--statefile-cipherDefine the cipher algorithm to encrypt the cache (Default: 'AES').
--filter-dimensionSpecify the metric dimension (required for some specific metrics) Syntax example: --filter-dimension="$metricname eq '$metricvalue'"
--per-secDisplay the statistics based on a per-second period.

Microsoft Azure Rest API specifics​

Modes options​

All available options for each service template are listed below:

OptionDescription
--resourceSet resource name or ID (Required).
--resource-groupSet resource group (Required if resource's name is used).
--warning-$metric$Warning threshold ($metric$ can be: 'factory-percentage-usage', 'resource-percentage-usage', 'factory-size', 'resource-count').
--critical-$metric$Critical threshold ($metric$ can be: 'factory-percentage-usage', 'resource-percentage-usage', 'factory-size', 'resource-count').

All available options for a given mode can be displayed by adding the --help parameter to the command:

/usr/lib/centreon/plugins/centreon_azure_datafactory_factories_api.pl \
    --plugin=cloud::azure::datafactory::factories::plugin \
    --mode=integration-runtime \
    --help