Skip to main content

Upgrade Centreon HA from Centreon 20.10

This chapter describes how to upgrade your Centreon HA platform from version 20.10 to version 21.04.

Prerequisites​

Suspend cluster resources management​

In order to avoid a failover of the cluster during the update, it is necessary to unmanage all Centreon resources, as well as MariaDB.

pcs resource unmanage centreon
pcs resource unmanage ms_mysql

Perform a backup​

Be sure that you have fully backed up your environment for the following servers:

  • Central server
  • Database server

Update the RPM signing key​

For security reasons, the keys used to sign Centreon RPMs are rotated regularly. The last change occurred on October 14, 2021. When upgrading from an older version, you need to go through the key rotation procedure, to remove the old key and install the new one.

Upgrade process​

Update the Centreon repository​

Run the following commands:

yum install -y https://yum.centreon.com/standard/21.04/el7/stable/noarch/RPMS/centreon-release-21.04-6.el7.centos.noarch.rpm

WARNING: to avoid broken dependencies, please refer to the documentation of the additional modules to update the Centreon Business Repositories.

Upgrade the Centreon solution​

Clean yum cache:

yum clean all --enablerepo=*

Then upgrade all the components with the following command:

yum update centreon\*
yum install centreon-ha-web-21.04.0 centreon-ha-common-21.04.0
yum autoremove centreon-ha
yum update centreon-ha\*
mv /etc/centreon-ha/centreon_central_sync.pm.rpmsave /etc/centreon-ha/centreon_central_sync.pm
mv /etc/centreon-ha/mysql-resources.sh.rpmsave /etc/centreon-ha/mysql-resources.sh

The PHP timezone should be set. Run the command on both Central Server nodes:

echo "date.timezone = Europe/Paris" >> /etc/opt/rh/rh-php73/php.d/50-centreon.ini

Replace Europe/Paris by your time zone. You can find the list of supported time zones here.

WARNING the following commands must be executed on only one node of the cluster.

pcs resource delete php7
pcs resource create "php7" \
    systemd:rh-php73-php-fpm \
    meta target-role="started" \
    op start interval="0s" timeout="30s" \
    stop interval="0s" timeout="30s" \
    monitor interval="5s" timeout="30s" \
    clone

Then to perform the WEB UI upgrade, please follow the official documentation Only on the active central node.

On the passive central node, move the "install" directory to avoid getting the "upgrade" screen in the WUI in the event of a further exchange of roles.

mv /usr/share/centreon/www/install /var/lib/centreon/installs/install-update-YYYY-MM-DD
sudo -u apache /usr/share/centreon/bin/console cache:clear

Removing cron jobs​

The RPM upgrade puts cron job back in place. Remove them to avoid concurrent executions: 

rm /etc/cron.d/centreon
rm /etc/cron.d/centstorage
rm /etc/cron.d/centreon-auto-disco

Clean broker memory files​

WARNING perform these commands only the active central node. Before resuming the cluster resources management, to avoid broker issues, cleanup all the .memory., .unprocessed. or .queue. files:

systemctl stop cbd-sql
rm -rf /var/lib/centreon-broker/central-broker-master.memory*
rm -rf /var/lib/centreon-broker/central-broker-master.queue*
rm -rf /var/lib/centreon-broker/central-broker-master.unprocessed*
systemctl start cbd-sql

Then perform these commands on the passive central node:

rm -rf /var/lib/centreon-broker/central-broker-master.memory*
rm -rf /var/lib/centreon-broker/central-broker-master.queue*
rm -rf /var/lib/centreon-broker/central-broker-master.unprocessed*

Restart Centreon process​

Then to restart all the processes on the active central node:

systemctl restart cbd-sql cbd gorgoned centengine

And on the passive central node:

systemctl restart cbd

Upgrade of MariaDB Databases​

The MariaDB components can now be upgraded.

Refer to the official MariaDB documentation to know more about this process:

https://mariadb.com/kb/en/upgrading-between-major-mariadb-versions/

WARNING the following commands must be executed firstly on the active database node. Once the active database node is in 10.5, you can upgrade the passive database node.

Upgrading MariaDB​

You have to uninstall then reinstall MariaDB to upgrade between major versions (i.e. to switch from version 10.3 to version 10.5).

  1. Stop the mariadb service:

    mysqladmin -p shutdown

  2. Uninstall the current version (MariaDB-shared is possibly not installed, remove it from this command if it's the case):

    rpm --erase --nodeps --verbose MariaDB-server MariaDB-client MariaDB-shared MariaDB-compat MariaDB-common

    or if MariaDB-shared not installed:

    rpm --erase --nodeps --verbose MariaDB-server MariaDB-client MariaDB-compat MariaDB-common

  3. Install version 10.5:

    yum install MariaDB-server-10.5\* MariaDB-client-10.5\* MariaDB-shared-10.5\* MariaDB-compat-10.5\* MariaDB-common-10.5\*

  4. Move the configuration file:

    mv /etc/my.cnf.d/server.cnf.rpmsave /etc/my.cnf.d/server.cnf

  5. Start the mariadb service:

    mysqld_safe &

  6. Launch the MariaDB upgrade process:

    mysql_upgrade

    If your database is password-protected, enter:

    mysql_upgrade -u <database_admin_user> -p

    Example: if your database_admin_user is root, enter:

    mysql_upgrade -u root -p

    Refer to the official documentation for more information or if errors occur during this last step.

Restart MariaDB Replication​

The replication thread will be down after the upgrade. To restart it Run this command on the secondary node:

mysqladmin -p shutdown

It is important to make sure that MariaDB is completely shut down. You will run this command and check that it returns no output:

ps -ef | grep mariadb[d]

Once the service is stopped on the secondary node, you will run the synchronization script from the primary node:

/usr/share/centreon-ha/bin/mysql-sync-bigdb.sh

the output of this command must display only OK results:

/usr/share/centreon-ha/bin/mysql-check-status.sh

The expected output is:

Connection Status '@CENTRAL_MASTER_NAME@' [OK]
Connection Status '@CENTRAL_SLAVE_NAME@' [OK]
Slave Thread Status [OK]
Position Status [OK]

Resuming the cluster resources management​

Now that the update is finished, the resources can be managed again:

pcs resource manage centreon
pcs resource manage ms_mysql

It can happen that the replication thread is not running right after installation. Restarting the ms_mysql resource may fix it.

pcs resource restart ms_mysql

Check cluster's health​

You can monitor the cluster's resources in real time using the crm_mon command:

Stack: corosync
Current DC: @CENTRAL_SLAVE_NAME@ (version 1.1.20-5.el7_7.2-3c4c782f70) - partition with quorum
Last updated: Thu Feb 20 13:14:17 2020
Last change: Thu Feb 20 09:25:54 2020 by root via crm_attribute on @CENTRAL_MASTER_NAME@

2 nodes configured
14 resources configured

Online: [ @CENTRAL_MASTER_NAME@ @CENTRAL_SLAVE_NAME@ ]

Active resources:

 Master/Slave Set: ms_mysql-master [ms_mysql]
     Masters: [ @CENTRAL_MASTER_NAME@ ]
     Slaves: [ @CENTRAL_SLAVE_NAME@ ]
 Clone Set: cbd_rrd-clone [cbd_rrd]
     Started: [ @CENTRAL_MASTER_NAME@ @CENTRAL_SLAVE_NAME@ ]
 Resource Group: centreon
     vip        (ocf::heartbeat:IPaddr2):   Started @CENTRAL_MASTER_NAME@
     http   (systemd:httpd24-httpd):        Started @CENTRAL_MASTER_NAME@
     gorgone    (systemd:gorgoned):     Started @CENTRAL_MASTER_NAME@
     centreon_central_sync  (systemd:centreon-central-sync):        Started @CENTRAL_MASTER_NAME@
     centreontrapd  (systemd:centreontrapd):        Started @CENTRAL_MASTER_NAME@
     snmptrapd  (systemd:snmptrapd):    Started @CENTRAL_MASTER_NAME@
     cbd_central_broker (systemd:cbd-sql):  Started @CENTRAL_MASTER_NAME@
     centengine (systemd:centengine):   Started @CENTRAL_MASTER_NAME@
 Clone Set: php7-clone [php7]
     Started: [ @CENTRAL_MASTER_NAME@ @CENTRAL_SLAVE_NAME@ ]

Verifying the platform stability​

You should now check that eveything works fine:

  • Access to the web UI menus.
  • Poller configuration generation + reload and restart method.
  • Schedule immediate check (Central + Pollers) and acknowledge, downtime etc.
  • Move resources or reboot active server and check again that everything is fine.