3-Node HA CentOS / RHEL Upgrade

The following covers upgrading the Morpheus App nodes in 3 Node HA configurations to v4.2.2.

Important

The following is only for “3 Node HA” Architecture configurations.

4.1.2+ -> v4.2.2 (rolling)

Note

All system services will automatically be stopped during the package install, and started during the reconfigure process. After the reconfigure has succeeded, tail the ui service to watch ui startup logs with morpheus-ctl tail morpheus-ui.

Important

Any externalized/non-system install services will not be stopped/started/upgraded/touched during package installs/upgrades or during reconfigures.

  1. Upgrade the RPM package on Node 1, then run a Reconfigure on Node 1

    [root@app-server-1 ~]# sudo wget https://packageUrl.morpheus-appliance-x.x.x-x.x86_64.rpm
    [root@app-server-1 ~]# sudo rpm -Uhv morpheus-appliance-x.x.x-x.x86_64.rpm
    [root@app-server-1 ~]# sudo morpheus-ctl reconfigure
    
  2. Once Node 1 upgrade has completed and the u is available, upgrade the RPM package on Node 2, then run a Reconfigure on Node 2.

    [root@app-server-2 ~]# sudo wget https://packageUrl.morpheus-appliance-x.x.x-x.x86_64.rpm
    [root@app-server-2 ~]# sudo rpm -Uhv morpheus-appliance-x.x.x-x.x86_64.rpm
    [root@app-server-2 ~]# sudo morpheus-ctl reconfigure
    
  3. Then upgrade the RPM package and run a Reconfigure on Node 3.

    [root@app-server-3 ~]# sudo wget https://packageUrl.morpheus-appliance-x.x.x-x.x86_64.rpm
    [root@app-server-3 ~]# sudo rpm -Uhv morpheus-appliance-x.x.x-x.x86_64.rpm
    [root@app-server-3 ~]# sudo morpheus-ctl reconfigure
    
  4. The upgrade is complete and the Morpheus-ui services should be running with clustered Elasticsearch and RabbitMQ services across the 3 nodes.

4.0.0, 4.1.0, 4.1.1 -> v4.2.2

  • Elasticsearch will be upgraded from 5.6 to 7.2.
  1. Starting with Node 3, on All App Nodes, stop all Morpheus services via morpheus-ctl stop. This will stop all system services. If any services timeout, run morpheus-ctl stop again.

    [root@app-server-3 ~]# morpheus-ctl stop
    
    [root@app-server-2 ~]# morpheus-ctl stop
    
    [root@app-server-1 ~]# morpheus-ctl stop
    
  2. Upgrade the RPM package on Node 1, then run a Reconfigure on Node 1

    [root@app-server-1 ~]# sudo wget https://packageUrl.morpheus-appliance-x.x.x-x.x86_64.rpm
    [root@app-server-1 ~]# sudo rpm -Uhv morpheus-appliance-x.x.x-x.x86_64.rpm
    [root@app-server-1 ~]# sudo morpheus-ctl reconfigure
    

    Note

    All services will automatically start during the reconfigure process. After the reconfigure has succeeded, tail the ui service to watch ui startup logs with morpheus-ctl tail morpheus-ui.

  3. Once Node 1 upgrade has completed and the u is available, upgrade the RPM package on Node 2, then run a Reconfigure on Node 2.

    [root@app-server-2 ~]# sudo wget https://packageUrl.morpheus-appliance-x.x.x-x.x86_64.rpm
    [root@app-server-2 ~]# sudo rpm -Uhv morpheus-appliance-x.x.x-x.x86_64.rpm
    [root@app-server-2 ~]# sudo morpheus-ctl reconfigure
    
  4. Then upgrade the RPM package on Node 3, then run a Reconfigure on Node 3

    [root@app-server-3 ~]# sudo wget https://packageUrl.morpheus-appliance-x.x.x-x.x86_64.rpm
    [root@app-server-3 ~]# sudo rpm -Uhv morpheus-appliance-x.x.x-x.x86_64.rpm
    [root@app-server-3 ~]# sudo morpheus-ctl reconfigure
    
  5. The upgrade is complete and the Morpheus-ui services should be running with clustered Elasticsearch and RabbitMQ services across the 3 nodes.

3.6.x -> v4.2.2

  • RabbitMQ will be upgraded from 3.5 to 3.7. On 3-Node configurations, the RabbitMQ queues and configuration will be dropped and the cluster will need to be configured and established again.
  • Elasticsearch will be upgraded from 5.6 to 7.2. Refer to Elasticsearch Upgrade Documentation for upgrading external ES Clusters.
  • Stop all morpheus services, not just the morpheus-ui, before the upgrade. Although the upgrade process will also stop the services, take this step to ensure they are stopped.
  • Warnings about missing files during the removal phase are expected and can be ignored.
  • The Morpheus package repo download location has changed to https://downloads.morpheusdata.com from https://downloads.gomorpheus.com. Update firewall and proxy ACLs when applicable.
  1. Starting with Node 3, on All App Nodes, stop all Morpheus services via morpheus-ctl stop. This will stop all system services. If any services timeout, run morpheus-ctl stop again.

    [root@app-server-3 ~]# morpheus-ctl stop
    
    [root@app-server-2 ~]# morpheus-ctl stop
    
    [root@app-server-1 ~]# morpheus-ctl stop
    
  2. Upgrade the RPM package on Node 1, then run a Reconfigure on Node 1

    [root@app-server-1 ~]# sudo wget https://packageUrl.morpheus-appliance-x.x.x-x.x86_64.rpm
    [root@app-server-1 ~]# sudo rpm -Uhv morpheus-appliance-x.x.x-x.x86_64.rpm
    [root@app-server-1 ~]# sudo morpheus-ctl reconfigure
    
  3. Upgrade the RPM package on Node 2, then run a Reconfigure on Node 2

    [root@app-server-2 ~]# sudo wget https://packageUrl.morpheus-appliance-x.x.x-x.x86_64.rpm
    [root@app-server-2 ~]# sudo rpm -Uhv morpheus-appliance-x.x.x-x.x86_64.rpm
    [root@app-server-2 ~]# sudo morpheus-ctl reconfigure
    
  4. Upgrade the RPM package on Node 3, then run a Reconfigure on Node 3

    [root@app-server-3 ~]# sudo wget https://packageUrl.morpheus-appliance-x.x.x-x.x86_64.rpm
    [root@app-server-3 ~]# sudo RPM -Uhv morpheus-appliance-x.x.x-x.x86_64.rpm
    [root@app-server-3 ~]# sudo morpheus-ctl reconfigure
    
  5. After reconfigure has completed on Node 1, apply the required ha-mode and expires policies to the morpheus vhost:

    [root@app-server-1 ~] source /opt/morpheus/embedded/rabbitmq/.profile
    [root@app-server-1 ~] rabbitmqctl set_policy -p morpheus --apply-to queues --priority 2 statCommands "statCommands.*" '{"expires":1800000, "ha-mode":"all"}'
    [root@app-server-1 ~] rabbitmqctl set_policy -p morpheus --apply-to queues --priority 2 morpheusAgentActions "morpheusAgentActions.*" '{"expires":1800000, "ha-mode":"all"}'
    [root@app-server-1 ~] rabbitmqctl set_policy -p morpheus --apply-to queues --priority 2 monitorJobs "monitorJobs.*" '{"expires":1800000, "ha-mode":"all"}'
    [root@app-server-1 ~] rabbitmqctl set_policy -p morpheus --apply-to all --priority 1 ha ".*" '{"ha-mode":"all"}'
    

    Important

    Failure to set the proper policies will result in degraded RabbitMQ performance, Java Heap issues, and/or refused RabbitMQ connections resulting in degraded Morpheus UI performance, unconsumed messages or UI failure.

  6. After reconfigure has completed on Nodes 2 and 3, stop the morpheus-ui service that was automatically started during the reconfigure process.

    [root@app-server-2 ~]# morpheus-ctl stop morpheus-ui
    
    [root@app-server-1 ~]# morpheus-ctl stop morpheus-ui
    
  7. Copy the secrets and erlang cookie from Node 1 to Nodes 2 and 3

    Begin by copying secrets from the Node 1 other nodes.

    [root@app-server-3 ~]# cat /etc/morpheus/morpheus-secrets.json
    
     "rabbitmq": {
       "morpheus_password": "***REDACTED***",
       "queue_user_password": "***REDACTED***",
       "cookie": "***REDACTED***"
     },
    

    Then copy the erlang.cookie from the Node 1 to Nodes 2 and 3

    [root@app-server-1 ~]# cat /opt/morpheus/embedded/rabbitmq/.erlang.cookie
    
    # 754363AD864649RD63D28
    
  8. Once this is done run a reconfigure on Nodes 2 & 3.

    [root@app-server-2 ~] morpheus-ctl reconfigure
    

    Note

    If the reconfigure fails or hangs it is ok. If the reconfigure hangs then use Ctrl+C to quit the reconfigure run and force a failure. Another reconfigure will be run after clustering.

  9. Next on Node 2, ensure the ui is stopped, then stop and start RabbitMQ and join the Node to the Cluster. Do not stop and start RabbitMQ on Node 1.

    Important

    The commands below must be run at root

    [root@app-server-1 ~]# morpheus-ctl stop morpheus-ui
    [root@app-server-2 ~]# morpheus-ctl stop rabbitmq
    [root@app-server-2 ~]# morpheus-ctl start rabbitmq
    [root@app-server-2 ~]# source /opt/morpheus/embedded/rabbitmq/.profile
    [root@app-server-2 ~]# rabbitmqctl stop_app
    
    Stopping node 'rabbit@app-server-2' ...
    
    [root@app-server-2 ~]# rabbitmqctl join_cluster rabbit@app-server-1
    
    Clustering node 'rabbit@app-server-2' with 'rabbit@app-server-1' ...
    
    [root@app-server-2 ~]# rabbitmqctl start_app
    
    Starting node 'rabbit@app-server-2' ...
    
  10. Perform the same steps on Node 3 to join the Node to the Cluster. Again, do not stop and start RabbitMQ on Node 1.

    Important

    The commands below must be run at root

    [root@app-server-3 ~]# morpheus-ctl stop rabbitmq
    [root@app-server-3 ~]# morpheus-ctl start rabbitmq
    [root@app-server-3 ~]# source /opt/morpheus/embedded/rabbitmq/.profile
    [root@app-server-3 ~]# rabbitmqctl stop_app
    
    Stopping node 'rabbit@app-server-3' ...
    
    [root@app-server-3 ~]# rabbitmqctl join_cluster rabbit@app-server-1
    
    Clustering node 'rabbit@app-server-3' with 'rabbit@app-server-1' ...
    
    [root@app-server-3 ~]# rabbitmqctl start_app
    
    Starting node 'rabbit@app-server-3' ...
    
  11. Next run a final reconfigure on Nodes 2 & 3 and start the Morpheus ui.

    [root@app-server-2 ~] morpheus-ctl reconfigure
    [root@app-server-2 ~] morpheus-ctl start morpheus-ui
    
    [root@app-server-3 ~] morpheus-ctl reconfigure
    [root@app-server-3 ~] morpheus-ctl start morpheus-ui
    
  12. You will be able to verify that the UI services have restarted properly by inspecting the logfiles. A standard practice after running a restart is to tail the UI log file.

    root@app-server-2 ~]# morpheus-ctl tail morpheus-ui