Upgrades & Maintenance

Upgrading Morpheus

The following covers upgrading Morpheus Appliances to Morpheus v4.1.1.

Important

Morpheus v3.6.0 or higher is required to upgrade to 4.x including v4.1.1.

Upgrade Requirements

3.6.x to v4.1.1 Upgrade

  • Only appliances running Morpheus v3.6.0 or higher can upgrade to 4.x.

  • MySQL will be upgraded to 5.7.x on Appliances with MySQL running on the app node (Single Node or “all-in-one” Appliances). Backup your database before running the upgrade.

    Important

    BACKUP YOUR DATABASE before the upgrade. You can use the appliance backup job in Morpheus, but make sure you download it before you do the upgrade.

  • RabbitMQ will be upgraded to v3.7 on Appliances with RabbitMQ running on the app node (Single Node or “all-in-one” Appliances). On 3-Node configurations, the RabbitMQ queues and configuration will be dropped and the cluster will need to be configured and established again.

  • Elasticsearch has a minor upgrade from 5.4 to 5.6 on Appliances with Elasticsearch running on the app node (Single Node or “all-in-one” Appliances)

  • Stop all morpheus services, not just the morpheus-ui, before the upgrade. Although the upgrade process will also stop the services, take this step to ensure they are stopped.

  • Warnings about missing files during the removal phase are expected and can be ignored.

  • The repo download location has changed to https://downloads.morpheusdata.com from https://downloads.gomorpheus.com so if a customer has an ACL on their firewall or proxy they will need to update the ACL.

Refer to v4.1.1 Compatibility for externalized MySQL, Elasticsearch and/or RabbitMQ version requirements.

Single Node Appliance Upgrade

The following covers upgrading single node (All-In-One) Appliance configurations.

Important

Only appliances running Morpheus v3.6.0 or higher can upgrade to 4.x.

3.6.x to v4.1.1 Upgrade

  • When upgrading from 3.6.x to v4.1.1, the following services will be automatically upgraded on Single Node (all-in-one) Appliances:
    • MySQL will be upgraded to v5.7.
      • Backup your database before running the upgrade.
    • Elasticsearch will be upgraded to v5.6.
    • RabbitMQ will be upgraded to v3.7.

Important

BACKUP YOUR DATABASE before the upgrade. You can use the appliance backup job in Morpheus, but make sure you download it before you do the upgrade.

  • Stop all morpheus services, not just the morpheus-ui, before the upgrade. Although the upgrade process will also stop the services, take this step to ensure they are stopped.
  • Warnings about missing files during the removal phase are expected and can be ignored.

Important

The Morpheus package repo download location has changed to https://downloads.morpheusdata.com from https://downloads.gomorpheus.com. Update firewall and proxy ACLs when applicable.

Debian / Ubuntu

To upgrade Morpheus running on Ubuntu/Debian, download new deb package, stop the morpheus-ui, install the new deb package, then reconfigure.

sudo wget https://packageUrl.morpheus-appliance_x.x.x-x.amd64.deb
sudo morpheus-ctl stop
sudo dpkg -i morpheus-appliance_x.x.x-1.amd64.deb
sudo morpheus-ctl reconfigure

Note

In 4.x services will be stopped during package installation and started during the reconfigure process, including the Morpheus-ui service.

All services will automatically start during the reconfigure process. After the reconfigure has succeeded, tail the ui service to watch ui startup logs with morpheus-ctl tail morpheus-ui.

After the morpheus-ui service finishes loading, the upgrade is complete.

CentOS / RHEL

To upgrade Morpheus running on CentOS/RHEL, download and install the new rpm package, stop the morpheus-ui, reconfigure and then start the morpheus-ui:

sudo wget https://packageUrl.morpheus-appliance-x.x.x-x.x86_64.rpm
sudo morpheus-ctl stop
sudo rpm -Uhv morpheus-appliance-x.x.x-x.x86_64.rpm
sudo morpheus-ctl reconfigure

Note

In 4.x services will be stopped during package installation and started during the reconfigure process, including the Morpheus-ui service.

All services will automatically start during the reconfigure process. After the reconfigure has succeeded, tail the ui service to watch ui startup logs with morpheus-ctl tail morpheus-ui.

After the morpheus-ui service finishes loading, the upgrade is complete.


4.x to v4.1.1 Upgrade

Debian / Ubuntu

To upgrade Morpheus running on Ubuntu/Debian, download new deb package, stop the morpheus-ui, install the new deb package, then reconfigure.

sudo wget https://packageUrl.morpheus-appliance_x.x.x-x.amd64.deb
sudo morpheus-ctl stop
sudo dpkg -i morpheus-appliance_x.x.x-1.amd64.deb
sudo morpheus-ctl reconfigure

Note

In 4.x services will be stopped during package installation and started during the reconfigure process, including the Morpheus-ui service.

All services will automatically start during the reconfigure process. After the reconfigure has succeeded, tail the ui service to watch ui startup logs with morpheus-ctl tail morpheus-ui.

After the morpheus-ui service finishes loading, the upgrade is complete.

CentOS / RHEL

To upgrade Morpheus running on CentOS/RHEL, download and install the new rpm package, stop the morpheus-ui, reconfigure and then start the morpheus-ui:

sudo wget https://packageUrl.morpheus-appliance-x.x.x-x.x86_64.rpm
sudo morpheus-ctl stop
sudo rpm -Uhv morpheus-appliance-x.x.x-x.x86_64.rpm
sudo morpheus-ctl reconfigure

Note

In 4.x services will be stopped during package installation and started during the reconfigure process, including the Morpheus-ui service.

All services will automatically start during the reconfigure process. After the reconfigure has succeeded, tail the ui service to watch ui startup logs with morpheus-ctl tail morpheus-ui.

After the morpheus-ui service finishes loading, the upgrade is complete.



3-Node HA Upgrade

3-Node HA Appliance represent 3 App nodes with local RabbitMQ and Elasticsearch services clustered across the app nodes, and an external Galera/Percona MySQL cluster.

When upgrading a 3-Node appliance from 3.6.x to v4.1.1 the following services will be upgraded:

  • RabbitMQ upgrade to v3.7
  • Elasticsearch upgrade to v5.6

The upgrade process will not upgrade the external MySQL node(s). Refer to v4.1.1 Compatibility for externalized database version requirements.

Due to RabbitMQ going from 3.4 to 3.7, which has no rolling upgrade path, the RabbitMQ queues and configuration will be dropped, and the cluster will need to be configured and established again. This also ensures new queues are created using our new declaration settings, and removes any old queues not in use anymore.

Important

Due to the RabbitMQ upgrade from 3.4 to 3.7, the RabbitMQ configuration will be dropped and the cluster will need to be configured and established again.

Debian / Ubuntu

The following covers upgrading the Morpheus App nodes in 3 Node HA configurations to v4.1.1.

Important

The following is only for 3 Node HA Architecture configurations, where MySQL services are external to the App nodes.

4.x -> v4.1.1
  1. Starting with Node 3, on All App Nodes, stop all Morpheus services via morpheus-ctl stop. This will stop all system services. If any services timeout, run morpheus-ctl stop again.

    [root@app-server-3 ~]# morpheus-ctl stop
    
    [root@app-server-2 ~]# morpheus-ctl stop
    
    [root@app-server-1 ~]# morpheus-ctl stop
    
  2. Upgrade the deb package on Node 1, then run a Reconfigure on Node 1

    [root@app-server-1 ~]# sudo wget https://packageUrl.morpheus-appliance_x.x.x-x.amd64.deb
    [root@app-server-1 ~]# sudo dpkg -i morpheus-appliance_x.x.x-1.amd64.deb
    [root@app-server-1 ~]# sudo morpheus-ctl reconfigure
    

    Note

    All services will automatically start during the upgrade process. After the reconfigure has succeeded, tail the ui service to watch ui startup logs with morpheus-ctl tail morpheus-ui.

  3. Once Node 1 upgrade has completed and the u is available, upgrade the deb package on Node 2, then run a Reconfigure on Node 2.

    [root@app-server-2 ~]# sudo wget https://packageUrl.morpheus-appliance_x.x.x-x.amd64.deb
    [root@app-server-2 ~]# sudo dpkg -i morpheus-appliance_x.x.x-1.amd64.deb
    [root@app-server-2 ~]# sudo morpheus-ctl reconfigure
    
  4. Then upgrade the deb package on Node 3, then run a Reconfigure on Node 3

    [root@app-server-3 ~]# sudo wget https://packageUrl.morpheus-appliance_x.x.x-x.amd64.deb
    [root@app-server-3 ~]# sudo dpkg -i morpheus-appliance_x.x.x-1.amd64.deb
    [root@app-server-3 ~]# sudo morpheus-ctl reconfigure
    
  5. The upgrade is complete and the Morpheus-ui services should be running with clustered Elasticsearch and RabbitMQ services across the 3 nodes.

3.6.x -> v4.1.1
  • RabbitMQ will be upgraded from 3.5 to 3.7. On 3-Node configurations, the RabbitMQ queues and configuration will be dropped and the cluster will need to be configured and established again.
  • Elasticsearch will be upgraded from 5.4 to 5.6.
  • Stop all morpheus services, not just the morpheus-ui, before the upgrade. Although the upgrade process will also stop the services, take this step to ensure they are stopped.
  • Warnings about missing files during the removal phase are expected and can be ignored.

Important

The Morpheus package repo download location has changed to https://downloads.morpheusdata.com from https://downloads.gomorpheus.com. Update firewall and proxy ACLs when applicable.

  1. Starting with Node 3, on All App Nodes, stop all Morpheus services via morpheus-ctl stop. This will stop all system services. If any services timeout, run morpheus-ctl stop again.

    [root@app-server-3 ~]# morpheus-ctl stop
    
    [root@app-server-2 ~]# morpheus-ctl stop
    
    [root@app-server-1 ~]# morpheus-ctl stop
    
  2. Upgrade the deb package on Node 1, then run a Reconfigure on Node 1

    [root@app-server-1 ~]# sudo wget https://packageUrl.morpheus-appliance_x.x.x-x.amd64.deb
    [root@app-server-1 ~]# sudo dpkg -i morpheus-appliance_x.x.x-1.amd64.deb
    [root@app-server-1 ~]# sudo morpheus-ctl reconfigure
    
  3. Upgrade the deb package on Node 2, then run a Reconfigure on Node 2

    [root@app-server-2 ~]# sudo wget https://packageUrl.morpheus-appliance_x.x.x-x.amd64.deb
    [root@app-server-2 ~]# sudo dpkg -i morpheus-appliance_x.x.x-1.amd64.deb
    [root@app-server-2 ~]# sudo morpheus-ctl reconfigure
    
  4. Upgrade the deb package on Node 3, then run a Reconfigure on Node 3

    [root@app-server-3 ~]# sudo wget https://packageUrl.morpheus-appliance_x.x.x-x.amd64.deb
    [root@app-server-3 ~]# sudo dpkg -i morpheus-appliance_x.x.x-1.amd64.deb
    [root@app-server-3 ~]# sudo morpheus-ctl reconfigure
    
  5. After reconfigure has completed on Node 1, apply the required ha-mode and expires policies to the morpheus vhost:

    [root@app-server-1 ~] source /opt/morpheus/embedded/rabbitmq/.profile
    [root@app-server-1 ~] rabbitmqctl set_policy -p morpheus --apply-to queues --priority 2 statCommands "statCommands.*" '{"expires":1800000, "ha-mode":"all"}'
    [root@app-server-1 ~] rabbitmqctl set_policy -p morpheus --apply-to queues --priority 2 morpheusAgentActions "morpheusAgentActions.*" '{"expires":1800000, "ha-mode":"all"}'
    [root@app-server-1 ~] rabbitmqctl set_policy -p morpheus --apply-to queues --priority 2 monitorJobs "monitorJobs.*" '{"expires":1800000, "ha-mode":"all"}'
    [root@app-server-1 ~] rabbitmqctl set_policy -p morpheus --apply-to all --priority 1 ha ".*" '{"ha-mode":"all"}'
    

    Important

    Failure to set the proper policies will result in degraded RabbitMQ performance, Java Heap issues, and/or refused RabbitMQ connections resulting in degraded Morpheus UI performance, unconsumed messages or UI failure.

  6. After reconfigure has completed on Nodes 2 and 3, stop the morpheus-ui service that was automatically started during the reconfigure process.

    [root@app-server-2 ~]# morpheus-ctl stop morpheus-ui
    
    [root@app-server-1 ~]# morpheus-ctl stop morpheus-ui
    
  7. Copy the secrets and erlang cookie from Node 1 to Nodes 2 and 3

    Begin by copying secrets from the Node 1 other nodes.

    [root@app-server-3 ~]# cat /etc/morpheus/morpheus-secrets.json
    
     "rabbitmq": {
       "morpheus_password": "***REDACTED***",
       "queue_user_password": "***REDACTED***",
       "cookie": "***REDACTED***"
     },
    

    Then copy the erlang.cookie from the Node 1 to Nodes 2 and 3

    [root@app-server-1 ~]# cat /opt/morpheus/embedded/rabbitmq/.erlang.cookie
    
    # 754363AD864649RD63D28
    
  8. Once this is done run a reconfigure on Nodes 2 & 3.

    [root@app-server-2 ~] morpheus-ctl reconfigure
    

    Note

    If the reconfigure fails or hangs it is ok. If the reconfigure hangs then use Ctrl+C to quit the reconfigure run and force a failure. Another reconfigure will be run after clustering.

  9. Next on Node 2, ensure the ui is stopped, then stop and start RabbitMQ and join the Node to the Cluster. Do not stop and start RabbitMQ on Node 1.

    Important

    The commands below must be run at root

    [root@app-server-1 ~]# morpheus-ctl stop morpheus-ui
    [root@app-server-2 ~]# morpheus-ctl stop rabbitmq
    [root@app-server-2 ~]# morpheus-ctl start rabbitmq
    [root@app-server-2 ~]# source /opt/morpheus/embedded/rabbitmq/.profile
    [root@app-server-2 ~]# rabbitmqctl stop_app
    
    Stopping node 'rabbit@app-server-2' ...
    
    [root@app-server-2 ~]# rabbitmqctl join_cluster rabbit@app-server-1
    
    Clustering node 'rabbit@app-server-2' with 'rabbit@app-server-1' ...
    
    [root@app-server-2 ~]# rabbitmqctl start_app
    
    Starting node 'rabbit@app-server-2' ...
    
  10. Perform the same steps on Node 3 to join the Node to the Cluster. Again, do not stop and start RabbitMQ on Node 1.

    Important

    The commands below must be run at root

    [root@app-server-3 ~]# morpheus-ctl stop rabbitmq
    [root@app-server-3 ~]# morpheus-ctl start rabbitmq
    [root@app-server-3 ~]# source /opt/morpheus/embedded/rabbitmq/.profile
    [root@app-server-3 ~]# rabbitmqctl stop_app
    
    Stopping node 'rabbit@app-server-3' ...
    
    [root@app-server-3 ~]# rabbitmqctl join_cluster rabbit@app-server-1
    
    Clustering node 'rabbit@app-server-3' with 'rabbit@app-server-1' ...
    
    [root@app-server-3 ~]# rabbitmqctl start_app
    
    Starting node 'rabbit@app-server-3' ...
    
  11. Next run a final reconfigure on Nodes 2 & 3 and start the Morpheus ui.

    [root@app-server-2 ~] morpheus-ctl reconfigure
    [root@app-server-2 ~] morpheus-ctl start morpheus-ui
    
    [root@app-server-3 ~] morpheus-ctl reconfigure
    [root@app-server-3 ~] morpheus-ctl start morpheus-ui
    
  12. You will be able to verify that the UI services have restarted properly by inspecting the logfiles. A standard practice after running a restart is to tail the UI log file.

    root@app-server-2 ~]# morpheus-ctl tail morpheus-ui
    

CentOS / RHEL

The following covers upgrading the Morpheus App nodes in 3 Node HA configurations to v4.1.1.

Important

The following is only for 3 Node HA Architecture configurations, where MySQL services are external to the App nodes.

4.x -> v4.1.1
  1. Starting with Node 3, on All App Nodes, stop all Morpheus services via morpheus-ctl stop. This will stop all system services. If any services timeout, run morpheus-ctl stop again.

    [root@app-server-3 ~]# morpheus-ctl stop
    
    [root@app-server-2 ~]# morpheus-ctl stop
    
    [root@app-server-1 ~]# morpheus-ctl stop
    
  2. Upgrade the rpm package on Node 1, then run a Reconfigure on Node 1

    [root@app-server-1 ~]# sudo wget https://packageUrl.morpheus-appliance-x.x.x-x.x86_64.rpm
    [root@app-server-1 ~]# sudo rpm -Uhv morpheus-appliance-x.x.x-x.x86_64.rpm
    [root@app-server-1 ~]# sudo morpheus-ctl reconfigure
    

    Note

    All services will automatically start during the reconfigure process. After the reconfigure has succeeded, tail the ui service to watch ui startup logs with morpheus-ctl tail morpheus-ui.

  3. Once Node 1 upgrade has completed and the u is available, upgrade the rpm package on Node 2, then run a Reconfigure on Node 2.

    [root@app-server-2 ~]# sudo wget https://packageUrl.morpheus-appliance-x.x.x-x.x86_64.rpm
    [root@app-server-2 ~]# sudo rpm -Uhv morpheus-appliance-x.x.x-x.x86_64.rpm
    [root@app-server-2 ~]# sudo morpheus-ctl reconfigure
    
  4. Then upgrade the rpm package on Node 3, then run a Reconfigure on Node 3

    [root@app-server-3 ~]# sudo wget https://packageUrl.morpheus-appliance-x.x.x-x.x86_64.rpm
    [root@app-server-3 ~]# sudo rpm -Uhv morpheus-appliance-x.x.x-x.x86_64.rpm
    [root@app-server-3 ~]# sudo morpheus-ctl reconfigure
    
  5. The upgrade is complete and the Morpheus-ui services should be running with clustered Elasticsearch and RabbitMQ services across the 3 nodes.

3.6.x -> v4.1.1
  • RabbitMQ will be upgraded from 3.5 to 3.7. On 3-Node configurations, the RabbitMQ queues and configuration will be dropped and the cluster will need to be configured and established again.
  • Elasticsearch will be upgraded from 5.4 to 5.6.
  • Stop all morpheus services, not just the morpheus-ui, before the upgrade. Although the upgrade process will also stop the services, take this step to ensure they are stopped.
  • Warnings about missing files during the removal phase are expected and can be ignored.
  • The Morpheus package repo download location has changed to https://downloads.morpheusdata.com from https://downloads.gomorpheus.com. Update firewall and proxy ACLs when applicable.
  1. Starting with Node 3, on All App Nodes, stop all Morpheus services via morpheus-ctl stop. This will stop all system services. If any services timeout, run morpheus-ctl stop again.

    [root@app-server-3 ~]# morpheus-ctl stop
    
    [root@app-server-2 ~]# morpheus-ctl stop
    
    [root@app-server-1 ~]# morpheus-ctl stop
    
  2. Upgrade the rpm package on Node 1, then run a Reconfigure on Node 1

    [root@app-server-1 ~]# sudo wget https://packageUrl.morpheus-appliance-x.x.x-x.x86_64.rpm
    [root@app-server-1 ~]# sudo rpm -Uhv morpheus-appliance-x.x.x-x.x86_64.rpm
    [root@app-server-1 ~]# sudo morpheus-ctl reconfigure
    
  3. Upgrade the rpm package on Node 2, then run a Reconfigure on Node 2

    [root@app-server-2 ~]# sudo wget https://packageUrl.morpheus-appliance-x.x.x-x.x86_64.rpm
    [root@app-server-2 ~]# sudo rpm -Uhv morpheus-appliance-x.x.x-x.x86_64.rpm
    [root@app-server-2 ~]# sudo morpheus-ctl reconfigure
    
  4. Upgrade the rpm package on Node 3, then run a Reconfigure on Node 3

    [root@app-server-3 ~]# sudo wget https://packageUrl.morpheus-appliance-x.x.x-x.x86_64.rpm
    [root@app-server-3 ~]# sudo rpm -Uhv morpheus-appliance-x.x.x-x.x86_64.rpm
    [root@app-server-3 ~]# sudo morpheus-ctl reconfigure
    
  5. After reconfigure has completed on Node 1, apply the required ha-mode and expires policies to the morpheus vhost:

    [root@app-server-1 ~] source /opt/morpheus/embedded/rabbitmq/.profile
    [root@app-server-1 ~] rabbitmqctl set_policy -p morpheus --apply-to queues --priority 2 statCommands "statCommands.*" '{"expires":1800000, "ha-mode":"all"}'
    [root@app-server-1 ~] rabbitmqctl set_policy -p morpheus --apply-to queues --priority 2 morpheusAgentActions "morpheusAgentActions.*" '{"expires":1800000, "ha-mode":"all"}'
    [root@app-server-1 ~] rabbitmqctl set_policy -p morpheus --apply-to queues --priority 2 monitorJobs "monitorJobs.*" '{"expires":1800000, "ha-mode":"all"}'
    [root@app-server-1 ~] rabbitmqctl set_policy -p morpheus --apply-to all --priority 1 ha ".*" '{"ha-mode":"all"}'
    

    Important

    Failure to set the proper policies will result in degraded RabbitMQ performance, Java Heap issues, and/or refused RabbitMQ connections resulting in degraded Morpheus UI performance, unconsumed messages or UI failure.

  6. After reconfigure has completed on Nodes 2 and 3, stop the morpheus-ui service that was automatically started during the reconfigure process.

    [root@app-server-2 ~]# morpheus-ctl stop morpheus-ui
    
    [root@app-server-1 ~]# morpheus-ctl stop morpheus-ui
    
  7. Copy the secrets and erlang cookie from Node 1 to Nodes 2 and 3

    Begin by copying secrets from the Node 1 other nodes.

    [root@app-server-3 ~]# cat /etc/morpheus/morpheus-secrets.json
    
     "rabbitmq": {
       "morpheus_password": "***REDACTED***",
       "queue_user_password": "***REDACTED***",
       "cookie": "***REDACTED***"
     },
    

    Then copy the erlang.cookie from the Node 1 to Nodes 2 and 3

    [root@app-server-1 ~]# cat /opt/morpheus/embedded/rabbitmq/.erlang.cookie
    
    # 754363AD864649RD63D28
    
  8. Once this is done run a reconfigure on Nodes 2 & 3.

    [root@app-server-2 ~] morpheus-ctl reconfigure
    

    Note

    If the reconfigure fails or hangs it is ok. If the reconfigure hangs then use Ctrl+C to quit the reconfigure run and force a failure. Another reconfigure will be run after clustering.

  9. Next on Node 2, ensure the ui is stopped, then stop and start RabbitMQ and join the Node to the Cluster. Do not stop and start RabbitMQ on Node 1.

    Important

    The commands below must be run at root

    [root@app-server-1 ~]# morpheus-ctl stop morpheus-ui
    [root@app-server-2 ~]# morpheus-ctl stop rabbitmq
    [root@app-server-2 ~]# morpheus-ctl start rabbitmq
    [root@app-server-2 ~]# source /opt/morpheus/embedded/rabbitmq/.profile
    [root@app-server-2 ~]# rabbitmqctl stop_app
    
    Stopping node 'rabbit@app-server-2' ...
    
    [root@app-server-2 ~]# rabbitmqctl join_cluster rabbit@app-server-1
    
    Clustering node 'rabbit@app-server-2' with 'rabbit@app-server-1' ...
    
    [root@app-server-2 ~]# rabbitmqctl start_app
    
    Starting node 'rabbit@app-server-2' ...
    
  10. Perform the same steps on Node 3 to join the Node to the Cluster. Again, do not stop and start RabbitMQ on Node 1.

    Important

    The commands below must be run at root

    [root@app-server-3 ~]# morpheus-ctl stop rabbitmq
    [root@app-server-3 ~]# morpheus-ctl start rabbitmq
    [root@app-server-3 ~]# source /opt/morpheus/embedded/rabbitmq/.profile
    [root@app-server-3 ~]# rabbitmqctl stop_app
    
    Stopping node 'rabbit@app-server-3' ...
    
    [root@app-server-3 ~]# rabbitmqctl join_cluster rabbit@app-server-1
    
    Clustering node 'rabbit@app-server-3' with 'rabbit@app-server-1' ...
    
    [root@app-server-3 ~]# rabbitmqctl start_app
    
    Starting node 'rabbit@app-server-3' ...
    
  11. Next run a final reconfigure on Nodes 2 & 3 and start the Morpheus ui.

    [root@app-server-2 ~] morpheus-ctl reconfigure
    [root@app-server-2 ~] morpheus-ctl start morpheus-ui
    
    [root@app-server-3 ~] morpheus-ctl reconfigure
    [root@app-server-3 ~] morpheus-ctl start morpheus-ui
    
  12. You will be able to verify that the UI services have restarted properly by inspecting the logfiles. A standard practice after running a restart is to tail the UI log file.

    root@app-server-2 ~]# morpheus-ctl tail morpheus-ui
    

Full HA Upgrade

When upgrading other Appliance Configurations from 3.6.x or 4.x to v4.1.1 only services local to the Morpheus App node(s) will be upgraded. For fully distributed configurations (Full HA), where MySQL, RabbitMQ and Elasticsearch are external clusters, the upgrade process will not upgrade these services.

Refer to v4.1.1 Compatibility for externalized MySQL, Elasticsearch and/or RabbitMQ version requirements.

Debian / Ubuntu

The following covers upgrading the Morpheus App nodes in Full HA Architecture configurations to v4.1.1.

Important

The following is only for Full HA Architecture configurations, where MySQL, Elasticsearch and RabbitMQ services are external to the App nodes.

4.x -> v4.1.1 Upgrade
  1. Starting with Node 3, on All App Nodes, stop all Morpheus services via morpheus-ctl stop. This will stop all system services. If any services timeout, run morpheus-ctl stop again.

    [root@app-server-3 ~]# morpheus-ctl stop
    
    [root@app-server-2 ~]# morpheus-ctl stop
    
    [root@app-server-1 ~]# morpheus-ctl stop
    
  2. Upgrade the deb package on Node 1, then run a Reconfigure on Node 1

    [root@app-server-1 ~]# sudo wget https://packageUrl.morpheus-appliance_x.x.x-x.amd64.deb
    [root@app-server-1 ~]# sudo dpkg -i morpheus-appliance_x.x.x-1.amd64.deb
    [root@app-server-1 ~]# sudo morpheus-ctl reconfigure
    

    Note

    All local services will automatically start during the reconfigure process. After the reconfigure has succeeded, tail the ui service to watch ui startup logs with morpheus-ctl tail morpheus-ui.

  3. Once Node 1 upgrade has completed and the ui is available, upgrade the deb package on Node 2, then run a Reconfigure on Node 2.

    [root@app-server-2 ~]# sudo wget https://packageUrl.morpheus-appliance_x.x.x-x.amd64.deb
    [root@app-server-2 ~]# sudo dpkg -i morpheus-appliance_x.x.x-1.amd64.deb
    [root@app-server-2 ~]# sudo morpheus-ctl reconfigure
    
  4. Then upgrade the deb package on Node 3, then run a Reconfigure on Node 3

    [root@app-server-3 ~]# sudo wget https://packageUrl.morpheus-appliance_x.x.x-x.amd64.deb
    [root@app-server-3 ~]# sudo dpkg -i morpheus-appliance_x.x.x-1.amd64.deb
    [root@app-server-3 ~]# sudo morpheus-ctl reconfigure
    
  5. After all morpheus-ui services have finished loading, the upgrade is complete.



3.6.x -> v4.1.1 Upgrade
  1. Starting with Node 3, on All App Nodes, stop all Morpheus services via morpheus-ctl stop. This will stop all system services. If any services timeout, run morpheus-ctl stop again.

    [root@app-server-3 ~]# morpheus-ctl stop
    
    [root@app-server-2 ~]# morpheus-ctl stop
    
    [root@app-server-1 ~]# morpheus-ctl stop
    
  2. Upgrade the deb package on Node 1, then run a Reconfigure on Node 1

    [root@app-server-1 ~]# sudo wget https://packageUrl.morpheus-appliance_x.x.x-x.amd64.deb
    [root@app-server-1 ~]# sudo dpkg -i morpheus-appliance_x.x.x-1.amd64.deb
    [root@app-server-1 ~]# sudo morpheus-ctl reconfigure
    

    Note

    All services will automatically start during the reconfigure process. After the reconfigure has succeeded, tail the ui service to watch ui startup logs with morpheus-ctl tail morpheus-ui.

  3. Once Node 1 upgrade has completed and the ui is available, upgrade the deb package on Node 2, then run a Reconfigure on Node 2.

    [root@app-server-2 ~]# sudo wget https://packageUrl.morpheus-appliance_x.x.x-x.amd64.deb
    [root@app-server-2 ~]# sudo dpkg -i morpheus-appliance_x.x.x-1.amd64.deb
    [root@app-server-2 ~]# sudo morpheus-ctl reconfigure
    
  4. Then upgrade the deb package on Node 3, then run a Reconfigure on Node 3

    [root@app-server-3 ~]# sudo wget https://packageUrl.morpheus-appliance_x.x.x-x.amd64.deb
    [root@app-server-3 ~]# sudo dpkg -i morpheus-appliance_x.x.x-1.amd64.deb
    [root@app-server-3 ~]# sudo morpheus-ctl reconfigure
    
  5. After all morpheus-ui services have finished loading, the upgrade is complete.



CentOS / RHEL

The following covers upgrading the Morpheus App nodes in Full HA Architecture configurations to v4.1.1.

Important

The following is only for Full HA Architecture configurations, where MySQL, Elasticsearch and RabbitMQ services are external to the App nodes.

4.x -> v4.1.1 Upgrade
  1. Starting with Node 3, on All App Nodes, stop all Morpheus services via morpheus-ctl stop. This will stop all system services. If any services timeout, run morpheus-ctl stop again.

    [root@app-server-3 ~]# morpheus-ctl stop
    
    [root@app-server-2 ~]# morpheus-ctl stop
    
    [root@app-server-1 ~]# morpheus-ctl stop
    
  2. Upgrade the rpm package on Node 1, then run a Reconfigure on Node 1

    [root@app-server-1 ~]# sudo wget https://packageUrl.morpheus-appliance-x.x.x-x.x86_64.rpm
    [root@app-server-1 ~]# sudo rpm -Uhv morpheus-appliance-x.x.x-x.x86_64.rpm
    [root@app-server-1 ~]# sudo morpheus-ctl reconfigure
    

    Note

    All local services will automatically start during the reconfigure process. After the reconfigure has succeeded, tail the ui service to watch ui startup logs with morpheus-ctl tail morpheus-ui.

  3. Once Node 1 upgrade has completed and the ui is available, upgrade the rpm package on Node 2, then run a Reconfigure on Node 2.

    [root@app-server-2 ~]# sudo wget https://packageUrl.morpheus-appliance-x.x.x-x.x86_64.rpm
    [root@app-server-2 ~]# sudo rpm -Uhv morpheus-appliance-x.x.x-x.x86_64.rpm
    [root@app-server-2 ~]# sudo morpheus-ctl reconfigure
    
  4. Then upgrade the rpm package on Node 3, then run a Reconfigure on Node 3

    [root@app-server-3 ~]# sudo wget https://packageUrl.morpheus-appliance-x.x.x-x.x86_64.rpm
    [root@app-server-3 ~]# sudo rpm -Uhv morpheus-appliance-x.x.x-x.x86_64.rpm
    [root@app-server-3 ~]# sudo morpheus-ctl reconfigure
    
  5. After all morpheus-ui services have finished loading, the upgrade is complete.



3.6.x -> v4.1.1
  1. Starting with Node 3, on All App Nodes, stop all Morpheus services via morpheus-ctl stop. This will stop all system services. If any services timeout, run morpheus-ctl stop again.

    [root@app-server-3 ~]# morpheus-ctl stop
    
    [root@app-server-2 ~]# morpheus-ctl stop
    
    [root@app-server-1 ~]# morpheus-ctl stop
    
  2. Upgrade the rpm package on Node 1, then run a Reconfigure on Node 1

    [root@app-server-1 ~]# sudo wget https://packageUrl.morpheus-appliance-x.x.x-x.x86_64.rpm
    [root@app-server-1 ~]# sudo rpm -Uhv morpheus-appliance-x.x.x-x.x86_64.rpm
    [root@app-server-1 ~]# sudo morpheus-ctl reconfigure
    

    Note

    All local services will automatically start during the reconfigure process. After the reconfigure has succeeded, tail the ui service to watch ui startup logs with morpheus-ctl tail morpheus-ui.

  3. Once Node 1 upgrade has completed and the ui is available, upgrade the rpm package on Node 2, then run a Reconfigure on Node 2.

    [root@app-server-2 ~]# sudo wget https://packageUrl.morpheus-appliance-x.x.x-x.x86_64.rpm
    [root@app-server-2 ~]# sudo rpm -Uhv morpheus-appliance-x.x.x-x.x86_64.rpm
    [root@app-server-2 ~]# sudo morpheus-ctl reconfigure
    
  4. Then upgrade the rpm package on Node 3, then run a Reconfigure on Node 3

    [root@app-server-3 ~]# sudo wget https://packageUrl.morpheus-appliance-x.x.x-x.x86_64.rpm
    [root@app-server-3 ~]# sudo rpm -Uhv morpheus-appliance-x.x.x-x.x86_64.rpm
    [root@app-server-3 ~]# sudo morpheus-ctl reconfigure
    
  5. After all morpheus-ui services have finished loading, the upgrade is complete.



morpheus-ctl tips

morpheus-ctl is useful beyond reconfigures and starting the ui, and many commands can be run across all services, or scoped to a singe service.

Some common commands include:

morpheus-ctl status
This list all the installed services and their current Status
morpheus-ctl start (service)

This starts all services if no service is specified, or starts the specified service. For example,

  • morpheus-ctl start/stop/restart/kill on an all-in-one appliance will start, stop, restart or kill mysql, elasticsearch, rabbitmq, check-server, redis, guacd and the morpheus-ui, one by one.
  • morpheus-ctl start/stop/restart/kill morpheus-ui will only start, stop, restart or kill the morpheus-ui service, leaving the other service in their current state. Same goes for morpheus-ctl start/stop/restart/kill mysql, morpheus-ctl start/stop/restart/kill elasticsearch etc.

morpheus-ctl commands:

General Commands:

  cleanse
    Delete *all* morpheus data, and start from scratch.
  help
    Print this help message.
  reconfigure
    Reconfigure the application.
  show-config
    Show the configuration that would be generated by reconfigure.
  uninstall
    Kill all processes and uninstall the process supervisor (data will be preserved).

 Service Management Commands:

   graceful-kill
     Attempt a graceful stop, then SIGKILL the entire process group.
   hup
     Send the services a HUP.
   int
     Send the services an INT.
   kill
     Send the services a KILL.
   once
     Start the services if they are down. Do not restart them if they stop.
   restart
     Stop the services if they are running, then start them again.
   service-list
     List all the services (enabled services appear with a *.)
   start
     Start services if they are down, and restart them if they stop.
   status
     Show the status of all the services.
   stop
     Stop the services, and do not restart them.
   tail
     Watch the service logs of all enabled services.
   term
     Send the services a TERM.

 Elasticsearch Commands:

   elastic-util
     Backup/Restore ElasticSearch data

 Firewall Commands:

   firewall-enable-blocking
     Enables firewall blocking mode.

Morpheus UI war files

Pre-release or patched versions of the Morpheus UI are sometimes provided. To deploy the ui war on a Morpheus Appliance:

  1. Download the war file to the target appliance

    wget https://url/war_file
    

    Note

    If the war file is provided via a droplr link, ensure a + is added to end of droplr url or the file will not download

  2. Backup current war file

    sudo mv /opt/morpheus/lib/morpheus/morpheus-ui.war /opt/morpheus/lib/morpheus/morpheus-ui.bak.`date +"%m-%d-%Y"`
    
  3. Move and rename new war file

    sudo mv <file> /opt/morpheus/lib/morpheus/morpheus-ui.war
    
  4. Ensure war is owned by morpheus-app

    sudo chown morpheus-app.morpheus-app /opt/morpheus/lib/morpheus/morpheus-ui.war
    
  5. Restart the Morpheus UI

    sudo morpheus-ctl restart morpheus-ui
    

The new ui war will load on startup!

Morpheus DB Migration

If your new installation is part of a migration or you need to move the data from your original Morpheus database, this is easily accomplished by using a stateful dump.

To begin this, stop the Morpheus UI on your original Morpheus server:

[root@app-server-old ~] morpheus-ctl stop morpheus-ui

Once this is done you can safely export. To access the MySQL shell we will need the password for the Morpheus DB user. We can find this in the morpheus-secrets file:

[root@app-server-old ~] cat /etc/morpheus/morpheus-secrets.json | grep morpheus_password
"morpheus_password": "451e122cr5d122asw3de5e1b", <---------------this one
"morpheus_password": "9b5vdj4de5awf87d",

Take note of the first morpheus_password as it will be used to invoke a dump. Morpheus provides embedded binaries for this task. Invoke it via the embedded path and specify the host. In this example we are using the morpheus database on the MySQL listening on localhost. Enter the password copied from the previous step when prompted:

[root@app-server-old ~] /opt/morpheus/embedded/mysql/bin/mysqldump -u morpheus -h 127.0.0.1 morpheus -p > /tmp/morpheus_backup.sql
Enter password:

This file needs to be pushed to the new Morpheus Installation’s backend. Depending on the GRANTS in the new MySQL backend, this will likely require moving this file to one of the new Morpheus frontend servers.

Once the file is in place it can be imported into the backend. Begin by ensuring the Morpheus UI service is stopped on all of the application servers:

[root@app-server-new ~] morpheus-ctl stop morpheus-ui

Then you can import the MySQL dump into the target database using the embedded MySQL binaries, specifying the database host, and entering the password for the morpheus user when prompted:

[root@app-server-new ~] /opt/morpheus/embedded/mysql/bin/mysql -u morpheus -h 10.1.2.2 morpheus -p < /tmp/morpheus_backup.sql
Enter password:

The data form the old appliance is now replicated on the new appliance. Simply start the UI to complete the process:

[root@app-server-new ~] morpheus-ctl start morpheus-ui

Scaling Morpheus Nodes

Morpheus App nodes can be scaled to accommodate additional load. Appliance nodes can be scaled vertically in centralized architectures, and both vertically and horizontally in distributed architectures.

Vertical Scaling

In all Appliance Architectures, Application nodes can be vertically scaled at any time, however a reconfigure must be performed for additional resources to be utilized bu Morpheus on a node, which will result in the morpheus-ui restarting on the reconfiguring node.

Morpheus configures memory/ram utilization for services during the reconfigure process. If additional memory/ram is added to a Host or VM running the Morpheus App, the additional memory/ram will not be utilized by the Morpheus Application until a morpheus-ctl reconfigure command is ran and the additional memory/ram is recognized.

When the morpheus-ctl reconfigure command detects changes on available memory/ram, it will trigger a morpheus-ui service restart.

Important

When the morpheus-ctl reconfigure command detects changes on available memory/ram, it will restart the morpheus-ui service.

The impact on Availability depends on the Morpheus Appliance Architecture.

  • Centralized Appliances
    Morpheus will be unavailable while the morpheus-ui restarts.
  • Distributed Appliances
    Zero-down time can be achieved by Reconfiguring one App Node at a time, with proper Front-End Load Balancer configuration.

Horizontal Scaling

Additional Morpheus App Nodes can be added at any time to Fully Distributed Architectures.

  • Configure Shared Storage paths for the new App Node(s)
  • Install, but do not run the morpheus-ctl reconfigure command on the new App Node(s), using the same Morpheus version as the existing Appliance nodes.
  • Copy the morphues.rb from an existing App Node to the new App Node(s)
  • Ensure permissions and network configuration for the new App Node(s) to access all MySQL and Elasticsearch nodes, and the RabbitMQ VIP.
  • Ensure permissions and network configuration for all required UI services and Integrations, such as network access to ESXi hosts over 443 for Hypervisor console and/or image transfers.
  • Add associated SSL files and configuration, of not on shared storage.
  • Reconfigure the new App Node(s) via morpheus-ctl reconfigure
  • Verify UI startup succeeded
  • Add New App Node(s) to Front End Morpheus UI Load Balancer pool.

During morpheus-ctl reconfigure, the new App Node(s) will validate and be configured to use the existing tiers for the UI service. Upon successful reconfigure, the Morpheus service will be available on the App Node(s) with consistent data and capabilities.

Note

No services, including morphues-ui, are required to be shut down on existing nodes when adding new App Nodes