Upgrades & Maintenance

Upgrading

4.x Upgrade Requirements

  • Only appliances running Morpheus v3.6.0 or higher can upgrade to 4.x.

  • MySQL will be upgraded to 5.7.x on Appliances with MySQL running on the app node (Single Node or “all-in-one” Appliances). Backup your database before running the upgrade.

    Important

    BACKUP YOUR DATABASE before the upgrade. You can use the appliance backup job in Morpheus, but make sure you download it before you do the upgrade.

  • RabbitMQ will be upgrade from 3.4.x to 3.7.x. On 3-Node configurations, the RabbitMQ queues and configuration will be dropped and the cluster will need to be configured and established again.

  • Stop all morpheus services, not just the morpheus-ui, before the upgrade. Although the upgrade process will also stop the services, take this step to ensure they are stopped.

  • Warnings about missing files during the removal phase are expected and can be ignored.

  1. The repo download location has changed to https://downloads.morpheusdata.com from https://downloads.gomorpheus.com so if a customer has an ACL on their firewall or proxy they will need to update the ACL.

When externalizing MySQL, Elasticsearch and/or RabbitMQ services, the following versions are compatible with Morpheus 4.1.0:

Service Compatible Branch 4.1.0 Installed Version
MySQL 5.7 5.7.27
Elasticsearch: 5.6 (5.6.16 installed) 5.6 5.6.16
RabbitMQ: 3.7 (3.7.16 installed) 3.7 3.7.16

Single Node Appliance Upgrade

When upgrading from 3.6.x to 4.0.0 or 4.1.0, the following services will be upgraded on Single Node (all-in-one) Appliances:

  • MySQL upgrade to v5.7
  • RabbitMQ upgrade to v3.7
  • Elasticsearch upgrade to v5.6

Debian / Ubuntu

To upgrade Morpheus running on Ubuntu/Debian, download and install the new deb package, stop the morpheus-ui, reconfigure and start the morpheus-ui:

sudo wget https://packageUrl.morpheus-appliance_x.x.x-x.amd64.deb
sudo morpheus-ctl stop
sudo dpkg -i morpheus-appliance_x.x.x-1.amd64.deb
sudo morpheus-ctl reconfigure

Note

In 4.x the morpheus-ui service will automatically start after reconfigure succeeds.

CentOS / RHEL

To upgrade Morpheus running on CentOS/RHEL, download and install the new rpm package, stop the morpheus-ui, reconfigure and then start the morpheus-ui:

sudo wget https://packageUrl.morpheus-appliance-x.x.x-x.x86_64.rpm
sudo morpheus-ctl stop
sudo rpm -Uhv morpheus-appliance-x.x.x-x.x86_64.rpm
sudo morpheus-ctl reconfigure

Note

In 4.x the morpheus-ui service will automatically start after reconfigure succeeds.

3-Node Appliance Upgrade

When upgrading a 3-Node appliance from 3.6.x to 4.0.0 or 4.1.0, the following services will be upgraded:

  • RabbitMQ upgrade to v3.7
  • Elasticsearch upgrade to v5.6

The upgrade process will not upgrade the external MySQL node(s). MySQL v5.7 is required for external databases.

Due to RabbitMQ going from 3.4.x to 3.7.x, which has no direct upgrade path, the RabbitMQ queues and configuration will be dropped, and the cluster will need to be configured and established again. This also ensures new queues are created using our new declaration settings, and removes any old queues not in use anymore.

Important

Due to the RabbitMQ upgrade from 3.4.x to 3.7.x, the RabbitMQ queues and configuration will be dropped and the cluster will need to be configured and established again.

  1. Stop all Morpheus services via morpheus-ctl stop on all Nodes
  2. Upgrade Node 1, then run a reconfigure on Node 1
  3. Upgrade Node 2, then run a reconfigure on Node 2
  4. Upgrade Node 3, then run a reconfigure on Node 3
  5. Establish the RabbitMQ cluster again using the steps from the 3 Node install guide.
  6. Start all services

Other Appliance Configurations Upgrades

When upgrading other Appliance Configurations from 3.6.x to 4.0.0 or 4.1.0, only services local to the Morpheus App node(s) will be upgraded. For fully distributed configurations, where MySQL, RabbitMQ and Elasticsearch are external, the upgrade process will not upgrade the external serviced.

When externalizing MySQL, Elasticsearch and/or RabbitMQ services, the following versions are compatible with Morpheus 4.1.0:

Service Compatible Branch 4.1.0 Installed Version
MySQL 5.7 5.7.27
Elasticsearch: 5.6 (5.6.16 installed) 5.6 5.6.16
RabbitMQ: 3.7 (3.7.16 installed) 3.7 3.7.16

Fix if Install Hangs

Some very old all-in-one appliances may hang during the mysql upgrade process during the 4.0.0 deb or rpm package upgrade.

To resolve, run the following in a separate session while the process is hanging:

  1. Create a file vi mysqlfix

  2. paste the following:

    export PATH=/opt/morpheus/sbin:/opt/morpheus/sbin:/opt/morpheus/embedded/sbin:/opt/morpheus/embedded/bin:$PATH
    MYSQL_ROOT=$(grep root /etc/morpheus/morpheus-secrets.json | awk '{print substr($2,2,length($2)-3)}')
    if [[ -z $MYSQL_ROOT ]]; then
      echo "Failed to lookup the MySQL root password, please enter it when prompted."
    else
      /opt/morpheus/embedded/bin/mysqld_safe --defaults-file=/opt/morpheus/embedded/mysql/my.cnf \
        --log-error=/var/log/morpheus/mysql/mysql_error.log --pid-file=/var/run/morpheus/mysqld/mysqld.pid &
      until [[ -S /var/run/morpheus/mysqld/mysqld.sock ]]; do
        sleep 1
      done
      /opt/morpheus/embedded/bin/mysql_upgrade --defaults-extra-file=/opt/morpheus/embedded/mysql/my.cnf -u root -p${MYSQL_ROOT}
      /opt/morpheus/embedded/bin/mysqladmin --defaults-extra-file=/opt/morpheus/embedded/mysql/my.cnf -u root -p${MYSQL_ROOT} shutdown
      touch /var/opt/morpheus/mysql/.mysql57_table_upgrade_done
    fi
    
  3. Run ./mysqlfix

The upgrade will then proceed.

Morpheus UI war files

Pre-release or patched versions of the Morpheus UI are sometimes provided. To deploy the ui war on a Morpheus Appliance:

  1. Download the war file to the target appliance

    wget https://url/war_file
    

    Note

    If the war file is provided via a droplr link, ensure a + is added to end of droplr url or the file will not download

  2. Backup current war file

    sudo mv /opt/morpheus/lib/morpheus/morpheus-ui.war /opt/morpheus/lib/morpheus/morpheus-ui.bak.`date +"%m-%d-%Y"`
    
  3. Move and rename new war file

    sudo mv <file> /opt/morpheus/lib/morpheus/morpheus-ui.war
    
  4. Ensure war is owned by morpheus-app

    sudo chown morpheus-app.morpheus-app /opt/morpheus/lib/morpheus/morpheus-ui.war
    
  5. Restart the Morpheus UI

    sudo morpheus-ctl restart morpheus-ui
    

The new ui war will load on startup!

morpheus-ctl tips

morpheus-ctl is useful beyond reconfigures and starting the ui, and many commands can be run across all services, or scoped to a singe service.

Some common commands include:

morpheus-ctl status
This list all the installed services and their current Status
morpheus-ctl start (service)

This starts all services if no service is specified, or starts the specified service. For example,

  • morpheus-ctl start/stop/restart/kill on an all-in-one appliance will start, stop, restart or kill mysql, elasticsearch, rabbitmq, check-server, redis, guacd and the morpheus-ui, one by one.
  • morpheus-ctl start/stop/restart/kill morpheus-ui will only start, stop, restart or kill the morpheus-ui service, leaving the other service in their current state. Same goes for morpheus-ctl start/stop/restart/kill mysql, morpheus-ctl start/stop/restart/kill elasticsearch etc.

morpheus-ctl commands:

General Commands:

  cleanse
    Delete *all* morpheus data, and start from scratch.
  help
    Print this help message.
  reconfigure
    Reconfigure the application.
  show-config
    Show the configuration that would be generated by reconfigure.
  uninstall
    Kill all processes and uninstall the process supervisor (data will be preserved).

 Service Management Commands:

   graceful-kill
     Attempt a graceful stop, then SIGKILL the entire process group.
   hup
     Send the services a HUP.
   int
     Send the services an INT.
   kill
     Send the services a KILL.
   once
     Start the services if they are down. Do not restart them if they stop.
   restart
     Stop the services if they are running, then start them again.
   service-list
     List all the services (enabled services appear with a *.)
   start
     Start services if they are down, and restart them if they stop.
   status
     Show the status of all the services.
   stop
     Stop the services, and do not restart them.
   tail
     Watch the service logs of all enabled services.
   term
     Send the services a TERM.

 Elasticsearch Commands:

   elastic-util
     Backup/Restore ElasticSearch data

 Firewall Commands:

   firewall-enable-blocking
     Enables firewall blocking mode.

Morpheus UI war files

Pre-release or patched versions of the Morpheus UI are sometimes provided. To deploy the ui war on a Morpheus Appliance:

  1. Download the war file to the target appliance

    wget https://url/war_file
    

    Note

    If the war file is provided via a droplr link, ensure a + is added to end of droplr url or the file will not download

  2. Backup current war file

    sudo mv /opt/morpheus/lib/morpheus/morpheus-ui.war /opt/morpheus/lib/morpheus/morpheus-ui.bak.`date +"%m-%d-%Y"`
    
  3. Move and rename new war file

    sudo mv <file> /opt/morpheus/lib/morpheus/morpheus-ui.war
    
  4. Ensure war is owned by morpheus-app

    sudo chown morpheus-app.morpheus-app /opt/morpheus/lib/morpheus/morpheus-ui.war
    
  5. Restart the Morpheus UI

    sudo morpheus-ctl restart morpheus-ui
    

The new ui war will load on startup!

Morpheus DB Migration

If your new installation is part of a migration or you need to move the data from your original Morpheus database, this is easily accomplished by using a stateful dump.

To begin this, stop the Morpheus UI on your original Morpheus server:

[root@app-server-old ~] morpheus-ctl stop morpheus-ui

Once this is done you can safely export. To access the MySQL shell we will need the password for the Morpheus DB user. We can find this in the morpheus-secrets file:

[root@app-server-old ~] cat /etc/morpheus/morpheus-secrets.json | grep morpheus_password
"morpheus_password": "451e122cr5d122asw3de5e1b", <---------------this one
"morpheus_password": "9b5vdj4de5awf87d",

Take note of the first morpheus_password as it will be used to invoke a dump. Morpheus provides embedded binaries for this task. Invoke it via the embedded path and specify the host. In this example we are using the morpheus database on the MySQL listening on localhost. Enter the password copied from the previous step when prompted:

[root@app-server-old ~] /opt/morpheus/embedded/mysql/bin/mysqldump -u morpheus -h 127.0.0.1 morpheus -p > /tmp/morpheus_backup.sql
Enter password:

This file needs to be pushed to the new Morpheus Installation’s backend. Depending on the GRANTS in the new MySQL backend, this will likely require moving this file to one of the new Morpheus frontend servers.

Once the file is in place it can be imported into the backend. Begin by ensuring the Morpheus UI service is stopped on all of the application servers:

[root@app-server-new ~] morpheus-ctl stop morpheus-ui

Then you can import the MySQL dump into the target database using the embedded MySQL binaries, specifying the database host, and entering the password for the morpheus user when prompted:

[root@app-server-new ~] /opt/morpheus/embedded/mysql/bin/mysql -u morpheus -h 10.1.2.2 morpheus -p < /tmp/morpheus_backup.sql
Enter password:

The data form the old appliance is now replicated on the new appliance. Simply start the UI to complete the process:

[root@app-server-new ~] morpheus-ctl start morpheus-ui

Scaling Morpheus Nodes

Morpheus App nodes can be scaled to accommodate additional load. Appliance nodes can be scaled vertically in centralized architectures, and both vertically and horizontally in distributed architectures.

Vertical Scaling

In all Appliance Architectures, Application nodes can be vertically scaled at any time, however a reconfigure must be performed for additional resources to be utilized bu Morpheus on a node, which will result in the morpheus-ui restarting on the reconfiguring node.

Morpheus configures memory/ram utilization for services during the reconfigure process. If additional memory/ram is added to a Host or VM running the Morpheus App, the additional memory/ram will not be utilized by the Morpheus Application until a morpheus-ctl reconfigure command is ran and the additional memory/ram is recognized.

When the morpheus-ctl reconfigure command detects changes on available memory/ram, it will trigger a morpheus-ui service restart.

Important

When the morpheus-ctl reconfigure command detects changes on available memory/ram, it will restart the morpheus-ui service.

The impact on Availability depends on the Morpheus Appliance Architecture.

  • Centralized Appliances
    Morpheus will be unavailable while the morpheus-ui restarts.
  • Distributed Appliances
    Zero-down time can be achieved by Reconfiguring one App Node at a time, with proper Front-End Load Balancer configuration.

Horizontal Scaling

Additional Morpheus App Nodes can be added at any time to Fully Distributed Architectures.

  • Configure Shared Storage paths for the new App Node(s)
  • Install, but do not run the morpheus-ctl reconfigure command on the new App Node(s), using the same Morpheus version as the existing Appliance nodes.
  • Copy the morphues.rb from an existing App Node to the new App Node(s)
  • Ensure permissions and network configuration for the new App Node(s) to access all MySQL and Elasticsearch nodes, and the RabbitMQ VIP.
  • Ensure permissions and network configuration for all required UI services and Integrations, such as network access to ESXi hosts over 443 for Hypervisor console and/or image transfers.
  • Add associated SSL files and configuration, of not on shared storage.
  • Reconfigure the new App Node(s) via morpheus-ctl reconfigure
  • Verify UI startup succeeded
  • Add New App Node(s) to Front End Morpheus UI Load Balancer pool.

During morpheus-ctl reconfigure, the new App Node(s) will validate and be configured to use the existing tiers for the UI service. Upon successful reconfigure, the Morpheus service will be available on the App Node(s) with consistent data and capabilities.

Note

No services, including morphues-ui, are required to be shut down on existing nodes when adding new App Nodes