Troubleshooting

Common Ports & Requirements

The following chart is useful for troubleshooting Agent install, Static IP assignment, Remote Console connectivity, and Image transfers.

Common Ports & Requirements
Feature Method OS Source Destination Port Requirement
Agent Communication All All Node Appliance 443 DNS Resolution from node to appliance url
Agent Install All Linux Node Appliance 80 Used for appliance yum and apt repos
  SSH Linux Appliance Node 22
DNS Resolution from node to appliance url
Virtual Images configured
SSH Enabled on Virtual Image
  WinRM Windows Appliance Node 5985
DNS Resolution from node to appliance url
Virtual Images configured
WinRM Enabled on Virtual Image(winrm quickconfig)
  Cloud-init Linux      
Cloud-init installed on template/image
Cloud-init settings populated in User Settings or in Admin –> Provisioning
Agent install mode set to Cloud-Init in Cloud Settings
  Cloudbase-init Windows      
Cloudbase-init installed on template/image
Cloud-init settings populated in User Settings or in Admin –> Provisioning
Agent install mode set to Cloud-Init in Cloud Settings
  VMtools All      
VMtools installed on template
Cloud-init settings populated in Morpheus user settings or in Administration –> Provisioning when using Static IP’s
Existing User credentials entered on Virtual Image when using DHCP
RPC mode set to VMtools in VMware cloud settings.
Static IP Assignment & IP Pools Cloud-Init All      
Network configured in Morpheus (Gateway, Primary and Secondary DNS, CIDR populated, DHCP disabled)
Cloud-init/Cloudbase-init installed on template/image
Cloud-init settings populated in Morpheus user settings or in Administration –> Provisioning
  VMware Tools All      
Network configured in Morpheus (Gateway, Primary and Secondary DNS, CIDR populated, DHCP disabled)
VMtools installed on Template/Virtual Image
Remote Console SSH Linux Applaince Node 22
ssh enabled on node
user/password set on VM or Host in Morpheus
  RDP WIdnows Appliance Node 3389
RDP Enabled on node
user/password set on VM or Host in Morpheus
  Hypervisor Console All Appliance ESXi Host 5900-6000+
GBB server opened on all ESXii host firewalls
*Port range req’s vary per env
ESXi host names resolvable by morpheus appliance
Morpheus Catalog Image Download   All Amazon S3 Appliance 443 Available space at /var/opt/morpheus/
Image Transfer Stream All Appliance Datastore 443 Hypervisor Host Names resolvable by Morpheus Appliance

Morpheus Agent Install Troubleshooting

When provisioning and instance, there are some network and configuration requirements to successfully install the morpheus agent. Typically when a vm instance is still in the provisioning phase long after the vm is up, the instance is unable to reach Morpheus , or depending on agent install mode, Morpheus is unable to reach the instance.

The most common reason an agent install fails is the provisioned instance cannot reach the Morpheus Appliance via the appliance_url set in Admin -> Settings over both 443 and 80. When an instance is provisioned from Morpheus, it must be able to reach the Morpheus appliance via the appliance_url or the agent will not be installed.

../_images/agent-7c9a2.png

In addition to the main appliance_url in Admin -> Settings, additional appliance_urls can be set per cloud in the Advanced options of the cloud configuration pane when creating or editing a cloud. When this field is populated, it will override the main appliance url for anything provisioned into that cloud.

Tip

The Morpheus UI current log, located at /var/log/morpheus/morpheus-ui/current, is very helpful when troubleshooting agent installations.

Agent Install Modes

There are 3 Agent install modes:

  • ssh/winrm
  • VMware Tools
  • cloud-init

For All Agent Install modes

When an instance is provisioned and the agent does not install, verify the following for any agent install mode:

  • The Morpheus appliance_url (Admin -> Settings) is both reachable and resolvable from the provisioned node.

  • The appliance_url begins with to https://, not http://.

    Note

    Be sure to use https:// even when using an ip address for the appliance.

  • Inbound connectivity access to the Morpheus Appliance from provisioned VM’s and container hosts on port 443 (needed for agent communication)

  • Private (non-morpheus provided) vm images/templates must have their credentials entered. These can be entered/edited in the Provisioning - Virtual Images section but clicking the Actions dropdown of an image and selecting Edit.

Note

Administrator user is required for Windows agent install.

  • The instance does not have an IP address assigned. For scenarios without a dhcp server, static IP information must be entered by selecting the Network Type: Static in the Advanced section during provisioning. IP Pools can also be created in the Infrastructure -> Networks -> IP Pools section and added to clouds network sections for IPAM.
  • DNS is not configured and the node cannot resolve the appliance. If dns cannot be configure, the ip address of the Morpheus appliance can be used as the main or cloud appliance.

SSH/Winrm

Linux Agent
  • Port 22 is open for Linux images, and ssh is enabled
  • Credentials have been entered on the image if using custom or synced image. Credentials can be entered on images in the Provisioning -> Virtual Images section.
Windows Agent
  • Port 5985 must be open and winRM enabled for Windows images.
  • Credentials have been entered on the image if using custom or synced image. Credentials can be entered on images in the Provisioning -> Virtual Images section.

Note

Administrator user is required for Windows agent install.

VMware tools (vmtools) rpc mode

  • VMware tools is installed on the template(s)
  • Credentials have been entered on the image if using custom or synced image. Credentials can be entered on images in the Provisioning -> Virtual Images section.

Cloud-Init agent install mode

  • Cloud-Init is configured in Admin -> Provisioning section
  • Provisioned image/template has Cloud-Init (linux) or Cloudbase-Init (windows) installed

Manually Installing a Morpheus Agent

While it should not be necessary to manually install an agent if the requirements are met, it is possible to manually install an agent on an instance. This can also be handy when troubleshooting an agent install.

Linux

  1. In Morpheus , go to the VM’s host detail page in Infrastructure->Hosts->Virtual Machines you will see an API Key that is unique to that host.

  2. As root user, run: (replacing ${} with the relevant information)

    curl -k -s "${opts.applianceUrl}api/server-script/agentInstall?apiKey=${opts.apiKey}" | bash
    
  3. This will pull the Morpheus Agent install script from the Morpheus appliance and run it.

  4. Once the agent is installed, run morpheus-node-ctl reconfigure to complete the manual process.

Restarting the Morpheus Agent

In some situations is may necessary to restart the morpheus agent on the host to re-sync communication from the agent to the Morpheus appliance.

Linux

On the target host, run sudo morpheus-node-ctl restart morphd and the Morpheus agent will restart. morpheus-node-ctl status will also show the agent status.

Windows

The Morpheus Windows Agent service can be restarted in Administrative Tools -> Services.

Tip

The Morpheus Remote Console is not dependent on agent communication and can be used to install or restart the Morpheus agent on an instance.

Uninstall Morpheus Agent

You can use the following to uninstall the linux agent:

sudo rm /etc/apt/sources.list.d/morpheus.list
sudo morpheus-node-ctl stop rsyslogd
sudo apt-get -y purge morpheus-vm-node
sudo rm -rf /opt/morpheus-node
sudo usermod -l morpheus-old morpheus-node
sudo killall runsv
sudo killall runsvdir
sudo killall morphd

centOS/RHEL 7 Images

For custom centOS 7 images we highly recommend setting up cloud-init and fixing the network device names. More information for custom centOS images can be found in the centOS 7 image guide.

Morpheus UI not loading after upgrade or reconfigure

Problem
The Morpheus ui does not load after performing an upgrade.
Causes
  • The morpheus-ui has not finished loading
  • The morpheus-ui was not fully stopped before reconfigure, or not started after reconfigure
  • Morpheus was forced to restart or shut down while the database schema was being migrated during an upgrade
Solutions
Cause

The morpheus-ui has not finished loading.

Note

After running morpheus-ctl start morpheus-ui, the Morpheus ui takes around 3 minutes to run depending on hardware.

Solution

An easy way to see when the ui is finished loading and running is to tail the ui current file and look for the morpheus logo with version and start time

morpheus-ctl tail morpheus-ui
Cause

The morpheus-ui was not fully stopped before reconfigure, or not started after reconfigure

The morpheus ui must be stopped prior to running morpheus-ctl reconfigure when upgrading. Sometimes running morpheus-ctl stop morpheus-ui will timeout and the ui is not actually stopped. If stopping the ui does timeout, run morpheus-ctl kill morpheus-ui prior to reconfigure, and be sure to run morpheus-ctl start morpheus-ui after reconfigure is completed.

Solution

If you ran a reconfigure before stopping the ui, run:

sudo morpheus-ctl kill morpheus-ui
sudo morpheus-ctl reconfigure
sudo morpheus-ctl start morpheus-ui

Wait for the ui to come up.

Cause

Morpheus was forced to restart or shut down while the database schema was being migrated during an upgrade

If the ui fails to start and you see the error Invocation of init method failed; nested exception is liquibase.exception.LockException: Could not acquire change log lock.  Currently locked by morpheus it likely means morpheus was forced to restart or shut down while the database schema was being migrated during an upgrade, and the lock was not released.

Solution

To release the lock, you will need to run a mysql query. You will need to install mysql-client on the morpheus appliance, and grab the password for morpheus mysql. The username and db name are both morpheus. The password to login to mysql can be found in the application.yml file located at /opt/morpheus/conf/application.yml

Then run the following:

mysql -u morpheus -p -h 127.0.0.1 morpheus

At the prompt, enter the mysql password from the application.yml

Then run:

DELETE FROM DATABASECHANGELOGLOCK;

Then restart morpheus-ui:

sudo morpheus-ctl restart morpheus-ui

If the restart timesout, run:

sudo morpheus-ctl kill morpheus-ui
sudo morpheus-ctl start morpheus-ui

Blank Dashboard

Problem
A blank dashboard or 500 error after installing morpheus

Note

A blank or 500 error on just the dashboard is different than the entire morpheus-ui not loading. Please see UI note loading article for troubleshooting the ui not loading after an upgrade.

Cause
Elasticsearch restarting prior to being fully bootstrapped during the initial install.
Solution
To fix, purge elasticsearch by running the following on the Morpheus Appliance:
curl -XDELETE http://localhost:9200/*" rel="nofollow noreferrer">http://localhost:9200/*
morpheus-ctl restart elasticsearch
morpheus-ctl restart morpheus-ui

Another option is:

sudo rm –rf /var/opt/|morpheus| /elasticsearch/data/morpheus
morpheus-ctl restart elasticsearch
morpheus-ctl restart morpheus-ui

If you get a term/timeout on ui restart, run

Note

The morpheus-ui may take a few minutes to load and be available after being restarted

Unable to Provision a Custom Image

Prior to provisioning an custom image, the image must be configured in the Provisioning -> Virtual Images section by selecting Edit on the Actions dropdown of the Virtual Image.

In the Edit Virtual Image pane:

  1. Select “Cloud Init Enabled?” only if the Virtual Image is a linux image with cloud init installed.
  2. Enter the username and password that are set on the Virtual Image.

Note

When using Static IP’s or IP Pools in VMware, VMware tools must also be installed on the template in order for Morpheus to set the static IP address when provisioning.

Note

Morpheus agents only support 64-bit vm’s prior to versions 2.12.3 and 3.0.2

VMware Hypervisor Console is not displaying

Morpheus features Remote Console support directly to VMware ESXi hypervisors. To enable this feature a few prerequisites must be met:

  • The Morpheus appliance must have network access to the ESXi hosts within vCenter.
  • Firewall settings need to be adjusted on each ESXi host. This can be done in vSphere under firewall configuration on the ESXi hosts. Simply check the gdbserver option for each required host, which will open up the necessary ports (starting at the 5900 range).
  • The Morpheus must be able to resolve the ESXi hostnames.

Now that the ESXi hosts are ready to utilize remote console, simply edit the cloud in Morpheus via Infrastructure → Clouds. Check the option that says Use Hypervisor Console. Morpheus is now able to use the VMware Remote Console without opening any ports on the Virtual Machines.

Restart a Morpheus Installation

If the initial reconfigure is stopped or your installation is damaged beyond reconfiguring again, it may be necessary to start over.

On the Morpheus appliance:

  1. Run morpheus-ctl cleanse

  2. Remove the Morpheus package

    • deb: `dpkg --purge morpheus-appliance... using the appropriate package name.
    • rpm: rpm -e (morpheus-appliance...) using the appropriate package name.

    Then Run

    rm -rf /etc/morpheus
    rm -rf /var/opt/morpheus
    rm -rf /var/run/morpheus
    rm -rf /var/log/morpheus
    rm -rf /opt/morpheus
    

    Re-install Morpheus

If the elasticsearch cluster is unhealthy and needs purged, run:

sudo morpheus-ctl stop elasticsearch`
sudo rm -rf /var/opt/morpheus/elasticsearch/data/morpheus`
sudo morpheus-ctl reconfigure`

If eleasticsearch does not restart during reconfigure:

sudo morpheus-ctl start elasticsearch

Variables

The following are the map structures passed to scripts and templates during provisioning. They can currently be used inside of a <%= %> block.

Important

Variables are case sensitive

PowerShell Example: $app_id = "<%= instance.metadata.app_id %>"

Bash Example: HOSTNAME="<%= container.server.hostname %>"

Note

customOptions are user defined as Option Types or Option Lists in custom Library items.

instance {
        instanceTypeName,
        instanceTypeCode,
        provisionType,
        instanceVersion,
        plan,
        name,
        displayName,
        description,
        environmentPrefix,
        hostname,
        domainName,
        firewallEnabled,
        status,
        userStatus,
        networkLevel,
        instanceLevel,
        deployGroup,
        instanceContext,
        autoScale,
        statusMessage,
        expireDate,
        tags,
        storage,
        memory,
        cores,
        configId,
        configGroup,
        configRole
        containers:[],
        metadata:[],
        evars:[]
}
container {
        containerTypeName,
        containerTypeCode,
        containerTypeShortName,
        provisionType,
        dataPath,
        logsPath,
        configPath,
        planCode,
        dateCreated,
        status,
        environmentPrefix,
        version,
        image,
        internalHostname,
        hostname,
        domainName,
        storage,
        memory,
        cores,
        internalIp,
        externalIp,
        sshHost,
        hostMountPoint,
        configId,
        configGroup,
        configRole,
        serverId,
        server:{}
}
server {
        serverTypeName,
        serverTypeCode,
        parentServerId,
        plan,
        visibility,
        osTypeCode,
        sourceImageId,
        name,
        displayName,
        internalName,
        category,
        description
        internalId,
        externalId,
        platform,
        platformVersion,
        agentVersion,
        nodePackageVersion,
        sshHost,
        sshPort,
        sshUsername,
        consoleType,
        consoleHost,
        consolePort,
        consoleUsername,
        internalSshUsername,
        internalIp,
        externalIp,
        osDevice,
        dataDevice,
        lvmEnabled,
        apiKey,
        softwareRaid,
        status,
        powerState,
        dateCreated,
        lastAgentUpdate,
        serverType,
        osType,
        commType,
        managed,
        agentInstalled,
        toolsInstalled,
        hostname,
        domainName,
        statusMessage,
        maxStorage,
        maxMemory,
        maxCores,
        macAddress,
        serverVendor,
        serverModel,
        serialNumber,
        tags,
        configId,
        configGroup,
        configRole
        volumes {
                name
                id
                deviceName
                maxStorage
                unitNumber
                displayOrder
                rootVolume
        }
}
cloud {
        name,
        code,
        location,
        cloudTypeName,
        cloudTypeCode,
        domainName,
        scalePriority,
        firewallEnabled,
        regionCode,
        agentMode,
        datacenterId
}
group {
        code,
        name,
        location,
        datacenterId
}
customOptions {
        customOptions.fieldName
}

Important

Variables are case sensitive