Troubleshooting

Common Ports & Requirements

The following chart is useful for troubleshooting Agent install, Static IP assignment, Remote Console connectivity, and Image transfers.

Common Ports & Requirements
Feature Method OS Source Destination Port Requirement
Agent Communication All All Node Appliance 443 DNS Resolution from node to appliance url
Agent Install All Linux Node Appliance 80 Used for appliance yum and apt repos
  SSH Linux Appliance Node 22
DNS Resolution from node to appliance url
Virtual Images configured
SSH Enabled on Virtual Image
  WinRM Windows Appliance Node 5985
DNS Resolution from node to appliance url
Virtual Images configured
WinRM Enabled on Virtual Image(winrm quickconfig)
  Cloud-init Linux      
Cloud-init installed on template/image
Cloud-init settings populated in User Settings or in Admin –> Provisioning
Agent install mode set to Cloud-Init in Cloud Settings
  Cloudbase-init Windows      
Cloudbase-init installed on template/image
Cloud-init settings populated in User Settings or in Admin –> Provisioning
Agent install mode set to Cloud-Init in Cloud Settings
  VMtools All      
VMtools installed on template
Cloud-init settings populated in Morpheus user settings or in Administration –> Provisioning when using Static IP’s
Existing User credentials entered on Virtual Image when using DHCP
RPC mode set to VMtools in VMware cloud settings.
Static IP Assignment & IP Pools Cloud-Init All      
Network configured in Morpheus (Gateway, Primary and Secondary DNS, CIDR populated, DHCP disabled)
Cloud-init/Cloudbase-init installed on template/image
Cloud-init settings populated in Morpheus user settings or in Administration –> Provisioning
  VMware Tools All      
Network configured in Morpheus (Gateway, Primary and Secondary DNS, CIDR populated, DHCP disabled)
VMtools installed on Template/Virtual Image
Remote Console SSH Linux Applaince Node 22
ssh enabled on node
user/password set on VM or Host in Morpheus
  RDP Windows Appliance Node 3389
RDP Enabled on node
user/password set on VM or Host in Morpheus
  Hypervisor Console All Appliance ESXi Host 5900-6000+
GBB server opened on all ESXii host firewalls
*Port range req’s vary per env
ESXi host names resolvable by morpheus appliance
Morpheus Catalog Image Download   All Appliance AWS S3 443 Available space at /var/opt/morpheus/
Image Transfer Stream All Appliance Datastore 443 Hypervisor Host Names resolvable by Morpheus Appliance

Morpheus Agent Install Troubleshooting

When provisioning and instance, there are some network and configuration requirements to successfully install the morpheus agent. Typically when a vm instance is still in the provisioning phase long after the vm is up, the instance is unable to reach Morpheus , or depending on agent install mode, Morpheus is unable to reach the instance.

The most common reason an agent install fails is the provisioned instance cannot reach the Morpheus Appliance via the appliance_url set in Admin -> Settings over both 443 and 80. When an instance is provisioned from Morpheus, it must be able to reach the Morpheus appliance via the appliance_url or the agent will not be installed.

../_images/agent-7c9a2.png

In addition to the main appliance_url in Admin -> Settings, additional appliance_urls can be set per cloud in the Advanced options of the cloud configuration pane when creating or editing a cloud. When this field is populated, it will override the main appliance url for anything provisioned into that cloud.

Tip

The Morpheus UI current log, located at /var/log/morpheus/morpheus-ui/current, is very helpful when troubleshooting agent installations.

Agent Install Modes

There are 3 Agent install modes:

  • ssh/winrm
  • VMware Tools
  • cloud-init

For All Agent Install modes

When an instance is provisioned and the agent does not install, verify the following for any agent install mode:

  • The Morpheus appliance_url (Admin -> Settings) is both reachable and resolvable from the provisioned node.

  • The appliance_url begins with to https://, not http://.

    Note

    Be sure to use https:// even when using an ip address for the appliance.

  • Inbound connectivity access to the Morpheus Appliance from provisioned VM’s and container hosts on port 443 (needed for agent communication)

  • Private (non-morpheus provided) vm images/templates must have their credentials entered. These can be entered/edited in the Provisioning - Virtual Images section but clicking the Actions dropdown of an image and selecting Edit.

Note

Administrator user is required for Windows agent install.

  • The instance does not have an IP address assigned. For scenarios without a dhcp server, static IP information must be entered by selecting the Network Type: Static in the Advanced section during provisioning. IP Pools can also be created in the Infrastructure -> Networks -> IP Pools section and added to clouds network sections for IPAM.
  • DNS is not configured and the node cannot resolve the appliance. If dns cannot be configure, the ip address of the Morpheus appliance can be used as the main or cloud appliance.

SSH/Winrm

Linux Agent
  • Port 22 is open for Linux images, and ssh is enabled
  • Credentials have been entered on the image if using custom or synced image. Credentials can be entered on images in the Provisioning -> Virtual Images section.
Windows Agent
  • Port 5985 must be open and winRM enabled for Windows images.
  • Credentials have been entered on the image if using custom or synced image. Credentials can be entered on images in the Provisioning -> Virtual Images section.

Note

Administrator user is required for Windows agent install.

VMware tools (vmtools) rpc mode

  • VMware tools is installed on the template(s)
  • Credentials have been entered on the image if using custom or synced image. Credentials can be entered on images in the Provisioning -> Virtual Images section.

Cloud-Init agent install mode

  • Cloud-Init is configured in Admin -> Provisioning section
  • Provisioned image/blueprint has Cloud-Init (linux) or Cloudbase-Init (windows) installed

Manually Installing a Morpheus Agent

While it should not be necessary to manually install an agent if the requirements are met, it is possible to manually install an agent on an instance. This can also be handy when troubleshooting an agent install.

Linux

  1. In Morpheus , go to the VM’s host detail page in Infrastructure->Hosts->Virtual Machines you will see an API Key that is unique to that host.

  2. As root user, run: (replacing ${} with the relevant information)

    curl -k -s "${opts.applianceUrl}api/server-script/agentInstall?apiKey=${opts.apiKey}" | bash
    
  3. This will pull the Morpheus Agent install script from the Morpheus appliance and run it.

  4. Once the agent is installed, run morpheus-node-ctl reconfigure to complete the manual process.

Windows

  • The windows agent setup can be downloaded at https://[morpheus-applaince-url]/msi/morpheus-agent/MorpheusAgentSetup.msi
  • On the Morpheus appliance package the windows agent is located at /var/opt/morpheus/package-repos/msi/morpheus-agent
  • WinRM, VMware Tools, or Cloudbase-Init can be used to install the agent from the Morpheus appliance
  • The initial windows installer is MorpheusAgentSetup.msi
  • Once the Windows agent is downloaded and installed with Morpheus AgentSetup.msi the agent is located and runs from /Program Files x86/Morphues/|morpheus| Windows Agent
  • Logs can be viewed in the Event Viewer under Applications and Services Logs -> Morpheus Windows Agent
  1. Replace the values for $apiKey and $applianceUrl in the script below.

  2. Execute this script on the Windows box in Powershell.

    $apiKey = "add VM apiKey here"
    $applianceUrl = "https://your_appliance_url.com/"
    
    $client = New-Object System.Net.WebClient
    $client.DownloadFile($applianceUrl + "/msi/morpheus-agent/MorpheusAgentSetup.msi", "C:\Program Files (x86)\Common Files\MorpheusAgentSetup.msi")
    Start-Sleep -Seconds 10
    cd ${env:commonprogramfiles(x86)}
    $serviceName = "Morpheus Windows Agent"
    if(Get-Service $serviceName -ErrorAction SilentlyContinue) {
    Stop-Service -displayname $serviceName -ErrorAction SilentlyContinue
    Stop-Process -Force -processname Morpheus* -ErrorAction SilentlyContinue
    Stop-Process -Force -processname Morpheus* -ErrorAction SilentlyContinue
    Start-Sleep -s 5
    $serviceId = (get-wmiobject Win32_Product -Filter "Name = 'Morpheus Windows Agent'" | Format-Wide -Property IdentifyingNumber | Out-String).Trim()
    cmd.exe /c "msiexec /x $serviceId /q"
    }
    [Console]::Out.Flush()
    [gc]::collect()
    try {
    Write-VolumeCache C
    }
    Catch {
    }
    $MSIArguments= @(
    "/i"
    "MorpheusAgentSetup.msi"
    "/qn"
    "/norestart"
    "/l*v"
    "morpheus_install.log"
    "apiKey=$apiKey"
    "host=$applianceUrl"
    "username=`".\LocalSystem`""
    "vmMode=`"true`""
    "logLevel=`"1`""
    )
    $installResults = Start-Process msiexec.exe -Verb runAs -Wait -ArgumentList $MSIArguments
    [Console]::Out.Flush()
    [gc]::collect()
    try {
    Write-VolumeCache C
    }
    Catch {
    }
    start-sleep -s 10
    $attempts = 0
    Do {
    try {
            Get-Service $serviceName -ea silentlycontinue -ErrorVariable err
            if([string]::isNullOrEmpty($err)) {
                    Break
            } else {
                    start-sleep -s 10
                    $attempts++
            }
    }
    Catch {
            start-sleep -s 10
            $attempts++
    }
    }
    While ($attempts -ne 6)
    Set-Service $serviceName -startuptype "automatic"
    $service = Get-WmiObject -Class Win32_Service -Filter "Name='$serviceName'"
    if ($service -And $service.State -ne "Running") {Restart-Service -displayname $serviceName}
    exit $installResults.ExitCode
    
  3. If the agent doesn’t install, logs can be found in the morpheus_install.log file located at C:\Program Files (x86)\Common Files\

Restarting the Morpheus Agent

In some situations is may necessary to restart the morpheus agent on the host to re-sync communication from the agent to the Morpheus appliance.

Linux

On the target host, run sudo morpheus-node-ctl restart morphd and the Morpheus agent will restart. morpheus-node-ctl status will also show the agent status.

Windows

The Morpheus Windows Agent service can be restarted in Administrative Tools -> Services.

Tip

The Morpheus Remote Console is not dependent on agent communication and can be used to install or restart the Morpheus agent on an instance.

Uninstall Morpheus Agent

You can use the following to uninstall the linux agent:

sudo rm /etc/apt/sources.list.d/morpheus.list
sudo morpheus-node-ctl stop rsyslogd
sudo apt-get -y purge morpheus-vm-node
sudo rm -rf /opt/morpheus-node
sudo usermod -l morpheus-old morpheus-node
sudo killall runsv
sudo killall runsvdir
sudo killall morphd

centOS/RHEL 7 Images

For custom centOS 7 images we highly recommend setting up cloud-init and fixing the network device names. More information for custom centOS images can be found in the centOS 7 image guide.

Remote Console

Morpheus has a built in Remote Console for Instances, Hosts, Virtual Machines and Bare Metal. The following information reviews the Roles Settings, Protocols, and Requirements necessary to configure and troubleshoot Remote Console access.

Role Settings

User Role settings determine if the Console tab or Open Console Action appear for a user, and if a login prompt is presented or the user is automatically logged in when using the Console.

  • Remote Console (None, Provisioned, Full)
    None

    The user will not have access to remote console.

    Provisioned

    The user will only have remote console access for Instances they provisioned.

    Full

    The user will have remote console access for all instances they have access to.

  • Remote Console: Auto Login (No, Yes)
    No

    A login prompt will be present in the console for Linux platforms, and the main login screen will present for Windows platforms.

    Yes

    Morpheus will automatically login to the remote console using the credentials defined on the VM or Host. For provisioned Instances, the credentials are defined either from the credentials defined on the Virtual Image used, added via cloud-init or VMware Tools using the global cloud-init settings (Administration - Provisioning) or the Linux or Windows settings defined in User Settings. For Instances created when converting a VM or Host to managed, the credentials are entered when converting to managed. These credentials can be changed by editing the underlying VM or Host of the Instance.

    Note

    If the credentials defined on the VM or Host are not valid, and the Remote Console: Auto Login Role setting is set to Yes, the console will not be able to connect and no console window or login prompt will be presented. The credentials on the underlying VM or Host must be edited or Remote Console: Auto Login Role setting can be set to No for a login prompt to present in the console. Credentials cannot be changed from an Instance view, only in the Infrastructure VM or Host view.

Protocols

Platform Type and Cloud Settings determines the protocol and port used for Remote Console connections.

  • SSH
    The SSH protocol will be used for Linux and OSX platform types, and 22 is the default port used.
  • RDP
    The RDP (Remote Desktop) protocol will be used for Windows platform types over port 3389 by default.
  • VNC (VMware Hypervisor Console)
    The VNC protocol will be used for all platform types in VMware Clouds with the Hypervisor Console option enabled in cloud settings. A unique port is assigned per per Virtual Machine starting at 5900.

Note

Alternative ports can be configured per VM or Host by editing the VM or Host and editing the Port field in the RPC host section.

SSH

For all Linux and OSX platform types, Morpheus will use the SSH protocol via port 22 by default for Remote Console connections, unless the Hypervisor Console` option is enabled for VMware type clouds.

Morpheus will SSH using the username, password, RPC Host IP address and Port defined in the VM or Host record.

Default Requirements for SSH Connectivity

  • SSH Enabled on the target VM or Host
  • Port 22 incoming open on the target VM or Host firewalls and security groups from the Morpheus Appliance (not from the users IP address)
  • An IP address defined on the VM or Host record that is routable from the Morpheus Appliance.
  • Valid credentials defined on the VM or Host record in the RPC host field.
  • Remote Console Role Permissions set to Provisioned or Full if the User provisioned the instance, or Full if the user did not provision the instance.

RDP

For all Windows platform types, Morpheus will use the RDP protocol via port 3389 by default for Remote Console connections, unless the Hypervisor Console` option is enabled for VMware type clouds.

Morpheus will RDP using the username, password, RPC Host IP address and Port defined in the VM or Host record.

Default Requirements for RDP Connectivity

  • Remote Access enabled on the target VM or Host and Remote Desktop enabled in the Windows Firewall settings. If the VM or Host is on a different network than the Morpheus appliance, public access for Remote Desktop must be enabled in the Firewall settings.
  • Port 3389 incoming open on the target VM or Host firewalls and security groups from the Morpheus Appliance (not from the users IP address)
  • An IP address defined on the VM or Host record that is routable from the Morpheus Appliance.
  • Valid credentials defined on the VM or Host record in the RPC host field.
  • Remote Console Role Permissions set to Provisioned or Full if the User provisioned the instance, or Full if the user did not provision the instance.

Note

If Remote Console: Auto Login is set to No in a users Role permissions, Allow connections only from computers running Remote Desktop with Network Level Authentication in the Windows System Properties -> Remote settings must be DISABLED for Remote Console to connect.

VNC (VMware Hypervisor Console)

When the Hypervisor Console option is enabled in VMware cloud settings, the VNC protocol will be used for all platform types that VMware Cloud.

When using VNC Hypervisor Console, the Morpheus Appliance connects directly to the ESXi host the VM is on, not directly to the VM. A VNC port is assigned per Virtual Machine starting at 5900 and a complex VNC password is generated. VNC is then enabled and the port and password are set on the VMs settings in vSphere.

Morpheus features Remote Console support directly to VMware ESXi hypervisors. To enable this feature a few prerequisites must be met:

  • The gdbserver Firewall setting needs to be enabled on each ESXi host the console will connect to. This can be done in vSphere under firewall configuration on the ESXi hosts by checking the gdbserver option on each required host, This will open up the necessary ports VNC ports on the ESXi host(s), starting at port 5900.

Note

The most common cause of Remote Console connection issues when using hypervisor console is gdbserver not being enabled on the ESXi hosts firewall settings.

  • The Morpheus Appliance must have network access to the ESXi hosts within vCenter, specifically to the host the VM is on when using Remote Console, over the VNC port range, typically 5900-6000. The connection does not go through the vCenter server(s).
  • The Morpheus Appliance must be able to resolve the ESXi hostnames.
  • VMware tools or equivalent must be installed on the VM.

Unlike SSH and RDP, valid credentials do not need to be set on the VM or Host records in Morpheus for VNC hypervisor console connections. An IP address is also not required on the VM or Host for VNC hypervisor console connections. Morpheus will be able to connect to the VVM or Host as soon as the Host (Hypervisor) record is set, which can be viewed in the Info section on the VM or Host detail page.

Note

Auto-login is not supported for Hypervisor Console. Auto-login role settings do not apply to console connecting when using Hypervisor Console. Please note Hypervisor Console sessions persist on the ESXi host and once a user manually logs in to the VM they will continue to be logged in, even if the console tab/window in Morpheus is closed, until they manually log out.

Note

Copy and Paste and Text selection in Linux terminals is not supported when using VNC (VMware Hypervisor Console).

Important

All VMs that are inventoried or provisioned into a VMware cloud with Hypervisor Console enabled will have their Console Type set to VNC. Disabling Hypervisor Console will only apply to newly inventoried or provisioned VM’s. VM’s inventoried or provisioned when Hypervisor Console was enabled will continue to use the VNC protocol even after Hypervisor Console is disabled. To revert a VM

Note

In Morpheus versions 3.2.0 and higher, a newer Guacamole version is installed that is not compatible with MacOS Platform Types over VNC. The

Copy and Paste

Copy and Paste for Text is supported for SSH and RDP protocols only.

To Copy text from the console:

  1. Select text in the Console window.
  2. Click the COPY button at the top of the Console window.
  3. The selected text is copied to the users clipboard.

To Paste text into console:

  1. Copy text on the local computer to you clipboard
  2. Right click into the “Paste Text Here” field at the top of the Console window. The field will the display “Text Copied, Use Console to Paste.”
  3. Right click into the console window.
  4. The text is pasted into the VM.

Guacamole

Overview

Morpheus uses Apache Guacamole, a clientless remote console. Guacamole is installed on the Morpheus Appliance during the initial reconfigure. In Morpheus versions 3.2.0 and higher, Guacamole 0.9.14 is automatically installed. On Morpheus versions older than 3.2.0, 0.9.9 is installed. The 0.9.14 version is required for VNC Hypervisor Console functionality on ESXi v6.5 and later.

The Guacamole proxy daemon, guacd, is used for all Remote Console connections and must be running for Remote Console functionality.

Troubleshooting guacd

If all console connections are not functioning, the Guacamole proxy daemon (guacd) process may not be running or have a stuck process preventing console connections. This is evident when only the header appears in the console tab/window, and no console window appears below the header and no connection status is show in the console header. The following commands can be used on the Morpheus Appliance to restore console functionality.

morpheus-ctl status
Lists all local Morpheus services including guacd and their states. If guacd is stopped, it will need to be started again for Remote Console to function.
morpheus-ctl start guacd
Starts the guacd process
morpheus-ctl stop guacd
Stops the guacd process
morpheus-ctl kill guacd
Forcefully kills the guacd process
morpheus-ctl restarts guacd
Restarts the guacd process
morpheus-ctl tail guacd
Tails the guacd current and state logs, located by default at /var/log/morpheus/guacd/. This log is useful when troubleshooting console connections, guacamole service status, and to determine the protocol being used for the Remote Console connection.

If guacd continues to stop even after being started, or if guacd is running and no properly configured console connections are functioning, there may be a stuck guacd or multiple guacd processes running, which will need to killed and guacd started again.

To kill all guacd processes on the Morpheus Appliance and start guacd again:

  1. Kill the morpheus gaucd proccess: morpheus-ctl kill guacd

  2. Grep for all running guacd processes: sudo ps -aux | grep guacd and note the guacd pid(s) (minus the process from the grep)

  3. Kill all running guacd processes: kill -9 pid replacing pid with the pid(s) of the target processes

  4. Start guacd again: morpheus-ctl start guacd

  5. Tail the guacd logs to verify guacd is started and listening: morpheus-ctl tail guacd The log output will resemble below when guacd is properly running:

    guacd[16899]: INFO:       Guacamole proxy daemon (guacd) version 0.9.14 started
    guacd[16899]: INFO:       Listening on host 127.0.0.1, port 4822
    
  6. Additional information in the guacd logs appears when Morpheus is making a console connection. A successful conneciton will resemble:

    guacd[24725]: INFO: Creating new client for protocol "ssh"
    guacd[24725]: INFO: Connection ID is "$24f67856-f050-4a17-83eb-9101g0cd8869"
    guacd[24743]: INFO: Current locale does not use UTF-8. Some characters may not render correctly.
    guacd[24743]: INFO: User "@63102f19-eff4-412e-b1f9-718405f55782" joined connection "$24f67856-f050-4a17-83eb-9101g0cd8869" (1 users now present)
    guacd[24743]: INFO: Auth key successfully imported.
    guacd[24743]: INFO: SSH connection successful.
    

Guacamole Version

In Morpheus versions 3.2.0 and higher, Guacamole version 0.9.14 is automatically installed. On Morpheus versions older than 3.2.0, 0.9.9 is installed. The 0.9.14 version is required for VNC Hypervisor Console functionality on ESXi v6.5 and later.

Note Guacamole version 0.9.14 is not compatible with MacOS Platform Types over VNC on ESXi v6.0 or prior (6.5 is supported). If necessary, the guacamole version can be reverted to 0.9.9.

To revert the guacamole version from 0.9.14 to 0.9.9.

  1. Kill guacd - morpheus-ctl kill guacd
  2. Check if any guacd processes are still running ps -aux | grep guac
  3. If so, kill the processes kill -9 pid with id being the actual process id, like 16101.
  4. Go to the guac 0.9.9 directory: cd /var/opt/morpheus/guacamole-server-0.9.9
  5. Run: make install
  6. Start guacd: morpheus-ctl start guacd

Cannot Login

Forgot password

If a user forgets their password, they can use the FORGOT PASSWORD? link on the login page. They can then enter their username or email address to send a reset password email to the email address defined on the user.

If the default or user added SMTP server is not functioning or blocked, a System Admin user can impersonate that user and update their password.

If the System Admin user password needs to be reset and the default or user added SMTP server is not functioning or blocked, please contact Morpheus support for assistance.

Sub-Tenant user cannot login after 3.4.0 upgrade

Morpheus v3.4.0 added support for all subtenant users to login via the main tenant url using subtenant id or subdomain prefix, ie tenantId\username or subdomain\username.

Note

Tenant subdomains can be defined by editing Tenant settings and updating the SUBDOMAIN field.

Important

Subtenant local users will no longer be able to login from main login url without using their subtenant id or subdomain prefix.

The login requirements were added in v3.4.0 to allow subtenant users with identity source integration generated user accounts to be able to login to the master tenant, gain API and CLI access, and remove the requirement for usernames to be unique across all tenants.

Previously subtenant users that had local/morpheus generated user accounts could login to their tenant via the master tenant url, while subtenant users that had identity source integration generated user accounts had to use the subtenant specific login url.

In v3.4.0+ all subtenant users can login via the master tenant url by specifying their tenant id or subdomain prefix, \, then username. Subtenants can still use the tenant specific login url as well.

Example:
I have a username subuser that belongs to a tenant with the subdomain acme and tenant id 58. When logging in from the main login url, I now need to enter in: acme\subuser and the password. Alternatively the tenant ID can be used, ie 58\subuser

Active Directory user suddenly cannot Login

In Morpheus v3.4.0 and prior, OU changes in Active Directory can disable logins for AD users who had previously authenticated/have existing user accounts in Morpheus. If an Active Directory user cannot login to Morpheus after their OU was changed in AD, please contact Morpheus support for a resolution. The OU association for the user(s) can also be manually updated in the database. This issue is resolved in Morpheus versions 3.4.1 and higher.

Morpheus UI not loading after upgrade or reconfigure

Problem
The Morpheus ui does not load after performing an upgrade.
Causes
  • The morpheus-ui has not finished loading
  • The morpheus-ui was not fully stopped before reconfigure, or not started after reconfigure
  • Morpheus was forced to restart or shut down while the database schema was being migrated during an upgrade
Solutions
Cause

The morpheus-ui has not finished loading.

Note

After running morpheus-ctl start morpheus-ui, the Morpheus ui takes around 3 minutes to run depending on hardware.

Solution

An easy way to see when the ui is finished loading and running is to tail the ui current file and look for the morpheus logo with version and start time

morpheus-ctl tail morpheus-ui
Cause

The morpheus-ui was not fully stopped before reconfigure, or not started after reconfigure

The morpheus ui must be stopped prior to running morpheus-ctl reconfigure when upgrading. Sometimes running morpheus-ctl stop morpheus-ui will timeout and the ui is not actually stopped. If stopping the ui does timeout, run morpheus-ctl kill morpheus-ui prior to reconfigure, and be sure to run morpheus-ctl start morpheus-ui after reconfigure is completed.

Solution

If you ran a reconfigure before stopping the ui, run:

sudo morpheus-ctl kill morpheus-ui
sudo morpheus-ctl reconfigure
sudo morpheus-ctl start morpheus-ui

Wait for the ui to come up.

Cause

Morpheus was forced to restart or shut down while the database schema was being migrated during an upgrade

If the ui fails to start and you see the error Invocation of init method failed; nested exception is liquibase.exception.LockException: Could not acquire change log lock.  Currently locked by morpheus it likely means morpheus was forced to restart or shut down while the database schema was being migrated during an upgrade, and the lock was not released.

Solution

To release the lock, you will need to run a mysql query. You will need to install mysql-client on the morpheus appliance, and grab the password for morpheus mysql. The username and db name are both morpheus. The password to login to mysql can be found in the application.yml file located at /opt/morpheus/conf/application.yml

Then run the following:

mysql -u morpheus -p -h 127.0.0.1 morpheus

At the prompt, enter the mysql password from the application.yml

Then run:

DELETE FROM DATABASECHANGELOGLOCK;

Then restart morpheus-ui:

sudo morpheus-ctl restart morpheus-ui

If the restart timesout, run:

sudo morpheus-ctl kill morpheus-ui
sudo morpheus-ctl start morpheus-ui

Unable to Provision a Custom Image

Prior to provisioning an custom image, the image must be configured in the Provisioning -> Virtual Images section by selecting Edit on the Actions dropdown of the Virtual Image.

In the Edit Virtual Image pane:

  1. Select “Cloud Init Enabled?” only if the Virtual Image is a linux image with cloud init installed.
  2. Enter the username and password that are set on the Virtual Image.

Note

When using Static IP’s or IP Pools in VMware, VMware tools must also be installed on the template in order for Morpheus to set the static IP address when provisioning.

Note

Morpheus agents only support 64-bit vm’s prior to versions 2.12.3 and 3.0.2

Restart a Morpheus Installation

If the initial reconfigure is stopped or your installation is damaged beyond reconfiguring again, it may be necessary to start over.

On the Morpheus appliance:

  1. Run morpheus-ctl cleanse

  2. Remove the Morpheus package

    • deb: `dpkg --purge morpheus-appliance... using the appropriate package name.
    • rpm: rpm -e (morpheus-appliance...) using the appropriate package name.

    Then Run

    rm -rf /etc/morpheus
    rm -rf /var/opt/morpheus
    rm -rf /var/run/morpheus
    rm -rf /var/log/morpheus
    rm -rf /opt/morpheus
    

    Re-install Morpheus

If the elasticsearch cluster is unhealthy and needs purged, run:

sudo morpheus-ctl stop elasticsearch`
sudo rm -rf /var/opt/morpheus/elasticsearch/data/morpheus`
sudo morpheus-ctl reconfigure`

If eleasticsearch does not restart during reconfigure:

sudo morpheus-ctl start elasticsearch

Variables

The following are the map structures passed to scripts and templates during provisioning inside of a <%= %> block.

Variables can also be passed in Naming Policies using ${ } block.

Important

Variables are case sensitive

PowerShell Example: $app_id = "<%= instance.metadata.app_id %>"

Bash Example: HOSTNAME="<%= container.server.hostname %>"

Instance Naming Policy example: ``${userInitials}-${cloudCode}-${platform == ‘windows’ ? ‘W’ : ‘L’}-${sequence}

Note

customOptions are user defined as Option Types or Option Lists in custom Library items.

cypher: <%=cypher.read('secret/hello')%>
customOptions: <%=customOptions.fieldName%>"
evars: <%=evars%>
instance.metadata: <%=instance.metadata%>
instance.instanceTypeName: <%=instance.instanceTypeName%>
instance.instanceTypeCode: <%=instance.instanceTypeCode%>
instance.provisionType: <%=instance.provisionType%>
instance.instanceVersion: <%=instance.instanceVersion%>
instance.plan: <%=instance.plan%>
instance.name: <%=instance.name%>
instance.displayName: <%=instance.displayName%>
instance.description: <%=instance.description%>
instance.environmentPrefix: <%=instance.environmentPrefix%>
instance.hostname: <%=instance.hostname%>
instance.domainName: <%=instance.domainName%>
instance.firewallEnabled: <%=instance.firewallEnabled%>
instance.status: <%=instance.status%>
instance.userStatus: <%=instance.userStatus%>
instance.networkLevel: <%=instance.networkLevel%>
instance.instanceLevel: <%=instance.instanceLevel%>
instance.deployGroup: <%=instance.deployGroup%>
instance.instanceContext: <%=instance.instanceContext%>
instance.autoScale: <%=instance.autoScale%>
instance.statusMessage: <%=instance.statusMessage%>
instance.expireDate: <%=instance.expireDate%>
instance.tags: <%=instance.tags%>
instance.storage: <%=instance.storage%>
instance.memory: <%=instance.memory%>
instance.cores: <%=instance.cores%>
instance.configId: <%=instance.configId%>
instance.configGroup: <%=instance.configGroup%>
instance.configRole: <%=instance.configRole%>
instance.containers[0]: <%=instance.containers[0].containerTypeName%>
instance.createdBYUsername: <%=instance.createdByUsername%>
instance.createdByEmail: <%=instance.createdByEmail %>
instance.createdByFirstName: <%=instance.createdByFirstName%>
instance.createdByLastName: <%=instance.createdByLastName%>
instance.createdById: <%=instance.createdById%>
container.containerTypeName: <%=container.containerTypeName%>
container.containerTypeCode: <%=container.containerTypeCode%>
container.containerTypeShortName: <%=container.containerTypeShortName%>
container.provisionType: <%=container.provisionType%>
container.dataPath: <%=container.dataPath%>
container.logsPath: <%=container.logsPath%>
container.configPath: <%=container.configPath%>
container.planCode: <%=container.planCode%>
container.dateCreated: <%=container.dateCreated%>
container.status: <%=container.status%>
container.environmentPrefix: <%=container.environmentPrefix%>
container.version: <%=container.version%>
container.image: <%=container.image%>
container.internalHostname: <%=container.internalHostname%>
container.hostname: <%=container.hostname%>
container.domainName: <%=container.domainName%>
container.storage: <%=container.storage%>
container.memory: <%=container.memory%>
container.cores: <%=container.cores%>
container.internalIp: <%=container.internalIp%>
container.externalIp: <%=container.externalIp%>
container.sshHost: <%=container.sshHost%>
container.hostMountPoint: <%=container.hostMountPoint%>
container.configId: <%=container.configId%>
container.configGroup: <%=container.configGroup%>
container.configRole: <%=container.configRole%>
container.serverId: <%=container.serverId%>
container.server: <%=container.server.serverTypeName%>
server.serverTypeName: <%=server.serverTypeName%>
server.serverTypeCode: <%=server.serverTypeCode%>
server.parentServerId: <%=server.parentServerId%>
server.plan: <%=server.plan%>
server.visibility: <%=server.visibility%>
server.osTypeCode: <%=server.osTypeCode%>
server.sourceImageId: <%=server.sourceImageId%>
server.name: <%=server.name%>
server.displayName: <%=server.displayName%>
server.internalName: <%=server.internalName%>
server.category: <%=server.category%>
server.description: <%=server.description%>
server.internalId: <%=server.internalId%>
server.externalId: <%=server.externalId%>
server.platform: <%=server.platform%>
server.platformVersion: <%=server.platformVersion%>
server.agentVersion: <%=server.agentVersion%>
server.nodePackageVersion: <%=server.nodePackageVersion%>
server.sshHost: <%=server.sshHost%>
server.sshPort: <%=server.sshPort%>
server.sshUsername: <%=server.sshUsername%>
server.consoleType: <%=server.consoleType%>
server.consoleHost: <%=server.consoleHost%>
server.consolePort: <%=server.consolePort%>
server.consoleUsername: <%=server.consoleUsername%>
server.internalSshUsername: <%=server.internalSshUsername%>
server.internalIp: <%=server.internalIp%>
server.externalIp: <%=server.externalIp%>
server.osDevice: <%=server.osDevice%>
server.dataDevice: <%=server.dataDevice%>
server.lvmEnabled: <%=server.lvmEnabled%>
server.apiKey: <%=server.apiKey%>
server.softwareRaid: <%=server.softwareRaid%>
server.status: <%=server.status%>
server.powerState: <%=server.powerState%>
server.dateCreated: <%=server.dateCreated%>
server.lastAgentUpdate: <%=server.lastAgentUpdate%>
server.serverType: <%=server.serverType%>
server.osType: <%=server.osType%>
server.commType: <%=server.commType%>
server.managed: <%=server.managed%>
server.agentInstalled: <%=server.agentInstalled%>
server.toolsInstalled: <%=server.toolsInstalled%>
server.hostname: <%=server.hostname%>
server.domainName: <%=server.domainName%>
server.statusMessage: <%=server.statusMessage%>
server.maxStorage: <%=server.maxStorage%>
server.maxMemory: <%=server.maxMemory%>
server.maxCores: <%=server.maxCores%>
server.macAddress: <%=server.macAddress%>
server.serverVendor: <%=server.serverVendor%>
server.serverModel: <%=server.serverModel%>
server.serialNumber: <%=server.serialNumber%>
server.tags: <%=server.tags%>
server.configId: <%=server.configId%>
server.configGroup: <%=server.configGroup%>
server.configRole: <%=server.configRole%>
zone.name: <%=zone.name%>
zone.code: <%=zone.code%>
zone.location: <%=zone.location%>
zone.cloudTypeName: <%=zone.cloudTypeName%>
zone.cloudTypeCode: <%=zone.cloudTypeCode%>
zone.domainName: <%=zone.domainName%>
zone.scalePriority: <%=zone.scalePriority%>
zone.firewallEnabled: <%=zone.firewallEnabled%>
zone.regionCode: <%=zone.regionCode%>
zone.agentMode: <%=zone.agentMode%>
zone.datacenterId: <%=zone.datacenterId%>
group.code: <%=group.code%>
group.name: <%=group.name%>
group.location: <%=group.location%>
group.datacenterId: <%=group.datacenterId%>
instance {
        instanceTypeName,
        instanceTypeCode,
        provisionType,
        instanceVersion,
        plan,
        name,
        displayName,
        description,
        environmentPrefix,
        hostname,
        domainName,
        firewallEnabled,
        status,
        userStatus,
        networkLevel,
        instanceLevel,
        deployGroup,
        instanceContext,
        autoScale,
        statusMessage,
        expireDate,
        tags,
        storage,
        memory,
        cores,
        configId,
        configGroup,
        configRole
        containers:[],
        metadata:[],
        evars:[]
}
container {
        containerTypeName,
        containerTypeCode,
        containerTypeShortName,
        provisionType,
        dataPath,
        logsPath,
        configPath,
        planCode,
        dateCreated,
        status,
        environmentPrefix,
        version,
        image,
        internalHostname,
        hostname,
        domainName,
        storage,
        memory,
        cores,
        internalIp,
        externalIp,
        sshHost,
        hostMountPoint,
        configId,
        configGroup,
        configRole,
        serverId,
        server:{}
}
server {
        serverTypeName,
        serverTypeCode,
        parentServerId,
        plan,
        visibility,
        osTypeCode,
        sourceImageId,
        name,
        displayName,
        internalName,
        category,
        description
        internalId,
        externalId,
        platform,
        platformVersion,
        agentVersion,
        nodePackageVersion,
        sshHost,
        sshPort,
        sshUsername,
        consoleType,
        consoleHost,
        consolePort,
        consoleUsername,
        internalSshUsername,
        internalIp,
        externalIp,
        osDevice,
        dataDevice,
        lvmEnabled,
        apiKey,
        softwareRaid,
        status,
        powerState,
        dateCreated,
        lastAgentUpdate,
        serverType,
        osType,
        commType,
        managed,
        agentInstalled,
        toolsInstalled,
        hostname,
        domainName,
        statusMessage,
        maxStorage,
        maxMemory,
        maxCores,
        macAddress,
        serverVendor,
        serverModel,
        serialNumber,
        tags,
        configId,
        configGroup,
        configRole
        volumes {
                name
                id
                deviceName
                maxStorage
                unitNumber
                displayOrder
                rootVolume
        }
}
cloud {
        name,
        code,
        location,
        cloudTypeName,
        cloudTypeCode,
        domainName,
        scalePriority,
        firewallEnabled,
        regionCode,
        agentMode,
        datacenterId
}
group {
        code,
        name,
        location,
        datacenterId
}
customOptions {
        customOptions.fieldName
}

Important

Variables are case sensitive