AWS

Overview

AWS is the Amazon public cloud, offering a full range of services and features across the globe in various datacenters. AWS provides businesses with a flexible, highly scalable, and low-cost way to deliver a variety of services using open standard technologies as well as proprietary solutions. This section of documentation will help you get Morpheus and AWS connected to utilize the features below:

Features

• Instance, Service, Infrastructure Provisioning & Synchronization
• EKS Cluster Creation & Synchronization
• Morpheus Kubernetes, Docker & KVM Cluster Creation
• ELB Classic Load Balancer Creation & Synchronization
• ELB Application Load Balancer (ALB) Creation & Synchronization
• Security Group Creation & Synchronization
• Security Group Rule Creation & Synchronization
• Network Synchronization
• VPC Creation & Synchronization
• CloudFormation Provisioning & Resource Synchronization
• Terraform Provisioning & Resource Synchronization
• Pricing & Costing Synchronization
• MetaData Tag Creation & Synchronization
• S3 Bucket Creation & Synchronization
• Route53 Automation & Synchronization
• IAM Profile Synchronization and Assignment
• RDS Support
• Backups / Snapshots
• Migrations
• Auto Scaling
• Remote Console (SSH & RDP)
• Lifecycle Management and Resize
• Restore from Snapshots
• Elastic IP Assignment
• Network Pools
• Enhanced Invoice Costing

Requirements

AWS IAM Security Credentials

Access Key Secret Key Sufficient User Privileges (see MinimumIAMPolicies section for more info)

Security Group Configuration for Agent Install, Script Execution, and Remote Console Access

• Typical Inbound ports open from Morpheus Appliance: 22, 5985, 3389 (22 & 3389 required for Console. 22 & 5985 required for agent-less comms)
• Typical Outbound to Morpheus Appliance: 443 (Required for Agent install & comms)

Note

These are required for Morpheus agent install, communication, and remote console access for windows and linux. Other configurations, such as docker instances, will need the appropriate ports opened as well. Cloud-init Agent Install mode does not require incoming access for port 22.

Network(s)

IP assignment required for Agent install, Script Execution, and Console if the Morpheus Appliance is not able to communicate with AWS instances private ip’s.

Note

Each AWS Cloud in Morpheus is scoped to an AWS Region and VPC. Multiple AWS Clouds can be added and even grouped if different region and VPC combinations are needed. It’s also recommended you verify Security Groups are properly configured in all regions Morpheus Clouds will scope to.

Adding an AWS Cloud

1. Navigate to Infrastructure -> Clouds
2. Select + Create Cloud
3. Select AWS
4. Enter the following:

Cloud Configuration

NAME

Name of the Cloud in Morpheus

CODE

Unique code used for api/cli, automation and policies.

LOCATION

Description field for adding notes on the cloud, such as location.

VISIBILITY

For setting cloud permissions in a multi-tenant environment. Not applicable in single tenant environments.

TENANT

If Visibility is set to Private, select the Tenant the Cloud resources will assigned to.

ENABLED

When disabled, automatic Cloud sync is paused and the Cloud will not be selectable for provisioning.

AUTOMATICALLY POWER ON VMS

When enabled, Morpheus will maintain the expected powered on state of managed VM’s and power on any managed VM’s in the cloud that have been shut down for unknown reasons (not powered off by Morpheus) to ensure availability of services.

Note

When “AUTOMATICALLY POWER ON VMS” is enabled, the power state of managed VM’s should be maintained in Morpheus. This setting is not applicable to discovered/unmanaged resources.

Details

REGION

Select AWS Region for the Cloud

ACCESS KEY

Access Key ID from AWS IAM User Security Credentials.

SECRET KEY

Secret Access Key associate with the Access Key ID.

USE HOST IAM CREDENTIALS

Check to use use Host IAM Credentials

ROLE ARN

Supports security token service (STS) to AssumeRole by entering an AWS Role ARN

INVENTORY

Basic

Morpheus will sync information on all EC2 Instances in the selected VPC the IAM user has access to, including Name, IP Addresses, Platform Type, Power Status, and overall resources sizing for Storage, CPU and RAM, every 5 minutes. Inventoried EC2 Instances will appear as Unmanaged VM’s.

Full

In addition to the information synced from Basic Inventory level, Morpheus will gather Resource Utilization metrics for Memory, Storage and CPU utilization per VM.

Off

Existing EC2 Instances will not be inventoried

Note

Cloud Watch must be configured in AWS for Morpheus to collect Memory and Storage utilization metrics on inventoried EC2 instances.

USE VPC

Specify if the target account is using EC2-VPC or EC2-Classic Platform. In almost all cases, VPC should be selected, and then select the target VPC from the synced available VPC’s list, or All VPC’s.

5. The AWS cloud is ready to be added to a group and saved. Additional configuration options available:

IMAGE TRANSFER STORE

S3 bucket for Image transfers, required for migrations into AWS.

EBS ENCRYPTION

Enable or disable encrytion of EBS Volumes

COSTING KEY

For Gov Cloud pricing only, key for standard managing cost account

COSTING SECRET

For Gov Cloud pricing only, secret for standard managing cost account

Advanced Options

DOMAIN

Specify a default domain for instances provisioned to this Cloud.

SCALE PRIORITY

Only affects Docker Provisioning. Specifies the priority with which an instance will scale into the cloud. A lower priority number means this cloud integration will take scale precedence over other cloud integrations in the group.

APPLIANCE URL

Alternate Appliance url for scenarios when the default Appliance URL (configured in admin -> settings) is not reachable or resolvable for Instances provisioned in this cloud. The Appliance URL is used for Agent install and reporting.

TIME ZONE

Configures the time zone on provisioned VM’s if necessary.

DATACENTER ID

Used for differentiating pricing among multiple datacenters. Leave blank unless prices are properly configured.

NETWORK MODE

Unmanaged or select a Network Integration (NSX, ACI etc)

LOCAL FIREWALL

On or Off. Enable to managed Host and VM firewall/IP Table rules (linux only)

SECURITY SERVER

Security Server setting is for Security Service Integrations such as ACI

TRUST PROVIDER

Select Internal (Morpheus) or an existing Trust Provider Integration

STORAGE MODE

Single Disk, LVM or Clustered

BACKUP PROVIDER

Select Internal Backups (Morpheus) or a Backup Integration

REPLICATION PROVIDER

Sets the default Replication Provider for the Cloud. Select an existing Replication Provider Integration

GUIDANCE

Enable Guidance recommendations on cloud resources.

COSTING

Enable for Morpheus to sync Costing data from the Cloud provider, when available. If your organization utilizes reserved instances and you want to pull in related pricing data, select Costing and Reservations. If this is not relevant, select Costing to save money on additional calls to the AWS Cost Explorer API or similar service for other clouds.

DNS INTEGRATION

Records for instances provisioned in this cloud will be added to selected DNS integration.

SERVICE REGISTRY

Services for instances provisioned in this cloud will be added to selected Service Registry integration.

CONFIG MANAGEMENT

Select a Chef, Salt, Ansible or Puppet integration to be used with this Cloud.

CMDB

Select CMDB Integration to automatically update selected CMDB.

CHANGE MANAGEMENT

Select an existing Change Management Integration to set on the Cloud. ex: Cherwell

AGENT INSTALL MODE

• SSH / WINRM: Morpheus will use SSH or WINRM for Agent install.
• Cloud Init / Unattend (when available): (DEFAULT) Morpheus will utilize Cloud-Init or Cloudbase-Init for agent install when provisioning images with Cloud-Init/Cloudbase-Init installed. Morpheus will fall back on SSH or WINRM if cloud-init is not installed on the provisioned image. Morpheus will also add Agent installation to Windows unattend.xml data when performing Guest Customizations or utilizing syspreped images.

API PROXY

Set a proxy for outbound communication from the Morpheus Appliance to the Cloud endpoints. Proxies can be added in the Infrastructure -> Networks -> Proxies tab.

INSTALL AGENT

Enable to have Agent Installation on by default for all provisioning into this Cloud. Disable for Agent Installation to be off by default for all provisioning into this Cloud.

Provisioning Options

PROXY

Set a proxy for inbound communication from Instances to the Morpheus Appliance. Proxies can be added in the Infrastructure -> Networks -> Proxies tab.

Bypass Proxy for Appliance URL

Enable to bypass proxy settings (if added) for Morpheus Agent communication to the Appliance URL.

USER DATA (LINUX)

Add cloud-init user data. Morpheus 4.1.0 and earlier assumes bash syntax. Morpheus 4.1.1 and later supports all User Data formats. Refer to https://cloudinit.readthedocs.io/en/latest/topics/format.html for more information.

Enhanced Invoice Costing Configuration

In addition to standard AWS costing data (Enabled by editing the AWS cloud integration and setting the COSTING value to “Costing” or “Costing and Reservations”), Morpheus can utilize highly-granular data from AWS Costing and Utilization Reports (CUR) in its integration. Once enabled, this data can be consumed by accessing Invoice endpoints of Morpheus API and eventually through the UI after a future update. Continue on with this section to enable these reports in the AWS web console and configure the Morpheus cloud integration to work with this report data.

Begin by logging into the AWS Billing Console, then click Create report.

Include a name for your report and mark the box to “Include resource IDs”. Morpheus uses these resource IDs to map costs to various resources. Click Next.

On the following page, begin by identifying an S3 bucket to house reports. Click Configure near the top of the page and select an existing bucket or create a new one.

After identifying the bucket, you must mark the box to accept the default policy being applied to the bucket. Click Save.

The default policy applied to the bucket is below:

{
  "Version": "2008-10-17",
  "Id": "SomeID",
  "Statement": [
    {
      "Sid": "SomeStmtID",
      "Effect": "Allow",
      "Principal": {
        "Service": "billingreports.amazonaws.com"
      },
      "Action": [
        "s3:GetBucketAcl",
        "s3:GetBucketPolicy"
      ],
      "Resource": "arn:aws:s3:::bucket-name"
    },
    {
      "Sid": "SomeStmtID",
      "Effect": "Allow",
      "Principal": {
        "Service": "billingreports.amazonaws.com"
      },
      "Action": [
        "s3:PutObject"
      ],
      "Resource": "arn:aws:s3:::bucket-name/*"
    }
  ]
}

After choosing a bucket, accepting the default policy, and saving the change, you’re brought back to the report delivery page. By default, CUR reports are saved to a folder at the path my-report-name/date-folder. If this bucket already contains CUR reports, you may want to specify a prefix path in the “Report path prefix” field. Outside of this field, use the default values as shown in the screenshot below, then click Next.

On the following page, make your final review and click Review and Complete. Following this, you will see your newly configured report in the list of CUR report(s).

In addition, the AWS cloud user associated with the integration in Morpheus needs IAM policy permission to access Cost Explorer. Attach a policy like the one below to this cloud user:

{
  "Version": "2012-10-17",
  "Id": "SomeID",
  "Statement": [
    {
      "Sid": "SomeStmtID",
      "Effect": "Allow",
      "Action": [
        "ce:DescribeReportDefinitions",
        "ce:DescribeCostCategoryDefinition",
        "ce:ListCostCategoryDefinitions"
      ],
      "Resource": [
        "*"
      ]
    }
  ]
}

Note

If the Cost Explorer permissions are granted at the master account level, the user will see all costs for each member account; if granted at the member account, only the costs for that member account are available.

With the AWS console configuration steps complete, we can move back into Morpheus. Keep in mind it is only necessary to set up one AWS cloud for Costing since we process all records in the CUR report.

Once back in Morpheus, add or edit the relevant AWS cloud integration (Infrastructure > Clouds > + ADD OR click the pencil icon in the row for the chosen AWS integration). Expand the Advanced Options drawer and complete the following fields:

• COSTING BUCKET: The S3 bucket name
• COSTING REGION: The region the bucket was created in
• COSTING FOLDER: This is the report path prefix if you configured one earlier
• COSTING REPORT NAME: The name given to your CUR report
• COSTING KEY: If the IAM user for this AWS cloud integration does not have access to the S3 bucket with the CUR data, enter the AWS Key ID for an IAM user with access
• COSTING SECRET: If the IAM user for this AWS cloud integration does not have access to the S3 bucket with the CUR data, enter the AWS Secret Key for the IAM account whose Key ID you entered in the previous field
• LINKED ACCOUNT ID: If the IAM user for this AWS cloud integration does not have access to the S3 bucket with the CUR data, enter the AWS account number that the IAM user from the above step resides in

Note

If the AWS cloud account is a GovCloud account, enter the COSTING KEY, COSTING SECRET, and LINKED ACCOUNT ID for the master commercial account your GovCloud account is associated with.

Save changes to your cloud integration.

Important

It may take as long as one hour for Morpheus to process the next CUR report.

Minimum AWS IAM Policies

Below are the AWS IAM Permissions covering the minimum access for Morpheus applying to all resources and services.

See http://docs.aws.amazon.com/IAM/latest/UserGuide/reference_policies.html for more information.

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "VisualEditor0",
            "Effect": "Allow",
            "Action": [
                "autoscaling:DescribeAutoScalingGroups",
                "ce:*",
                "cloudwatch:GetMetricStatistics",
                "ec2:AllocateAddress",
                "ec2:AssignPrivateIpAddresses",
                "ec2:AssociateAddress",
                "ec2:AttachVolume",
                "ec2:AuthorizeSecurityGroupEgress",
                "ec2:AuthorizeSecurityGroupIngress",
                "ec2:CancelExportTask",
                "ec2:CancelImportTask",
                "ec2:CopyImage",
                "ec2:CopySnapshot",
                "ec2:CreateImage",
                "ec2:CreateInstanceExportTask",
                "ec2:CreateKeyPair",
                "ec2:CreateNetworkAcl",
                "ec2:CreateNetworkAclEntry",
                "ec2:CreateNetworkInterface",
                "ec2:CreateSecurityGroup",
                "ec2:CreateSnapshot",
                "ec2:CreateTags",
                "ec2:CreateVolume",
                "ec2:DeleteKeyPair",
                "ec2:DeleteNetworkAcl",
                "ec2:DeleteNetworkAclEntry",
                "ec2:DeleteNetworkInterface",
                "ec2:DeleteSecurityGroup",
                "ec2:DeleteSnapshot",
                "ec2:DeleteTags",
                "ec2:DeleteVolume",
                "ec2:DeregisterImage",
                "ec2:DescribeAccountAttributes",
                "ec2:DescribeAddresses",
                "ec2:DescribeAvailabilityZones",
                "ec2:DescribeClassicLinkInstances",
                "ec2:DescribeConversionTasks",
                "ec2:DescribeExportTasks",
                "ec2:DescribeImageAttribute",
                "ec2:DescribeImages",
                "ec2:DescribeImportImageTasks",
                "ec2:DescribeImportSnapshotTasks",
                "ec2:DescribeInstances",
                "ec2:DescribeInstanceStatus",
                "ec2:DescribeKeyPairs",
                "ec2:DescribeNetworkAcls",
                "ec2:DescribeNetworkInterfaceAttribute",
                "ec2:DescribeNetworkInterfaces",
                "ec2:DescribeRegions",
                "ec2:DescribeSecurityGroupReferences",
                "ec2:DescribeSecurityGroups",
                "ec2:DescribeSnapshotAttribute",
                "ec2:DescribeSnapshots",
                "ec2:DescribeStaleSecurityGroups",
                "ec2:DescribeSubnets",
                "ec2:DescribeTags",
                "ec2:DescribeVolumeAttribute",
                "ec2:DescribeVolumes",
                "ec2:DescribeVolumeStatus",
                "ec2:DescribeVpcAttribute",
                "ec2:DescribeVpcClassicLink",
                "ec2:DescribeVpcClassicLinkDnsSupport",
                "ec2:DescribeVpcEndpoints",
                "ec2:DescribeVpcEndpointServices",
                "ec2:DescribeVpcs",
                "ec2:DetachNetworkInterface",
                "ec2:DetachVolume",
                "ec2:DisassociateAddress",
                "ec2:ImportImage",
                "ec2:ImportInstance",
                "ec2:ImportKeyPair",
                "ec2:ImportSnapshot",
                "ec2:ImportVolume",
                "ec2:ModifyImageAttribute",
                "ec2:ModifyInstanceAttribute",
                "ec2:ModifyNetworkInterfaceAttribute",
                "ec2:ModifySnapshotAttribute",
                "ec2:ModifyVolumeAttribute",
                "ec2:RebootInstances",
                "ec2:RegisterImage",
                "ec2:ReleaseAddress",
                "ec2:ReplaceNetworkAclAssociation",
                "ec2:ReplaceNetworkAclEntry",
                "ec2:ResetImageAttribute",
                "ec2:ResetInstanceAttribute",
                "ec2:ResetNetworkInterfaceAttribute",
                "ec2:ResetSnapshotAttribute",
                "ec2:RevokeSecurityGroupEgress",
                "ec2:RevokeSecurityGroupIngress",
                "ec2:RunInstances",
                "ec2:StartInstances",
                "ec2:StopInstances",
                "ec2:TerminateInstances",
                "ec2:UnassignPrivateIpAddresses",
                "eks:*",
                "iam:ListGroups",
                "iam:ListInstanceProfiles",
                "iam:ListRoles",
                "rds:AddRoleToDBCluster",
                "rds:AddTagsToResource",
                "rds:ApplyPendingMaintenanceAction",
                "rds:AuthorizeDBSecurityGroupIngress",
                "rds:CopyDBClusterSnapshot",
                "rds:CopyDBParameterGroup",
                "rds:CopyDBSnapshot",
                "rds:CreateDBCluster",
                "rds:CreateDBClusterSnapshot",
                "rds:CreateDBInstance",
                "rds:CreateDBInstanceReadReplica",
                "rds:CreateDBSecurityGroup",
                "rds:CreateDBSnapshot",
                "rds:DeleteDBCluster",
                "rds:DeleteDBInstance",
                "rds:DeleteDBSecurityGroup",
                "rds:DeleteDBSnapshot",
                "rds:DescribeAccountAttributes",
                "rds:DescribeCertificates",
                "rds:DescribeDBClusterParameterGroups",
                "rds:DescribeDBClusterParameters",
                "rds:DescribeDBClusters",
                "rds:DescribeDBClusterSnapshotAttributes",
                "rds:DescribeDBClusterSnapshots",
                "rds:DescribeDBEngineVersions",
                "rds:DescribeDBInstances",
                "rds:DescribeDBLogFiles",
                "rds:DescribeDBParameterGroups",
                "rds:DescribeDBParameters",
                "rds:DescribeDBSecurityGroups",
                "rds:DescribeDBSnapshotAttributes",
                "rds:DescribeDBSnapshots",
                "rds:DescribeDBSubnetGroups",
                "rds:DescribeEngineDefaultClusterParameters",
                "rds:DescribeEngineDefaultParameters",
                "rds:DescribeEventCategories",
                "rds:DescribeEvents",
                "rds:DescribeOptionGroupOptions",
                "rds:DescribeOptionGroups",
                "rds:DescribeOrderableDBInstanceOptions",
                "rds:ListTagsForResource",
                "rds:ModifyDBCluster",
                "rds:ModifyDBClusterParameterGroup",
                "rds:ModifyDBClusterSnapshotAttribute",
                "rds:ModifyDBInstance",
                "rds:ModifyDBParameterGroup",
                "rds:ModifyDBSnapshotAttribute",
                "rds:PromoteReadReplica",
                "rds:RebootDBInstance",
                "rds:RemoveTagsFromResource",
                "rds:RestoreDBClusterFromSnapshot",
                "rds:RestoreDBClusterToPointInTime",
                "rds:RestoreDBInstanceFromDBSnapshot",
                "rds:RestoreDBInstanceToPointInTime",
                "rds:RevokeDBSecurityGroupIngress",
                "route53:GetHostedZone",
                "route53:ListHostedZones",
                "route53:ListResourceRecordSets",
                "s3:AbortMultipartUpload",
                "s3:CreateBucket",
                "s3:DeleteBucket",
                "s3:DeleteObject",
                "s3:DeleteObjectVersion",
                "s3:GetBucketLocation",
                "s3:GetObject",
                "s3:GetObjectVersion",
                "s3:ListAllMyBuckets",
                "s3:ListBucket",
                "s3:ListBucketMultipartUploads",
                "s3:ListBucketVersions",
                "s3:ListMultipartUploadParts",
                "s3:PutObject"
            ],
            "Resource": "*"
        }
    ]
}

If you need to limit actions based on filters you have to pull out the action and put it in a resource based policy since not all the actions support resource filters.

See http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ec2-supported-iam-actions-resources.html for more info on limiting resources by filter.

Resource filter example:

{
  "Effect": "Allow",
  "Action": [
   "ec2:StopInstances",
   "ec2:StartInstances"
  ],
  "Resource": *
 },
 {
  "Effect": "Allow",
  "Action": "ec2:TerminateInstances",
  "Resource": "arn:aws:ec2:us-east-1:123456789012:instance/*",
  "Condition": {
    "StringEquals": {
       "ec2:ResourceTag/purpose": "test"
     }
   }
 }