ECS

ECS - Amazon Elastic Container Service

https://docs.aws.amazon.com/AmazonECS/latest/developerguide/Welcome.html

Really good blog post about how ECS manages CPU and memory resources: https://aws.amazon.com/blogs/containers/how-amazon-ecs-manages-cpu-and-memory-resources/

AWS provides an Optimized AMI to run ECS on it (recommended), which simplifies the setup steps needed to run ECS on a normal instance, but you can use your own AMI to run the ECS Agent and have your customized ECS AMI.
- Doesn't apply to Fargate, as Fargate abstracts the Container Instances outside the customer control.

Container Instance States:

Active

- Container instance can accept run task requests.
- Active container instance is the one that will be used to place tasks scheduled by a Scheduler or the user.

Draining

- https://docs.aws.amazon.com/AmazonECS/latest/developerguide/container-instance-draining.html
- Container instance draining enables you to remove a container instance from a cluster without impacting tasks in your cluster.
- When you set a container instance to DRAINING, Amazon ECS prevents new tasks from being scheduled for placement on the container instance. Service tasks on the draining container instance that are in the PENDING state are stopped immediately. If there are container instances in the cluster that are available, replacement service tasks are started on them.
- Service tasks on the container instance that are in the RUNNING state are stopped and replaced according to the service's deployment configuration parameters, minimumHealthyPercent and maximumPercent.
- To gracefully stop tasks on an instance.
- No new Tasks are placed on the instance.
- Active Services are terminated and relocated to other instances.

Inactive

- If you deregister or terminate a container instance, the container instance status changes to INACTIVE immediately.

Bootstrapping

In general, bootstrapping refers to a self-starting process that is supposed to proceed without any external input.

Container instances can be bootstrapped to run configurations on startup by leveraging supported user-data formats: https://docs.aws.amazon.com/AmazonECS/latest/developerguide/bootstrap_container_instance.html

User Data script
- User-data scripts run only during the first boot cycle.
- Mainly used to pass configuration options to the ECS Agent. The most common example is to join an instance to the cluster or even pull a pre-configured ecs.config file from an S3 bucket.
- The ECS agent starts later in the boot process, hence the shell-script passed through user data will be executed before the ECS agent starts.
- The Cloud-init service will save the scripts passed through user data in directory /var/lib/cloud/instances/<instance ID>/scritps directory. These scripts are executed in the final stage of the boot.
- Any supported configuration option that needs to be applied to ECS agent can be passed in this way.
- Docker daemon is started before the shell-script's in uesr-data is executed.
- Example of using user data to configure ECS agent:
  - Here we are passing the ECS cluster name and ECS agent log level configuration. When the ECS agent starts, it will read the cluster name from this file and registers itself with the specified cluster.

#!/bin/bash
echo "ECS_CLUSTER=MyCluster" >> /etc/ecs/ecs.config
echo "ECS_LOGLEVEL=debug" >> /etc/ecs/ecs.config

Cloud-boothook
- Cloud-boothook is the earliest "hook" available. User-data scripts run much later in the process !!!!!!!!
- Cloud boothook runs on every instance boot and its frequency can be controlled by cloud-init-per to control the frequency of execution.
- To configure Docker daemon before the daemon starts, use this. The content in boothook is saved in /var/lib/cloud directory and is executed immediately.
- If we check the order of execution in /etc/init we can see that cloud-init is executed before the docker service is started. Hence passing Docker configuration through boothook will make sure the Docker configuration file is updated as soon as cloud-init detects it and before Docker daemon is started.
- A common example of passing configuration settings to the docker daemon would be the modification of the base device size. By default, docker limits the size of images and containers to 10GB. There may be some users that will need to increase the dm.basesize

#cloud-boothook
cloud-init-per once docker_options echo
'OPTIONS="${OPTIONS} --storage-opt
dm.basesize=20G"' >> /etc/sysconfig/docker

MIME multipart
- This format allows you to specify more than one type of data. I.e. user-data script and cloud-boothook type. Example where both the ECS container agent is configured to register the instance to a cluster and also configures the base device size to 20GB: See Specifying Multiple User Data Blocks Using a MIME Multi Part Archive here
- To configure Docker daemon you use cloud-boothook, and to configure ecs agent you would want to use shell script format.
- In this example, we can see a MIME multi part file consists "Content-Type" attribute with value "multipart/mixed", which indicates that the file consists of multiple user data blocks. There are boundaries indicating the start and end of the whole MIME file and also boundaries indicating the start and end of each block within the file.
- Within each block there is again a "Content-Type" attribute indicating what type of user data format it represents. In the example we can see that there is a Cloud-boothook format and shell script fromat combined together.
- MIME multi part file example shown below will configure Docker daemon to set container base size to 200GB and it also sets ECS agent configuration options.

Content-Type: multipart/mixed;
boundary="==BOUNDARY=="
MIME-Version: 1.0
--==BOUNDARY==
Content-Type: text/cloud-boothook; charset="us-ascii"
# Set Docker Daemon options
cloud-init-per once docker_options echo 'Options="${OPTIONS} --storage-opt dm.basesize=200GB"'>> /etc/sysconfig/docker
--==BOUNDARY==
Content-Type: text/x-shellscript; charset="us-ascii"
#!/bin/bash
# Set any ECS agent configuration options
echo ECS_CLUSTER=MyCluster >> /etc/ecs/ecs.config
echo ECS_LOGLEVEL=debug >> /etc/ecs/ecs.config
--==BOUNDARY==--

Click here to learn how to bootstrap an ECS container instance to mount an existing EFS volume to the instance.

Container agent

Application running on ECS Container instances that performs the "bridge" between the local Docker Daemon API and the ECS Endpoint.

Scheduler

ECS mechanism that determines the task placement on specific container instances based on a set algorithm that takes into account several variables.

On ECS you can use the ECS Scheduler, but as we provide an APi to start/run tasks, you can write your own scheduler in any language supported by the AWS SDK.
Also you can use the BLOX open source scheduler project with ECS.
Mesos also has a driver provided by AWS for scheduling tasks.

How to create a custom scheduler for ECS: https://aws.amazon.com/blogs/compute/how-to-create-a-custom-scheduler-for-amazon-ecs/

Task Definition

JSON formatted template describing what the associated Task will run. Contains one or more container specifications Docker run parameters. Also defines the Networking Mode, Task Placement constraints and volume mounts associated with the task.

https://docs.aws.amazon.com/AmazonECS/latest/developerguide/task_definitions.html

Before you can run a task on your ECS cluster, you must register a task definition. Task definitions are lists of containers grouped together.

Aside from using the console, you can also register a Task Definition by sending using a JSON formatted file or input to the AWS CLI command.

Task

Good Read: https://aws.amazon.com/blogs/containers/how-amazon-ecs-manages-cpu-and-memory-resources/

In ECS, the basic unit of a deployment is a task, a logical construct that models one or more containers. This means that the ECS APIs operate on tasks rather than individual containers. In ECS, you can’t run a container: rather, you run a task, which, in turns, run your container(s). A task contains (no pun intended) one or more containers. An instantiation of a Task Definition which will launch the container or containers listed within the Task Definition.

In other words, a task is an instantiation of a Task Definition which will launch the container or containers listed within the Task Definition.

Placement can be defined by the scheduler or specified by the API.
Ideal for short live tasks or batch processing.

Scheduled Tasks

https://docs.aws.amazon.com/AmazonECS/latest/developerguide/scheduling_tasks.html

Cron like tasks
CloudWatch Events

ECS Scheduled Tasks provide the avility to run ECS tasks on a regular, scheduled basis and in response to CloudWatch Events Rules. This makes it easier to launch Docker containers that you need to run only at certain times.

ECS scheduled tasks can be run on a specific interval called, rate, or on a ,ore detailed pattern using a cron expression.
Scheduled tasks are managed through the CloudWatch Events service events API and not ECS API calls.
CloudWatch Events also allows you to run ECS Tasks as a target of event based rules.

Rate: the minimum amount of time allowed to schedule a Task is 1 minute.

Task definitions that use the awsvpc network mode are not supported for scheduled tasks.

Common Use cases:

Batch processing: launch a Task to process a specific file that is uploaded into a S3 bucket everyday at the same time.
Retrieving log files in a most cost-effective way: Running task that are long and not suitable for Lambda functions, like gathering and processing big logs from an RDS Data Base, using Spot Container Instance to make it cost-effective

How to Run Scheduled Tasks:

Before you can use ECS Scheduled Tasks you need to be sure you have the right permissions in place, as the CloudWatch Events service needs permission to run ECS tasks on your behalf.
These permissions are provided by the CloudWatch Events IAM role, called the ecsEventsRole. A new specific IAM role is automatically created in the AWS Management Console when a scheduled task is configured, but it needs to be created manually if you want to configure the same role for all your Scheduled Tasks or if you want to run them programmatically. Click here for more info.

Schedule Expressions for Rules:
For a rule to be triggered every day at 12:00pm UTC:

cron(0 12 * * ? *)

For a rule to be triggered every day, at 5 and 35 minutes past 2:00pm UTC:

cron(5,35 14 * * ? *)

For a rule to be triggered at 09:10am UTC on the last Friday of each month during the years 2013 to 2018:

cron(10 9 ? * 6L 2013-2018)

You can't specify the day-of-month and day-of-week fields in the same cron expression.
If you specify a value or an asterisk in one of the fields, you must use a question mark in the other.
Also, cron expressions that lead to rates faster than 1 minute are not supported.

For a rule to be triggered every 5 minutes:

rate(5 minutes)

For a rule to be triggered every 1 hour:

rate(1 hour)

For a rule to be triggered every 1 day:

rate(1 day)

On rate expressions, if the value is equal to 1, then the unit must be singular. Similarly for values greater than 1, the unit must be plural.

Troubleshooting ECS Scheduled Tasks:

Troubleshooting ECS scheduled tasks starts with checking the state of the task, and determining whether the task is in running or in stopped state.
If the Task should be running but is stopped or failing to start, the troubleshooting is done on ECS side.
If the task was not running on the ECS cluster, then it needs to be investigated why it was not invoked by CloudWatch events. So check:
- Failed Invocations
- Invocations metrics.

One of the most common causes for a Failed Invocation Task is the lack of permissions. If the scheduled task requires a specific task execution role or a task role override, then the iam:PassRole permissions need to be added to the CloudWatch Events IAM role, for the CloudWatch Event rule to be able to "forward" the role to the ECS Scheduled Task.

Solution: A new policy needs to be created and attached to the CloudWatch Events IAM role. The policy needs to specify the full ARN of the task execution role or task role override. This is an example of a policy for an ECS Scheduled Task that needs a specific task role.

Other Common issues include:

Cron Validation Error
- Scheduled task using a cron expression fails with a validation error on the scheduled tasks syntax, even though the syntax is correct according to documentation, because it is not possible to specify the day-of-month and day-of-week fields in the same cron expression. To solve this, if a value or a * is specified in one of the fields, a question must (?) must be used in the other.
Task Definition Misconfiguration
- Task definitions that use the awsvpc network mode are not supported for scheduled tasks. For this reason, the Fargate launch type is not supported either. To solve this a different network mode like "Bridge" should be selected.
The rule did not trigger at the expected time
- CloudWatch Events doesn't support setting an exact start time when you create a rule to run every time period. The count down to run time begins as soon as you create the rule.

Pending Task

PENDING tasks count = the number of tasks scheduled to launch which are yet to receive the task result state from ECS agent.

ECS Scheduler will examine the cluster for task placement.

ECS service increments PENDING task count after acknowledgement, clears PENDING task count after receiving tasks result state.

The ECS Scheduler will MAINTAIN the PENDING task count on the following scenarios:

The ECS agent received the scheduled task and task launch was already initiated.
The ECS agent is experiencing an issue while updating the task result state to Scheduler due to issues local to the instance.
The ECS agent is experiencing a problem in the middle of task launch and lost connectivity with Scheduler due to external factors.
The ECS agent is in the process of stopping the existing tasks, and a new task will not start until the stopping task has fully stopped.

The ECS Scheduler will CLEAR the PENDING task count on following scenarios:

ECS agent successfully launched the task and updated the Scheduler with task in RUNNING state.
The container instance where the task scheduled to launch is terminated.

If the ECS Schedyler continues to show PENDING status for a while, then it could indicate a problem at the instance side, whcih impacts the ECS agent to update the ECS scheduler with result state of the task launch. The following are common areas that could affect the ECS agent in updating the task result state to the ECS scheduler:

ECS Instance
- Network congestion
  - The ECS agent running on a contianer instance depends on outbound internet connectivity over HTTPS to perform following on container instance.
  - The following are issues that could affect the container instance Network connectivity in reaching out to the ECS service endpoint or dependent service like ECR:
    - NAT Instance - ENI bandiwdth or CPU credits
    - NAT gatway - ENI bandwidth
    - VPC Endpoint - ENI bandwidth
    - VPN Gateway issues
    - HTTP Proxy issues.
- Resource issues
  - The ECS agent and task requires system resources for its execution, and if the container instance experiences any resource challenges in the middle of ECS agent task placement, that will delay the task launch or agent polling operation to scheudler. The scheduler will keep the positive value in PENDING task until it receives task launch state information form the ECS agent.
  - The following are the issues that could affect the resource allocation:
    - Container instances (T2) CPU utilization exceeds CPU credit balance for a longer duration or ran out of CPU credits. Analyzing the following CW instance metrics will be helpful in confirming the issue:
      - CPUUtilization
      - CPUCreditUsage
      - CPUCreditBalance
    - Container Instances (GP) disk IOPS utilization exceeds IO credit balance for a longer duration or ran out of IO credits. CW metrics to look at:
      - BurstBalance
      - VolumeReadOps
      - VolumeWriteOps
      - VolumeThroughputPercentage
ECS agent
- IAM permission
  - The "ecs:Submit*" IAM permission is crucial for ECS agent to send a result state of task launch to scheduler, to avoid task hung up in PENDING status.
  - You will either "error - access denied" stuff in the ECS agent logs
- Agent Status Issues
  - If the ECS agent disconnects and fails to reconnect to cluster/scheduler in the middle of task launch that will cause a PENDING task issue. Several reasons that might impact the ECS agent status and connectivity:
    - The ECS agent service status on container instance is not UP, perhaps due to crash or stopped by user.
    - ECS agent may be impacted by a bug affecting a current or old version. (Review github for info)
    - Network issues
- Configuration Issues
  - If Cx tweaks the configuration variable 'ECS_CONTAINER_STOP_TIMEOUT' to a higher value in ECS agent configuration that may lead to PENDING tasks issue.
  - The 'ECS_CONTAINER_STOP_TIMEOUT' configuration instructs Docker service to wait for the given period to allow the container to exist gracefully and 'SIGKILL' signal will be sent if the cotnainer does not exit within the stop timeout period.
  - When ECS agent STOPS the task, the equivalent of docker stop is issued to the containers running in the task. It results in a SIGTERM and a default 30-second timeout, after which SIGKILL is sent and the containers are forcibly stopped. If the container handles the SIGTERM gracefully and exits within 30 seconds from receiving it, no SIGKILL is sent.
  - Example: if we use 30 minutes as the stop timeout and assume container experiences issue to exit gracefully. ECS agent has to wait up to 30 minutes before applying SIGKILL and to update the result of 'container stop' to ECS scheduler. ECS scheduler will keep the tasks scheduled on the container instance in PENDING status until it receives an update from ECS agent.
Image registry slowness.
- The ECS agent will run a task on container instance once it is received the schedule from the ECS service. Task launch depends on image pull from ECR or DockerHub or Private Image registry.
- The network congestion between container instance to image repository will slow down the image download. The default 'net.ipv4.tcp_keepalive_time' is 7200 seconds so container to image repository connection will continue to be active up to a maximum of 2 hours and this will delay the task launch. The ECS service will keep the progressing task in PENDING status until docker run completes or timed out.
- The following are may affect image download speed:
  - Image registry performance issues
  - S3 performance issues (ECR depends on S3)
  - VPC Endpoint Issues - ENI bandwidth

Click here to learn various ways on how to find PENDING tasks count per cluster, service and instance.

Service

Allows you to run and maintain a specified number of instances of a task definition. Are capable of integrating with ELB/ALB and of being scaled up or down based on specific CloudWatch Metrics by leveraging Application Auto Scaling.

Container

A Docker container that was launched as part of a task.

ECS Optimized AMI

It's optimized, meaning there's a minimal number of packages installed, and it is an EC2 image preconfigured and tested by AWS engineers.

ECS Optimized AMI components:

Minimalized Amazon Linux AMI
ECS Agent
Docker
ECS-init
Pre-configured Storage

Launching options to launch an instance that uses the ECS Optimized AMI:

First Run Wizard
EC2 Console (via the Marketplace and Community AMIs)
AWS CLI/SDK
Cloudformation
OpsWorks

Storage:
ECS Optimized AMIs come with 30 GB of storage by default. Older AMIs will differe on how the sotrage is configured.

AMI version 2015.09c and older had a single 30 GB EBS volume.
Ami versions 2015.09d and newer split it ito two EBS volumes:
- 8GB volume for root
- 22GB volume for Docker image and metadata.

Container Agent

ECS container agent:

Is an open source application and programmed in Go.

ECS agent is essentially what allows the instances themselves to communicate with the cluster, and it processes ECS commands and turns them into Docker commands. This instructs the EC2 instances to start and stop containers, and monitor available resources that it has across the cluster.

Running the ECS container agent outside of Amazon EC2 is not supported.

ECS agent is published on Github and you can click on this link for more information: https://github.com/aws/amazon-ecs-agent/

Click here for information about Agent Debugging.

Included in ECS Optimized AMI.

ECS Agent states:

Connected
- ECS Agent CONNECTED status indicates the agent is able to communicate with the ECS Scheduler on AWS.
Disconnected
- ECS Agent DISCONNECTED status indicates there is a problem with the agent. It can be networking or permissions generally.
- Agents periodically disconnects and reconnects (several times per hour) as a part of its normal operation. This is done to ensure that the connectivity can be established, and that there are no issues with the connectivity.
  - If an agent is kept in disconnected state for a long period, this indicates a problem with the agent, instance, permissions, connectivity, or the service.

Agent Configuration

The Amazon ECS container agent supports a number of configuration options, most of which should be set through environment variables. The following environment variables are available, and all of them are optional.

The ECS Agent will search for configuration options from the /etc/ecs/ecs.config file.

Example: To register an EC2 instance with ECS cluster:

Attach IAM role to instance
add the attribute ECS_CLUSTER to the file:

$ cat /etc/ecs/ecs.config 
ECS_CLUSTER=FirstCluster
ECS_LOGLEVEL=debug

Here are the available parameters: https://docs.aws.amazon.com/AmazonECS/latest/developerguide/ecs-agent-config.html#ecs-agent-availparam

Amazon/Linux AMI

On Amazon or other Linux AMIs, ECS container agent will most likely be run in a Docker container on an EC2 instance.

The ECS container agent may also be run outside of a Docker container as a Go binary. However, this is not recommended for production on Linux, but it can be useful for development or easier integration with your local Go tools.

Windows Server 2016

On Windows Server 2016, the container agent runs as a service on the host. Unlike Linux, the agent may not run inside a container as it uses the host's registry and the named pipe at \\.\pipe\docker_engine to communicate with the Docker daemon.

API

Also, ECS container agent provides an API for gathering details about the container instance.

You can use the curl command from within the container instance to query the Amazon ECS container agent (port 51678) and return container instance metadata or task information.

Agent Debugging

There are logs available from the Amazon ECS container agent and the ecs-init service that controls the state of the agent (start/stop) on the container instance.

You can increase the verbosity of the container agent logs by setting ECS_LOGLEVEL=debug and restarting the container agent.

If you are unsure how to collect all of the various logs on your container instances, you can use the Amazon ECS log collector: https://docs.aws.amazon.com/AmazonECS/latest/developerguide/ecs-logs-collector.html

Click here for the log file locations.

Updating the container agent

Updating the Amazon ECS Container Agent https://docs.aws.amazon.com/AmazonECS/latest/developerguide/ecs-agent-update.html

Checking Your Amazon ECS Container Agent Version https://docs.aws.amazon.com/AmazonECS/latest/developerguide/ecs-agent-update.html#checking_agent_version

Updating the Amazon ECS Container Agent on an Amazon ECS-optimized AMI https://docs.aws.amazon.com/AmazonECS/latest/developerguide/agent-update-ecs-ami.html

Manually Updating the Amazon ECS Container Agent (for Non-Amazon ECS-Optimized AMIs) https://docs.aws.amazon.com/AmazonECS/latest/developerguide/manually_update_agent.html

How to stop/start the ecs agent

$ sudo stop ecs
$ sudo start ecs

# OR

$ sudo systemctl stop ecs
$ sudo systemctl start ecs

How to check ECS Agent status and connectivity

Commands to verify ECS Agent Status and Connectivity:

$ sudo status ecs
$ sudo docker ps -filter name=ecs-agent

Commands to view ECS container instance metadata & running tasks info:

$ curl http://localhost:51678/v1/metadata
$ curl http://localhost:51678/v1/tasks

How to verify the Instance to ECS/TCS/ACS endpoing connectivity:

$ telnet ecs.<region>.amazonaws.com 443
$ telnet ecs-t-1.<region>.amazonaws.com 443
$ telnet ecs-a-1.<region>.amazonaws.com 443

The ecs-init service

ecs-init service

Controls the starting and stopping of the agent at boot and shutdown.
Also handles starting the agent if it crashes.

Amazon ECS RPM

The Amazon ECS RPM is packaged for RPM-based systems that utilize Upstart as the init system.
- The upstart script installed by the ECS container service RPM can be started or stopped like so: see here.
RPM Logs info:
- Logs from the RPM are available at /var/log/ecs/ecs-init.log, while logs from the Amazon ECS container agent are available at /var/log/ecs/ecs-agent.log.
- The Amazon ECS RPM makes the Amazon ECS Container Agent introspection endpoint available at http://127.0.0.1:51678/v1 or http://localhost:51678/v1
- Configuration for the Amazon ECS container agent is read from /etc/ecs/ecs.config.

Ecs-init Github source code: https://github.com/aws/amazon-ecs-init

Configuration and log files

Configuration for the Amazon ECS container agent is read from /etc/ecs/ecs.config.

Logs from the RPM are available at /var/log/ecs/ecs-init.log

Logs from the Amazon ECS container agent are available at /var/log/ecs/ecs-agent.log.

The Amazon ECS RPM makes the Amazon ECS Container Agent introspection endpoint available at http://172.0.0.1:51678/v1.

For Windows: both ECS agent and ecs-init logs are located at: C:\ProgramData\Amazon\ECS\log

The ECS CLI configuration file: ~/.ecs/config

The ECS CLI credentials file: ~/.ecs/credentials

The checkpoint file: /var/lib/ecs/data/ecs_agent_data.json or /var/lib/ecs/data/agent.db.
- Delete this file if you have made changes to the /etc/ecs/ecs.config, and you want to restart the agent.

Autoscaling in ECS

There are two levels of autoscaling in ECS:

Instance level: autoscaling groups
Container level: Application autoscaling

Scale out is not a problem, as you add instances and add tasks.

Scale in: ASG will stop any instance, by default, without ECS knowing which one is being terminated.

Solution: Implement instance draining in Lambda, allowing ECS to stop the tasks before the instance is terminated.

Recommneded, but not mandatory: Adapt alarms so Application Autoscaling Scale In first, then the Instances.

https://aws.amazon.com/blogs/compute/how-to-automate-container-instance-draining-in-amazon-ecs/

Load Balancers in ECS

Dynamic Port Mapping:

Allows you to run more than one task on the same host.
Supported by ALB and NLB only.
HostPort on the task definition must be set to '0'.

https://aws.amazon.com/premiumsupport/knowledge-center/dynamic-port-mapping-ecs/

Classic support:
Classic support more than 1 port per service on the load balancer. You can map as many port as you want on each container (http and https on the same container for example).

Support Load balancers

ALB
NLB
CLB

Application and Network Support Dynamic Port

You can only map 1 port per service/target group.
With a NLB, you cannot register instances by instance ID if they have the following instance types: C1, CC1, CC2, CG1, CG2, CR1, CS1, G1, G2, HI1, HS1, M1, M2, M3, and T1. You can register instances of these types by IP address instead.

Task Networking (awsvpc)

https://docs.aws.amazon.com/AmazonECS/latest/developerguide/task-networking.html

ENI is attached to the Task:

One ENI per task, not per container; all the containers in the task share the same ENI.

Task Networking...

Improves security - Security groups are attached to the ENI not the instance.
Only supported on Optimized ECS AMI or Amazon Linux with ecs-init >= 1.15.0-4
Linked Service Role is required -

$ aws iam create-service-linked-role --aws-service-name ecs.amazonaws.com

Task networking considerations: https://docs.aws.amazon.com/AmazonECS/latest/developerguide/task-networking.html#task-networking-considerations

The awsvpc network mode does not provide task elastic network interfaces with public IP addresses. You need to use NAT or load balancer to allow the task to access resources outside the VPC. Load balancer is a better choice overall as it proves to provide a better scaling option.
awsvpc is only supported on Linux (ECS Optimized AMI or Amazon Linux with ecs-init >= 1.15.0-4)
The limit of ENI depends on the EC2 instance in use, i.e. c4.large can have up to 3 ENI.

Some considierations with AWSVPC mode:

No dynamic-port mapping
No Public IP with awsvpc mode in EC2 Launch Type

ENI Limit per instance:

Instances have ENI limits
- Primary ENI counts as 1
Example: c5.large -> 2 ENIs in addition to primary ENI
- Only 2 tasks can be placed on it.
awsvpcTrunking to rescue
- With awsvpcTrunking, c5.large instance can have up to 10 tasks
- New ENI Limit is 12
  - 1 Primary ENI
  - 1 Trunk ENI
More info: https://docs.aws.amazon.com/AmazonECS/latest/developerguide/container-instance-eni.html#eni-trunking-supported-instance-types

Container Storage

You can create volumes in a Docker container to store and/or share information with other containers.

In the Linux ECS Optimized AMI the volumes will take space on the xvda volume and not the xvdcz
On Winodws they will be created inside C: drive

By default volumes in a container are managed by Docker and are empty on start and removed on container removal.

To use persistent storage you may use EBS or EFS

Use EFS to store the data.
- Tutorial: Using Amazon EFS File Systems with Amazon ECS: https://docs.aws.amazon.com/AmazonECS/latest/developerguide/tutorial-efs-volumes.html
For EBS you can add an additional volume to the instance, mount those volumes before Docker is started, and use volumes to store the data on the instance file system.
Docker volume driver plugins are now supported by ECS.

ECS CLI

ECS CLI provides the ability to create, update and monitor clusters and tasks.

ECS CLI can be used to push/pull Docker images from Amazon ECR repository.

ECS CLI supports Docker Compose with limits. Docker compose is used to define and run multi-container applications. It is used as an alternative to AWS Management console.

This section will walk you through the tutorial:

Creating a Cluster with an EC2 Task Using the Amazon ECS CLI

https://docs.aws.amazon.com/AmazonECS/latest/developerguide/ecs-cli-tutorial-ec2.html

How to install the ECS CLI

https://docs.aws.amazon.com/AmazonECS/latest/developerguide/ECS_CLI_installation.html

How to configure the Amazon ECS CLI

https://docs.aws.amazon.com/AmazonECS/latest/developerguide/ECS_CLI_Configuration.html

The Amazon ECS CLI requires some basic configuration information before you can use it, such as:

AWS credentials
AWS region in which to create your cluster
Amazon ECS cluster name

These configurations are passed on using $ ecs-cli configure command.

The settings are then stored in the ~/.ecs/config file.

There are three ways to set the AWS Credentials to be used with the $ ecs-cli configure command:

You can set the environment variable AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY on your development computer, then when you call the ecs-cli command, those values will be passed to the command.
You can use a named profile from the ~/.aws/credentials, both the access key and secret key will then be passed to the ecs-cli command.
The last method is to pass the access key and secret key as arguments to the ecs-cli command using the --access-key and --secret-key options.

How to set environment variables: https://docs.aws.amazon.com/cli/latest/userguide/cli-configure-envvars.html#envvars-set

How to configure:

$ ecs-cli configure profile --profile-name main_ecs --access-key $AWS_ACCESS_KEY_ID --secret-key $AWS_SECRET_ACCESS_KEY
$ ecs-cli configure --cluster SecondCluster --default-launch-type EC2 --region us-east-1 --config-name SecondCluster_config

ECS CLI configuration file: ~/.ecs/config

$ cat ~/.ecs/config
version: v1
default: SecondCluster_config
clusters:
  SecondCluster_config:
    cluster: SecondCluster
    region: us-east-1
    default_launch_type: EC2

ECS CLI credentials file: ~/.ecs/credentials

$ cat ~/.ecs/credentials 
version: v1
default: main_ecs
ecs_profiles:
  main_ecs:
    aws_access_key_id: AKIAZOEEYIZHRY376S4W
    aws_secret_access_key: 4B09uH11KLSWEtVz8aDYWiNpx8ZlVuRcihu8Xod2

How to create a cluster

$ ecs-cli up --keypair [your_key] --capability-iam --size 2 --instance-type t2.large
INFO[0000] Using recommended Amazon Linux 2 AMI with ECS Agent 1.40.0 and Docker version 19.03.6-ce 
INFO[0000] Created cluster                               cluster=SecondCluster region=us-east-1
INFO[0000] Waiting for your cluster resources to be created... 
INFO[0000] Cloudformation stack status                   stackStatus=CREATE_IN_PROGRESS
INFO[0061] Cloudformation stack status                   stackStatus=CREATE_IN_PROGRESS
INFO[0121] Cloudformation stack status                   stackStatus=CREATE_IN_PROGRESS
VPC created: vpc-00e137a44a3d76999
Security Group created: sg-07f3954ba17662632
Subnet created: subnet-0f250960e3e51e10a
Subnet created: subnet-021053587cc96f053
Cluster creation succeeded.

2 instances were launched from the command above.
By default port 80 would be open on your new security group, if you prefer a different port, you can set it by using the flag --port.

How to deploy the Compose File to a Cluster

After you create the compose file, you can deploy it to your cluster with the ecs-cli compose up command. By default the command looks for a file called docker-compose.yml in the current directory, but you can specify a different file with the --file option. By default, the resources created by this command have the current directory in the title, but you can override that with the '--project-name project_name' option.

A Docker Compose file that creates 2 containers, WordPress and MySQL database:

$ sudo vi docker-compose.yml
version: "2"
services:
  wordpress:
    cpu_shares: 100
    image: wordpress
    links:
      - mysql
    mem_limit: 524288000
    ports:
      - "80:80"
  mysql:
    cpu_shares: 100
    environment:
      MYSQL_ROOT_PASSWORD: password
    image: "mysql:5.7"
    mem_limit: 524288000

Run: $ ecs-cli compose --file docker-compose.yml up to deploy the Compose File to a Cluster.

$ ecs-cli compose --file docker-compose.yml up
INFO[0000] Using ECS task definition                     TaskDefinition="ec2-user:1"
INFO[0000] Starting container...                         container=1824492b-1422-495d-b66d-8185c111bda6/mysql
INFO[0000] Starting container...                         container=1824492b-1422-495d-b66d-8185c111bda6/wordpress
INFO[0000] Describe ECS container status                 container=1824492b-1422-495d-b66d-8185c111bda6/mysql desiredStatus=RUNNING lastStatus=PENDING taskDefinition="ec2-user:1"
INFO[0000] Describe ECS container status                 container=1824492b-1422-495d-b66d-8185c111bda6/wordpress desiredStatus=RUNNING lastStatus=PENDING taskDefinition="ec2-user:1"
INFO[0012] Describe ECS container status                 container=1824492b-1422-495d-b66d-8185c111bda6/mysql desiredStatus=RUNNING lastStatus=PENDING taskDefinition="ec2-user:1"
INFO[0012] Describe ECS container status                 container=1824492b-1422-495d-b66d-8185c111bda6/wordpress desiredStatus=RUNNING lastStatus=PENDING taskDefinition="ec2-user:1"
INFO[0024] Describe ECS container status                 container=1824492b-1422-495d-b66d-8185c111bda6/mysql desiredStatus=RUNNING lastStatus=PENDING taskDefinition="ec2-user:1"
INFO[0024] Describe ECS container status                 container=1824492b-1422-495d-b66d-8185c111bda6/wordpress desiredStatus=RUNNING lastStatus=PENDING taskDefinition="ec2-user:1"
INFO[0030] Started container...                          container=1824492b-1422-495d-b66d-8185c111bda6/mysql desiredStatus=RUNNING lastStatus=RUNNING taskDefinition="ec2-user:1"
INFO[0030] Started container...                          container=1824492b-1422-495d-b66d-8185c111bda6/wordpress desiredStatus=RUNNING lastStatus=RUNNING taskDefinition="ec2-user:1"

How to view the running containers on the cluster

After you deploy the compose file, you can view the containers that are running on your cluster with the $ ecs-cli ps command.

Run: $ ecs-cli ps

$ ecs-cli ps
Name                                            State    Ports                     TaskDefinition  Health
1824492b-1422-495d-b66d-8185c111bda6/mysql      RUNNING                            ec2-user:1      UNKNOWN
1824492b-1422-495d-b66d-8185c111bda6/wordpress  RUNNING  34.236.254.87:80->80/tcp  ec2-user:1      UNKNOWN

How to scale the Tasks on a Cluster

You can scale your task count up so you could have more instances of your application with the $ ecs-cli compose scale command. In this example, you can increase the count of your application to two.

$ ecs-cli compose scale 2 --cluster-config SecondCluster_config --ecs-profile main_ecs
INFO[0000] Starting container...                         container=5e0a1741-5f4b-4621-8ed6-d69aa6e13a6f/mysql
INFO[0000] Starting container...                         container=5e0a1741-5f4b-4621-8ed6-d69aa6e13a6f/wordpress
INFO[0000] Describe ECS container status                 container=5e0a1741-5f4b-4621-8ed6-d69aa6e13a6f/wordpress desiredStatus=RUNNING lastStatus=PENDING taskDefinition="ec2-user:1"
INFO[0000] Describe ECS container status                 container=5e0a1741-5f4b-4621-8ed6-d69aa6e13a6f/mysql desiredStatus=RUNNING lastStatus=PENDING taskDefinition="ec2-user:1"
=INFO[0012] Describe ECS container status                 container=5e0a1741-5f4b-4621-8ed6-d69aa6e13a6f/wordpress desiredStatus=RUNNING lastStatus=PENDING taskDefinition="ec2-user:1"
INFO[0012] Describe ECS container status                 container=5e0a1741-5f4b-4621-8ed6-d69aa6e13a6f/mysql desiredStatus=RUNNING lastStatus=PENDING taskDefinition="ec2-user:1"
INFO[0024] Describe ECS container status                 container=5e0a1741-5f4b-4621-8ed6-d69aa6e13a6f/wordpress desiredStatus=RUNNING lastStatus=PENDING taskDefinition="ec2-user:1"
INFO[0024] Describe ECS container status                 container=5e0a1741-5f4b-4621-8ed6-d69aa6e13a6f/mysql desiredStatus=RUNNING lastStatus=PENDING taskDefinition="ec2-user:1"
INFO[0030] Started container...                          container=5e0a1741-5f4b-4621-8ed6-d69aa6e13a6f/wordpress desiredStatus=RUNNING lastStatus=RUNNING taskDefinition="ec2-user:1"
INFO[0030] Started container...                          container=5e0a1741-5f4b-4621-8ed6-d69aa6e13a6f/mysql desiredStatus=RUNNING lastStatus=RUNNING taskDefinition="ec2-user:1"

Now you should see two more containers in your cluster by running: $ ecs-cli ps
Example 1:

$ ecs-cli ps
Name                                            State                Ports                     TaskDefinition  Health
1824492b-1422-495d-b66d-8185c111bda6/mysql      RUNNING                                        ec2-user:1      UNKNOWN
1824492b-1422-495d-b66d-8185c111bda6/wordpress  RUNNING              34.236.254.87:80->80/tcp  ec2-user:1      UNKNOWN
5e0a1741-5f4b-4621-8ed6-d69aa6e13a6f/wordpress  RUNNING              3.88.22.104:80->80/tcp    ec2-user:1      UNKNOWN
5e0a1741-5f4b-4621-8ed6-d69aa6e13a6f/mysql      RUNNING                                        ec2-user:1      UNKNOWN

Example 2:

$ ecs-cli ps --cluster-config SecondCluster_config --ecs-profile main_ecs
Name                                            State                Ports                     TaskDefinition  Health
1824492b-1422-495d-b66d-8185c111bda6/mysql      RUNNING                                        ec2-user:1      UNKNOWN
1824492b-1422-495d-b66d-8185c111bda6/wordpress  RUNNING              34.236.254.87:80->80/tcp  ec2-user:1      UNKNOWN
5e0a1741-5f4b-4621-8ed6-d69aa6e13a6f/wordpress  RUNNING              3.88.22.104:80->80/tcp    ec2-user:1      UNKNOWN
5e0a1741-5f4b-4621-8ed6-d69aa6e13a6f/mysql      RUNNING                                        ec2-user:1      UNKNOWN

How to create an ECS Service from a Compose File

Now that you know that your containers work properly, you can make sure that they are replaced if they fail or stop. You can do this by creating a service from your compose file with the $ ecs-cli compose service up command. This command creates a task definition from the latest compose file (if it does not already exist) and creates an ECS service with it, with a desired count of 1.

Before starting your service, stop the containers from your compose file with the $ ecs-cli compose down command so that you have an empty cluster to work with. Example 1:

$ ecs-cli compose --file docker-compose.yml down

Example 2:

$ ecs-cli compose down --cluster-config SecondCluster_config --ecs-profile main_ecs
INFO[0000] Stopping container...                         container=1824492b-1422-495d-b66d-8185c111bda6/mysql
INFO[0000] Stopping container...                         container=1824492b-1422-495d-b66d-8185c111bda6/wordpress
INFO[0000] Stopping container...                         container=5e0a1741-5f4b-4621-8ed6-d69aa6e13a6f/wordpress
INFO[0000] Stopping container...                         container=5e0a1741-5f4b-4621-8ed6-d69aa6e13a6f/mysql
INFO[0000] Describe ECS container status                 container=1824492b-1422-495d-b66d-8185c111bda6/mysql desiredStatus=STOPPED lastStatus=RUNNING taskDefinition="ec2-user:1"
INFO[0000] Describe ECS container status                 container=1824492b-1422-495d-b66d-8185c111bda6/wordpress desiredStatus=STOPPED lastStatus=RUNNING taskDefinition="ec2-user:1"
INFO[0000] Describe ECS container status                 container=5e0a1741-5f4b-4621-8ed6-d69aa6e13a6f/wordpress desiredStatus=STOPPED lastStatus=RUNNING taskDefinition="ec2-user:1"
INFO[0000] Describe ECS container status                 container=5e0a1741-5f4b-4621-8ed6-d69aa6e13a6f/mysql desiredStatus=STOPPED lastStatus=RUNNING taskDefinition="ec2-user:1"
INFO[0006] Stopped container...                          container=1824492b-1422-495d-b66d-8185c111bda6/mysql desiredStatus=STOPPED lastStatus=STOPPED taskDefinition="ec2-user:1"
INFO[0006] Stopped container...                          container=1824492b-1422-495d-b66d-8185c111bda6/wordpress desiredStatus=STOPPED lastStatus=STOPPED taskDefinition="ec2-user:1"
INFO[0006] Stopped container...                          container=5e0a1741-5f4b-4621-8ed6-d69aa6e13a6f/wordpress desiredStatus=STOPPED lastStatus=STOPPED taskDefinition="ec2-user:1"
INFO[0006] Stopped container...                          container=5e0a1741-5f4b-4621-8ed6-d69aa6e13a6f/mysql desiredStatus=STOPPED lastStatus=STOPPED taskDefinition="ec2-user:1"

$ ecs-cli ps
Name                                            State                Ports                     TaskDefinition  Health
1824492b-1422-495d-b66d-8185c111bda6/mysql      STOPPED ExitCode: 0                            ec2-user:1      UNKNOWN
1824492b-1422-495d-b66d-8185c111bda6/wordpress  STOPPED ExitCode: 0  34.236.254.87:80->80/tcp  ec2-user:1      UNKNOWN
5e0a1741-5f4b-4621-8ed6-d69aa6e13a6f/wordpress  STOPPED ExitCode: 0  3.88.22.104:80->80/tcp    ec2-user:1      UNKNOWN
5e0a1741-5f4b-4621-8ed6-d69aa6e13a6f/mysql      STOPPED ExitCode: 0                            ec2-user:1      UNKNOWN

Now you can create your service using the $ ecs-cli compose service up command. Once ran you should be able to see the service from the ECS console:

$ ecs-cli compose service up --cluster-config SecondCluster_config --ecs-profile main_ecs
INFO[0000] Using ECS task definition                     TaskDefinition="ec2-user:1"
INFO[0005] (service ec2-user) has started 1 tasks: (task 3a18eb82-2d13-4697-b326-9024bd4fb497).  timestamp="2020-06-15 21:39:16 +0000 UTC"
INFO[0010] Service status                                desiredCount=1 runningCount=1 serviceName=ec2-user
INFO[0010] ECS Service has reached a stable state        desiredCount=1 runningCount=1 serviceName=ec2-user
INFO[0010] Created an ECS service                        service=ec2-user taskDefinition="ec2-user:1"

How to clean up resources

To delete the service so that it stops the existing containers and does not try to run any more tasks using $ ecs-cli compose service rm command:
Example 1:

$ ecs-cli compose --file docker-compose.yml service rm

Example 2:

$ ecs-cli compose service rm --cluster-config SecondCluster_config --ecs-profile main_ecs

Now, take down your cluster, which cleans up the resources that you created earlier with $ ecs-cli up.
Example 1:

$ ecs-cli down --force
INFO[0000] Waiting for your cluster resources to be deleted... 
INFO[0000] Cloudformation stack status                   stackStatus=DELETE_IN_PROGRESS
INFO[0060] Cloudformation stack status                   stackStatus=DELETE_IN_PROGRESS
INFO[0121] Cloudformation stack status                   stackStatus=DELETE_IN_PROGRESS
INFO[0151] Deleted cluster                               cluster=SecondCluster

Example 2:

$ ecs-cli down --force --cluster-config SecondCluster_config --ecs-profile main_ecs

Other Notes

Troubleshooting Basics

Amazon ECS Troubleshooting: https://docs.aws.amazon.com/AmazonECS/latest/developerguide/troubleshooting.html

When troubleshooting a container instance, some log analysis might be required. Here are the most common logs:

/var/log/messages
/var/log/dmesg
/var/log/cloud-init(-output).log
docker logs <container id> - note these logs are only available when using the default 'json-file' log driver. If you are using 'awslogs' as many customers do, the CloudWatch log group and stream (as configured in the task definition) can be checked.

All logs can be collected for analysis using the ecs-log-collector tool: https://docs.aws.amazon.com/AmazonECS/latest/developerguide/ecs-logs-collector.html

Say, you noticed that the ECS container agent was not 'connected', so we need to see why. There are many reasons why the agent may not be connected to the ECS service:

Is the instance currently running?
Can you SSH to the container instance?
Is the Docker daemon running?

$ ps aux | grep -i docker
$ docker ps

- If that is the case and you need to further troubleshoot the docker daemon, you could set the docker logs to debug and restart the service. Enable Debugging (ecs-optmized) by adding line below, and then restart docker:

$ sudo vi /etc/sysconfig/docker
...
OPTIONS="-D --default-ulimit nofile=1024:4096"

$ sudo service docker restart

Click here to view more Docker troubleshooting tips.

If the ECS container agent is running, but you still see it as not connected, run $ docker ps and check it. If it shows the task is only up for 1 second that means it was just restarted. If you seeing it like this constantly then this suggests that the ECS container agent is constnatly restarting.

Where to check what task a local ecs-agent is tracking

$ curl http://localhost:51678/v1/metadata | python -mjson.tool
$ /var/log/ecs/ecs-agent.log*
$ /var/lib/ecs/data/ecs_agent_data.json
$ /var/lib/ecs/data/agent.db # replaces the above in later agent versions

Useful 'aws ecs' AWS CLI commands

How to describe your cluster using $ aws ecs list-container-instances:

$ aws ecs list-container-instances --cluster default
{
    "containerInstanceArns": [
        "arn:aws:ecs:us-east-1:648818476623:container-instance/1542e693-2d21-4ccd-b476-455faef68e68",
        "arn:aws:ecs:us-east-1:648818476623:container-instance/99c833f7-5561-4aa3-92d6-ae18608f06e5"
    ]
}

Now that you have the ARN or ID of a container instance, you can use the $ describe-container-instances command to get valuable information on the instance, such as remaining and registered CPU and memory resources:

$ aws ecs describe-container-instances --container-instances 1542e693-2d21-4ccd-b476-455faef68e68
{
    "containerInstances": [
        {
            "containerInstanceArn": "arn:aws:ecs:us-east-1:648818476623:container-instance/1542e693-2d21-4ccd-b476-455faef68e68",
            "ec2InstanceId": "i-0c567d7b38900c18c",
            "version": 95,
            "versionInfo": {
                "agentVersion": "1.40.0",
                "agentHash": "17e8d834",
                "dockerVersion": "DockerVersion: 19.03.6-ce"
            },
            "remainingResources": [
                {
                    "name": "CPU",
                    "type": "INTEGER",
                    "doubleValue": 0.0,
                    "longValue": 0,
                    "integerValue": 1024
                }, 
                {
                    "name": "MEMORY",
                    "type": "INTEGER", 
[......]

You can list task definitions by using the list-task-definitions command. The output of this command shows the family and revision values that you can use together when calling run-task or start-task:

$ aws ecs list-task-definitions

How to bootstrap an ECS contianer instance to mount an existing EFS volume to the instance

An EFS file system can be mounted on to container instances and the mount points from host instance can be shared with the containers running on the host.

Using EFS file system with Docker containers provides an advantage of not running out of host volume space and the same EFS file system can be shared with containers running on multiple hosts.

Boothook example:
- The EFS file system is mounted through cloud-boothook. EFS file system is mounted to directoy /efs on the host instance. When the instance is up we can SSH into the instance and create a couple empty files in /efs directory, for example use the below command to create 2 files named "123" and "abc" -

$ touch 123 abc

- You can then launch an nginx contianer and mount the /efs directory to the directory /mnt on the container as shown below. Inside the container we should be able to see the 2 example files that we created in /mnt directory.

#cloud-boothook
# Install nfs-utils
cloud-init-per once yum-update yum update -y
cloud-init-per once install_nfs_utils yum install -y nfs-utils

# Create /efs folder 
cloud-init-per once mkdir_efs
mkdir /efs

# Mount /efs
cloud-init-per once mount_efs echo -e 'fs2aa70323.efs.us-east-1.amazonaws.com:/ /efs nfs4 nfsver=4.1,rsize=1048576,wsize=1048576,hard,timeo=600,retrans=2 0 0' >> /etc/fstab
mount -a

#Then you can run:
$ docker run -it -v /efs:/mnt nginx bash
# cd /mnt/
# ls
123  abc

How to find PENDING tasks count per cluster, service and instance? (using CLI)

Describe the CLUSTER to find the number of tasks in the CLUSTER that are in the RUNNING and PENDING state:

$ aws ecs describe-clusters --cluster default --region us-east-1 --query "clusters[].[pendingTasksCount,runningTasksCount]"
[
   [
      0,
      2
   ]
]

Describe a CONTAINER INSTANCE to find the number of tasks on the CONTAINER instance that are in RUNNING and PENDING status:

$ aws ecs describe-container-instances --cluster default --container-instances 2131234-3243243-af34-af23-vda3-1431gds --region us-east-1 --query "containerInstances[].[runningTasksCount,pendingTasksCount]"
[
   [
      1,
      0
   ]
]

Describe an ECS SERVICE to find the number of desired tasks in the service as well as the actual number of RUNNING and PENDING state:

Capacity Providers

MUST READ/ALWAYS READ THIS:: https://aws.amazon.com/blogs/containers/deep-dive-on-amazon-ecs-cluster-auto-scaling/

https://docs.aws.amazon.com/AmazonECS/latest/developerguide/cluster-capacity-providers.html

WHAT IS A CAPACITY PROVIDER?
[+] capacity providers are used to manage the infrastructure the tasks in your clusters use.

[+] Per Gilbert "Service auto scaling is responsible for scaling additional tasks when CPU utilization is 60%, the capacity provider simply scales the cluster in relation to the "desired tasks" number."

[+] A capacity provider is associated with a cluster and is used in a capacity provider strategy to determine the infrastructure that a task runs on.

[+] For Amazon ECS on Amazon EC2 users, a capacity provider consists of a capacity provider name, an Auto Scaling group, and the settings for managed scaling and managed termination protection. With managed scaling, Amazon ECS manage the scale-in and scale-out actions of the Auto Scaling group which provides auto scaling for your cluster's infrastructure.

AUTO SCALING GROUP CAPACITY PROVIDERS:
[+] Amazon ECS capacity providers can use Auto Scaling groups to manage the Amazon EC2 instances registered to their clusters. You can use the managed scaling feature to have Amazon ECS manage the scale-in and scale-out actions of the Auto Scaling group or you can manage the scaling actions yourself.

AUTO SCALING GROUP CAPACITY PROVIDERS CONSIDERATIONS
[+] It is recommended that you create a new empty Auto Scaling group to use with a capacity provider rather than using an existing one. If you use an existing Auto Scaling group, any Amazon EC2 instances associated with the group that were already running and registered to an Amazon ECS cluster prior to the Auto Scaling group being used to create a capacity provider may not be properly registered with the capacity provider. This may cause issues when using the capacity provider in a capacity provider strategy. The DescribeContainerInstances API can confirm whether a container instance is associated with a capacity provider or not.

[+] When using managed termination protection, managed scaling must be enabled otherwise managed termination protection will not work.

[+] When using managed scaling, the Auto Scaling group shouldn't have any scaling policies attached to it other than the ones Amazon ECS creates, otherwise the Amazon ECS created scaling plans will receive an ActiveWithProblems error.

https://docs.aws.amazon.com/AmazonECS/latest/developerguide/asg-capacity-providers-update-capacity-provider.html:

For Target capacity %, if managed scaling is enabled, specify an integer between 1 and 100. The target capacity value is used as the target value for the CloudWatch metric used in the Amazon ECS-managed target tracking scaling policy. This target capacity value is matched on a best effort basis. For example, a value of 100 will result in the Amazon EC2 instances in your Auto Scaling group being completely utilized and any instances not running any tasks will be scaled in, but this behavior is not guaranteed at all times.

Other notes from Nitheesha:

so 80% utilization/reservation. 80% can be 8 instances more or less, depending on how much memory, cpu they have left
if we use 100% capacity, it means you are saying capacity provider to use 100% of resources available, dont leave any excess. only launch more when required and utilize the reosurces to 100%.
if you say 50% target capacity, you are instructing capacity provider to make sure the cpu/memory combined reservation is 50% and you aleays have 50% excess available in ASG.
This cpu/memory combined is called caapcity provider reservation

ECS

Contents

ECS Components

Cluster

Container Instance

Active

Draining

Inactive

Bootstrapping

Container agent

Scheduler

Task Definition

Task

Scheduled Tasks

Pending Task

Service

Container

ECS Optimized AMI

Container Agent

Agent Configuration

Amazon/Linux AMI

Windows Server 2016

API

Agent Debugging

Updating the container agent

How to stop/start the ecs agent

How to check ECS Agent status and connectivity

The ecs-init service

Configuration and log files

Autoscaling in ECS

Load Balancers in ECS

Task Networking (awsvpc)

Container Storage

ECS CLI

How to install the ECS CLI

How to configure the Amazon ECS CLI

How to create a cluster

How to deploy the Compose File to a Cluster

How to view the running containers on the cluster

How to scale the Tasks on a Cluster

How to create an ECS Service from a Compose File

How to clean up resources

Other Notes

Troubleshooting Basics

Where to check what task a local ecs-agent is tracking

Useful 'aws ecs' AWS CLI commands

How to bootstrap an ECS contianer instance to mount an existing EFS volume to the instance

How to find PENDING tasks count per cluster, service and instance? (using CLI)

Capacity Providers