EC2

From Dikapedia
Jump to: navigation, search

the default user name for the AMI that you used to launch your instance:

  • For Amazon Linux 2 or the Amazon Linux AMI, the user name is ec2-user.
  • For a CentOS AMI, the user name is centos.
  • For a Debian AMI, the user name is admin.
  • For a Fedora AMI, the user name is ec2-user or fedora.
  • For a RHEL AMI, the user name is ec2-user or root.
  • For a SUSE AMI, the user name is ec2-user or root.
  • For an Ubuntu AMI, the user name is ubuntu.
  • Otherwise, if ec2-user and root don't work, check with the AMI provider.



Hypervisors

https://docs.aws.amazon.com/whitepapers/latest/aws-overview-security-processes/hypervisor.html

Amazon EC2 currently utilizes a highly customized version of the Xen hypervisor, taking advantage of paravirtualization (in the case of Linux guests). Because para-virtualized guests rely on the hypervisor to provide support for operations that normally require privileged access, the guest OS has no elevated access to the CPU. The CPU provides four separate privilege modes: 0-3, called rings. Ring 0 is the most privileged and 3 the least. The host OS executes in Ring 0. However, rather than executing in Ring 0 as most operating systems do, the guest OS runs in a lesser-privileged Ring 1 and applications in the least privileged Ring 3. This explicit virtualization of the physical resources leads to a clear separation between guest and hypervisor, resulting in additional security separation between the two.


Traditionally, hypervisors protect the physical hardware and bios, virtualize the CPU, storage, networking, and provide a rich set of management capabilities. With the Nitro System, we are able to break apart those functions, offload them to dedicated hardware and software, and reduce costs by delivering all of the resources of a server to your instances.


The Nitro Hypervisor provides consistent performance and increased compute and memory resources for EC2 virtualized instances by removing host system software components. It allows AWS to offer larger instance sizes (like c5.18xlarge) that provide practically all of the resources from the server to customers. Previously, C3 and C4 instances each eliminated software components by moving VPC and EBS functionality to hardware designed and built by AWS. This hardware enables the Nitro Hypervisor to be very small and uninvolved in data processing tasks for networking and storage.


Nevertheless, as AWS expands its global cloud infrastructure, Amazon EC2’s use of its Xen-based hypervisor will also continue to grow. Xen will remain a core component of EC2 instances for the foreseeable future.


Troubleshooting


Things to check if instance is failing to boot up or failing instance status checks
  • First thing to check is console output and screenshot
  • Look for any errors regarding
    • out of memory
    • no disk space
    • auditd
    • mounting or dependency issues/fstab/emergency mode
[ TIME ] Timed out waiting for device dev-mapper-appvg2\x2dlvol0.device.
[DEPEND] Dependency failed for /u02.
.
.
.
You are in  emergency mode. After logging in, type "journalctl -xb" to view
system logs, "systemctl reboot" to reboot, "systemctl default" or "exit"
to boot into default mode. 

Cannot open access to console, the root account is locked. 
See sulogin(8) man page for more details.
  • cloud-init-output, cloud-init ran?
  • dhclient exists?
  • ifcfg-eth0, Hardcoded MAC address?


Instance Meta-data (metadata)

To view the private IPv4 address, public IPv4 address, and all other categories of instance metadata from within a running instance, use the following URL: 

$ curl http://169.254.169.254/latest/meta-data/
ami-id
ami-launch-index
ami-manifest-path
block-device-mapping/
events/
hostname
iam/
identity-credentials/
instance-action
instance-id
instance-life-cycle
instance-type
local-hostname
local-ipv4
mac
metrics/
network/
placement/
profile
public-hostname
public-ipv4
public-keys/
reservation-id
security-groups

Couple of ways:


Get instance ID:
$ curl http://169.254.169.254/latest/meta-data/instance-id
i-0dcecd45293806bb7


Get Instance IAM role
$  curl http://169.254.169.254/latest/meta-data/iam/security-credentials
AmazonSSMRoleForInstancesQuickSetup
  • You will get 404 not found for requests for /iam meta-data if there is no IAM role attached to the instance.

Then you can get more info on that specific role: 

$  curl http://169.254.169.254/latest/meta-data/iam/security-credentials/AmazonSSMRoleForInstancesQuickSetup
{
  "Code" : "Success",
  "LastUpdated" : "2021-04-03T18:47:17Z",
  "Type" : "AWS-HMAC",
  "AccessKeyId" : "XXXXXXXXXXXXX",
  "SecretAccessKey" : "YYYYYYYYYYYYYYYYYYYYY",
  "Token" : "ZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZ==",
  "Expiration" : "2021-04-04T01:22:43Z"


How to view billing product code

Per our docs: 

curl http://169.254.169.254/latest/meta-data/product-codes

Other ways that work: 

$  curl 169.254.169.254/latest/dynamic/instance-identity/document
{
  "accountId" : "648818476623",
  "architecture" : "x86_64",
  "availabilityZone" : "eu-west-1b",
  "billingProducts" : [ "bp-6fa54006" ],
  "devpayProductCodes" : null,
  "marketplaceProductCodes" : null,
  "imageId" : "ami-065ec1e661d619058",
  "instanceId" : "i-0f1ee0db536f6181d",
  "instanceType" : "t2.micro",
  "kernelId" : null,
  "pendingTime" : "2020-09-15T15:21:00Z",
  "privateIp" : "172.31.47.114",
  "ramdiskId" : null,
  "region" : "eu-west-1",
  "version" : "2017-09-30"

On SUSE/Debian/Ubuntu, you can also do this: 

ec2metadata --document --api latest | grep billing
"billingProducts" : [ "bp-6ca54005" ],

ec2metadata --api latest | grep billing


How to update kernel version

https://ubuntu.pkgs.org/18.04/ubuntu-updates-main-amd64/linux-tools-5.4.0-1025-aws_5.4.0-1025.25~18.04.1_amd64.deb.html 

$ sudo apt-get update
$ sudo apt-get install linux-tools-5.4.0-1025-aws
$ reboot

$ uname -r
5.4.0-1025-aws


Kernel Panic

Kernel Panic: 

[    2.268531] md: Waiting for all devices to be available before autodetect
[    2.273870] md: If you don't use raid, use raid=noautodetect
[    2.278974] md: Autodetecting RAID arrays.
[    2.283349] md: autorun ...
[    2.286885] md: ... autorun DONE.
[    2.290647] List of all partitions: 
[    2.294490] No filesystem could mount root, tried: 
[    2.299832] Kernel panic - not syncing: VFS: Unable to mount root fs on unknown-block(0,0)
[    2.300826] CPU: 1 PID: 1 Comm: swapper/0 Not tainted 3.10.0-1127.19.1.el7.x86_64 #1
[    2.300826] Hardware name: Xen HVM domU, BIOS 4.2.amazon 08/24/2006
[    2.300826] Call Trace:
[    2.300826]  [<ffffffffa197ffa5>] dump_stack+0x19/0x1b
[    2.300826]  [<ffffffffa1979541>] panic+0xe8/0x21f
[    2.300826]  [<ffffffffa1f8b794>] mount_block_root+0x291/0x2a0
[    2.300826]  [<ffffffffa1f8b7f6>] mount_root+0x53/0x56
[    2.300826]  [<ffffffffa1f8b935>] prepare_namespace+0x13c/0x174
[    2.300826]  [<ffffffffa1f8b412>] kernel_init_freeable+0x222/0x249
[    2.300826]  [<ffffffffa1f8ab28>] ? initcall_blacklist+0xb0/0xb0
[    2.300826]  [<ffffffffa196e6b0>] ? rest_init+0x80/0x80
[    2.300826]  [<ffffffffa196e6be>] kernel_init+0xe/0x100
[    2.300826]  [<ffffffffa1992d37>] ret_from_fork_nospec_begin+0x21/0x21
[    2.300826]  [<ffffffffa196e6b0>] ? rest_init+0x80/0x80
  • Possible causes can be due to bad kernel update occurring.
  • Couple solutions:
    • Restore from a backup prior to the kernel update.
    • Follow one of these articles that can help you fix this issue:

https://aws.amazon.com/premiumsupport/knowledge-center/ec2-linux-kernel-panic-unable-mount/ https://access.redhat.com/solutions/57018


How to roll back the kernel version

https://aws.amazon.com/premiumsupport/knowledge-center/ec2-linux-kernel-panic-unable-mount/

Use EC2 Rescue instance to rollback to earlier kernel version.

Steps where /dev/xvdf2 is root FS: 

mount /dev/xvdf2 /mnt
mount -o bind /proc /mnt/proc && mount -o bind /dev /mnt/dev && mount -o bind /sys /mnt/sys
chroot /mnt
vi /etc/default/grub (Change to GRUB_DEFAULT=1)
grub2-mkconfig -o /boot/grub2/grub.cfg
exit
umount /mnt/{proc,sys,dev} && umount /mnt


Spot Instances

To use Spot Instances, you create a Spot Instance request that includes the number of instances, the instance type, the Availability Zone, and the maximum price that you are willing to pay per instance hour. If your maximum price exceeds the current Spot price, Amazon EC2 fulfills your request immediately if capacity is available. Otherwise, Amazon EC2 waits until your request can be fulfilled or until you cancel the request.


How to launch RHEL 6 on Nitro-instance

Yes, it is possible to run RHEL 6 on Nitro-isntances (M5, C5, T3). I have tested this out using AMI: ami-0351faf7328fdb373 (RHEL 6.10 - HVM - Red Hat Provided, ENA: no).


1) First make sure NVMe driver is there and check the driver version using the following command. If the instance has the NVMe driver, the command returns information about the driver. 

$ modinfo nvme


2) Install and Enable ENA: https://aws.amazon.com/premiumsupport/knowledge-center/install-ena-driver-rhel-ec2/

A. Update the kernel and reboot the system so that the latest kernel takes effect: 

sudo yum upgrade kernel -y && sudo reboot

B. Install the development package for building kernel modules to match the kernel: 

sudo yum install kernel-devel-$(uname -r) gcc git patch rpm-build wget -y
cd /usr/src/
sudo wget https://github.com/amzn/amzn-drivers/archive/master.zip
sudo unzip master.zip
cd amzn-drivers-master/kernel/linux/ena
sudo make

C. Copy the module to the modules directory: 

sudo cp ena.ko /lib/modules/$(uname -r)/

D. Regenerate the kernel module dependency map files: 

sudo depmod

E. Use the modinfo command to confirm that the ENA module is present: 

modinfo ena

The modinfo command output shows the ENA driver information.

Note: The ENA driver version might be newer than 2.2.11g while you compile and install it on your system. 

filename:       /lib/modules/2.6.32-754.33.1.el6.x86_64/ena.ko
version:        2.2.11g
license:        GPL
description:    Elastic Network Adapter (ENA)
author:         Amazon.com, Inc. or its affiliates
retpoline:      Y
srcversion:     17C7CD1CEAD3F0ADB3A5E5E
alias:          pci:v00001D0Fd0000EC21sv*sd*bc*sc*i*
alias:          pci:v00001D0Fd0000EC20sv*sd*bc*sc*i*
alias:          pci:v00001D0Fd00001EC2sv*sd*bc*sc*i*
alias:          pci:v00001D0Fd00000EC2sv*sd*bc*sc*i*
alias:          pci:v00001D0Fd00000051sv*sd*bc*sc*i*
depends:        
vermagic:       2.6.32-754.33.1.el6.x86_64 SMP mod_unload modversions 
parm:           debug:Debug level (0=none,...,16=all) (int)
parm:           rx_queue_size:Rx queue size. The size should be a power of 2. Max value is 8K
(int)
parm:           force_large_llq_header:Increases maximum supported header size in LLQ mode to 224  bytes, while reducing the maximum TX queue size by half.
(int)
parm:           num_io_queues:Sets number of RX/TX queues to allocate to device. The maximum value depends on the device and number of online CPUs.
(int)

F. Append net.ifnames=0 to /boot/grub/grub.conf to disable network interface naming: 

sudo sed -i '/^kernel/s/$/ net.ifnames=0/' /boot/grub/grub.conf

G. Stop the instance.

H. Enable enhanced network support at the instance level. The following example modifies the instance's attribute from the AWS Command Line Interface (AWS CLI). 

aws ec2 modify-instance-attribute --instance-id i-xxxxxxxxxxxxxxxxx --ena-support --region xx-xxxxx-x

I. Change the instance type to one of the ENA supported instance types.

J. Start the instance, connect to the instance using SSH, and then run the ethtool command: 

ethtool -i eth0
driver: ena
version: 2.4.1g
firmware-version: 
bus-info: 0000:00:05.0
supports-statistics: yes
supports-test: no
supports-eeprom-access: no
supports-register-dump: no
supports-priv-flags: no


Steps to convert SLES15 SP3 AMI to boot in UEFI mode

Steps to convert the "regular" SLES 15 SP3 AMI to boot in UEFI mode, since there's no AWS-provided AMI for UEFI boot with x86 proc. (Only w/ ARM proc.):

(1) Launch an instance from the regular SLES 15 SP3 AMI (I used ami-08895422b5f3aa64a in us-east-1).

(2) I then ran updates before creating the AMI.

(3) Take snapshot of root volume. Note the snapshot ID.

(4) Run the following command at AWSCLI, replacing the snapshot ID and Region as necessary: aws --region $REGION ec2 register-image \ 

 --description "sle-15-sp3-uefi" \
 --name "sle-15-sp3-uefi" \
 --block-device-mappings "DeviceName=/dev/sda1,Ebs={SnapshotId=$SNAPSHOT_ID_GOES_HERE,DeleteOnTermination=true}" \
 --architecture x86_64 \
 --root-device-name /dev/sda1 \
 --virtualization-type hvm \
 --ena-support \
 --boot-mode uefi

(5) Launch a new instance from the new AMI. To check boot mode: [ -d /sys/firmware/efi ] && echo "UEFI Boot Detected" || echo "Legacy BIOS Boot Detected" dmesg | grep -i "EFI


Amazon Linux 1 AMI

Available through CLI: 

ami-00514a528eadbc95b
Retrieved from "https://dikapedia.com/wiki/index.php?title=EC2&oldid=2605"