File Gateway
File Gateway
File Gateway
o EC2
o IAM
o Lambda
o S3
o EFS
o EBS
o S3 glacier
o RDS
o Synamo DB
o Route 53
o Cloud watch
o Cloud trail
o Aws auto scaling
o Cognito
o Aws cost explorer
o Aws marketplace
o EKS
o ECS
o Code build
o Code deploy
o Code pipeline
o Code artifact
o Cloud formation
Cached volumes – You store your data in Amazon Simple Storage Service (Amazon
S3) and retain a copy of frequently accessed data subsets locally. Cached volumes
offer a substantial cost savings on primary storage and minimize the need to scale
your storage on-premises. You also retain low-latency access to your frequently
accessed data.
Stored volumes – If you need low-latency access to your entire dataset, first
configure your on-premises gateway to store all your data locally. Then
asynchronously back up point-in-time snapshots of this data to Amazon S3. This
configuration provides durable and inexpensive offsite backups that you can recover
to your local data center or Amazon Elastic Compute Cloud (Amazon EC2). For
example, if you need replacement capacity for disaster recovery, you can recover the
backups to Amazon EC2.
A Tape Gateway- A tape gateway provides cloud-backed virtual tape storage. The tape gateway
is deployed into your on-premises environment as a VM running on VMware ESXi, KVM, or
Microsoft Hyper-V hypervisor.
With a tape gateway, you can cost-effectively and durably archive backup data in GLACIER
A VPC endpoint enables connections between a virtual private cloud (VPC) and supported
services, without requiring that you use an internet gateway, NAT device, VPN connection, or
AWS Direct Connect connection. Therefore, your VPC is not exposed to the public internet.
VPC endpoints are virtual devices. They are horizontally scaled, redundant, and highly available
VPC components. The following are the different types of VPC endpoints. You create the type
of VPC endpoint that's required by the supported service.
Interface endpoints
An interface endpoint is an elastic network interface with a private IP address from the IP
address range of your subnet. It serves as an entry point for traffic destined to a service that is
owned by AWS or owned by an AWS customer or partner. For a list of AWS services that
integrate with AWS PrivateLink, see AWS services that integrate with AWS PrivateLink.
You are billed for hourly usage and data processing charges. For more information, see Interface
endpoint pricing.
A Gateway Load Balancer endpoint is an elastic network interface with a private IP address from
the IP address range of your subnet. It serves as an entry point to intercept traffic and route it to a
network or security service that you've configured using a Gateway Load Balancer. You specify
a Gateway Load Balancer endpoint as a target for a route in a route table. Gateway Load
Balancer endpoints are supported only for endpoint services that are configured using a Gateway
Load Balancer.
You are billed for hourly usage and data processing charges. For more information, see Gateway
Load Balancer endpoint pricing.
Gateway endpoints
A gateway endpoint is a gateway that is a target for a route in your route table used for traffic
destined to either Amazon S3 or DynamoDB.
10) Many questions in RDS and difference between dynamo db and other database
RDS makes it easy to set up, operate, and scale a relational database. DynamoDB is an
AWS fully-managed, high-performance, NoSQL database
Amazon Redshift is a fully-managed petabyte-scale cloud based data
warehouse product designed for large scale data set storage and analysis. It
is also used to perform large scale database migrations.
A VPC peering connection is a networking connection between two VPCs that enables you to route
traffic between them using private IPv4 addresses or IPv6 addresses.
Instances in either VPC can communicate with each other as if they are within the same network.
You can create a VPC peering connection between your own VPCs, or with a VPC in another AWS
account.
The VPCs can be in different regions (also known as an inter-region VPC peering connection).
Conditions:
->A Classic Load Balancer makes routing decisions at either the transport layer (TCP/SSL) or the
application layer (HTTP/HTTPS).
->An Application Load Balancer makes routing decisions at the application layer (HTTP/HTTPS),
supports path-based routing, and can route requests to one or more ports on each container instance in
your cluster.
->A Network Load Balancer makes routing decisions at the transport layer (TCP/SSL). It can handle
millions of requests per second. After the load balancer receives a connection, it selects a target from the
target group for the default rule using a flow hash routing algorithm. It attempts to open a TCP connection
to the selected target on the port specified in the listener configuration. It forwards the request without
modifying the headers
Spin up one more EC2 instance1 in another availability zone (az2) with http port open in Security
group
Add the below script and launch the instance.
#!/bin/bash
yum update -y
yum install httpd -y
It’s a traditional upload where you will upload the object in one part
A single operation upload can upload the file up to 5GB in size.
Upload object in parts:
Using multipart upload, you can upload the large objects up to 5TB.
You can use multipart upload for the objects from 5MB to 5TB in size.
Limitation of S3 bucket:
Only 100 buckets can be created per account.
Can hold unlimited objects
S3 Storage classes:
Standard:
Designed for general- and all-purpose storage
Default storage option
99.999999999% object durability
99.99% object availability
Most expensive storage class.
Reduced Redundancy storage
Designed for non-critical objects
99.99% object durability
99.99% object availability
Less expensive than standard
Infrequent access
Designed for less frequently accessed objects.
99.999999999% object durability
99.99% object availability
Less expensive than reduced redundancy storage
Glacier
Designed for long term archival storage
May take several hours to retrieve the objects from this storage
Cheapest s3 storage class
When you associate an Elastic IP address with an EC2 instance, it replaces the default public IP
address. If an external hostname was allocated to the instance from your launch settings, it will also
replace this hostname; otherwise, it will create one for the instance. The Elastic IP address remains in
place through events that normally cause the address to change, such as stopping or restarting the
instance.
Amazon S3
Amazon S3 is a simple storage service offered by Amazon and it is useful for hosting
website images and videos, data analytics, etc. S3 is an object-level data storage that
distributes the data objects across several machines and allows the users to access the
storage via the internet from any corner of the world.
Amazon EBS
Unlike Amazon S3, Amazon EBS is a block-level data storage offered by Amazon.
Block storage stores files in multiple volumes called blocks, which act as separate hard
drives, and this storage is not accessible via the internet. Use cases include business
continuity, transactional and NO SQL database, software testing, etc
Move object
-> aws s3 mv s3://bucket-name/example s3://my-bucket/
$The following example moves a local file from your current working directory to the Amazon -
S3 bucket with the s3 cp command.
$ aws s3 mv filename.txt s3://bucket-name
--The following example moves a file from your Amazon S3 bucket to your current working
directory, where ./ specifies your current working directory.
$ aws s3 mv s3://bucket-name/filename.txt ./
Copy object
-> aws s3 cp s3://bucket-name/example s3://my-bucket/
Sync object
-> aws s3 sync . s3://my-bucket/path
CNAME Record:
Maps hostname to another host name: us-east.2.elb.amazonaws.com to
myapp.mydomain.com
Alias Record: points a host name to AWS Resource ex: myapp.mydomain.com to us-
east.2.elb.amazonaws.com
29) Write a script to list all AWS S3 buckets in a given region and total used capacity of the bucket.
Select "Your VPCs" in the left sidebar and the dashboard will display all of your VPCs in a
particular AWS Region, including the default VPC. A region can only have one default VPC.
Although you can have up to five VPCs in a region, only the initial VPC that AWS creates for
you can be the default VPC.
Every VPC is associated with an IP address range that is part of a Classless Inter-Domain
Routing (CIDR) block which will be used to allocated private IP addresses to EC2 instances.
AWS recommends that VPCs use private ranges that are defined in RFC 1918. These private
ranges are reserved by the Internet Assigned Numbers Authority (IANA) and cannot be routed to
the Internet. Different sized ranges with different allocations of IP addresses can be assigned to a
VPC depending on need.
All default VPCs will be associated with an IPv4 CIDR block with a 172.31.0.0/16 address
range. This will give you 65,536 possible IP addresses, minus some AWS reserved addresses.
VPCs can be created with smaller CIDR blocks, such as a /20, which would yield 4091 possible
addresses.
2)Subnet
Next, if you go to the "Subnets" screen, you will see that multiple default subnets have already
been assigned to your default VPC, one subnet for each availability zone.
A subnet is always associated with a single availability zone and cannot span multiple zones.
However, an availability zone can host multiple subnets. Each subnet in a VPC is associated with
an IPv4 CIDR block that is a subset of the /16 CIDR block of its VPC.
In a default VPC, each default subnet is associated with /20 CIDR block address range which
will have 4091 possible IP addresses minus the five addresses AWS always reserves. Note that
two subnets cannot have overlapping address ranges.
When you launch an EC2 instance into a default VPC without specifying a specific subnet, it’s
automatically launched in one of the default subnets. Every instance in a default subnet receives
a private IP address from the pool of addresses associated with that subnet and also a private
DNS hostname.
In a default subnet, an instance will also receive a public IP address from the pool of addresses
owned by AWS along with a public DNS hostname, which will facilitate Internet access for your
instances.
3)Gateways
Frequently, your EC2 instances will require connectivity outside of AWS to the Internet or to a
user's corporate network via the use of gateways.
For communication with the Internet, a VPC must be attached to an Internet gateway. An
Internet gateway is a fully managed AWS service that performs bi-direction source and
destination network address translation for your EC2 instances. Optionally, a VPC may use a
virtual private gateway to grant instances secure access to a user's corporate network via VPN or
direct connect links. Instances in a subnet can also be granted outbound only Internet access
through a NAT gateway.
A subnet that provides its instances a route to an Internet gateway is considered a public subnet.
A private subnet may be in a VPC with an attached Internet gateway but will not have a route to
that gateway. In a default VPC, all default subnets are public subnets and will have a route to a
default gateway.
4)Route table
I've mentioned routing several times while talking about the Internet gateway. Every VPC is
attached to an implicit router. This router is not visible to the user and is fully managed and
scaled by AWS. What is visible is the route table associated with each subnet, which is used by
the VPC router to determine the allowed routes for outbound network traffic leaving a subnet.
Note from the screenshot below that every route table contains a default local route to facilitate
communication between instances in the same VPC, even across subnets. This intra-VPC local
route is implied and cannot be changed. In the case of the main route table that is associated with
a default subnet, there will also be a route out to the Internet via the default gateway for the VPC.
Also note that every subnet must be associated with a route table. If the association is not
explicitly defined, then a subnet will be implicitly associated with the main route table.
5)Network security(network ACLS)
One concern you may have is network security, particularly if all default subnets in a default
VPC are public and open to Internet traffic. AWS provides security mechanisms for your
instances in the form of network ACLs and security groups. These two mechanisms can work
together to provide layered protection for your EC2 instances.
An ACL acts as a firewall that controls network traffic in and out of a subnet. You create rules
for allowing or denying network traffic for specific protocols, through specific ports and for
specific IP address ranges.
Network ACLs are stateless and have separate inbound and outbound rules. This means both
inbound and outbound rules have to be created to allow certain network traffic to enter the subnet
and for responses to go back through.
A number is assigned to each rule and all rules are evaluated starting with the lowest numbered
rule. When traffic hits the firewall, it is evaluated against the rules in ascending order. As soon as
a rule is evaluated that matches the traffic being considered, it is applied regardless of what is
indicated in a subsequent rule.
As indicated above, the default ACL in a default VPC is configured with lower-numbered rules
for both inbound and outbound traffic which combine to explicitly allow bi-directional
communication for any protocol, through any port and to and from any source or destination.
You can associate an ACL with multiple subnets but any single subnet can only be associated
with one ACL. If you don't specifically associate an ACL with a subnet, the subnet is
automatically associated with the default ACL. This is the case with your default VPCs which
have all subnets associated with the default ACL.
6)Security groups
Security groups are considered the first line of defense and consist of a firewall that’s applied at
the instance level. This means only instances explicitly associated with a security group will be
subject to its rules while all instances in a subnet are impacted by the network ACL applied to
that subnet.
Similar to ACLs, you create inbound and outbound traffic rules based on protocol, port and
source or destination IP. However, there are some differences:
You can specify rules to allow network traffic but cannot create rules to deny specific types of
traffic. In essence, all traffic is denied except for traffic you explicitly allow.
Security groups are stateful, so if you create a rule to allow a certain type of traffic in, then
outbound traffic in response is also allowed even if there is no explicit outbound rule to allow
such traffic.
Every instance must be associated with a security group and if a security group is not specified at
launch time, then that instance will be associated with a default security group.
You can see from the screenshot above that a default security group will have a rule that only
allows inbound traffic from other instances that are associated with the same default security
group. No other inbound traffic is allowed.
Looking at the outbound rules above, all network traffic out is allowed by the default security
group. This includes traffic out to the Internet since a default VPC will have a route to a default
internet gateway.
a. Click here to open the Elastic IPs part of the EC2 console in a new
window and click Allocate New Address.
b. Set EIP used in: to VPC and click Yes, Allocate.
Note: There is no charge for Elastic IP addresses (EIPs) that are connected to
running instances. If you remove the instance (e.g. the EIP is no longer
connected to a running instance) then there is a cost of $0.005/hr for the EIP).
32) I have aplication in mumbai region and suddenly the region is goes down, application will be
impacted, how to resolve/avoid this.
33) have you written any terraform file to create AWS resources?
EFS is:
Generally Available (out of preview), but may not yet be available in your region
Network filesystem (that means it may have bigger latency but it can be shared across several
instances; even between regions)
It is expensive compared to EBS (~10x more) but it gives extra features.
It's a highly available service.
It's a managed service
You can attach the EFS storage to an EC2 Instance
Can be accessed by multiple EC2 instances simultaneously
Since 2016.dec.20 it's possible to attach your EFS storage directly to on-premise servers via
Direct Connect. ()
EBS is:
A block storage (so you need to format it). This means you are able to choose which type of file
system you want.
As it's a block storage, you can use Raid 1 (or 0 or 10) with multiple block storages
It is really fast
It is relatively cheap
With the new announcements from Amazon, you can store up to 16TB data per storage on SSD-
s.
You can snapshot an EBS (while it's still running) for backup reasons
But it only exists in a particular region. Although you can migrate it to another region, you
cannot just access it across regions (only if you share it via the EC2; but that means you have a
file server)
You need an EC2 instance to attach it to
New feature (2017.Feb.15): You can now increase volume size, adjust performance, or change
the volume type while the volume is in use. You can continue to use your application while the
change takes effect.
It supports only allow rules, and by It supports both allow and deny rules,
default, all the rules are denied. You and by default, all the rules are denied.
cannot deny the rule for establishing You need to add the rule which you can
a connection. either allow or deny it.
It is a stateful means that any It is a stateless means that any changes
changes made in the inbound rule made in the inbound rule will not reflect
will be automatically reflected in the the outbound rule, i.e., you need to add
outbound rule. For example, If you the outbound rule separately. For
are allowing an incoming port 80, example, if you add an inbound rule port
then you also have to add the number 80, then you also have to
outbound rule explicitly. explicitly add the outbound rule.
All the rules are evaluated before Rules are evaluated in order, starting from
deciding whether to allow the traffic. the lowest number.
The main difference is the route for 0.0.0.0/0 in the associated route table.
A private subnet sets that route to a NAT instance. Private subnet instances only need a
private ip and internet traffic is routed through the NAT in the public subnet. You could
also have no route to 0.0.0.0/0 to make it a truly private subnet with no internet access
in or out.
A public subnet routes 0.0.0.0/0 through an Internet Gateway (igw). Instances in a public
subnet require public IPs to talk to the internet.
39) How can someone excess sitting on-site access your instance inside your private subnet
Bastionhost:
A bastion host is a server whose purpose is to provide access to a private network from an external
network.
VPN:
Is mainly used to establish a secure and private tunnel from you network or device to aws network
• Aws site-to-site vpn: enables you to securely connect your on-premises network to your
vpc.
• AWS client vpn : enables you to securely connect users to AWS or on premises network.
40) How will you you establish connection bw public and private subnet
a NAT gateway will allow the EC2 instances in the private subnets to connect to the
internet and achieve high availability.
Amazon RDS Read Replicas enable you to create one or more read-only copies
of your database instance within the same AWS Region or in a different AWS
Region. Updates made to the source database are then asynchronously copied to
your Read Replicas.
To create a DB instance
1. Sign in to the AWS Management Console and open the Amazon RDS console
at https://console.aws.amazon.com/rds/.
2. In the upper-right corner of the Amazon RDS console, choose the AWS Region in which you
want to create the DB instance.
3. In the navigation pane, choose Databases.
4. Choose Create database.
5. In Choose a database creation method, select Standard Create.
6. In Engine options, choose the engine type: MariaDB, Microsoft SQL Server, MySQL, Oracle,
or PostgreSQL. Microsoft SQL Server is shown here.
7. For Edition, if you're using Oracle or SQL Server choose the DB engine edition that you want to
use.
MySQL has only one option for the edition, and MariaDB and PostgreSQL have none.
8. For Version, choose the engine version.
9. In Templates, choose the template that matches your use case. If you choose Production, the
following are preselected in a later step:
Multi-AZ failover option
Provisioned IOPS storage option
Enable deletion protection option
By default, the new DB instance uses an automatically generated password for the master user.
11. For the remaining sections, specify your DB instance settings. For information about each
setting, see Settings for DB instances.
12. Choose Create database.
If you chose to use an automatically generated password, the View credential details button
appears on the Databases page.
On the RDS console, the details for the new DB instance appear. The DB instance has a status
of creating until the DB instance is created and ready for use. When the state changes
to available, you can connect to the DB instance. Depending on the DB instance class and
storage allocated, it can take several minutes for the new instance to be available.
● IAM provides access to accounts services where we can manage User, Roles, Groups & Policy password
policy.
● It applies globally to all AWS regions.
Users: we create users and assign necessary permissions to them in the form of policies.
Groups: We can create groups for ex. Dev QA etc. and attach policies at the group level.
Policy:
A policy is a set of permission
Always explicit deny overrides explicit allow
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": *,
"Resource": "*"
},
Policy types:
1. Identity Based Policy: Applicable on users, groups of users, and roles
● AWS Managed policy:
● Custom Managed Policy:
● Inline Policy
2. Resource Based policy: Attach to a resource such as an Amazon S3 bucket
3. Session based Policy: create a temporary session for a role or federated user
Imp Notes:
More than one policy can be attached to a user or a group at the same time.
Policies can’t be attached directly to resources like EC2 instance, S3 bucket etc.,
Basic Policy structure:
6. Set the new size for your EBS volume (in this case i extended an
8GB volume to 20GB)
7. Click on modify.
Now, we need to extend the partition itself.
SSH to the EC2 instance where the EBS we’ve just extended is
attached to.
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT xvda 202:0 0 20G 0 disk └─xvda1 202:1
0 8G 0 part /
As you can see size of the root volume reflects the new size, 20GB,
the size of the partition reflects the original size, 8 GB, and must be
extended before you can extend the file system.
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT xvda 202:0 0 20G 0 disk └─xvda1 202:1
0 20G 0 part /
Last but not least, we need to extend the filesystem itself.
If your filesystem is an ext2, ext3, or ext4, type:
[ec2-user ~]$ sudo resize2fs /dev/xvda1
If your filesystem is an XFS, then type:
[ec2-user ~]$ sudo xfs_growfs /dev/xvda1
Finally we can check our extended filesystem by typing:
[ec2-user ~]$ df -h
If everything went right, we should be able to see our effective
filesystem extended size:
Filesystem Size Used Avail Use% Mounted on devtmpfs 980M 0 980M 0% /dev tmpfs
997M 0 997M 0% /dev/shm tmpfs 997M 440K 997M 1% /run tmpfs 997M 0 997M 0%
/sys/fs/cgroup /dev/xvda1 20G 1,4G 19G 7% /
You have just extended your EBS volume size with 0 downtime, enjoy!
OS-level settings
1. To optimize network and OS performance, configure the following settings
in /etc/sysctl.conf file of Linux. These settings specify a larger port range, a more effective
TCP connection timeout value, and a number of other important parameters at the OS-
level.
It is not recommended to use net.ipv4.tcp_tw_recycle = 1 when working with network address
translation (NAT), such as if you are deploying products in EC2 or any other
environment configured with NAT.
net.ipv4.tcp_fin_timeout = 30
fs.file-max = 2097152
net.ipv4.tcp_tw_recycle = 1
net.ipv4.tcp_tw_reuse = 1
net.core.rmem_default = 524288
net.core.wmem_default = 524288
net.core.rmem_max = 67108864
net.core.wmem_max = 67108864
2. To alter the number of allowed open files for system users, configure the following
settings in /etc/security/limits.conf file of Linux (be sure to include the leading * character).
* soft nofile 4096
Memory settings
Memory allocated for ELB should be increased to the following. Note that you need to modify
the <ELB_HOME>/bin/wso2server.sh file to make this change.
-Xms1500m -Xmx3000m
-XX:PermSize=256m -XX:MaxPermSize=512m
worker_pool_size_core=400
worker_pool_size_max=500
io_buffer_size=16384
52) What is ALB,ELB
ELB (Elastic load balancing) helps to make your applications highly available by using
health checks and distribute the incoming traffic multiple targets such as Amazon EC2
instances, containers, IP addresses, and Lambda functions.
ALB (Application Load Balancer) is a type of ELB that is mainly designed for web
applications with HTTP and HTTPS. This is named Application Load Balancer
because this load balancer works at the Application layer. In ALB, we register targets
in target groups and route it routes requests to its target groups.
Amazon S3 has a simple web services interface that you can use to store and retrieve any amount
of data, at any time, from anywhere on the web.
It’s a traditional upload where you will upload the object in one part
A single operation upload can upload the file up to 5GB in size.
Upload object in parts:
Using multipart upload, you can upload the large objects up to 5TB.
You can use multipart upload for the objects from 5MB to 5TB in size.
Limitation of S3 bucket:
Only 100 buckets can be created per account.
Can hold unlimited objects
S3 Storage classes:
Standard:
Designed for general- and all-purpose storage
Default storage option
99.999999999% object durability
99.99% object availability
Most expensive storage class.
Reduced Redundancy storage
Designed for non-critical objects
99.99% object durability
99.99% object availability
Less expensive than standard
Infrequent access
Designed for less frequently accessed objects.
99.999999999% object durability
99.99% object availability
Less expensive than reduced redundancy storage
Glacier
Designed for long term archival storage
May take several hours to retrieve the objects from this storage
Cheapest s3 storage class