HPC on AWS

Question 1

Data Transfer - HPC

Answer 1

Snowball, snowmobile (terabytes/petabytes worth of data

AWSDataSync to store on s3,EFS,FSx for Windows etc

Direct Connect - dedicated private line from your data center to AWS. reducce your network costs, increase bandwidth throughput, and provide a more consistent network experience than internet connections

Question 2

Compute & Networking - HPC

Answer 2

EC2 instances that are cPU or GPU optimized

EC2 fleets ( spot instances or spot fleets)

Placement Groups ( Cluster placement groups)

Enhanced Networking
Elastic Network Adapters - higher i/o lower CPU.
It usses single root I/O virtualization (SR-IOV) to provide high performance networking capabilities .. higher i/o lower CPU. No additional charge for enhanced networking.

Elastic Fabric Adapters - Lower or more consistency and higher throughput.
OS-Bypass
is a network device that you attach to your ec2 instance to accelerate HPC and machine learning applications.

Question 3

Storage services that allow HPC

Answer 3

Instance attached storage:
EBS: scales up to 64,000 iops with provisioned iops.
instance store : scales up to millions of iops; low latency.

Network storage:
Amazon s3: Distributed object based storage not a file system.

Amazon EFS: Scales iops based on total size or use provisioned iops

Amazon Fsx for lustre = HPC optimized distributed file system; millions of iops; which is also backed by s3.

Question 4

Orchestration and Automation services

Answer 4

AWS Batch :

which lets you run thousands of batch computing jobs.
allows to run a single job which spans in multiple ec2 instances
you can easily schedule jobs and launch Ec2 instances according to your needs

AWS Parallel Cluster:

Open source cluster management tool that makes it easy for you to deploy and manage hPC clusters on AWS
PARALLEL CLUSTERS USES A SIMPLE FILE TO MODEL AND PROVISION ALL THE RESOURCES NEEDED.

AUTOMATES CREATION OF VPC, SUBNET, CLUSTER TYPES AND INSTACNE STYPES