Data Engineering

Question 1

Amazon Kinesis Data Streams

Answer 1

Collect and store streaming data in real-time
1. Retention up to 365 days (can’t be deleted until it expires)
2. Data ordering guarantee

Question 2

Kinesis Data Streams - Capacity Modes

Answer 2

Provisioned mode

• Choose number of shards
• Each shard gets 1000 records per second
• Scale manually
• Pay per shard provisioned per hour

On demand mode:

• Default capacity provisiones (4000 records per second)
• Scale based on observed throughput peaks
• Pay per stream per hour & data in/out

Question 3

Amazon Data Firehose - AWS Destinations

Answer 3

• Amazon S3: Supports compression
• Amazon Redshift (copy through S3)
• Amazon OpenSearch

Question 4

Amazon Data Firehose

Answer 4

1. Receive records (up to 1MB) from producers
2. Can make data transformation with Lambda functions
3. Batch writes to destinations based on buffer time and size, Near Real Time
4. All or failed data can be backup in S3

Question 5

Firehose Buffer Sizing

Answer 5

Firehose accumulates records in a buffer and is flushed based on time and size rules

Question 6

Amazon Managed Service for Apache Flink

Answer 6

Framework for processing data streams. Can’t read from Amazon Data Firehose.

Question 7

Amazon Managed Streaming For Apache Kafka

Answer 7

Amazon MSK creates & manages Kafka brokers nodes.
* Deployed in your VPC, multi-AZ
* Data stores on EBS volumes

Question 8

Kinesis Data Streams vs Amazon MSK

Answer 8

Kinesis Data Streams:
* 1 MB message size limit
* 12 months maximum retention
* Shard splitting and merging

Amazon MSK:
* Configure for bigger messages
* No retention limit
* Can only add partitions to a topic

Question 9

AWS Batch

Answer 9

Run batch jobs as Docker images

Question 10

AWS Batch - Multi node mode

Answer 10

Leverages multiple EC2 / ECS instances. One main node and multiple childs. Doesn’t work with Spot Instances.

Question 11

Amazon Elastic MapReduce

Answer 11

EMR creates Hadoop clusters in a single AZ to analyze and process vast amount of data.
* Can access data in DynamoDB and S3

Question 12

EMR File System

Answer 12

EMRFS stores persistent data in Amazon S3 while providing data encryption

Question 13

EMR node types

Answer 13

• Master node: Manage the cluster
• Core node: Run tasks and store data
• Task node: Justo to run tasks, usually uses spot

Question 14

EMR instance configuration

Answer 14

• Uniform instance groups: select a single instance type and purchasing option for each node. Has auto scaling.
• Instance fleet: Select target capacity, mix instance types and purchasing options. No auto scaling.

Question 15

AWS Glue

Answer 15

Managed extract, transform and load service

Question 16

AWS Glue Data Catalog

Answer 16

Data crawler writes metadata of databases and supportes storages to the data catalog

Question 17

Redshift

Answer 17

Online analytical processing
* Columnar storage of data: easier to aggregate
* Massively Parallel Query Execution
* Not all clusters support multi AZ

Question 18

Redshift nodes

Answer 18

Leader node: query planning and results aggregation
Compute node: performing queries and send results to leader

Question 19

Redshift Enhanced VPC Routing

Answer 19

Copy and unload goes through VPC

Question 20

Redshift snapshots

Answer 20

Point in time backups stored in S3
* Incremental
* Restore into a new cluster
* Automated or manual
* Copy automatically to another region

Question 21

Redshift spectrum

Answer 21

Query data that is in S3 without loading it
* Must have a Redshift cluster to start the query

Question 22

Redshift resource link

Answer 22

Data catalog object which is linked to a database, allows integrated services to run queries on the database. These services will not be able to access directly across cross accounts.

Question 23

Redshift Workload Management

Answer 23

Flexibly manage queries priorities within workloads
* Multiple query queues
* Route queries to the appropriate queue

Question 24

Redshift EVEN distribution

Answer 24

The leader node distributes the rows across the slices in a round-robin fashion, regardless of the values. Is appropriate when a table doesn’t participate in joins

Question 25

Redshift KEY distribution

Answer 25

The leader node places matching values in one column on the same node slice. Matching values from the common columns are physically stored together.

Question 26

Redshift ALL distribution

Answer 26

A copy of the entire table is distributed to every node. Ensures that every row is collocated for every join. Appropriate only for relatively slow moving tables

Question 27

DocumentDB

Answer 27

Fully managed MongoDB used to store, query and index JSON data * Replication across 3 AZ * Automatically grows in increments of 10GB

Question 28

DocumentDB Pricing

Answer 28

* On-demand instances per second * Database I/O * Database storage per GB * Backup storage per GB

Question 29

Amazon Timestream

Answer 29

Fully managed time series database * Faster and cost effective comparing with relational database * SQL compatibility * Recent stored in memory, historical cost optimized storage * Built-in time series analytics functions * You ccan configure schedule queries

Question 30

Timestream Data Stores

Answer 30

1. **Memory store:** high throughput data writes, and reads performed around current time 2. **Magnetic store:** optimize cost, optimized for large volume data reads

Question 31

Amazon Athena

Answer 31

**Serverless** query service to **analyze** data stored in **S3** * Commonly used with Quicksight

Question 32

Athena Performance Improvement

Answer 32

* Columnar data for cost savings. Parquet or ORC is recommended * Compress data for small retrievals * Partition datasets in S3 (Partioning pruning) * Use larger files to minimize overhead * Predicate pushdown to filter data at source

Question 33

Amazon QuickSight

Answer 33

Serverless business intelligence service to create interactive dashboards

Question 34

Amazon Forecast

Answer 34

Fully managed service that uses statistical and machine learning algorithms to deliver highly accurate time-series forecasts. To import your data you must store it in an S3 bucket.