Compute & Distributed Processing

Question 1

What is the main goal of distributed data processing frameworks?

Answer 1

To process large datasets by splitting work across multiple machines so that jobs complete faster than on a single node.

Question 2

What is data parallelism in distributed processing?

Answer 2

Running the same operation on different partitions of the data in parallel across multiple workers.

Question 3

What is task parallelism in distributed processing?

Answer 3

Running different tasks or stages in parallel, possibly on the same or different data, to increase throughput.

Question 4

What is the basic idea of MapReduce?

Answer 4

Apply a map function to transform records into key-value pairs, then a reduce function to aggregate values for each key across the cluster.

Question 5

Why was MapReduce historically important for big data?

Answer 5

It provided a simple programming model and fault-tolerant execution over large clusters for batch jobs on file systems like HDFS.

Question 6

What are the main limitations of classic MapReduce?

Answer 6

Disk-heavy execution with repeated writes between stages, limited expressiveness for complex DAGs, and relatively high latency.

Question 7

What is a DAG (Directed Acyclic Graph) in modern data engines?

Answer 7

A graph of computation stages where nodes are operations and edges represent data dependencies, with no cycles.

Question 8

Why do engines like Spark use DAGs instead of fixed map/reduce stages?

Answer 8

DAGs allow more complex multi-stage workflows and better global optimization across operations.

Question 9

What is an execution engine in a distributed system?

Answer 9

The component that schedules tasks, coordinates workers, manages data locality, and handles retries and failures for jobs.

Question 10

What is data locality and why does it matter?

Answer 10

Processing data close to where it is stored to minimize network I/O and improve performance.

Question 11

What is Apache Spark conceptually?

Answer 11

A distributed data processing engine that supports batch, streaming, SQL, and ML workloads using DAG-based execution and in-memory computation.

Question 12

What is an RDD in Spark?

Answer 12

A Resilient Distributed Dataset representing an immutable, partitioned collection of records that can be processed in parallel.

Question 13

Why did Spark introduce higher-level APIs like DataFrames?

Answer 13

DataFrames provide schema-aware, declarative operations that allow the optimizer to generate more efficient execution plans than RDD code.

Question 14

What is a DataFrame in Spark-like systems?

Answer 14

A distributed, tabular data structure with named columns and a schema, similar to a table in a relational database.

Question 15

Why are DataFrames generally preferred over raw RDDs for most workloads?

Answer 15

They enable query planning, optimization, and code generation, often yielding better performance with less boilerplate code.

Question 16

What is lazy evaluation in Spark or similar engines?

Answer 16

Transformations build a logical plan but are not executed until an action is invoked, allowing optimizations before execution.

Question 17

What is a transformation vs an action in Spark-style APIs?

Answer 17

Transformations (e.g., map, filter) define new datasets from existing ones; actions (e.g., count, collect) trigger execution and return results.

Question 18

What is shuffle in distributed processing?

Answer 18

A data movement stage where data is redistributed across the cluster, typically to group records by key for joins or aggregations.

Question 19

Why are shuffles expensive?

Answer 19

They involve network I/O, disk writes, and coordination across nodes, often becoming the bottleneck in distributed jobs.

Question 20

How can you reduce the cost of shuffles?

Answer 20

Reduce data volume before shuffles, pick join strategies carefully, partition data appropriately, and avoid unnecessary group-by operations.

Question 21

What is a wide transformation?

Answer 21

A transformation that requires data from many partitions, such as groupByKey, that typically triggers a shuffle.

Question 22

What is a narrow transformation?

Answer 22

A transformation like map or filter where each output partition depends on data from a single input partition, avoiding shuffles.

Question 23

What is vectorized execution in query engines?

Answer 23

Processing batches of values per column at a time using CPU-friendly operations, improving cache utilization and throughput.

Question 24

Why is vectorized execution faster than row-at-a-time processing?

Answer 24

It reduces per-row overhead, uses CPU instructions more efficiently, and benefits from modern CPU pipelines and caching.

Question 25

What is the difference between an interpreter-based and codegen-based execution engine?

Answer 25

Interpreter-based engines interpret expressions dynamically; codegen-based engines compile expressions to bytecode or native code for faster execution.

Question 26

How do modern engines combine columnar storage with vectorized execution?

Answer 26

They read column chunks into columnar in-memory structures and apply vectorized operators over these batches during query execution.

Question 27

What is a worker node in a distributed cluster?

Answer 27

A machine that executes tasks assigned by the driver or master node and holds partitions of data in memory or disk.

Question 28

What is a driver (or master) node?

Answer 28

The component that coordinates job execution, constructs DAGs, schedules tasks, and aggregates results from workers.

Question 29

Why is fault tolerance necessary in distributed processing?

Answer 29

With many nodes, failures are common; fault-tolerant systems can recompute or recover data without losing the entire job.

Question 30

How do RDDs provide fault tolerance?

Answer 30

By tracking the lineage of transformations so lost partitions can be recomputed from original data and operations.

Question 31

What is checkpointing in distributed systems?

Answer 31

Persisting intermediate results to durable storage so recomputation on failure does not have to start from the very beginning.

Question 32

Why is memory management critical in engines like Spark?

Answer 32

Too little memory leads to spilling to disk or out-of-memory errors; using memory efficiently is key to performance and stability.

Question 33

What is spilling in distributed processing?

Answer 33

Writing intermediate data from memory to disk when memory is insufficient, typically during shuffles or aggregations.

Question 34

Why can too many small partitions hurt performance?

Answer 34

They cause overhead in task scheduling, context switching, and bookkeeping, reducing efficiency.

Question 35

Why can too few partitions also hurt performance?

Answer 35

Work may not be evenly distributed, and some tasks become hotspots while other cores sit idle.

Question 36

What is a reasonable guideline for partition sizing?

Answer 36

Aim for enough partitions to keep all cores busy, with partition sizes large enough to amortize overhead, often tens to hundreds of MB per partition.

Question 37

What is data skew in distributed processing?

Answer 37

An uneven distribution of data across partitions or keys, where some partitions or keys have much more data than others.

Question 38

Why is data skew problematic?

Answer 38

Skewed partitions or keys create straggler tasks that delay job completion and may exhaust resources on specific workers.

Question 39

What strategies help mitigate skew?

Answer 39

Key salting, pre-aggregation, splitting hot keys, or redesigning data and queries to distribute work more evenly.

Question 40

What is a broadcast join?

Answer 40

A join strategy where a small table is sent to all workers so that each partition can join locally without a shuffle of both sides.

Question 41

When is a broadcast join appropriate?

Answer 41

When one side of the join is small enough to fit in memory on each worker, significantly reducing shuffle cost.

Question 42

What is a sort-merge join?

Answer 42

A join algorithm that sorts both inputs on the join key and then merges them, often used for large datasets when properly partitioned.

Question 43

Why is understanding join strategies important for distributed processing?

Answer 43

Joins are common and expensive; choosing the right strategy significantly impacts performance and resource usage.

Question 44

What is a good one-sentence mental model for distributed compute engines?

Answer 44

They break data into partitions, build an optimized DAG of operations, and schedule tasks across a cluster to transform partitions in parallel while minimizing shuffles and skew.