What is Cloud Dataflow?
- A fully managed service for transforming and enriching data in stream (real-time) and batch (historical) modes with equal reliability and expressiveness.
- Serverless approach to resource provisioning and management provides limitless capacity for the biggest data processing challenges, pay only for what you use.
What are some Use Cases for Cloud Dataflow?
- Clickstream, Point-of-Sale and segmentation analysis in retail
- Check fraud detection in financial services
- Check personalized user experience in gaming
- Check IoT analytics in manufacturing, healthcare and logistics
What are the advantages to using Cloud Dataflow?
Particularly for out-of-order processing and real-time session management.
1) Accelerate development for batch and streaming: provides simple pipeline development via Java and Python API’s in the Apache Beam SDK, code reuse.
2) Simplify operations and management: serverless approach removes operational overhead.
3) Integrates with Stackdriver for a unified logging and monitoring and alerting solution.
4) Builds on a foundation for machine learning, by adding ML models and API’s to your data processing pipelines.
5) Integrates seamlessly with GCP services for streaming events ingestion (Pub/Sub), data-warehousing (BigQuery), machine learning (Cloud Machine Learning), and more. Beam-based SDK allows customer extenstions for alternative execution engines like Apache Spark via Cloud Dataproc or on-premises.
What are features of Cloud Dataflow?
- Automated resource management
- Dynamic work re-balancing
- Reliable and consistent ‘exactly-once’ processing, fault-tolerant execution
- Horizontal auto-scaling, results in better price-to-performance
- Unified programming model (Apache Bean SDK) provides MapReduce-like operations, powerful data-windowing and fine-grained correctness control for streaming and batch data
- Community-driven Innovation: Dataflow can be extended by forking or contributing to Apache Beam
What are the different types of Triggers provided by Dataflow?
- Time-based (Default)
- operate on a time reference, either event time or processing time
- AfterWatermark, AFterProcessingTime
- Data-driven
- examines data as it arrives and fires on a specified data condition, for example after a set number
- AfterPane.elementCountAtLeast
- Composite
- comines time-based and data-driven, fires when all triggers are met
What are the two basic primitives in Dataflow?
- PCollections: represents data-sets, across which parallel transformations may be performed. The ‘P’ is for parallel.
- PTransforms: applied to PCollections to create new PCollections. The perform Element tranformations, aggregate multiple elements together, or may be a composite combination of other PTransforms.
What does Triggering provide for Windowing and Watermarks?
- Tackles the problem of Watermarks being too fast or too slow
- Allows control of the materialization for a Window given some regular interval or other data condition
- Too slow might use processing time to trigger
- Too fast might use element or transaction counts to trigger
What are some differences between Dataflow and Spark?
- Spark uses per processing-time window, or per micro-batch, reflects events as they arrived at the pipeline, lack of event-time windowing support means emulating with available API’s, additional code and duplicated logic.
- Dataflow uses per event-time window with Watermarks, reflects events as they happened. Clear and modular.
What are the four major concepts when you think about data processing with Dataflow?
- Pipelines
- PCollections
- Transforms
- I/O Sources and Sinks
What is a Pipeline?
A pipeline encapsulates an entire series of computations that accepts some input data from external sources, transforms that data to provide some useful intelligence, and produces some output data. That output data is often written to an external data sink. The input source and output sink can be the same, or they can be of different types, allowing you to easily convert data from one format to another.
Each pipeline represents a single, potentially repeatable job, from start to finish, in the Dataflow service.
What is a PCollection?
A PCollection represents a set of data in your pipeline. The Dataflow PCollection classes are specialized container classes that can represent data sets of virtually unlimited size. A PCollection can hold a data set of a fixed size (such as data from a text file or a BigQuery table), or an unbounded data set from a continuously updating data source (such as a subscription from Google Cloud Pub/Sub).
PCollections are the inputs and outputs for each step in your pipeline.
PCollections are optimized for parallelism, unlike the standard JDK Collection class.
What is a Transform?
A transform is a data processing operation, or a step, in your pipeline. A transform takes one or more PCollections as input, performs a processing function that you provide on the elements of that PCollection, and produces an output PCollection.
Your transforms don’t need to be in a strict linear sequence within your pipeline. You can use conditionals, loops, and other common programming structures to create a branching pipeline or a pipeline with repeated structures. You can think of your pipeline as a directed graph of steps, rather than a linear sequence.
In a Dataflow pipeline, a transform represents a step, or a processing operation that transforms data. A transform can perform nearly any kind of processing operation, including performing mathematical computations on data, converting data from one format to another, grouping data together, reading and writing data, filtering data to output only the elements you want, or combining data elements into single values.
Transforms in the Dataflow model can be nested—that is, transforms can contain and invoke other transforms, thus forming composite transforms.
What are I/O Sources and Sinks?
The Dataflow SDKs provide data source and data sink APIs for pipeline I/O. You use the source APIs to read data into your pipeline, and the sink APIs to write output data from your pipeline. These source and sink operations represent the roots and endpoints of your pipeline.
The Dataflow source and sink APIs let your pipeline work with data from a number of different data storage formats, such as files in Google Cloud Storage, BigQuery tables, and more. You can also use a custom data source (or sink) by teaching Dataflow how to read from (or write to) it in parallel.
What are the Dataflow SDK options and advantages?
For SDK 2.x:
Language: Java and Python
Automatic patches and updates by Google
Issue and new version communications
Tested by Google
Eclipse Plugin Support
What questions should you consider when designing your pipeline?
- Where is your input data stored?
- What does your data look like?
- What do you want to do with your data?
- What does your output data look like, where should it go?
What is the basic design behind a Pipeline?
A simple pipeline is linear: Input > Transform(PCollection) > Output
But can be much more complex, a pipeline represents a Directed Acyclic Graph of steps. It can have multiple input sources, multiple output sinks, and its operations (transforms) can output multiple PCollections.
It’s important to understand that transforms do not consume PCollections; instead, they consider each individual element of a PCollection and create a new PCollection as output. This way, you can do different things to different elements in the same PCollection.
What are the parts of a Pipeline?
A pipeline consists of two parts:
- Data - PCollection for input, intermediate and output
- Transforms applied to that data
- Core Transforms - basic processing operation
- The Dataflow SDKs supply core transforms such as ParDo and GroupByKey, Combine and Flatten(merge), as well as other core transforms for combining, merging, and splitting data sets.
- Composite Transforms - combining multiple transforms into larger composite transforms. Cloud SDK also provides composite tranforms for combining data, map/shuffle/reduce and statistical analysis.
- Root Transforms - often used to create the initial PCollection, reading data from a data source - pipeline i/o
- Core Transforms - basic processing operation
How does Dataflow perform data encoding?
When you create or output pipeline data, you’ll need to specify how the elements in your PCollections are encoded and decoded to and from byte strings. Byte strings are used for intermediate storage as well reading from sources and writing to sinks. The Dataflow SDKs use objects called coders to describe how the elements of a given PCollectionshould be encoded and decoded.
You typically need to specify a coder when reading data into your pipeline from an external source (or creating pipeline data from local data), and also when you output pipeline data to an external sink.
You set the Coder by calling the method .withCoderwhen you apply your pipeline’s Read or Write transform.
What are the PCollection limitations?
- A PCollection is immutable. Once created, you cannot add, remove, or change individual elements. (new PCollection is used)
- A PCollection does not support random access to individual elements.
- A PCollection belongs to the pipeline in which it is created. You cannot share a PCollection between Pipeline objects.
What is meant by bounded and unbounded PCollections, and what data-sources/sinks create and accept them?
- Bounded
- fixed data set, known size that does not change
- server logs from a month, orders from last week
- TextIO, BigQueryIO, DataStoreIO and Custom Bounded IO produce and accept bounded PCollections
- Unbounded
- continuously updating dat set, steaming data
- server logs as they are generated; all new orders as they are processed
- PubsubIO, Custom Unbounded IO produce
- PubsubIO, BigQueryIO accept
What is Dataflow’s default Windowing behavior?
Dataflow’s default windowing behavior is to assign all elements of a PCollection to a single, global window, even for unbounded PCollections. Before you use a grouping transform such as GroupByKey on an unbounded PCollection, you must set a non-global windowing function.
If you don’t set a non-global windowing function for your unbounded PCollection and subsequently use a grouping transform such as GroupByKey or Combine, your pipeline will generate an error upon construction and your Dataflow job will fail.
You can alternatively set a non-default Trigger for a PCollection to allow the global window to emit “early” results under some other conditions. Triggers can also be used to allow for conditions such as late-arriving data.
What I/O API’s (read and write transforms) are available in the Dataflow SDK’s?
- Avro files
- BigQuery files
- Bigtable
- Datastore
- Pub/Sub
- Text files
- Custom data sources and sinks
What security does Dataflow provide to keep your data secure and private?
- Pipeline Submission - project level account permission (email address based); auth required for gcloud and then pipelines submitted over HTTPS.
- Pipeline Evaluation - temporary data stored in Cloud Storage are encrypted at rest and do not persist after pipeline eval concludes.
- Cloud Logging only shows what your code produces
- In-Flight Data - communications between sources, workers and sinks are encrypted and carried over HTTPS.
- Data Locality - Zone selection and VPN provide narrow and secure options for pipeline logic evaluation on the workers or compute instances
How is a Datflow Pipeline constructed, what are the general steps?
- Create a Pipeline object
- Use a Read or Create tansform to create one or more PCollections for your pipeline data.
- Apply transforms to each PCollection to change, filter, group, analyze or process elements in a PCollection. Each transform creates a new output PCollection, to which additional transforms may be applied until processing is complete.
- Write or otherwise output the final, transformed PCollection(s)
- Run the pipeline
————– code example ———————————————
// Start by defining the options for the pipeline. PipelineOptions options = PipelineOptionsFactory.create();
// Then create the pipeline. Pipeline p = Pipeline.create(options);
// Read the data
PCollection<string> lines = p.apply(<br> TextIO.Read.named("ReadMyFile").from("gs://some/inputData.txt"));</string>// Apply transforms PCollection<string> words = ...;<br>PCollection<string> reversedWords = words.apply(new ReverseWords());</string></string>
// Write out final pipeline data
PCollection<string> filteredWords = ...;<br>filteredWords.apply(TextIO.Write.named("WriteMyFile").to("gs://some/outputData.txt"));</string>// Run the pipeline p.run();
