Data Wrangling

Question 1

Data wrangling consists of…

Answer 1

Data preparation, Data Munging, Data Transformation

Question 2

What is the curse of dimensionality and how to solve it?

Answer 2

• if there are many more dimensions than data points, data become sparse
• Huge areas of the multi-dimensional space are empty
• Hard to draw conclusions
• SOLUTION —> dimensionality reduction: flattening the points onto a few clever chosen dimensions in this huge space.

Question 3

Wrangling Stages

Answer 3

a. Raw: data ingestion and discovery — “unboxing”
- Data unboxing
 What data do I have? What do I want to do with it?
Basic tools
- UNIX command line
- SublimeText editor
Trifacta: free visual data wrangling tool
- Codifies some good practices you can also follow by hand
Python’s Pandas library
b. Refined: curating data for reuse
- What: data warehousing, canonical models
- Who: data curators, IT engineers, actuaries …
c. Production: Ensuring feeds and workflows
 - What: recurrent, automated use cases
 - Who: often involves SW engineers and IT/ops folks

Question 4

Data Wrangling issues

Answer 4

• Structure: the shape of the data
• Regular data: easy to access, filter, tabulate; Two variants: Relations (tables, data frames); Matrices (manipulate with algebra)
• Granularity: how fine/coarse is each datum
• Faithfulness: how well does the data capture “reality”; Outliers; Functional dependencies; Correlations; Good data cleaning
• Temporality: how is the data situated in time
• Scope: how (in)complete is the data