1. Code Club(Wrangling Data With Python)
Tony Hirst & Sam Leon
@psychemedia / tony.hirst @ okfn.org
Week 4

2. Week4–LearningObjectives
By the end of this session, you will be able to:
• Recall and make use of what we did previously…
• Be able to merge data from different data
frames
• Be familiar with the idea of “tidy data”
• Have taken first steps in being able to
reshape datasets (long-wide data
transforms, pivots)
• Be able to start cleaning datasets using a
variety of strategies

3. Weeks1-3–Recap
In previous weeks, you have learned…
• How to getting data into and out of pandas from a
variety of file types (CSV, XSLSX) on your computer
and the web, as well as from HTML wen pages and
the World Bank Indicators API
• How Python can use lists to represent data and
how pandas represents tabular data using a data
frame
• How to filter, sort and generally manipulate tabular
data using pandas
• How to process data columns and derive new ones

4. Why Code?

5. Ref:PythonforFinance
Efficiency and Productivity Through Python
From Prototyping to Production
Financial industry context
- “quants” develop proof-of-concept models in
eg Matlab or R
- developers translate applications into
production code (C++, Java)
Inefficiencies – prototype code not reusable
Diverse skill sets – different programming languages & regimes
Legacy code – maintenance and development becomes complex

6. Ref:PythonforFinance
Efficiency and Productivity Through Python
Shorter time to results
- eg ability to download data directly source API
- eg ability to perform vectorised, column based
operations,
such as cumulative sum operations
- eg ability to reshape datasets
- eg ability to generate charts directly from
appropriately shaped data

7. ?pd.read_csv
FindingHelp…
pd.<TAB>
df. <TAB>
pd.read_csv.<TAB>

8. Ref:PythonforFinance
Efficiency and Productivity Through Python
A consistent technological framework
“Python has the potential to provide
a single, powerful, consistent
framework with which to streamline
end to end development and
production efforts…”

9. Fromthecommandline…
cd “/path/to/my file”
(no ! required.....)

10. Merging
Data

11. .reset_index()

12. Mergiongdata…
pd.merge()

13. Reshaping data
Wide and long format data
“Tidy data”
.melt() and .pivot()

15. Hadley Wickham, “Tidy Data”, Journal of Statistical Software
http://vita.had.co.nz/papers/tidy-data.pdf

16. Reshapingdata:melt
pd.melt()
Index COL_A COL_B COL_C
ROW1 VAL_A1 VAL_B1 VAL_C1
ROW2 VAL_A2 VAL_B2 VAL_C2
Index variable value
ROW1 COL_A VAL_A1
ROW2 COL_A VAL_A2
ROW1 COL_B VAL_B1
ROW2 COL_B VAL_B2
ROW1 COL_C VAL_C1
ROW2 COL_C VAL_C2
pd.melt( df, id_vars=“COL_A”, value_vars=[“COL_B”,”COL_C”])

17. Reshapingdata:melt
pd.melt()
Reshape:
R rows
M index cols (id_vars=[])
N value cols (value_vars=[])
to:
M index cols
1 variable col
1 value col
Lots of rows…

18. Reshapingdata:melt
pd.pivot()
Index COL_A COL_B COL_C
ROW1 VAL_A1 VAL_B1 VAL_C1
ROW2 VAL_A2 VAL_B2 VAL_C2
df.pivot( index=‘Index’, variable=‘variable’, value=‘value’)
Index variable value
ROW1 COL_A VAL_A1
ROW2 COL_A VAL_A2
ROW1 COL_B VAL_B1
ROW2 COL_B VAL_B2
ROW1 COL_C VAL_C1
ROW2 COL_C VAL_C2

19. Reshapingdata:melt&pivot

20. Cleaning
Data

21. Cleaningoperations
Handle empty rows/cols
Handle duplicate rows
(“deduping”)
Clean up text strings
Map one value to another

22. Drop empty rows
df.dropna(axis=0,how=‘all’)
Drop empty columns
df.dropna(axis=1,how=‘all’)
Emptyandduplciatedata
Remove duplicate rows
df.drop_duplicates()
df.drop_duplicates([columns])

23. Tidywithincolumns
df3['incomeLevel'].str.split(':')

24. StringOperations
http://pandas.pydata.org/pandas-docs/dev/text.html

25. AnewPythondatastructure:dict()
Dictionaries / “dicts” / {}
“associatively addressed lists”

26. Translatingorconvertingvalues
map()