This document discusses how small data can be effectively analyzed using simple command line tools and scripting languages like Ruby. The key points made are: 1) Most data is small in size and does not require large Hadoop clusters for processing - command line tools and scripting are often much faster for small data workloads. 2) The Unix shell is a powerful programming environment that allows stringing together simple commands into powerful pipelines for analyzing and transforming data in flexible ways. 3) Ruby is a great fit for scripting small data tasks and integrating with Unix tools due to its clean syntax, large standard library, and ability to be used for one-liners or full scripts.

1. Mini Munging:
the Joy of Small Data
Rob Miller
https://robm.me.uk/ | @robmil

2. Who are you?

4. What do you do?

5. I mung data

6. mung (mʌŋ)
verb
To transform data from
one form into another,
unrecognisable one.

7. My Toolkit
• Hadoop
• Elasticsearch
• Cassandra

8. My Toolkit
• Hadoop
• Elasticsearch
• Cassandra

9. My data
is small

10. Your data is
(probably)
small too

11. The Small
Data Toolkit
• The command line
• Pipelines
• Ruby!

12. The shell is a
programming
environment

13. Text is a
universal interface

14. Pipelines are
incredibly powerful

15. $ head -1 log.csv
fred@example.com,login,2015-07-20 13:10:11
$ cat log.csv | cut -d, -f1 | sort | uniq -c
25 fred@example.com
107 bob@example.net

16. $ cat log.csv | grep '^bob@example.net,' |
cut -d, -f3 | cut -d' ' -f2 | sort | uniq -c
61 2015-01-20
42 2015-06-18
4 2015-07-20

17. Free functionality,
free parallelism,
composable & modular

18. ”For the same amount of data
I was able to use my laptop
to get the results in about
12 seconds (270MB/sec), while
the Hadoop cluster took about
26 minutes (1.14MB/sec)”
Adam Drake, “Command-line tools can be 235x faster
than your Hadoop cluster”, http://bit.ly/1sS01aP

19. And Ruby fits here too!

20. $ ruby -e
$ ruby -ne
$ ruby -pe
$ ruby -F -ane
$ ruby -r

21. $ head -1 log.csv
fred@example.com,login,2015-07-20 13:10:11
$ cat log.csv | cut -d, -f1 | cut -d@ -f2 |
ruby -rresolv -ne 'puts Resolv.getaddress(chomp)' |
sort | uniq -c | sort -rn
24 10.0.42.1
3 10.27.100.8

22. Start with coreutils,
then throw in Ruby

23. $ ruby -rcsv -rbitly -e 'b =
Bitly.new("user", "foo");
CSV.filter { |r| r.each { |f|
f.replace b.shorten(f).short_url
if f =~ /^https?:/ } }'
urls.csv > urls-shortened.csv

24. % nokogiri -e 'puts @doc.css("img").map { |i|
"https:" + i["src"] }'
https://en.wikipedia.org/wiki/Unix |
xargs -n1 -P4 wget

25. Your shell is a
programming
environment

26. Ruby fits into
this world perfectly

27. Most data is small

28. Go and mung stuff!

29. The most useful
bits of coreutils
cat
grep
head
tail
split
wc
sort
shuff
uniq
comm
cut
paste
join
tr
column

30. Text Processing
with Ruby
• Published by
Pragmatic Bookshelf
• Currently in beta
• https://pragprog.com/
book/rmtpruby/text-
processing-with-ruby