Introduction to Web Scraping in Python

1. Web Scraping with Python
by @sauravtom
(work is progress …)

2. Data Scraping
Automated Process
Specify css or xml path
grab the content
store it in a database

3. Who uses Scrapers ?
Scrapers as backbone of Big Data
Importance in Industry level as well as indie
projects.

4. Why choose python ?
Robust, flexible and powerful
Relatively lesser development time
Easy to learn and use
Huge standard library, thorough documentation and
helpful community.

5. Scraping libraries in python
lxml
BS4
Scrapy
Mechanize
twill
...

6. Scraper Demonstration in bs4
Inspect the element
Find the node
Plug it in
(some code and pictures)

7. Making Scrapers faster
Thread and Queues
(some code ...)

8. Detecting bottlenecks
Introduction to profiling in python
(some code)

9. Making Scrapers even faster
Using memcache to reduce redundant
scraping
(some code)

10. Thats it !!
(links to the code present in these slides)