Python Tutorial-Mining imgur images

Curiosity Bits - curiositybits.com
• This tutorial is created for social scientists interested in grabbing data from
the image-hosting site, Imgur (imgur.com).
• Find out more about Python for mining the social web, please visit Curiosity
Bits (curiositybits.com).
• Social-Metrics.org also hosts a series of Python tutorials on aggregating
and analyzingTwitter/Facebook data. More at social-metrics.org
CURIOSITY BITS©
Get Imgur Data through Python

This tutorial shows you how to download images and the images’ meta-
data (i.e., image title, description, source, upload time, etc.).
CURIOSITY BITS©
Imgur.com is a popular site for
sharing selfies and images intended
for persuasion.

CURIOSITY BITS©
Images will be saved in a
designated folder.
The metadata will be saved in a SQLite database

CURIOSITY BITS©
Imgur images are available through keyword search, reddit timelines and public
albums.
Here are the examples:
• http://imgur.com/search?q=hillary images from the search using the keyword “hillary”
• http://imgur.com/r/transtimelines images from a public reddit timeline called
transtimelines
• http://imgur.com/gallery/ROYAZ images from a public album called ROYAZ
Three sources of Imgur images

Final note before we start: for simplicity, I am laying out only the most
essential steps. Previous tutorials provide details about how to set up a
Python programing environment, please visit curiositybits.com and click the
PYTHON tab.
CURIOSITY BITS©

1. Install Anaconda Python (with Spyder and Ipython Notebook)
2. Install SQLite Browser
3. Install four essential Python packages (imgurpython, sqlalchemy, urllib, sqlite3)
4. Register a Imgur client to get client ID and client secret
5. Create a SQLite database for images from keyword search
6. Download images from the keyword search
7. Create a SQLite database for images from a reddit timeline
8. Download images from a reddit timeline
9. Create a SQLite database for images from a public album
10. Download images from a public album
CURIOSITY BITS©
Steps

CURIOSITY BITS©
Install Anaconda Python
• store.continuum.io/cshop/anaconda/
You can run Python codes in Spyder,
which is a component in Anaconda
Python
Or use IPython Notebook, a web-
based interactive Python
environment

CURIOSITY BITS©
This is what Spyder looks like.The left side of
the window displays codes.The one at the
bottom right shows you the results.
IPython Notebook runs your codes on
the web and displays results in your
browser.
Install Anaconda Python

CURIOSITY BITS©
http://sqlitebrowser.org
Install SQLite Browser

CURIOSITY BITS©
• In the Command Prompt, use the command line pip install, followed by the
package name, to install necessary packages.
Required packages
• Imgurpython
• Sqlalchemy
• Urllib
• sqlite3
Install four essential Python packages

CURIOSITY BITS©
type the following command lines in the command prompt:
pip install imgurpython
pip install sqlalchemy
pip install urllib
pip install sqlite3
Install four essential Python packages

Imgur FetcherV1 is a collection of
codes used in this tutorial.You
can download them at
https://github.com/cosmopolitan
van/imgur_curiositybits_v1
On the site, you can also find
three examples of SQLite
database.The database files end
with the extention .sqlite
CURIOSITY BITS©
The Python codes used in the tutorial

CURIOSITY BITS©
Sign in your Imgur account,
go to
https://api.imgur.com/oauth
2/addclient and get your
“security clearance”.
Register a Imgur client

CURIOSITY BITS©
You need Client ID and Client
secret to grab data from
Imgur API
Register a Imgur client

CURIOSITY BITS©
Let’s get images with the keyword “hilliary”
http://imgur.com/search?q=hillary
We will download all available images, but at first, let’s get their metadata. By
metadata, I mean, attributes related to each image. Examples of the metadata
include image title, image description, image upload date, image link (which is
what we will use to download images).
Create a SQLite database for images from keyword search

You can right-click the
link and download the file
with the extension .py.
Open the file in Spyder.
Or, copy the entire block
of codes into Ipython
Notebook.
CURIOSITY BITS©
Use the code named Imgur search v1.py

CURIOSITY BITS©
You don’t have to
change anything in this
block of codes. It is
used for importing
necessary Python
packages.Think of
packages as apps
running on iOS.
Open Imgur search v1.py in Spyder

CURIOSITY BITS©
This is where you enter
the keyword(s) you
want to apply to the
search.You can have
multiple keywords,
wrapped in
parenthesis, and
separated by a comma.

CURIOSITY BITS©
This is where you enter
the client ID and client
secret generated in the
previous step.

CURIOSITY BITS©
No tweaking is needed
for this block. But it
gives you a sense of
what data are to be
collected.

CURIOSITY BITS©
Go to line 136, this is where you specify the name of the SQLite database
to be saved. If no absolute file path is given, the database will be saved in
the same folder with your Python script (Imgur search v1.py).

CURIOSITY BITS©
Imgur images are indexed into multiple pages. Here, 5 means that we are to get five
pages of images from the keyword search.You can put 3 or 1 or 2, just play around to
see what number gives you the most adequate, while at the same time manageable
amount of data.

CURIOSITY BITS©
Now, the tweaking is done! From the
menu, chose Run-Configure and hit RUN

CURIOSITY BITS©
This is the SQLite database that contains all
the metadata.

CURIOSITY BITS©
Notice that the last column is called multiple_images.
Most image links will end with an extension name of a
picture (.jpg, .png, .,bmp, .gif). If that is the case, the
value on the column will be NO.
But, some links contain multiple images, and you will
see “this link contains multiple images…” warning in
the column.

CURIOSITY BITS©
Now, we have a saved SQLite database with image links.The next step is to
use Python to download images from those image links
Use the .py file named Imgur search_downloader v1.py
Download images from the keyword search

CURIOSITY BITS©
Go to line 72, and make sure the filename
there matches the SQLite database we
have just created.

CURIOSITY BITS©
Please create a new folder and
enter the folder path here.
Image filename is comprised of
the image’s unique identifier id
from Imgur API.

CURIOSITY BITS©
Again, this script downloads
images from links with the
extension of .jpg, .png, .gif,
.bmp
If an image link ends with
anything but an image
extension, the script will prompt
a reminder message.

CURIOSITY BITS©
Getting image from a reddit timeline is very similar to getting images from keyword
search.This time, we will try executing the Python code in Ipython Notebook.
Use the script called Imgur reddit timeline v1.py
Create a SQLite database for images from a reddit timeline

CURIOSITY BITS©
Create a new notebook in IPython Notebook and copy and paste the code from Imgur
reddit timeline v1.py
Create a SQLite database for images from a reddit timeline

CURIOSITY BITS©
Enter the name of a reddit timeline.
In the URL term, a reddit timeline displayed
as
http://imgur.com/r/buffalo
Enter your client ID and client secret.

CURIOSITY BITS©
Click Run Cell
The output will be saved in your
IPython Notebooks folder. By
default, it is
…DocumentsIPython Notebooks

CURIOSITY BITS©
Load the code in IPython Notebooks, Like what you do to download images from
keyword search, make sure the file path in the code matches the database you have just
created.

CURIOSITY BITS©
Create a SQLite database for images from a public album
Exactly as what we do in previous steps.There are only a few places in the script that
need tweaking.You need to enter client ID, client secret, the name of the album, the file
path of the database, the number of pages to be grabbed.Then you are all good to go!
To create a SQLite database for images from a public album, use the script named
Imgur album v1.py

CURIOSITY BITS©
Download images from a public album
To download images from a public album, use the script named Imgur
alubm_downloader v1.py

Python Tutorial-Mining imgur images

Recomendados

Recomendados

Más contenido relacionado

La actualidad más candente

La actualidad más candente (10)

Similar a Python Tutorial-Mining imgur images

Similar a Python Tutorial-Mining imgur images (20)

Más de Weiai Wayne Xu

Más de Weiai Wayne Xu (10)

Último

Último (20)

Python Tutorial-Mining imgur images