Puppeteer is more than a node library. It comes with a powerful toolset that enable you to test every feature in every environment of your application.

1. Puppeteer
A WebScraping and UI Testing tool
help
1

2. Who am I?
Miki Lombardi
Full Stack Developer at Plansoft s.r.l
Endless curious, computer science lover, cinema addicted,
sport maniac | Married w Pizza
@thejoin95
2

3. What is it?
“ Puppeteer is a Node library which
provides a high-level API to control
Chrome or Chromium over the
DevTools Protocol. Puppeteer runs
headless by default, but can be
configured to run full (non-headless)
Chrome or Chromium.
What can I do?
Scrape a website in headless mode
Page automation
UI Testing environment
Timeline trace (performance)
CSS & JS Usage stats
Most things that you can do
manually in the browser and more!
3

4. Wow! But.. what it means "headless"?
4

5. Why Puppeteer?
Google vs Other
Powered by Google
Javascript & NPM (easy to setup)
Easy to use/learn APIs
Fast and high reliability
Easy to integrate with Test Framework
Visual Regression Test compatible
Offline Mode
Network request interceptor
Debuggable
Advantages Disadvantages
Just one language
Just one platform (puppeteer-firefox in
beta)
No native video recording in non-
headless mode
5

6. Why Puppeteer?
Google vs Other
Selenium & PhantomJS Cypress & Scrapy
Maintenance issue
Low Learn curve
Setup issue
Disadvantages
Selenium is multi-language
and multi platform
Great user community
Advantages
Single language
Only Testing or scraping purpose
Few system integration
Disadvantages
Cypress is a great test service
Great user community
(scrapy) Python language
Advantages
6

7. Getting started
npm install puppeteer
const puppeteer = require('puppeteer');
(async () => {
const browser = await puppeteer.launch();
const page = await browser.newPage();
await page.goto('https://www.plansoft.it/');
await page.screenshot({path: 'hp.png'});
await browser.close();
})();
1
2
3
4
5
6
7
8
9
10
Take a screenshot of a webpage
Install & Setup Puppeteer
Demo console: https://try-puppeteer.appspot.com/
7

8. Ok, cool. Let's get serious
8

9. Testing & Scraping
const puppeteer = require('puppeteer');
(async () => {
const browser = await puppeteer.launch();
const page = await browser.newPage();
await page.goto('https://www.ebay.it/.....')
const price = await page.$eval('.vi-price', div => div.textContent);
console.log(price);
await browser.close();
})();
/**
* Print out: EUR 79,99
*/
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
We have to test (and maybe trace the price change) the product page of Ebay.it.
How you do it? First of all.. get that price!
9 . 1

10. Testing & Scraping
const puppeteer = require('puppeteer');
(async () => {
const browser = await puppeteer.launch();
const page = await browser.newPage();
await page.goto('https://www.ebay.it/.....')
// Wait until the selector is being visible on the page
await page.waitForSelector('.vi-price');
const price = await page.$eval('.vi-price', div => div.textContent);
console.log(price);
await browser.close();
})();
/**
* If the price will be loaded via AJAX we can have these results:
*
* Without the waitForSelector: Null or Throw an error
*
* With the waitForSelector it print out: EUR 79,99
*/
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
Price loaded over ajax request - Example
9 . 2

11. // Dependencies
// More tests
const browser = await puppeteer.launch();
it('should exists the buy now button', async () => {
const context = await browser.createIncognitoBrowserContext();
const page = await context.newPage();
await page.goto('https://www.ebay.it/.....')
expect(await page.$('#binBtn_btn')).toBeTruthy();
await context.close();
})();
// More tests
await browser.close();
1
2
3
4
5
6
7
8
9
10
11
12
13
14
Testing & Scraping
// Dependencies
// More tests
it('should exists the buy now button', async () => {
const browser = await puppeteer.launch();
const page = await browser.newPage();
await page.goto('https://www.ebay.it/.....')
expect(await page.$('#binBtn_btn')).toBeTruthy();
await browser.close();
})();
// More tests
1
2
3
4
5
6
7
8
9
10
11
12
Test if exist the "Buy Now" button
9 . 3

12. Testing: request & async
it('should pay', async () => {
const context = browser.createIncognitoBrowserContext();
const browser = await context.launch();
const page = await context.newPage();
await page.emulate(puppeteer.devices['iPhone 8']);
await page.goto('https://www.ebay.it/.....')
await page.click('#binBtn_btn');
const response = page.waitForResponse(res => res.url().endsWith('paypal'));
// Wait to catch the response of the payment request
await response;
console.log(response); // {success: true, etc...}
assert(await page.$('.payment-success-popup'));
await context.close();
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
We can test also the payment feature by clicking on the "Buy now" button and wait for the request
response.
If the request fail or the success popup did not show, the test fail.
Note the emulate method, that is emulating an iPhone 8.
The waitFor* method are very powerful and useful for e2e testing:
waitForSelector
waitForRequest
waitForResponse
waitForFunction
waitForNavigation
waitForTarget
waitForXpath
Methods:
note: this is just an example. Ebay works differently
10 . 1

13. Service Worker
it('should cache the logo image', async () => {
const context = browser.createIncognitoBrowserContext();
const browser = await context.launch();
const page = await context.newPage();
await page.goto('https://pptr.dev')
const swTarget = await context.waitForTarget(target => {
return target.type() === 'service_worker';
});
const serviceWorker = await swTarget.worker();
const isCached = await serviceWorker.evaluate(async () => {
return !!(
await caches.match(
'https://user-images.githubusercontent.com/10379601/29446482-04f7036a-841f-11e7-9872-91d1fc2ea683.png'
)
);
});
expect(isCached).toBe(true);
await context.close();
})();
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
Test the service worker with Puppeteer:
10 . 2

14. Testing: geolocation
it('should have the price in pound', async () => {
...
await context.overridePermissions('https://www.ebay.it', [
'geolocation'
]);
const page = await context.newPage();
await page.setGeolocation({
latitude: 51.50,
longitude: 0
});
await page.goto('https://www.ebay.it/.....')
const price = await page.$eval('.vi-price', div => div.textContent);
expect(price).toBe('£79.99');
})();
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
We can set also the geolocation for testing the application in different language and currency:
note: this is just an example. Ebay works differently
11

15. Testing: request interceptor
...
await page.setRequestInterception(true);
page.on('request', request => {
if(request.resourceType() === 'image')
request.respond({body: 'cat.jpg'});
else
request.continue();
});
...
1
2
3
4
5
6
7
8
9
10
11
12
We can intercept the requests and change the content, or abort it, with the following example.
Imagine an ebay of cats:
note: this is just an example. Ebay does not sell cats
12

16. Performance Testing
....
const metrics = await page.metrics();
console.log(metrics);
{
...
Frames: 4,
JSEventListeners: 353,
Nodes: 160,
LayoutCount: 7,
ScriptDuration: 0.20300213,
JSHeapUsedSize: 78309013,
JSHeapTotalSize: 11689984
...
}
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
Using the DevTools backend it possible to retrieve the metrics, the css & js coverage and all the
other useful data from the google developer console
The following example will print out the page metrics:
13 . 1

17. Performance Testing
await page.tracing.start({path: '/metrics/trace.json'});
await page.coverage.startJSCoverage()
await page.coverage.startCSSCoverage()
await page.goto('https://www.ebay.it/...');
// Some actions here
await page.tracing.stop();
await page.coverage.stopJSCoverage()
await page.coverage.stopCSSCoverage()
1
2
3
4
5
6
7
8
9
10
11
The page trace follow the whole navigation of the browser, from the point where the tracing was
started. This is a very useful feature if there is a necessary to record the performance of a single
page or a single feature instead of all.
The following example will record the JS and CSS coverage and the navigation trace:
The trace and the JS & CSS coverages are the same of the Google Developer Console, but
in JSON.
13 . 2

18. Performance Testing
[
{
"url": "some file .css",
"ranges": [
{
"start": 4,
"end": 58
},
{
"start": 102,
"end": 130
}
],
"text": " div {n background: red;n color: white;n }nn .aqua {n color: aqua;n }nn"
}
]
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
Here an example of the CSS coverage:
13 . 3

19. Mouse & Keyboard event
It's possible to emulate the keyPress and the mouseClick events by using the following methods:
await page.click('#element');
await page.type('#element', 'some text');
await page.select('#element', 'value');
await page.keyboard.press('Enter');
await page.keyboard.press('Shift');
1
2
3
4
5
6
7
8
9
14

20. Skyscanner Scraper
....
await this.page.click('#departure-fsc-datepicker-button');
await this.page.waitForSelector('.departure-datepicker'); // is a popup element loaded by an AJAX request
await page.select('select[name="months"]', '2019-10');
let day = '20';
await page.evaluate(day => {
document.querySelectorAll(
'[class*="BpkCalendarDate_bpk-calendar-date"]:not([class*="BpkCalendarDate_bpk-calendar-date--outside"])'
)[(parseInt(day) - 1)]
.click();
);
...
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
I made a simple SkyScanner Scraper that use Puppeteer, so you can see a "real usage".
The code is available at:
Here are some implementation of the feature as show in this presentation:
https://github.com/TheJoin95/skyscanner-flights-scraping
Here is another great project:
Example: &
https://github.com/social-manager-tools
Login Auth
15

21. Questions?
16

22. Thank you
17