www.drupalfacile.org

1. Optimiser les performances
Drupal depuis les tranchées
Saturday, May 26, 12

2. Qui suis-je?
Client Advisor
• Produits : Managed Cloud, Dev Cloud, Drupal Gardens,
Acquia Commons, Dev Desktop...
• Offres : conseil, support et expertise Drupal
• Nos clients : Twitter, Intel, Ebay, Paypal, Al Jazeera, World
Economic Forum, nombreux sites de gouvernements, institutions,
organisations, etc..
• Tutoriels vidéos Drupal 7+ en français
• 500+ visites par jour / 1000+ abonnés
• 600+ abonnés YouTube / 200k+ vues
www.drupalfacile.org
@DrupalFacile
Saturday, May 26, 12

3. Quelques faits
• Un problème de cache est presque toujours la cause d'un lancement raté
• Seul un load test rigoureux vous dira avec exactitude quel traffic vous pouvez absorber
• Il y a virtuellement beaucoup d'argent à perdre pour avoir voulu en économiser un peu
auparavant
• Les (très) grands comptes ne font pas nécessairement mieux que les "petits"
• Mieux vaut une équipe humble qui suit les “bonnes pratiques” Drupal que d’excellents
développeurs PHP qui réinventent la roue
Saturday, May 26, 12

4. Causes principales d’une pointe de traffic ?
• Lancement de site
• Evénement planifié (webinar, sortie d’un nouveau produit, fait divers, soldes,
événement annuel...)
• Buzz innatendu (réseau social, médias, polémique...)
• Attaque massive (DDoS, DoS...)
• ...
Saturday, May 26, 12

5. 0 à 1M+ pages vues en 3 heures
Saturday, May 26, 12

6. Anatomie d'une attaque DDoS*
Traffic illégitime
*Distributed Denial of Service
Saturday, May 26, 12

7. Comment faire face à une
pointe de traffic ?
Saturday, May 26, 12

8. Effectuez un test de montée en charge !
AB
Saturday, May 26, 12

9. Gestion du traffic anonyme et authentifié
nginx (serveur HTTP et reverse proxy)
Varnish (reverse proxy cache)
memcached (mémoire cache distribuée)
Stack LAMP (Linux, Apache, MySQL, PHP)
(+ APC)
Saturday, May 26, 12

10. Une mine d’info : les headers HTTP
Quels outils utiliser ?
Firebug pour Firefox
Webkit Inspector
Expiration du cache : 1h
Ou via cURL...
$ curl -s -D /dev/stderr http://site.com
Varnish HITS
Saturday, May 26, 12

11. Le problème avec Drupal 6...
Solution :
Cookie de session • Supprime le cookie de session de D6
• Gère le cache externe (Varnish)
• Mise en cache des alias d’URL
• Cookie cache bypass (forms, CAPTCHAs...)
• Backport de SimpleTest
• Réplication de base de données
• ...
Varnish MISS
...ou Drupal 7 !
Saturday, May 26, 12

12. Checklist des caches
Cache de page, cache de blocs, aggrégation CSS et JavaScript, cache des modules (Views,
Panels, Date...)
Pour que Pressflow fonctionne avec un reverse proxy cache choisissez le cache externe
Jusqu’à Drupal 7.4, pour faire fonctionner Varnish vous devez ajouter la ligne
suivante dans settings.php : $conf['page_cache_invoke_hooks'] = FALSE;
Monitorez vos HITS Varnish avec Firebug, Webkit Inspector our cURL
Monitorez les get_hits et get_misses memcache avec la commande :
$ watch "(echo stats ; echo quit ) | nc SERVER_ID 11211"
NE PURGEZ PAS LES CACHES (Drupal, Varnish) AUX HEURES DE POINTE !
Saturday, May 26, 12

13. Acquia Insight - Quel score auriez-vous ?
Recommandations diverses
• Performances
• Sécurité
• Bonnes pratiques Drupal
• SEO Grader (partenariat Volacci)
Analyse des données
• Examination de la configuration
• Analyse du code (hacks, updates...)
Saturday, May 26, 12

14. Le tuning de votre serveur est vital !
• Réservez 60% de la RAM pour MySQL
Monitorez la santé de vos serveurs
• Préférez Percona MySQL
$ iostat -mx 1
• Utilisez PHP FastCGI (mod_fcgi) $ dstat -lcm 5
• Limitez le pool Apache à ~60 procs $ mytop -d mysql
$ varnishtop -i TxHeader
• Surveillez les logs (Apache, MySQL...)
La recette du (7500*.6)+(128x20) = 7060
sysadmin ?
(TotalRAM*.AllocMySQL)+(memory_limit*PHP procs) = RAMutil
Saturday, May 26, 12

15. Débugger grâce à la ligne de commande...
Vérification rapide de modules à désactiver en prod :
$ drush pml --type=module --status=enabled | egrep '(backup_migrate|
boost|dblog|devel|diff|masquerade|migrate|performance|statistics|_ui)'
Compter le nombre de pages 404 renvoyées par Apache :
$ grep "" 4[0-9][0-9] " access.log | wc -l
Obtenir un rapport du nombre de réponses HTTP 503 (Page Temporary Unavailable) par URL :
$ awk '{ if ($9 == 503) print $0 }' access.log | awk '{print $7}' |
sort | uniq -c | sort -rn | head -n 20
Compter et classer le nombre de HITS Apache par site :
$ wc -l /var/log/sites/*/logs/SERVEUR/access.log | sort -n
Déterminer l’impact des crawlers sur votre traffic :
$ grep "bot" access.log | awk '{print $14}' | sort | uniq -c | sort -rn
| head -n 20
Saturday, May 26, 12

16. Les bonnes pratiques en place,
qu’en est-il de la base de données ?
Saturday, May 26, 12

17. Ceci est une requête SQL...
SELECT DISTINCT "user" AS content_type, u.name AS content_title, u.uid AS content_author, u.created AS content_date, u.picture AS
content_teaser, u.uid AS content_id, (points.points*100000+101) AS content_score, 0 AS score FROM users u INNER JOIN (select * from
profile_values where (value LIKE '%ICT%') )pv1 ON u.uid = pv1.uid INNER JOIN (select * from profile_values where (value LIKE '%for%') )pv3 ON
u.uid = pv3.uid INNER JOIN (select * from profile_values where (value LIKE '%lifelong%') )pv5 ON u.uid = pv5.uid INNER JOIN (select * from
profile_values where (value LIKE '%learning%') )pv7 ON u.uid = pv7.uid LEFT JOIN userpoints points ON points.uid=u.uid UNION ALL SELECT
"comment" AS content_type, c.subject AS content_title, c.uid AS content_author, c.jdoestamp AS content_date, c.comment AS content_teaser, c.cid
AS content_id, t.voting content_score, 0 AS score FROM comments c left JOIN (SELECT content_id, AVG(value) as voting FROM votingapi_vote
where content_type = 'comment' GROUP BY content_id) as t on c.cid = t.content_id left join term_comment tc on tc.cid = c.cid left join term_data td on
tc.tid = td.tid where ( (td.name like '%ICT%' or c.subject like '%ICT%') AND (td.name like '%for%' or c.subject like '%for%') AND (td.name like
Temps d’exécution moyen ?
'%lifelong%' or c.subject like '%lifelong%') AND (td.name like '%learning%' or c.subject like '%learning%')) GROUP BY c.cid UNION ALL SELECT
n.type AS content_type, n.title AS content_title, n.uid AS content_author, n.changed AS content_date, nr.teaser AS content_teaser, i.sid AS content_id,
AVG(votes.value) AS content_score, 5 * (1.0E-6 * SUM(i.score * t.count)) + 5 * POW(2, (GREATEST(MAX(n.created), MAX(n.changed),
10+ secondes...
MAX(c.last_comment_jdoestamp)) - 1323851948) * 6.43e- + 5 * (2.0 - 2.0 / (1.0 + MAX(c.comment_count) * 0.019230769230769)) + 5 * (2.0 - 2.0 /
(1.0 + MAX(nc.totalcount) * 8.7926001477157E-6)) AS score FROM search_index i INNER JOIN search_total t ON i.word = t.word INNER JOIN node
n ON n.nid = i.sid INNER JOIN search_dataset d ON i.sid = d.sid AND i.type = d.type LEFT JOIN node_comment_statistics c ON c.nid = i.sid LEFT
JOIN node_counter nc ON nc.nid = i.sid INNER JOIN node_revisions nr ON nr.nid=n.nid INNER JOIN votingapi_vote votes ON votes.content_id=n.nid
AND votes.content_type='node' WHERE n.status = 1 AND ( n.type NOT IN
('award','award_category','award_jury_assignment','award_jury_criterion','award_vote','banner_type','book','cases_awards_submission_type','cases_d
ocumentation_type','cases_awards_type','cases_contact_type','cases_related_type','event','event_type','journal_fixed_type','mailout_template','newsle
tter','newsletter_edition','newsletter_section','page','poll','professional_data','resources_type','webform','wiki','workshop_type','cases_reference_type','e
banner','phpcode_type','workshop_highlighted_cases','workshop_participate','workshop_presentation_and_docs','workshop_registration_settings','wor
kshop_speaker') ) AND (i.word LIKE 'ict%' OR i.word LIKE 'for%' OR i.word LIKE 'lifelong%' OR i.word LIKE 'learning%') AND i.type = 'node' AND
(d.data LIKE 'ict%' AND d.data LIKE 'for%' AND d.data LIKE 'lifelong%' AND d.data LIKE 'learning%') GROUP BY i.type, i.sid HAVING COUNT(*) >= 4
ORDER BY content_date DESC LIMIT 0, 10
Saturday, May 26, 12

18. Morceau choisi...
SELECT DISTINCT "user" AS content_type, u.name AS content_title, u.uid AS content_author, u.created AS content_date, u.picture AS
content_teaser, u.uid AS content_id, (points.points*100000+101) AS content_score, 0 AS score FROM users u INNER JOIN (select * from
profile_values where (value LIKE '%ICT%') )pv1 ON u.uid = pv1.uid INNER JOIN (select * from profile_values where (value LIKE '%for%') )pv3 ON
SELECT n.type AS content_type,
u.uid = pv3.uid INNER JOIN (select * from profile_values where (value LIKE '%lifelong%') )pv5 ON u.uid = pv5.uid INNER JOIN (select * from
profile_values where (value AS content_title,
n.title LIKE '%learning%') )pv7 ON u.uid = pv7.uid LEFT JOIN userpoints points ON points.uid=u.uid UNION ALL SELECT
"comment" AS content_type, c.subject AS content_title, c.uid AS content_author, c.jdoestamp AS content_date, c.comment AS content_teaser, c.cid
n.uid AS content_author,
AS content_id, t.voting content_score, 0 AS score FROM comments c left JOIN (SELECT content_id, AVG(value) as voting FROM votingapi_vote
n.changed AS content_date,
where content_type = 'comment' GROUP BY content_id) as t on c.cid = t.content_id left join term_comment tc on tc.cid = c.cid left join term_data td on
tc.tid = td.tid where nr.teaser AS content_teaser, like '%ICT%') AND (td.name like '%for%' or c.subject like '%for%') AND (td.name like
( (td.name like '%ICT%' or c.subject
'%lifelong%' or c.subject like '%lifelong%') AND (td.name like '%learning%' or c.subject like '%learning%')) GROUP BY c.cid UNION ALL SELECT
i.sid ASAS content_title, n.uid AS content_author, n.changed AS content_date, nr.teaser AS content_teaser, i.sid AS content_id,
n.type AS content_type, n.title
content_id,
AVG(votes.value) AS content_score, 5 * AS content_score, * t.count)) + 5 SUM(i.score * t.count)) + 5 *
AVG(votes.value) (1.0E-6 * SUM(i.score 5 * (1.0E-6 * * POW(2, (GREATEST(MAX(n.created), MAX(n.changed),
MAX(c.last_comment_jdoestamp)) - 1323851948) * 6.43e- + 5 * (2.0 - 2.0 / (1.0 + MAX(c.comment_count) * 0.019230769230769)) + 5 * (2.0 - 2.0 /
POW(2, (GREATEST(MAX(n.created),
(1.0 + MAX(nc.totalcount) * 8.7926001477157E-6)) AS score FROM search_index i INNER JOIN search_total t ON i.word = t.word INNER JOIN node
MAX(n.changed),
n ON n.nid = i.sid INNER JOIN search_dataset d ON i.sid = d.sid AND i.type = d.type LEFT JOIN node_comment_statistics c ON c.nid = i.sid LEFT
JOIN node_counter nc ON nc.nid = i.sid INNER JOIN node_revisions nr ON nr.nid=n.nid INNER JOIN votingapi_vote2.0 / (1.0 +
MAX(c.last_comment_jdoestamp)) - 1323851948) * 6.43e- + 5 * (2.0 - votes ON votes.content_id=n.nid
AND v o t e s .MAX(c.comment_count) * 0.019230769230769)) + 5=* (2.0 - 2.0 / D
content_type='node' WHERE n.status 1 A N (1.0 ( + n.type NOT IN
('award','award_category','award_jury_assignment','award_jury_criterion','award_vote','banner_type','book','cases_awards_submission_type','cases_d
MAX(nc.totalcount) * 8.7926001477157E-6)) AS score
ocumentation_type','cases_awards_type','cases_contact_type','cases_related_type','event','event_type','journal_fixed_type','mailout_template','newsle
FROM search_index i;
tter','newsletter_edition','newsletter_section','page','poll','professional_data','resources_type','webform','wiki','workshop_type','cases_reference_type','e
banner','phpcode_type','workshop_highlighted_cases','workshop_participate','workshop_presentation_and_docs','workshop_registration_settings','wor
kshop_speaker') ) AND (i.word LIKE 'ict%' OR i.word LIKE 'for%' OR i.word LIKE 'lifelong%' OR i.word LIKE 'learning%') AND i.type = 'node' AND
(d.data LIKE 'ict%' AND d.data LIKE 'for%' AND d.data LIKE 'lifelong%' AND d.data LIKE 'learning%') GROUP BY i.type, i.sid HAVING COUNT(*) >= 4
ORDER BY content_date DESC LIMIT 0, 10
Saturday, May 26, 12

19. Requêtes MySQL lentes - Rapport New Relic
6+ secondes !
Insertions WATCHDOG
en base de données
Saturday, May 26, 12

20. Temps de chargement d’une page
10 secondes pour charger
la page dont 7 rien que pour
les insertions WATCHDOG
Saturday, May 26, 12

21. Maatkit / Percona Toolkit
1. Statistiques de la requête # Query 1: 0.03 QPS, 0.08x concurrency, ID 0x70761915D5D15769 at byte
1429910
1
# This item is included in the report because it matches --limit.
# Attribute pct total min max avg 95% stddev median
# ========= ====== ======= ======= ======= ======= ======= ======= =======
# Count 24 2548
2. Nombre de lignes examinées # Exec jdoe
# Lock jdoe
# Rows sent
28
14
0
7131s
96ms
7.46k
2s
0
3
11s
4ms
3
3s
37us
3
5s
0
3
958ms
368us
0
2s
0
3
# Rows exam 41 4.83G 1.94M 1.94M 1.94M 1.86M 0.03 1.86M
# Users 1 johndoe
3. Temps d’exécution moyen
# Hosts
# Databases 2
9 ded-662.pr... (565/22%)... 8 more
1 johndoe
# Jdoe range 2012-03-06 06:33:47 to 2012-03-07 06:47:05
# bytes 2 179.16k 72 72 72 72 0 72
# Rows affe 0 0 0 0 0 0 0 0
# Rows read 0 7.46k 3 3 3 3 0 3
4. Requête SQL # 1us
# 10us
3
# Query_jdoe distribution
# 100us
# 1ms
# 10ms
# 100ms
#
4
1s ################################################################
$ mk-query-digest mysqld-slow.log > slow.txt # 10s+ #
# Tables
ou # SHOW TABLE STATUS FROM `johndoe ` LIKE 'url_alias'G
# SHOW CREATE TABLE `johndoe `.`url_alias`G
# EXPLAIN /*!50100 PARTITIONS*/
SELECT SUBSTRING_INDEX(src, '/', 1) AS path FROM url_alias GROUP BY pathG
$ pt-query-digest mysqld-slow.log > slow.txt
Saturday, May 26, 12

22. Checklist de base de données
Cf. checklist des caches - Plus les pages Drupal sont mises en cache, plus la base de
données peut se concentrer sur d’autres tâches...
Utilisez le moteur InnoDB (row-level locking) plutôt que MyISAM (table-level locking)
Suivez les bonnes pratiques MySQL (index, suppression de rand(), SELECT *...)
Utilisez les rapports New Relic pour savoir quelles requêtes SQL doivent être
optimisées ou supprimées
Sauvegardez vos bases de données depuis un hot-spare MySQL
Explorez les alternatives à MySQL : MongoDB, Cassandra...
Saturday, May 26, 12

23. Chaque détail compte.
Évident ? Pas tant que ça...
Saturday, May 26, 12

24. Exemple #1 - Redirections .htaccess
• Un fichier .htaccess ne devrait jamais contenir plus de 100 redirections
• Pensez à Global Redirect, Path Redirect...
• Préférez les redirections visibles de votre fournisseur DNS plutôt que des RewriteRules
RewriteCond %{HTTP_HOST} ^(www.)?site.com$ [NC]
RewriteCond %{REQUEST_URI} ^/quality$ [NC]
RewriteRule ^(.*)$ http://newsite.com/services/max-rehab/
• 150k+ redirections...1mn pour ouvrir le fichier ! quality [L,R=301]
• Taille du fichier ? 90M+ ! RewriteCond %{HTTP_HOST} ^(www.)?site.com$ [NC]
RewriteCond %{REQUEST_URI} ^/satisfaction$ [NC]
RewriteRule ^(.*)$ http://newsite.com/services/addons/
• Le fichier .htaccess n’est pas mis en cache satisfaction [L,R=301]
RewriteCond %{HTTP_HOST} ^(www.)?site.com$ [NC]
• Apache à genoux, CPU et I/O saturés... RewriteCond %{REQUEST_URI} ^/support$ [NC]
RewriteRule ^(.*)$ http://newsite.com/customers/products/
support [L,R=301]
.htaccess
Saturday, May 26, 12

25. Exemple #2 - Utilisation de LOWER()
• Drupal 5, 6 et 7 ont tour à tour utilisé LOWER() car les opérations LIKE de PostgreSQL
sont sensibles à la casse et cela maximisait ainsi la portabilité du code entre SGBD
• Pressflow - qui ne supporte que MySQL - à le premier supprimé LOWER()
• Beaucoup de modules de contrib continuent malheureusement à l’utiliser
/**
* Custom validation for user login form
*
* @ingroup logintoboggan_form
*/
function logintoboggan_user_login_validate($form, &$form_state) {
if (isset($form_state['values']['name']) && $form_state['values']['name']) {
if ($name = db_result(db_query("SELECT name FROM {users} WHERE LOWER(mail) = LOWER('%s')",
$form_state['values']['name']))) {
form_set_value($form['name'], $name, $form_state);
}
}
}
Saturday, May 26, 12

26. Exemple #3 - Dumps MySQL
# cat /mnt/files/backup/DB.sync.sh
#!/usr/bin/env bash
mysqldump -cK --single-transaction PROD675 -u USER -pPASS -h server.domain | mysql STGdb1 -u USER -pPASS;
mysqldump
mysqldump
-cK
-cK
--single-transaction
--single-transaction
PROD740
PROD742
-u
-u
USER
USER
-pPASS
-pPASS
-h
-h Taille de chaque base ?
server.domain
server.domain
|
|
mysql
mysql
STGdb2 -u USER -pPASS;
STGdb3 -u USER -pPASS;
mysqldump -cK --single-transaction PROD693 -u USER -pPASS -h server.domain | mysql STGdb4 -u USER -pPASS;
mysqldump
mysqldump
-cK
-cK
--single-transaction
--single-transaction
PROD681
PROD764
-u
-u
USER
USER
-pPASS
-pPASS
-h
-h
server.domain
server.domain
2GB minimum...
|
|
mysql
mysql
STGdb5 -u USER -pPASS;
STGdb6 -u USER -pPASS;
mysqldump -cK --single-transaction PROD762 -u USER -pPASS -h server.domain | mysql STGdb7 -u USER -pPASS;
21 dumps MySQL
mysqldump
mysqldump
-cK
-cK
--single-transaction
--single-transaction
PROD778
PROD780
-u
-u
USER
USER
-pPASS
-pPASS
-h
-h
server.domain
server.domain
|
|
mysql
mysql
STGdb8 -u USER -pPASS;
STGdb9 -u USER -pPASS;
mysqldump
mysqldump
concurrents
-cK
-cK
--single-transaction
--single-transaction
PROD670
PROD738
-u
-u
USER
USER
-pPASS
-pPASS
-h
-h
server.domain
server.domain
|
|
mysql
mysql
STGdb17 -u USER -pPASS;
STGdb18 -u USER -pPASS;
mysqldump -cK --single-transaction PROD736 -u USER -pPASS -h server.domain | mysql STGdb19 -u USER -pPASS;
mysqldump -cK --single-transaction PROD679 -u USER -pPASS -h server.domain | mysql STGdb20 -u USER -pPASS;
mysqldump -cK --single-transaction PROD744 -u USER -pPASS -h server.domain | mysql STGdb21 -u USER -pPASS;
mysqldump
mysqldump
-cK
-cK
--single-transaction
--single-transaction
PROD746
PROD677
-u
-u
USER
USER
-pPASS
-pPASS
-h
-h
server.domain
server.domain • I/O saturés
|
|
mysql
mysql
STGdb22 -u USER -pPASS;
STGdb23 -u USER -pPASS;
mysqldump
mysqldump
-cK
-cK
--single-transaction
--single-transaction
PROD768
PROD766
-u
-u
USER
USER
-pPASS
-pPASS
-h
-h
server.domain
server.domain • SWAP permanent
|
|
mysql
mysql
STGdb24 -u USER -pPASS;
STGdb25 -u USER -pPASS;
A travers le réseau
mysqldump -cK --single-transaction PROD683 -u USER -pPASS -h server.domain | mysql STGdb26 -u USER -pPASS;
mysqldump
mysqldump
-cK
-cK
--single-transaction
--single-transaction
PROD961
PROD691
-u
-u
USER
USER
-pPASS
-pPASS
-h
-h
server.domain
server.domain
• CPU lockups
|
|
mysql
mysql
STGdb27 -u USER -pPASS;
STGdb28 -u USER -pPASS;
echo 'The sync of your databases from prod to stage is complete at' `date`| mail -s
'Database Sync Complete' -a "From:root@source.com" root@dest.com;
Saturday, May 26, 12

27. Exemple #4 - Organisation de fichiers
• L’arrivé du Cloud Computing change les habitudes de stockage
• NAS, SAN, GlusterFS ? Le stockage réseau est lent (liste des fichiers, téléchargement...)
• Adoptez une stratégie d’organisation de fichiers AAAA-MM-JJ
• Limitez-vous à 1000 fichiers par répertoire
[15:51:50] root@web-1.prod:/mnt/gfs/jdoe/files# ls -l | wc -l
204718
[15:53:12] root@web-1.prod:/mnt/gfs/jdoe/files# ls -lh | grep 'M' | grep 'pdf' | head -n 5
-rw-r--r-- 1 jdoe jdoe 2.1M 2011-09-20 22:33 081010_ns_1.pdf
-rw-r--r-- 1 jdoe jdoe 2.1M 2011-09-20 22:42 081010_ns_1v2.pdf
-rw-r--r-- 1 jdoe jdoe 1.5M 2011-09-20 15:52 090911_g2_digiblazes.pdf
-rwxr-xr-x 1 jdoe jdoe 1.6M 2011-09-15 17:07 101112_g5_1.pdf
-rw-r--r-- 1 jdoe jdoe 2.0M 2011-09-20 20:17 110304_quiz5-6.pdf
SELECT ws.uid, f.* FROM file_managed f
Cocktail INNER JOIN webform_submitted_data wsd ON f.fid = wsd.data
détonnant INNER JOIN webform_submissions ws ON ws.sid = wsd.sid
WHERE f.uri = 'public://files/[FILENAME].pdf'G
Saturday, May 26, 12

28. Merci. Questions ?
Saturday, May 26, 12

29. Aurélien Navarre
@AurelienNavarre
@DrupalFacile
Saturday, May 26, 12