Author: Fabrizio Branca Date: 2013-10-23

1. Multithreaded
XML Import
…for Magento
San Francisco Magento Meetup Group - October 23, 2013

2. Fabrizio
Branca
Lead System Developer at

3. E-Commerce:
Magento
CMS:
TYPO3
Global
Enterprise
Projects
Portals:
ZF, FLOW,…
High
Performance
/Scale
Mobile
Searchperience:
SOLR
120 people
in 7 offices
world-wide

4. Aoe_Import
github.com/AOEmedia/Aoe_Import
git clone --recursive …

5. Will Aoe_Import be the fastest
product importer around?
YES, of course!
Well, maybe…
Actually, Aoe_Import is only a
XML Importer “Framework”. It’s
up to you to decide how to handle
the xml snippets…

6. for large XML files
XML! Not CSV.
full flexibility in
processor implementation
Aoe_Import
multi-thread support!
Subscribe your
“Processors” to xpaths
Stream processing
(XMLReader)
“event” driven

7. memory
single product
Problem
Memory limit
time

8. Memory limit
time
memory
memory
Trivial Solution
Memory limit
time

9. Beat the memory Leak
by forking
Waiting for other
thread to terminate
Threading
overhead
Process
import

10. Forking?
In PHP?
$pid = pcntl_fork();
if ($pid) {
// parent process runs what is here
echo "parentn";
} else {
// child process runs what is here
echo "childn";
}

11. Threadi
github.com/AOEmedia/Threadi

12. Clean OOP interface for PHP to
forking and process management
Threadi

13. Batch Processor
Collect a bunch of imports …
…fork…
…and process them in
a child process.

14. No imports are processed in the main thread.
So there’s no memory leak happing here
Main thread
memory
Memory limit
time
Create process
collection
Waiting for other thread to terminate
Threading Process imports in process collection
overhead
Forks
Every fork starts with the low
memory footprint of the main thread
Find the number of imports
that can be processed at a
time without hitting the memory limit

15. Multi-threading? Sure!
Number of
threads
processed
in parallel
Number of items in a batch

16. Problems?
Database Connection
Database
connection
doesn’t like to
be cloned!
Mage::getSingleton('core/resource')
->getConnection('core_write')
->closeConnection();

17. Problems?
Thread Safety

18. Problems?
Thread Safety
--- a/app/code/core/Enterprise/Catalog/Model/Index/Action/Catalog/Category/Product/Refresh.php
+++ b/app/code/core/Enterprise/Catalog/Model/Index/Action/Catalog/Category/Product/Refresh.php
@@ -326,7 +326,7 @@ class Enterprise_Catalog_Model_Index_Action_Catalog_Category_Product_Refresh
->setComment('Catalog Category Product Index Tmp');
$this->_connection->dropTable($this->_getMainTmpTable());
$this->_connection->createTable($table);
$this->_connection->createTemporaryTable($table);
+
}
/**

19. Other Use-Cases?
Scheduler
Queue processing
Indexes
Everything
that’s batchable

20. Thank you!
Any questions?
My blog
http://www.aoemedia.com
http://www.fabrizio-branca.de
@fbrnc
Follow me on twitter!