General consensus amongst developers playing with current Magento 2 development versions is that is appears to be much slower than Magento 1.x. Unfortunately before the public release of performance toolkit package for both platforms, it was quite hard to quantify performance regression in Magento 2. Performance toolkit was released last year, therefore I did the first benchmark using official tools for my Meet Magento Poland 2014 talk. Performance results were not impressive, but since Magento 2 just started to take shape, that was OK.

Recently I had the privilege of presenting benchmark results for current state of Magento 2 code to significant part of Magento community. In communication with some of the community members, I promised to publish detailed benchmark results, hence this article.

How I tested

Hardware

Hardware used for these benchmarks tests was as follows:

• CPU: 4 x Intel(R) Core(TM) i3 CPU 550 @ 3.20GHz
• Memory: 4 x 2GiB Samsung M378B5673EH1-CH9
• Storage: SSD - Kingston SH103S3240G

Software

Software used for these benchmarks tests was as follows:

• Ubuntu 14.04 x86_64, PHP 5.5 with OpCache, Apache 2.4 MPM prefork, Varnish 3.0.6 configured with .vcl exported from Magento 2 admin.
• Magento 2 beta 3, production mode, compiled with non-singletenant compiler.
• Magento EE 1.14.1.0, production mode.
• Apache jMeter 2.12 r1636949.
• Magento performance toolkit - M2 beta 3 bundled version, Enterprise Edition 1.14 version from official GitHub repository.

Methodology

Besides configuring settings which are appropriate for hardware configuration I had at hand and executing bundled test plan, I used no special methodology for creating these benchmarks. Trial and error showed that setting 10 concurrent users, 10 seconds ramp period and 10 loops to be executed, gave results in reasonable amount of time for all tests on all platforms. If I had more powerful hardware, this is definitely something I'd scale up because results would then be even more accurate, as outlined by the Magento community.

All important parameters of jMeter test plan are as follows (thanks Denys for pointing not all of them were outlined):

ramp_period                         ${__P(ramp_period,10)}
orders                              ${__P(orders,0)}
users                               ${__P(users,10)}
view_product_add_to_cart_percent    ${__P(view_product_add_to_cart_percent,60)}
view_catalog_percent                ${__P(view_catalog_percent,20)}
guest_checkout_percent              ${__P(guest_checkout_percent,10)}
customer_checkout_percent           ${__P(customer_checkout_percent,10)}
loops                               ${__P(loops,10)}

I executed benchmarks at both Magento 2 and Magento 1 in the following configurations:

• #1 All caches disabled - idea was to determine how fat these frameworks are. This isn't a real life test.
• #2 All caches enabled, Magento 2 with Built-In page cache, Magento Enterprise Edition with Full Page Cache - idea was to see how would these frameworks handle load out the box.
• #3 All caches enabled, Magento 2 behind Varnish, Magento Enterprise Edition with Full Page Cache - idea was to see how would these frameworks handle load with preferred page cache solution.

Benchmark results

Every benchmark result provides following information sources:

• Charts @ MageMeter.com displaying aggregated data visually.
• jMeter CSV - jMeter output presenting raw benchmark results for both platforms.

#1 All caches disabled

• Combined charts @ MageMeter.com
• Magento 2 No Cache jMeter CSV
• Magento EE No Cache jMeter CSV

We can observe that average Magento 2 response time for all tests was 6749ms, while the same metric for Magento EE was 2110ms. This makes Magento 2 beta 3 on average 3.2 times slower than Magento 1.14.1.0 Enterprise Edition when running without cache. I must mention feedback about Magento 2 being much more cache dependent, although I'm not sure whether this is by design.

#2 All caches enabled, Magento 2 w/ Built-In page cache, Magento Enterprise Edition w/ Full Page Cache

• Combined charts @ MageMeter.com
• Magento 2 Built-In jMeter CSV
• Magento EE FPC jMeter CSV

We can observe that average Magento 2 response time for all tests was 1232ms, while the same metric for Magento EE was 661ms. This makes Magento 2 beta 3 on average 1.9 times slower than Magento 1.14.1.0 Enterprise Edition when running with default page cache solution.

#3 All caches enabled, Magento 2 w/ Varnish, Magento Enterprise Edition w/ Full Page Cache

• Combined charts @ MageMeter.com
• Magento 2 Varnish jMeter CSV
• Magento EE FPC jMeter CSV

We can observe that average Magento 2 response time for all tests was 1180ms, while the same metric for Magento EE was 661ms. This makes Magento 2 beta 3 on average 1.8 times slower than Magento 1.14.1.0 Enterprise Edition when running with preferred page cache solution. I must point out that routes being served by Varnish are lightning fast due to cached requests not even reaching Magento 2.

Conclusion

Looking at the benchmark results I just outlined, Magento 2 codebase in its current state is no match for Magento Enterprise Edition performance-wise. Regardless to the updated stack and framework features, without providing clear performance improvement with the next generation Magento platform, its position inside current eCommerce ecosystem will be questionable. Luckily, Magento team is aware of the problem, so lets hope they'll succeed in addressing it before the general availability of Magento 2.0.

Stay tuned for more Magento 2 performance benchmarks in the future.