During the COVID-19 Pandemic in late March, one of our Magento 2 Commerce clients had some massive traffic and sales. In partnership with Creatuity, the development agency for the client, we worked to scale up their site to handle this surge in visitors\u2026<\/p>\n
This blog post will look at the steps we took on the hosting end to meet this demand.<\/p>\n
Identifying Bottlenecks<\/span><\/strong><\/h3>\nVisitors to the client\u2019s website went through the roof from news exposure and marketing. From one night to the next, concentrated visits increased 5 to 10 times, which was an issue.<\/p>\n
The hosting cluster set up for their store was designed to handle large amounts of traffic and orders. Scaling up php nodes would allow for increased traffic. But the surge in traffic all at once when commercials aired nationwide, combined with the media attention, and everyone at home watching TV was quite the challenge. We found that bottlenecks were quickly reached.<\/p>\n
Here is what the New Relic graph looked like when these traffic spikes occurred initially:<\/p>\n
![\"(click](\"https:\/\/www.lexiconn.com\/blog\/wp-content\/uploads\/2020\/05\/mypillow_bad_spikes-300x119.png\")
<\/a><\/p>\n(click image to enlarge)<\/p>\n<\/div>\n
As you can see, performance dropped precipitously. We quickly identified that Redis was the first bottleneck we had to overcome (see the next section for specific fixes we implemented). After Redis, we found that a single MySQL database for everything in Magento was the next hurdle. In between Redis and MySQL were also many other tweaks and improvements that were made.<\/p>\n
Implementing Structural Changes<\/span><\/strong><\/h3>\nRedis:<\/strong><\/span><\/p>\nWe quickly discovered that Redis uses A LOT of data transfer under heavy traffic. In fact, it completely saturated the 1 Gbps private network card (which led to php and website actions backing up quickly). We had to split Redis off to its own server (away from MySQL), and more importantly, upgrade the private network to a 10 Gbps system.<\/p>\n
We did this quickly, and Redis was now bursting in the 1-2.5 Gbps range comfortably at peak traffic. No more slowdowns due to Redis reaching network saturation.<\/p>\n
MySQL:<\/strong><\/span><\/p>\nOnce Redis was running optimally, we found that when traffic spiked all at once, MySQL was having a hard time processing regular Ajax web requests, customer logins, add to cart actions, and checkouts all at the same time. It was quickly determined that splitting the database into three databases (Main, Cart, and Checkout) with Magento Commerce would pay big dividends.<\/p>\n
We worked with Creatuity to migrate the code to Commerce and split out the databases. Once that was done, MySQL could comfortably scale with high levels of traffic and orders without getting bogged down during peak traffic. It also now supports replicated databases for read-only access to further increase throughput.<\/p>\n
Misc. Changes:<\/strong><\/span><\/p>\nIn addition to the big changes with Redis and MySQL, we worked on improving performance in a number of areas:<\/p>\n
\n- Tweaking network parameters (TCP\/IP level) to improve throughput and private network performance (bare metal servers and network is a distinct advantage in this area)<\/li>\n
- Memory adjustments to MySQL, Redis, etc\u2026 to make more efficient use of in-memory utilization<\/li>\n
- PHP-FPM configuration changes to perform faster at scale<\/li>\n
- Load Balancer adjustments to even out performance across the web nodes<\/li>\n
- Better use of the CDN for static assets to offload NFS and load balancer utilization<\/li>\n<\/ul>\n
The End Result:<\/span><\/strong><\/h3>\nAfter all of these changes were made in a short period of time, we were able to easily handle over 12,000 simultaneous visitors<\/strong><\/span> on the site. The infrastructure was able to support a peak order volume of 5 to 6 orders per second<\/strong><\/span>! The New Relic graph now shows a system that is working as it should:<\/p>\n
![\"(click](\"https:\/\/www.lexiconn.com\/blog\/wp-content\/uploads\/2020\/05\/mypillow_bad_spikes.png\")
<\/a><\/p>\n(click image to enlarge)<\/p>\n<\/div>\n
As you can see, performance dropped precipitously. We quickly identified that Redis was the first bottleneck we had to overcome (see the next section for specific fixes we implemented). After Redis, we found that a single MySQL database for everything in Magento was the next hurdle. In between Redis and MySQL were also many other tweaks and improvements that were made.<\/p>\n
Implementing Structural Changes<\/span><\/strong><\/h3>\nRedis:<\/strong><\/span><\/p>\nWe quickly discovered that Redis uses A LOT of data transfer under heavy traffic. In fact, it completely saturated the 1 Gbps private network card (which led to php and website actions backing up quickly). We had to split Redis off to its own server (away from MySQL), and more importantly, upgrade the private network to a 10 Gbps system.<\/p>\n
We did this quickly, and Redis was now bursting in the 1-2.5 Gbps range comfortably at peak traffic. No more slowdowns due to Redis reaching network saturation.<\/p>\n
MySQL:<\/strong><\/span><\/p>\nOnce Redis was running optimally, we found that when traffic spiked all at once, MySQL was having a hard time processing regular Ajax web requests, customer logins, add to cart actions, and checkouts all at the same time. It was quickly determined that splitting the database into three databases (Main, Cart, and Checkout) with Magento Commerce would pay big dividends.<\/p>\n
We worked with Creatuity to migrate the code to Commerce and split out the databases. Once that was done, MySQL could comfortably scale with high levels of traffic and orders without getting bogged down during peak traffic. It also now supports replicated databases for read-only access to further increase throughput.<\/p>\n
Misc. Changes:<\/strong><\/span><\/p>\nIn addition to the big changes with Redis and MySQL, we worked on improving performance in a number of areas:<\/p>\n
\n- Tweaking network parameters (TCP\/IP level) to improve throughput and private network performance (bare metal servers and network is a distinct advantage in this area)<\/li>\n
- Memory adjustments to MySQL, Redis, etc\u2026 to make more efficient use of in-memory utilization<\/li>\n
- PHP-FPM configuration changes to perform faster at scale<\/li>\n
- Load Balancer adjustments to even out performance across the web nodes<\/li>\n
- Better use of the CDN for static assets to offload NFS and load balancer utilization<\/li>\n<\/ul>\n
The End Result:<\/span><\/strong><\/h3>\nAfter all of these changes were made in a short period of time, we were able to easily handle over 12,000 simultaneous visitors<\/strong><\/span> on the site. The infrastructure was able to support a peak order volume of 5 to 6 orders per second<\/strong><\/span>! The New Relic graph now shows a system that is working as it should:<\/p>\n
We quickly discovered that Redis uses A LOT of data transfer under heavy traffic. In fact, it completely saturated the 1 Gbps private network card (which led to php and website actions backing up quickly). We had to split Redis off to its own server (away from MySQL), and more importantly, upgrade the private network to a 10 Gbps system.<\/p>\n
We did this quickly, and Redis was now bursting in the 1-2.5 Gbps range comfortably at peak traffic. No more slowdowns due to Redis reaching network saturation.<\/p>\n
MySQL:<\/strong><\/span><\/p>\n Once Redis was running optimally, we found that when traffic spiked all at once, MySQL was having a hard time processing regular Ajax web requests, customer logins, add to cart actions, and checkouts all at the same time. It was quickly determined that splitting the database into three databases (Main, Cart, and Checkout) with Magento Commerce would pay big dividends.<\/p>\n We worked with Creatuity to migrate the code to Commerce and split out the databases. Once that was done, MySQL could comfortably scale with high levels of traffic and orders without getting bogged down during peak traffic. It also now supports replicated databases for read-only access to further increase throughput.<\/p>\n Misc. Changes:<\/strong><\/span><\/p>\n In addition to the big changes with Redis and MySQL, we worked on improving performance in a number of areas:<\/p>\n After all of these changes were made in a short period of time, we were able to easily handle over 12,000 simultaneous visitors<\/strong><\/span> on the site. The infrastructure was able to support a peak order volume of 5 to 6 orders per second<\/strong><\/span>! The New Relic graph now shows a system that is working as it should:<\/p>\n\n
The End Result:<\/span><\/strong><\/h3>\n