Load testing your cloud app - Tools comparison

If you’re planning to load test your cloud app, the comparison below might be handy.

The research I have done is on these service providers:

1.          Soasta CloudTest,
2.          Blitz,
3.          BlazeMeter,
4.          LoadStorm,
5.          and MicrosoftCloud-based Load test tool.

Soasta – CloudTest

Pros

•          A complete environment to setup tests
•          Add test users locations seamlessly
•          a good coverage around the globe
•          A comprehensive real-time interface to metrics
•          Drill down analysis on web requests
•          A Lite (Free) version is available with a limit of 100 concurrent users

Cons

•          Requires some setup: you have to spin up a VM to load the test environment

Pricing

“CloudTest is sold based on server hours, including support. With no limits on the number of testers or software access, you select

Plans based on how many test server hours you’ll run. Plans are available for coverage of a few tests each month up to thousands of tests each year” - http://www.soasta.com/wp/wp-content/uploads/2012/10/DS-CloudTest-100812.pdf

Blitz

Pros

•          Capacity planning tools, Optimization, Performance monitoring are out-of-the-box
•          Nothing to install/setup, all cloud-based
•          Simulate up to 5,000 concurrent virtual users
•          It has a chrome/firefox plugin to monitor app performance on develop environment
•          Integrated with Google Analytics
•          Performance Monitoring Alerts  (Email, Twitter, SMS, PagerDuty)

Cons

•          Only 8 test locations around the globe (Australia, Brazil, California Ireland, Japan, Oregon, Singapore, Virginia)
•          Relatively less sophisticated interface (compared to Soasta)

Pricing

•          Works with “Credits”
•          1 credit = 1 minute and 1,000 users - https://www.blitz.io/pricing
•          40 Credits = $40
•          150 Credits = $135
•          300 Credits = $240

BlazeMeter

Pros

•          JMeter compliant
•          Easy to use test creation interface
•          Nice interface for test results
•          You can export test results
•          Compare two test runs
•          Has a server in Australia (yes, we’re load-testing apps in Oz)
•          Test scheduling
•          Integrated with Google Analytics
•          VPN is supported
•          Chrome extension to record JMeter scripts and upload it to BlazeMeter
•          It has a free version that can simulate up to 50 concurrent users
•          A good customer base

Cons

•          Limited Geo coverage (California, Oregon and Virginia USA, Tokyo, Ireland, Brazil, Singapore and Australia)

Pricing

•          Monthly plans – Starting $199 (1,000 users, 20 hours) up to “Call Us”
•          On-Demand (Per hour) – Starting $19 (1,000 users, 1 server) to $299 (40,000 users, 40 servers)
•          http://blazemeter.com/pricing

LoadStorm

Pros

•          Scalable up to 300,000 users
•          A testing data centre in Sydney, Australia
•          Supports automatic crawling (Spider)
•          No scripting language is involved
•          Nothing to install

Cons

•          Currently they only cover Virginia, California, Oregon, Ireland, Sydney, Tokyo, Singapore, Sao Paulo
•          No built-in feature to collect server performance metrics

Pricing

•          Monthly plans – Starting Free (100 users, 25 users per test) up to “Call Us”
•          On-Demand (Per hour) – Starting $39 (1,000 users per test) to “Call” (10,000+ users)
•          http://loadstorm.com/load-testing-cost/

Microsoft Cloud-based Load test tool

Pros

•          MS offers a free version of TFS services (up to 5 developers)
•          Reuse on-premise tests
•          Easy to set-up

Cons

•          Requires Visual Studio™ Ultimate (Pricy)

Pricing

•          It’s still vague - varies according to your MSDN subscription
•          If you don’t have an MSDN subscription, you’ll have to buy VS Ultimate (around AU$19,274) and then subscribe for TFS Services