On-demand software testing service pays off for three startups

By Colleen Frye, News Writer 16 Jun 2008 | SearchSoftwareQuality.com

Software quality news and advice Digg This!     StumbleUpon     Del.icio.us

Exposing your applications to the globe via the Internet can be downright scary, but exposing your applications to a global community of testers on-demand can be downright effective, according to three companies that recently completed pilots with uTest.

uTest gives us the opportunity to be exposed to a variety of people. It's important to have people around world give feedback--it maximizes the ability to find unique bugs. Sagi Richberg CEO, GroupGain.com

Founded in 2007, Ashland, Mass.-based uTest offers a hosted infrastructure for QA and usability testing with a pay-per-bug business model. uTest provides a global community of testers, dubbed "uTesters," leveraging a social networking-type model. As of May, uTest had signed up more than 7,000 testers, from a range of countries and with a range of experience levels.

The diversity of uTest's community is a key benefit, according to Anson Kuo, vice president of business development for Scottsdale, Ariz.-based PerfSpot.com, an international Web portal for
people sharing their interests and favorite things on the Web.

"Our QA is based in Scottsdale, so we only get a one-dimensional perspective," he said. "uTest has testers all over world. The bugs they found were complications from different platforms, different operating systems, different locations, different firewalls. They provide a more thorough testing perspective."

PerfSpot, which launched near the end of 2006, just hit over 17 million members, Kuo said. The uTest pilot tested the live site, primarily the technical and functional requirements of the registration process.

"They found a few technical bugs, and we were very happy they were able to do that. These were issues that stopped the site from working," he said. "To us, that's golden. They were also able to come up with some functional bugs or issues that didn't stop the site but were big enough of a deal where users who were Internet-savvy would see it and know it's not quite right."

Finding financial transaction bugs
For Sagi Richberg, CEO of GroupGain.com, based in Ashland, Mass., uTest's ability to leverage social networking and provide a global pool of testers offers a diverse perspective.

"It gives us the opportunity to be exposed to a variety of people. It's important to have people around world give feedback -- it maximizes the ability to find unique bugs," he said. "We were impressed with people finding what we didn't think about."

Founded in 2007, GroupGain enables buyers to collaborate to drive down a product's price through economy of scale. For example, a GroupGain feature called BuyerPower lets the buyer form a group around a product, then sellers can bid for the group's business.

uTest's software testing and QA service Through uTest, companies can get on-demand QA and software testing services from a global community of testers. >> Read more about uTest's services

uTest tested the GroupGain platform for usability and webflow, Richberg said. The service tested for "bugs relating to financial transactions, which are most important to us, including credit card transactions, and they were testing a lot of internal things that had to do with intercommunication like chat, email, phone." Richberg said the back-office system that manages sellers, buyers, deal authorization, and the BuyerPower mechanism was also tested.

Previously, GroupGain did QA in-house, "semi-successfully," Richberg said. "We had just one internal guy, so just one way of thinking. I think the diversity [of the uTest community] really did a great job."

In addition to QA, GroupGain had uTest do some usability testing. "We were looking for a diversity of people, from China, etc. We had 20 people give us great remarks worth paying for," Richberg said.

GroupGain hopes to go live by the end of June.

"They found a lot of bugs, and we had to deal with them. Some of them were major. You can't find every variation [on your own.] There is a range of different browsers, different patches to different widgets and DLLs in each operating system. One of the greatest things was finding a major security issue that had to do with SQL injection. That would've been a great security compromise, so by itself that was worth it to us."

A complement to internal QA resources
For Waltham, Mass.-based Second Rotation, uTest helped extend and complement internal resources, according to James McElhiney, chief technology officer. Second Rotation, founded in 2006, buys and sells previously owned consumer electronics and gadgets, and it recycles e-waste if the item has no market value.

The startup is an agile development shop, updating the site every two weeks with new releases, McElhiney explained. Previous to the uTest pilot, Second Rotation had one QA person, then they'd "try to grab anyone from the company to do more testing. That's not really their job and don't know how to be antagonistic, so we didn't get the best results. Everyone is super busy, so they have several things they're working on. It's hard to say 'test this application for couple hours.' People would help, but sometimes we'd release with bugs and find out later through customer support."

McElhiney continued, "That's where the value of uTest is — these people are professional testers, and they know how to be antagonistic. So it really complements the resources we have. And the nice thing about their model is you only have to pay for things they find." Also, he added, the uTest model enables the company to "get tons of people when you need them [to test a release], and when you don't need them you don't have to pay. It's hard to get that infrastructure ourselves."

While all three companies are startups, they said that uTest was something they envisioned that could scale with them.

"I truly see it as a virtual QA team," Richberg said. McElhiney agrees, with this caveat: "We might need more people to manage the [uTest] relationship, but we can be more cost-effective using the uTest model."

All three companies are now also clients. For other organizations considering the uTest Service, Richberg offered this advice: "It's important to define exactly what they want. We specifically said what we wanted them to test. For example, we said we don't want them to tell us about mistakes in spelling."

McElhiney echoed that advice. "Try to make sure you have test plans in place. It is a bit of work to monitor the bugs as they come in. We have an internal bug tracking system, so we have to have a sync-up mechanism. Also, not every bug is an actual bug, so have an upfront list of known defects; otherwise you end up having to pay for those. These might be little things you decided not to fix yet, but make sure the uTest tester doesn't report that."

He added: "It helped us make our release better; it was very cost-effective. It's a relatively low-cost barrier to entry to give it a try. I absolutely think it's worth it."

Tags: Software testing tools and frameworks,  VIEW ALL TAGS

Digg This!     StumbleUpon     Del.icio.us