In Part 1, I discussed setting up a Mechanical Turk (MTurk) survey as the first stage of a two-stage process of getting feedback on a proposed UX change to Ingerchat.

At the end of the first stage, I had a list of 10 people in our target segment. To reach them for the second stage, I used two features on Mechanical Turk: messages and bonuses.

The identities of MTurk workers are anonymized, but consistent. It’s possible to message individual workers who have completed your task, and to pay them a bonus. So I created a followup survey on SurveyMonkey, messaged each worker in my segment, and offered a $1 bonus for them to complete the survey.

An aside: social science researchers often use MTurk to do surveys, but use software such as Qualtrics that provides a “reward code” at the end. This integrates more easily into the MTurk system, but Qualtrics was too expensive for my budget.

For each worker, I game them an individualized “access code” (the last few characters of their Amazon worker ID), which they entered at the beginning of the survey. In addition, as a backup, each participant got a separate web link to the survey (Survey Monkey lets you create multiple web links, and track which ones have been used), though that proved to be unnecessary.

The survey itself consisted of a number of images. Each image was a combination of two screenshots side-by-side, with an “A” above the left and a “B” above the right.  The question was “Which screen do you prefer?”, with the choice of “A” or “B”.

In the world of MTurk, $1 is a fairly high price for a survey. Of the 10 invitations I sent out, 7 responded, and quickly.  In the message I sent, I said they would be paid within 12 hours, but once I saw that an individual had filled out a survey, I used the bonus mechanism to pay them their $1 promptly.

The result? Remember that the question was: should Ingerchat use icons instead of text for the drawing panel? My instinct was that icons would be the preferred choice, but in fact 6 out of 7 respondents preferred the text buttons! So the text buttons stayed. It’s true that 7 responses is not a lot relatively speaking, but with that strong a signal, I decided that was good enough.

My conclusion is that MTurk can be a good research tool for doing some simple A/B testing. The advantage is that you get direct responses from real people, quickly, without coding or shipping a new product (this is especially important with the iPhone environment, where shipping new versions is a heavyweight process). And it’s relatively inexpensive.

The disadvantage is that the MTurk system is cumbersome to set up. MTurk out of the box is not designed to take demographics into account. And while there is some support for dual-level surveys, that support assumes the use of expensive third-party survey software. There’s probably a startup opportunity in using MTurk’s API to automate this process for UX researchers.