Cyborg Method

What is the Cyborg Method for crowdsourced data?

The M Turk Cyborg Method is a strategy designed to improve the quality of data collected on Mechanical Turk (M Turk).

The Cyborg Method proposes that high quality data can be collected from M Turk when using

  • Automated screening tools that identify "bad actors"

  • Human review of responses

Using only one of these methods will increase the quality of your data, but it will miss a subset of "bad actors" and ultimately lower the quality of your data. Using both methods significantly increases the quality of your data and protects your budget. That is, you will pay for responses that are of high quality.

Setting up cyborg method

Automated Evaluation

Integrate an Automated IP Check

We have found IPHub.Info and IPQualityscore to perform well.

These services will determine if a user is suspected of using a VPN/VPS or BOT.

They can be integrated into Qualtrics via their APIs.

In order to use these services, you will need to create an account with them and get your unique API key. At the time of this writing, accounts were free to create and had a limited number of IP checks per day. However, we found their free accounts to be sufficient for a modest research study.

  1. The process described below does the following:

  2. Notifies participant that their IP address will be evaluated

  3. Submits their IP address (and only their IP address) to an external IP address check

  4. Records the results from the external IP address evaluation to your Qualtrics dataset

  5. Make a determination as to whether or not the participant will be allowed to proceed

Step 1

We recommend creating using a text "question" for participants that warns them you are going to evaluate their IP address.

This will allow them to turn off any VPN services they might be using.

Step 2

Next, navigate to survey flow. Find your warning question and after your warning question, add a new element and select web service

Step 3 (iPHub)

In the URL box enter :${loc://IPAddress}

For Method use GET

Add Custom Header to Send to Web Service

Then the header to the web service is called X-Key

Set a Value Now is your API key.

Then, set your Embedded Data of IP_Block = block and IP_Country = countryName

It should look like the following:

Step 3 (IPQualityScore)

For IPQualityscore

In the URL box enter :${loc://IPAddress}

For Method use GET

Then you must set the embedded data, the information you want Qualtrics to pull from IPQualityScore. This service has a number of elements, but we have found that the best indicator was Fraud Score.

It should look like the following:

Step 4

Finally, add branching logic below the IP Check to block participants who fail the check from progressing

  • fraud_score > 85

  • IP_country is not equal to United States

  • IP_block = 1

Participants that meet these criteria should be displayed a message that tells them they are blocked and then have the survey ended.

Human Evaluation

Include a self-report measure that requires a written response. We have found the Life Events Checklist - Part 2 to be an excellent measure. It requires participants to describe a traumatic event that is relevant to them.

Upon completion of the survey, but prior to providing compensation, review written response and determine if it is a valid response.

Unique Key for Compensation

At the conclusion of the survey, generate a unique key that the participant will enter into M Turk. This allows you to know which individuals should receive compensation in M Turk.

We use a combination of four random words generated by random word generator. The service we use is Wordnik. Wordnik has an API that can be integrated into Qualtrics to generate a per-participant-key of several words.

Here is an example of how to integrate this using a webservice in Qualtrics.