Tools & resources

The Cyborg Method

A strategy to improve the quality of data collected on crowdsourcing platforms like Amazon Mechanical Turk by combining automated screening with human review.

Set it up Read the paper

All responsesFrom MTurk or another crowdsourcing platform
Automated IP checkBlocks VPNs, servers, bots, and out-of-country IPs
Human reviewA reviewer reads a short written answer
Valid sampleOnly these participants are paid

Two layers, because each one misses some

The “cyborg” half

Automated screening

IP-address checks that flag VPNs, VPS servers, bots, and out-of-country responses before anyone reaches your survey.

The human half

Human review

A trained reviewer reads short-answer responses before compensation is approved, catching the bad actors that slip past the IP check.

Using only one of these methods will increase the quality of your data, but it will miss a subset of “bad actors.” Using both significantly increases data quality and protects your budget: you only pay for high-quality responses.

What it saves

We collected two samples the same way. In Sample 1 we recorded the Cyborg Method checks but didn't enforce them, and paid everyone who finished. In Sample 2 we stopped participants the method flagged as invalid, and didn't pay them.

With the Cyborg Method$4.86

Without it$19.53

Budget: $4.80 per valid participant

Valid responses look different

We compared PTSD (PCL-5) and depression (PHQ-8) scores between valid and invalid participants. Valid participants scored significantly lower. They answer honestly, so in a community sample most report few symptoms: a positively skewed distribution. Invalid participants appear to answer at random, which pulls their scores toward the middle of each scale (about 40 on the PCL-5 and 13 on the PHQ-8).

The method is also replicable. Distributions from two samples collected at separate times overlap closely, suggesting the method consistently yields a valid sample.

Density plots of PCL-5 and PHQ-8 total scores for valid and invalid cases in Sample 1 and Sample 2 — Valid Invalid · PCL-5 and PHQ-8 scores for valid and invalid cases in two independent samples.

Setting up the Cyborg Method

Everything runs inside Qualtrics. It takes three parts: an automated IP check at the start, a written question for human review, and a unique key for compensation at the end.

Automated IP evaluation

We have found IPHub.info and IPQualityScore to perform well. Both flag VPNs, VPS servers and bots, and connect to Qualtrics through their APIs. You'll need a free account and an API key; at the time of writing, free accounts allowed enough daily checks for a modest study.

Warn participants
Add a text “question” telling participants their IP address will be evaluated, so they can turn off any VPN.
Add a Web Service element
Open Survey Flow. After your warning question, add a new element and choose Web Service.
Configure the IP check: IPHub

Method GET, with this URL:

http://v2.api.iphub.info/ip/${loc://IPAddress}

Add a custom header named X-Key whose value is your API key. Set embedded data IP_Block = block and IP_Country = countryName.
…or IPQualityScore

Method GET, with this URL (put in your key):

https://ipqualityscore.com/api/json/ip/YOUR_API_KEY_HERE/${loc://IPAddress}

Then set the embedded data you want Qualtrics to pull. We found fraud_score to be the best indicator.
Block failed checks

Below the IP check, add branch logic that ends the survey (with a message explaining they're blocked) when any of these are true:
- fraud_score > 85
- IP_Country is not equal to United States
- IP_Block = 1

Human evaluation

Include a self-report measure that requires a written response. We have found the Life Events Checklist – Part 2 to be excellent: it asks participants to describe a traumatic event relevant to them. After the survey, but before compensation, review each written response and decide whether it is valid.

Unique key for compensation

At the end of the survey, generate a unique key that participants enter into MTurk so you know whom to compensate. We combine four random words from Wordnik, whose API can be called from a Qualtrics web service to create a key for each participant (0.word to 3.word). Add or remove words through embedded data.

Addressing IRB concerns

Describe the components of the Cyborg Method in any protocol reviewed by an Institutional Review Board. An IRB's main question will be whether sending an IP address to a third-party service raises the risk to participants above minimal risk. It does not necessarily do so: an IP address identifies a device, not a person. Identifying someone would take additional information, held by internet service providers, that most research teams cannot access. IP addresses are also publicly visible (more on IP addresses).

IP addresses are, however, considered Protected Health Information (PHI) under HIPAA, so participants must be told that this information will be collected and how it will be used. Dennis, Goodson, and Pearson (2019) provide language you can use:

“Prior research demonstrates how participants recruited through online platforms such as MTurk use virtual private servers to fraudulently circumvent sample screening procedures. We intend to collect and evaluate IP address data for potential fraudulent responses. Collection of this data does not affect the minimum level of risk to participants in the study, as this data does not readily allow individuals to be identified without other identifying information (e.g., user IDs, domain login records) that we will not collect – and will not seek to access.”

Read the paper