Survey data review in the AI era

Criterion applied: Duplicates agree on every selected item. Near duplicates differ on at most one item, among varying profiles; at most 800 rows are compared.

Matching profiles prompt a check of record IDs and data entry. Excess copies differ from all records belonging to duplicate groups.

Assumptions and limitations: A match is a count, not a fraud test. Short questionnaires, few response categories or shared opinions can produce legitimate matches.

• Kuriakose & Robbins (2016): Don't get duped
• Simmons et al. / Pew Research Center (2016): sensitivity of the MPM threshold

Criterion applied: Count profiles with MPM > 85%, both before and after excluding comparison profiles dominated ≥ 70% by one response; at most 600 rows per comparison.

MPM is agreement with the closest profile in the evaluated set. The 85% reference is inherited; the 70% filter is a tool-specific choice.

Assumptions and limitations: Pew shows that one threshold is sensitive to question count, categories, sample size and homogeneity. Exceeding 85% does not establish fabrication or a near-zero chance probability.

• Kuriakose & Robbins (2016): Don't get duped
• Simmons et al. / Pew Research Center (2016): sensitivity of the MPM threshold

Criterion applied: Uniform profile: within-row SD = 0. Low variation: within-row SD ≤ 0.3; only complete valid profiles are used.

Within-row SD describes response variation. The 0.3 cutoff is inherited from the original tool, without calibration for questionnaire length or scale range.

Assumptions and limitations: Do not exclude automatically. Similar wording or consistent opinions can yield low SD; examine reverse-worded items and questionnaire context.

• Curran (2016): Methods for the detection of carelessly invalid responses in survey data
• Meade & Craig (2012): Identifying careless responses in survey data

Criterion applied: Same-level run ≥ max(3, floor(k/2)); endpoint/midpoint use > 70% of items; agreement with a two-level alternating pattern ≥ 90%. k is the selected item count.

IRV is within-profile SD; runs depend on item order. These are tool-specific descriptive rules, not universal cutoffs established by the references.

Assumptions and limitations: Endpoints and midpoint use the configured scale; an even scale has no integer midpoint. Items are not automatically reverse-coded.

• Curran (2016): Methods for the detection of carelessly invalid responses in survey data
• Ward & Meade (2023): Prevention, identification and recommended best practices

Criterion applied: Blank cells are separate from unparseable or out-of-range values. Cell percentages use rows × selected items as the denominator.

Separate missingness from coding errors before analysis. Complete-case counts may differ across checks.

Assumptions and limitations: Missing values do not establish low effort, and complete data are not suspicious by themselves. MCAR/MAR/MNAR mechanisms are not inferred and values are not imputed.

• Ward & Meade (2023): Prevention, identification and recommended best practices

Criterion applied: Inspect n, mean, SD, skewness, excess kurtosis and frequencies. H = −Σp ln(p), maximum ln(L); H > 0.92 ln(L) is counted by a tool-specific rule.

L is the number of scale categories. Descriptive statistics summarize distribution shape and category use for coding review.

Assumptions and limitations: Uniformity, normality and skewness do not establish response origin. The 92% rule is not a validated fabrication-detection cutoff.

• R stats: correlation, variance and covariance
• Shannon (1948): A mathematical theory of communication

Criterion applied: Compute Pearson correlations from valid pairs, then summarize their mean, SD and range within and between declared item groups.

Correlations describe shared variation. Name-based groups organize the checks but do not validate a theoretical structure.

Assumptions and limitations: There is no ‘too uniform means fabricated’ threshold. Ordinality, reverse coding, restricted range and pairwise sample sizes affect r; these summaries do not replace CFA.

• R stats: correlation, variance and covariance

Criterion applied: α = k/(k−1) × [1 − ΣVar(item)/Var(total score)], using complete profiles within each group. No authenticity cutoff is applied to alpha.

Alpha summarizes consistency among items intended to measure one construct; item grouping and coding direction matter.

Assumptions and limitations: High alpha does not prove unidimensionality or validity. Similar group alphas do not establish simulation; pooled multi-construct alpha is descriptive only.

• Cronbach (1951): Coefficient alpha and the internal structure of tests

Criterion applied: PC1 uses each group's correlation matrix; explained variance = λ₁/k. Require at least two varying items and complete n ≥ k + 5.

PC1 describes concentration of variation along one common direction, organized by declared item group.

Assumptions and limitations: No PC1 authenticity cutoff is applied. PCA does not replace EFA/CFA; constants and conceptually unjustified groups limit computation or interpretation.

• R stats: principal components analysis

Criterion applied: D² = (x−μ)′S⁻¹(x−μ); flag D² > χ² quantile (0.999; k), α = 0.001. Require complete n > k + 5, ≤ 60 items and nonsingular S.

This distance accounts for covariance between items. The chi-square cutoff is an approximate multivariate-outlier reference.

Assumptions and limitations: Continuous/normal assumptions, estimation on the same sample and multiple comparisons limit inference. Outliers or their absence do not verify data origin.

• R stats: squared Mahalanobis distance
• Meade & Craig (2012): Identifying careless responses in survey data

Criterion applied: Skewness and kurtosis tests use covariance normalized by n, with skewness correction for n < 20. At a 0.05 reading level, p < 0.05 suggests departure from multivariate normality.

Mardia concerns distribution assumptions and estimator choice. Require n > k + 5, ≤ 60 items; at most 600 complete rows are used.

Assumptions and limitations: Likert items are ordinal. p ≥ 0.05 proves neither normality nor authenticity. An evenly spaced subset is not a probability sample of all respondents.

• Korkmaz, Goksuluk & Zararsiz (2014), The R Journal: MVN

Criterion applied: Correlate each person's profile with the item means of everyone else; require ≥ 3 complete profiles. Negative correlations are counted descriptively.

This leave-one-person-out profile correlation is an implementation of individual/group agreement, not an independent authenticity test.

Assumptions and limitations: It is not an exact reproduction of the original Donlon–Fischer person-fit index. Negative r can reflect genuine opinion differences, reverse coding or nearly equal item means.

• Donlon & Fischer (1968): agreement with group-determined item difficulties

Criterion applied: Select same-sample pairs with r ≥ 0.6 or r ≤ −0.6; require ≥ 3 pairs in each set to describe within-profile agreement.

The |r| ≥ 0.6 selection is a tool-specific rule for finding empirical item relationships to review against the questionnaire.

Assumptions and limitations: Pairs are not established semantic synonyms/antonyms and may overlap. No validated individual-score cutoff is supplied for automatic exclusion.

• Curran (2016): Methods for the detection of carelessly invalid responses in survey data

Criterion applied: Order items by descending mean, omit tied pairs, then count profiles where the later item has a higher response than the preceding adjacent item.

This exploratory Guttman-inspired count reports the actual number of compared adjacent pairs as its denominator.

Assumptions and limitations: It is not a validated polytomous Guttman scale. Interpret only with coherent, aligned items; few reversals do not establish fabrication.

• Curran (2016): Methods for the detection of carelessly invalid responses in survey data

Criterion applied: Require ≥ 3 valid timestamps; sort them to summarize intervals, CV and clusters of ≥ 3 submissions/minute. No automatic decision cutoff is applied.

Use an explicitly named timestamp column or the first column if > 80% valid dates; only valid Date/clear ISO values are accepted.

Assumptions and limitations: Submission time is not completion duration. Batch entry, classrooms, clock resolution or exports can legitimately produce clustered timestamps.

• Ward & Meade (2023): Prevention, identification and recommended best practices

Criterion applied: Search terms such as simulated/synthetic in at most 4,000 cells per selected sheet; report the total hits and up to 25 excerpts.

Sheet, row, column and original excerpts make workbook notes traceable. This is a tool-specific text-search rule.

Assumptions and limitations: Terms can occur in instructions, negations or references. A keyword does not establish the origin of all response records.

• Ward & Meade (2023): Prevention, identification and recommended best practices

Criterion applied: Count non-item columns with one distinct value among ≥ max(10, floor(50% of rows)) nonblank cells; attention/consent/flag-like names are codebook hints only.

A constant can represent a version code, eligibility condition, screening result or system field.

Assumptions and limitations: All-consenting records can be a legitimate inclusion rule. A name alone does not identify an attention-check item.

• Ward & Meade (2023): Prevention, identification and recommended best practices

Criterion applied: Cramér's V = √[χ²/(n·min(r−1,c−1))]. Small-association description: V < 0.1 and p > 0.05; at most 10 candidate columns, ≥ 30 complete pairs.

Candidates have 2–20 levels outside selected items. p is omitted if minimum expected count < 1 or > 20% of cells have expected counts < 5.

Assumptions and limitations: p > 0.05 does not prove independence; a small association does not prove independent generation. Multiple testing is unadjusted and demographic meaning is not inferred.

• R stats: Pearson chi-squared contingency-table test

Criterion applied: Map each input level to its modal output; report ≥ 98% consistency, ≥ 30 complete records and ≥ 2 levels on each side; at most 14 candidates.

A consistent mapping suggests one field may derive from another. The 98% reference is an original-tool heuristic, not an origin test.

Assumptions and limitations: Derived variables, age grouping, relabelling and legitimate recoding can give high consistency. Check the codebook before treating this as a problem.

• Ward & Meade (2023): Prevention, identification and recommended best practices

Criterion applied: For non-items, require ≥ max(50, 50% of rows) numeric values and ≥ 8 levels. Report endpoints with ≥ 5%, ≥ 3×(100/levels), and ≥ 2.5× the adjacent level's share.

These three tool-specific conditions find a jump between an endpoint and its nearest observed level.

Assumptions and limitations: Real measurement bounds, sampling rules, rounding or legitimate clipping can produce pileups. These cutoffs are not a validated falsification test.

• Ward & Meade (2023): Prevention, identification and recommended best practices