Data Analysis & Validity (amj-data-analysis)

When to trigger

• Data are collected and it is time to estimate and report
• You are unsure whether your estimator matches your design (nested data, latent constructs, panel)
• Reviewers will probe measurement validity, common-method bias, or endogeneity
• Interaction/mediation effects need correct testing and reporting
• A reviewer says "the analysis does not support the inference" or "validity is not established"

Establish measurement before estimation

AMJ reviewers expect the measurement model to be defended first:

• Reliability: Cronbach's alpha and/or composite reliability for each multi-item scale.
• Confirmatory factor analysis (CFA): report fit (e.g., CFI, TLI, RMSEA, SRMR) and show the hypothesized factor structure fits better than plausible alternatives (one-factor, combined-factor models).
• Convergent & discriminant validity: AVE per construct; AVE > inter-construct squared correlations (or HTMT). Report the correlation matrix with reliabilities on the diagonal.
• Aggregation (multilevel): justify with ICC(1), ICC(2), and r_wg(j) before aggregating to a higher level.
• Qualitative analysis: where the design is qualitative, "validity" becomes trustworthiness — present a Gioia-style data structure (first-order codes → second-order themes → aggregate dimensions), an audit trail, and representative quotations so the path from raw data to constructs is traceable.