High R² Does Not Mean Good Prediction: Reading PLSpredict and CVPAT in PLS-SEM
Most PLS-SEM reports stop at R² and structural path p-values, assessing explanation within the collected sample. PLSpredict and CVPAT address a different question: does the model predict data it has not seen, and is its improvement over simple linear regression large enough to be statistically significant?
Start readingPrediction needs observations held out from estimation
- Split observations
Set folds and repetitions
- Fit
Use the training observations
- Evaluate
RMSE, MAE and Q²predict
- Benchmark
Read IA, LM and CVPAT in their roles
Conceptual diagram. Use consistent data splits when comparing prediction benchmarks.
Source: Lê Minh · Conceptual illustration, not software output.1. Why R² Does Not Answer the Prediction Question
A PLS-SEM model with R² = 0.70 for its main dependent variable is often considered good. But R² is calculated on the same data used for estimation, so it indicates fit to observed data rather than performance for a new respondent outside the sample. Shmueli and Koppius (2011) describe the distinction between explanatory modeling (modeling to test and explain theory) and predictive modeling (modeling to predict new data). These objectives are compatible, but high standards for one do not automatically establish high standards for the other.
Jumani (2026) illustrates this distinction. The extended TPB model achieved R² = 0.706 for consumption behavior toward Islamic brands (CBIB), a relatively high value for consumer behavior research. To assess whether it could predict new customers' behavior, however, the author separately ran PLSpredict and CVPAT, the tools discussed here.
Explanation asks whether the model fits the data and its relationships are significant within the existing sample. Prediction asks whether it produces lower error than a benchmark on unseen data. A complete PLS-SEM report should address both using separate tools.
2. PLSpredict: Interpreting Q²predict, RMSE, and MAE
PLSpredict (Shmueli et al., 2019) uses k-fold validation: divide the sample into k parts, 10 by default in SmartPLS, estimate the model on k-1 parts, and predict the remaining part. Repeating this process, 10 times by default, stabilizes results. For each endogenous indicator, PLSpredict provides Q²predict, RMSE (root mean square error), and MAE (mean absolute error), alongside RMSE and MAE for a simple linear model (LM) using the same direct antecedents as a benchmark.
Follow the sequence proposed by Shmueli et al. (2019): first check whether Q²predict is positive. Negative Q²predict indicates predictions worse than the training-sample mean and therefore no predictive value. For positive indicators, compare PLS-SEM and LM RMSE, or MAE when errors are strongly skewed, following Chai and Draxler (2014). If PLS-SEM has lower prediction error than LM for all target indicators, predictive power is high; for a majority but not all, medium; for a minority, low; and for none, predictive power is not established by this rule. This is a descriptive classification of out-of-sample errors, not a conclusion from in-sample R².
| Endogenous variable | Q²predict | RMSE (PLS-SEM) | RMSE (LM) |
|---|---|---|---|
| Islamic Brand Health Benefits (IBH) | 0.56 | ≈ 0.67 (indicator average) | Higher for most indicators |
| Subjective Norms (SN) | 0.52 | ≈ 0.69 | Higher for most indicators |
| Perceived Behavioural Control (PBC) | 0.32 | ≈ 0.83 | Similar; count indicator-level wins over LM before classifying |
| Consumption Behaviour (CBIB) | 0.52 | ≈ 0.70 | Higher for most indicators |
Condensed from Table 12 (PLSpredict) in Jumani (2026). The original compares each indicator's RMSE directly with its corresponding LM RMSE. This construct-level presentation illustrates interpretation and does not replace the indicator-level detail required in a complete report.
3. PLSpredict's Limitation
PLSpredict's limitation lies in its decision rule: count indicators with lower PLS-SEM than LM RMSE and label predictive ability high, medium, or low. This is a heuristic rather than a statistical test. A model beating LM on 6 of 9 indicators by very small margins is labeled medium under this counting rule, although those differences may simply be random noise in a particular sample.
PLSpredict does not establish whether PLS-SEM and LM RMSE differ significantly. Models with identical counts, such as wins on 7/9 indicators, may support very different levels of confidence depending on sample size and error variability. CVPAT addresses this gap.
4. CVPAT: A Formal Statistical Test
CVPAT, the Cross-Validated Predictive Ability Test, was introduced by Liengaard et al. (2021), with extended practical guidance by Sharma et al. (2022). It replaces counting with a formal hypothesis test. CVPAT computes prediction loss for PLS-SEM on each cross-validation fold and compares average loss against indicator average (IA, each target indicator predicted by its own training-sample mean) and linear model (LM, the same regression benchmark used in PLSpredict). It produces an average loss difference and p-value for each comparison.
A negative loss difference indicates lower average PLS-SEM prediction error than the benchmark. A small p-value, commonly below 0.05, means the difference is difficult to explain by chance and is statistically significant. Negative loss differences with p < 0.05 against both IA and LM establish predictive validity statistically rather than through counts alone.
| Comparison against | Average loss difference | p-value | Conclusion |
|---|---|---|---|
| Indicator Average (IA) | -0.40 | < 0.001 | PLS-SEM clearly outperforms the benchmark |
| Linear Model (LM) | -0.02 | < 0.001 | PLS-SEM outperforms the benchmark despite a much smaller absolute difference |
The Overall row in Table 13 (CVPAT) of Jumani (2026). The absolute difference against LM is much smaller than against IA, as expected for a stronger benchmark, but both remain statistically significant at p < 0.001.
After PLSpredict, run CVPAT before concluding predictive ability, especially when high or superior prediction is intended as a statistically supported claim. PLSpredict provides indicator-level descriptive detail; CVPAT provides a tested overall conclusion. They complement each other.
5. Running the Analysis in SmartPLS 4
Step 1: Establish measurement model quality
PLSpredict and CVPAT cannot repair a weak measurement model. Loadings, composite reliability (CR), AVE, and discriminant validity should meet requirements first, because poor measurement affects subsequent predictive error measures.
Step 2: Run PLSpredict
Select Calculate > PLSpredict. Retain the default folds (10) and repetitions (10) unless there is a specific reason to change them. Examine each endogenous variable's Q²predict first, then compare indicator-level RMSE or MAE against LM.
Step 3: Run CVPAT if PLSpredict is encouraging
If Q²predict is positive and PLS-SEM beats LM for most indicators, select Calculate > CVPAT to obtain average loss differences and p-values against IA and LM.
Step 4: Report both fully, without selecting favorable findings
A complete report presents PLSpredict's Q²predict/RMSE/MAE and CVPAT's loss differences/p-values, even if one is less convincing than the other.
Example Results wording
PLSpredict showed Q²predict = [value] for [endogenous variable], with lower PLS-SEM than LM RMSE for [X/Y] indicators, indicating [low/medium/high] predictive ability. CVPAT (Sharma et al., 2022) supported this finding, with an average loss difference of [value] against [IA/LM] (p [=/<] [value]), indicating statistically significant predictive superiority of the extended model.
6. Reporting in a Thesis or Article
Report the measurement model first, followed by the structural model (R², paths, and f²), then prediction. State the PLSpredict settings, folds, and repetitions; provide Q²predict by endogenous variable; and interpret PLSpredict before presenting CVPAT. Avoid replacing both tables with a single claim of good prediction. Reviewers and supervisors often want the actual Q²predict, indicator RMSE, and CVPAT p-values to assess the evidence themselves.
7. Common Mistakes
| Common wording | Why it is wrong | Better wording |
|---|---|---|
| "R² = 0.70, so the model predicts well." | R² measures in-sample explanation rather than out-of-sample prediction. | Present R² as explanatory evidence and Q²predict/CVPAT separately as predictive evidence. |
| "PLS-SEM RMSE is below LM for every indicator, so the model is certainly superior." | PLSpredict does not test the significance of those differences. | Treat PLSpredict as initial evidence and confirm it with a CVPAT p-value before claiming superiority. |
| "Positive Q²predict is sufficient to conclude good prediction." | Positive Q²predict establishes only minimal predictive value, without distinguishing low, medium, or high ability. | Also compare the proportion of indicators where PLS-SEM beats LM on RMSE, following Shmueli et al. (2019). |
- State PLSpredict folds and repetitions rather than merely saying the procedure was run.
- Report Q²predict for all main endogenous variables, beyond the final dependent variable.
- Compare RMSE or MAE against LM by indicator rather than only by construct.
- Run CVPAT and report loss differences and p-values for both IA and LM.
- Avoid terms such as superior or strong predictive ability without accompanying CVPAT p-values.
Conclusion
R² and significant paths indicate fit to collected data, which differs from predicting people outside the sample. PLSpredict gives indicator-level detail through Q²predict, RMSE, and MAE, but its win-loss decision rule remains a heuristic. CVPAT adds a statistical test with a p-value, allowing more rigorous claims about PLS-SEM's predictive superiority over a benchmark.
Jumani (2026) demonstrates their practical complementarity: PLSpredict established positive Q²predict for all main endogenous variables, and CVPAT supported significant differences at p < 0.001 against both benchmarks. A complete PLS-SEM report, especially for journals familiar with predictive assessment, should present both steps rather than stop at R².
References
Primary Sources on PLSpredict and CVPAT
- Shmueli, G., Ray, S., Estrada, J.M.V. and Chatla, S.B. (2019). Predictive model assessment in PLS-SEM: Guidelines for using PLSpredict. European Journal of Marketing, 53(11), 2322-2347. https://doi.org/10.1108/EJM-02-2019-0189
- Liengaard, B.D., Sharma, P.N., Hult, G.T.M., Jensen, M.B., Sarstedt, M., Hair, J.F. and Ringle, C.M. (2021). Prediction: Coveted, yet forsaken? Introducing a cross-validated predictive ability test in partial least squares path modeling. Decision Sciences, 52(2), 362-392. https://doi.org/10.1111/deci.12445
- Sharma, P.N., Liengaard, B.D., Hair, J.F., Sarstedt, M. and Ringle, C.M. (2022). Predictive model assessment and selection in composite-based modeling using PLS-SEM: Extensions and guidelines for using CVPAT. European Journal of Marketing, 57(6), 1662-1677. https://doi.org/10.1108/EJM-08-2020-0636
- Shmueli, G. and Koppius, O.R. (2011). Predictive analytics in information systems research. MIS Quarterly, 35(3), 553-572.
- Chai, T. and Draxler, R.R. (2014). Root mean square error (RMSE) or mean absolute error (MAE)? Arguments against avoiding RMSE in the literature. Geoscientific Model Development, 7(3), 1247-1250. https://doi.org/10.5194/gmd-7-1247-2014
Illustration and Implementation Guidance
- Jumani, Z.A. (2026). Predictive validation of an extended TPB model in Islamic branding: PLSpredict and CVPAT evidence from Muslim and non-Muslim Thai consumers. Journal of Islamic Marketing, ahead-of-print. https://doi.org/10.1108/JIMA-09-2025-0574
- Hair, J.F., Hult, G.T.M., Ringle, C.M., Sarstedt, M. and Danks, N. (2022). Partial least squares structural equation modeling (PLS-SEM) using SmartPLS 4. Sage.
Academic note: Illustrative figures come from Jumani (2026), published in Journal of Islamic Marketing. Some tables condense results to the construct level for illustration. For your own data, report complete indicator-level results as provided in SmartPLS.