"Men's satisfaction is 75%, women's is 80% — women are more satisfied" goes into the report, and a senior reviewer asks: "Is that difference actually significant?" Everyone hits this moment at some point. Reading the numbers in an aggregation table and judging whether the difference is meaningful are two different jobs. The first anyone can do; the second is a separate craft surprisingly few field researchers run cleanly.
This piece walks through why aggregation and significance testing must be treated as separate steps, when to use GT (single-variable) aggregation versus cross-tabulation, the five cross-tab patterns that show up in practice, the chi-square test workflow, why p-values alone aren't enough (and what effect sizes contribute), and the editorial pitfalls we always check for. As the fourth installment of the question-quality series (wording → pilot → cleaning), this article covers the "design → verify → prepare → analyze" arc.
1. Why aggregation and significance testing are separate steps
"Looks like a difference" vs. "is a difference"
Spotting "Men 75% / Women 80%" in a cross-tab and concluding "there's a difference" is premature. With a small sample, that 5-point gap is within sampling noise; with a large one, it's reliably significant. Same numbers, opposite conclusions depending on N.
Agresti (2018) Statistical Methods for the Social Sciences lays this out as a foundation for social-science survey analysis: always check whether the observed difference is within sampling error first. Reading the table without that check is statistically equivalent to declaring a random result.
Splitting the work
| Step | What it does | Output |
|---|---|---|
| Aggregation | Organize the numbers (GT, cross-tab) | Tables, charts |
| Significance testing | Judge whether the difference is random | p-value, effect size |
| Interpretation | Translate statistical results into decisions | Report, recommended actions |
Concluding from aggregation alone is like diagnosing heatstroke without checking a thermometer because "it feels hot today." Make the testing step mandatory.
2. GT vs. cross-tabulation
GT aggregation (single-variable, Grand Total)
The most basic — for each question, how many respondents picked each option.
- Purpose: capture overall trends
- Use when: opening "what's the overall picture?" sections of reports, distribution checks per question
- Limit: doesn't show segment differences
Cross-tabulation
Crosses two questions (or attributes) to show segment-level patterns.
- Purpose: compare across attributes or groups
- Use when: "gender × satisfaction," "age band × purchase intent," etc.
- Limit: max 2 axes (3+ becomes hard to interpret without external tools)
Choosing between them
| Question you're trying to answer | Recommended aggregation |
|---|---|
| "What's the overall result?" | GT |
| "Are there differences across segments?" | Cross-tab |
| "What's the result for this specific subset?" | Filtered GT |
| "Combined effects of multiple attributes?" | Three-way cross-tab or multivariate analysis (external) |
3. Five cross-tabulation patterns to know
Practical cross-tab work breaks into roughly five patterns.
Pattern 1: Demographic comparison
"Gender × satisfaction," "age × purchase intent" — segmenting by demographic attributes. The most common pattern by far.
Pattern 2: Time-series comparison
Comparing the same question across time points (2025 vs. 2026). The bread and butter of tracking studies.
Pattern 3: Group comparison (experiment vs. control)
A/B tests or pre-/post- comparisons looking at "condition × outcome." How marketing impact gets measured.
Pattern 4: Three-way cross-tab
"Gender × age × satisfaction" — three axes. Cells get thin fast; recommended only at N=300+.
Pattern 5: Filtered (conditional) GT
GT after filtering ("only respondents who bought product X," "only users with 6+ months tenure"). Often a cleaner alternative to cross-tabs.
Row % vs. column %
Cross-tabs offer two kinds of percentage views:
- Row % — each row sums to 100% (e.g., distribution of satisfaction within "men")
- Column % — each column sums to 100% (e.g., gender breakdown among "very satisfied")
Pick the one matching your question. The same table can flip your conclusion if you read it the wrong way.
4. The chi-square test workflow
The standard test for "are these segment differences random or significant" in a cross-tab is the chi-square test of independence.
The basics
- Null hypothesis (H0): the two variables are independent (no relationship)
- Alternative hypothesis (H1): the two variables are related (there's a relationship)
- Decision: reject H0 when the p-value is below your pre-set significance level (typically 0.05)
Field workflow
- Build the cross-tab (e.g., gender × satisfaction)
- Run a chi-square test in R / Python / SPSS / Excel
- Check both the p-value and the effect size (Cramér's V)
- Confirm no cells have an expected count under 5
The expected-count constraint
Chi-square assumes each cell's expected count is 5 or more. When too many cells fall below:
- Switch to Fisher's exact test (better for sparse tables)
- Collapse cells (group "20s/30s," "40s/50s," "60+" instead of fine bands)
- Increase sample
Field (2018) Discovering Statistics notes that test reliability degrades meaningfully when more than 20% of cells have expected counts under 5.
5. Significance vs. effect size — why p < 0.05 alone is insufficient
Big N makes tiny differences "significant"
The single biggest pitfall in chi-square. With large samples, even practically meaningless differences show as statistically significant.
Example: at N=10,000, "men 50% / women 51% purchase intent" can come out at p < 0.001. Is the 1-point gap actionable for business decisions? Almost never.
The ASA Statement on p-values
Wasserstein & Lazar (2016) The ASA Statement on p-Values: Context, Process, and Purpose — the American Statistical Association's official position that p-values alone should not drive conclusions. Interpretation requires:
- Effect size
- Confidence intervals
- Substantive significance
These three together, alongside the p-value.
What effect size tells you
A statistical measure of "how big is the difference." Common ones for cross-tabs:
- Cramér's V — overall association strength in a contingency table (0–1; 0.1 weak, 0.3 medium, 0.5 strong)
- Cohen's d — standardized mean difference between two groups (continuous variables; 0.2 small, 0.5 medium, 0.8 large)
- Odds ratio / risk ratio — group-to-group effect in 2×2 tables
Sullivan & Feinn (2012) Using Effect Size — or Why the P Value Is Not Enough recommends always reporting p-value and effect size together in papers and reports.
A practical decision matrix
| p-value | Effect size | Interpretation |
|---|---|---|
| p < 0.05 | Large | Meaningful difference — take action |
| p < 0.05 | Small | Statistically significant but weak in substance — interpret cautiously |
| p ≥ 0.05 | Large | Possibly under-powered — increase N or argue from effect size |
| p ≥ 0.05 | Small | No real difference — report as null |
6. Editorial view — five pitfalls we always watch for
From the literature and field practice, the five things we'd push hard on.
1. Over-reading low-N cells. Once a cross-tab cell drops below n≈30, the percentages bounce around. Before writing "90% of women in their 20s are satisfied," always check the cell's n. At N=10, one respondent moves the % by 10 points — interpretive credibility is essentially zero.
2. The multiple-comparisons trap. "Run a bunch of cross-tabs, only report the significant ones" is structurally p-hacking. 5 random tests will reliably yield 1 with p < 0.05 by chance. Increase the number of comparisons and false positives scale with it. Pre-register the hypotheses you'll test before opening the data.
3. Concluding from p < 0.05 alone. The single most common pitfall in the wild. Always pair the p-value with an effect size. A report that just says "p < 0.05, significant difference" has done half the statistical job. Sullivan & Feinn (2012) is worth circulating to executives so the conversation shifts to "how big is the difference."
4. Confusing correlation with causation. "Service users have higher satisfaction" in a cross-tab does not justify "using the service raises satisfaction." Cross-tabs show correlation, not causation. Causal claims need experimental designs (A/B tests, quasi-experiments).
5. Cherry-picking the cross axis. Which axis you cross by reshapes "what the data shows." Write an analysis plan beforehand and lock the axes. Hunting for "interesting" axes after the fact biases conclusions toward whatever you find narratively convenient.
7. Aggregation operations in the Survey Tool Kicue
Kicue ships the aggregation foundations as standard.
GT and cross-tab
GT aggregation shows single-variable summaries for every question on a single screen, with question-type-aware tables (SA / MA / matrix / scale).
Cross-tabulation generates 2-axis cross-tabs in real time. Row % / column % toggle in one click, so you read the table the right way for your question.
URL parameters as cross axes
URL parameters — referrer, campaign ID, customer ID — are usable as cross axes. "Email vs. SNS satisfaction" type analyses work without extra implementation.
Raw data export for significance testing
Chi-square and effect-size calculations don't run inside Kicue. The standard pattern is to use raw data export (CSV / Excel) to push data into R / Python / SPSS and run chisq.test() and cramersV() there.
Combine with fraud filtering
Toggle "Exclude flagged responses" in the analytics view, with flag management confirming your fraud cases — gives you cleaning → aggregation → testing as a single in-tool flow.
Choosing the right tool — Free plan limits, branching support, AI capabilities, and CSV export vary widely across tools. See our free survey tool comparison to find the right fit for this approach.
Summary
A checklist for aggregation and significance testing:
- Aggregation and testing are separate steps — never conclude from the table alone.
- GT (overall) vs. cross-tab (segments) — match the aggregation to the question.
- Five cross-tab patterns — demographic, time-series, group, three-way, filtered.
- Chi-square for testing differences. Watch the expected-count ≥5 constraint.
- Don't conclude from p alone — always report effect size (Cramér's V, Cohen's d). See ASA Statement (2016).
- Five pitfalls — low-N over-reading, multiple comparisons, p-only reporting, correlation/causation confusion, cherry-picking axes.
- Kicue covers GT and cross-tab natively; significance testing happens in R / Python after export.
Aggregation organizes the numbers; testing asks whether they mean anything. Run both, and only then do survey results become decision material. The four-part question-quality series (wording → pilot → cleaning → aggregation/analysis) closes here.
References (9)
Academic and methodological
- Agresti, A. (2018). Statistical Methods for the Social Sciences (5th ed.). Pearson.
- Field, A. (2018). Discovering Statistics Using IBM SPSS Statistics (5th ed.). SAGE.
- Cohen, J. (1988). Statistical Power Analysis for the Behavioral Sciences (2nd ed.). Routledge.
- Wasserstein, R. L., & Lazar, N. A. (2016). The ASA Statement on p-Values: Context, Process, and Purpose. The American Statistician, 70(2), 129–133.
- Sullivan, G. M., & Feinn, R. (2012). Using Effect Size — or Why the P Value Is Not Enough. Journal of Graduate Medical Education, 4(3), 279–282.
Standards bodies and methodology centers
- AAPOR (American Association for Public Opinion Research): Standard Definitions.
- Pew Research Center: Our Survey Methodology in Detail.
Industry guides (treated as practitioner observations)
Want to take aggregation through to significance testing in one workflow? Try Kicue — a free survey tool. GT and cross-tabulation, URL-parameter segment analysis, and raw data export ship out of the box — Kicue handles the aggregation, R / Python handles the testing.