Flaw Finder Slides FLAW FINDER
Anatomy of a Flawed Study: Scrutinizing Research Methodology
Fact or Fiction?
"The tongue has distinct regions for tasting sweet, sour, salty, and bitter."
You've seen the map. You've heard the 'fact'.
It's a Myth.
Based on a 1901 paper with a mistranslated data interpretation error that lasted for 74 years.
The Anatomy of Failure
Unrepresentative Samples
When the group being studied doesn't actually represent the population it claims to describe (e.g., studying only men for heart disease).
Conflict of Interest
Who paid for the study? If a chocolate company funds a study saying chocolate makes you live longer, be skeptical.
P-Hacking
Manipulating data or running different statistical tests until you find something that looks "significant" by chance.
Small Sample Sizes
Making grand claims based on 5 people. Outliers can easily skew the results in small groups.
Case Study: "Power Posing"
The Original Claim (2010):
Standing in "high-power" poses for 2 minutes increases testosterone, lowers cortisol, and improves performance in job interviews.
The Reality (2015-2017):
Multiple large-scale attempts failed to replicate the results. One of the original authors eventually admitted the effect wasn't real.
What went wrong?
Small sample size (n=42).
Aggressive p-hacking of data.
Publication bias (only publishing the "successes").
The Debater's Shield
In a debate, don't just accept a statistic. Ask how they got it. If the methodology is broken, the evidence is worthless.
SCRUTINIZE
The sample size and population.
QUESTION
The funding and affiliations.
VERIFY
If the results have been replicated.
Case Study Investigator Worksheet Case Study Investigator
Topic Research & Critical Analysis
Investigator
Date
In debate, evidence is only as strong as its foundation. Your task is to act as a forensic researcher. Analyze the following summary of a (fictional) influential study that was recently called into question. Identify the methodological failures and explain why this evidence would be vulnerable in a cross-examination.
Case File #772: The "Sleep-Study Efficiency" Report
Status: Contested
The Claim
"Students who sleep only 4 hours per night show a 15% increase in creative problem-solving skills compared to those who sleep 8 hours."
Funding Source
The study was commissioned by Midnight Fuel Energy Drinks Inc.
Methodology Details
Researchers recruited 12 college-aged volunteers through a flyer in the University Science Building. The participants were monitored for three days. On the final day, the group was given a standardized creativity test. The researchers noted that while physical coordination dropped, "innovative thinking scores" (a metric designed specifically for this study) rose significantly for 10 of the 12 participants. The 2 participants whose scores dropped were excluded from the final data set because they "did not follow the experimental protocol correctly."
1
Identify the "Red Flags"
List at least three specific methodological flaws present in the "Sleep-Study Efficiency" report.
Flaw #1:
Flaw #2:
Flaw #3:
2
The "Why" Behind the Flaw
Select one flaw from above. Explain how that specific flaw likely skewed the results in favor of the researchers' claim.
3
Cross-Examination Strategy
Imagine an opponent uses this study to argue against school start time adjustments. Write one "Challenge Question" that would undermine the credibility of this evidence during a debate.
"
Research Red Flags Guide Research Red Flags
Teacher Reference & Discussion Guide
"Debaters often treat published research as an objective truth. This guide helps you teach students how to peek behind the curtain of 'scientific' authority to find the structural weaknesses that make evidence vulnerable to attack."
THE FUNDING FAIL
Conflict of Interest
What to look for:
The research is funded by an organization that has a direct financial or political stake in a specific outcome.
Debate Attack: "Is it possible that the conclusions of this study were influenced by the fact that the researchers' entire budget was provided by the very industry they were evaluating?"
SAMPLE SMOKE & MIRRORS
Selection Bias
What to look for:
The participants were not chosen randomly. They might be "WEIRD" (Western, Educated, Industrialized, Rich, and Democratic) or have a specific trait that guarantees the result.
Debate Attack: "You're applying a study performed on a group of elite 20-year-old athletes to the general population of elderly citizens. Why should we believe these results are transferable?"
THE CHERRY BASKET
Data Exclusion
What to look for:
The researchers threw out "outliers" or participants who didn't fit the desired curve without a valid, pre-disclosed scientific reason.
Debate Attack: "The study notes that 20% of the participants were removed from the final analysis. If their data had been included, would the 'significant' increase still exist?"
METRIC MANIPULATION
Proxy Variables
What to look for:
Instead of measuring what they claim to measure, they measure a "proxy." E.g., claiming to measure "intelligence" but only measuring how fast someone can solve a specific puzzle.
Debate Attack: "Does this study actually measure 'happiness,' or does it simply measure how many times a person smiled during a five-minute window?"
Discussion Starters for Lesson 1
• "If a study is published in a major journal, does that automatically mean it's flawless? Why or why not?"
• "Why might a researcher be tempted to 'p-hack' their data? Is it always intentional?"
• "In a debate, which is more effective: attacking the claim directly, or attacking the study used to support it?"
Selective Truth Slides SELECTIVE TRUTH
Identifying Cherry-Picking & Contextomy
The Power of the Ellipsis...
"The Senator stated that he would support a tax increase..."
The Original Sentence:
"The Senator stated that he would support a tax increase only if every other alternative was proven to be a total disaster first."
Vocabulary of Deception
1. Contextomy
The act of quoting someone out of context in a way that distorts their intended meaning.
Example: Taking a critic's words "This movie is a spectacular failure" and quoting it as "Spectacular!" on the poster.
2. Cherry-Picking
Supplying only the data or examples that support your argument while ignoring all the data that contradicts it.
Example: A tech company claiming "90% of our customers are happy!" but only sending the survey to 10 hand-picked people.
The Detective's Toolkit
Trace the Source
Where did the quote come from? Find the paragraph before and after.
Check the Trend
Is this data point an outlier, or does it represent the whole set?
Ask "What Else?"
What information is conspicuously missing from their presentation?
GOAL: Reveal the full picture that your opponent is hiding.
Practice: Spot the Spin
Claim:
"Global temperatures dropped in 1998, proving that climate change is a hoax."
The Hidden Context:
1998 was an exceptionally hot El Niño year. While it dropped slightly in 1999, the 10-year trend showed consistent rising. Using 1998 as a starting point is cherry-picking a peak.
Your Mission
Open your "Quote Detective" sheet. You have 3 minutes to find the "missing piece" for each quote provided.
Quote Detective Sheet Quote Detective
Unmasking Contextomy & Cherry-Picking
Agent Name
Below are three "Evidence Snippets" taken from real and fictional sources. Your goal is to identify how they have been manipulated. Read the Claim (the "spin"), compare it to the Original Text , and diagnose the fallacy.
CASE #1
The Movie Critic's "Review"
The Claim (Ad Poster)
"...a performance of staggering intensity that you will never forget!"
The Original Source
"The lead actor delivered a performance of staggering intensity, yet it was so loud and misplaced that it resulted in a headache you will never forget for all the wrong reasons."
Analysis Notes
Diagnosis:
Contextomy
Cherry-Picking
Explain the distortion:
CASE #2
The "Winning" Season
The Claim (Team Memo)
"Our team is unbeatable at home, having won 100% of our Friday night games this season!"
The Full Data Set
Season Record: 3 Wins, 15 Losses.
Analysis Notes
Diagnosis:
Contextomy
Cherry-Picking
Explain the distortion:
The Rebuttal Strategy
Select ONE of the cases above. Write a short rebuttal (2-3 sentences) you would use in a debate to expose this manipulation to the judge.
Interrogator Arsenal Slides THE ARSENAL
Formulating Cross-Examination Questions
Why Ask Questions?
1. Clarification
Ensuring you (and the judge) understand their evidence exactly. (Low threat)
2. Exposure
Revealing weaknesses, biases, or logical leaps in their research. (High threat)
"The best cross-ex question is one you already know the answer to."
— Trial Lawyer Wisdom
The Question Categories
Credibility
Attacks the source of the information.
Example:
"Is it true your primary source for these statistics is a blog post written by a high school student?"
Relevance
Attacks how the evidence applies to the topic.
Example:
"This study was conducted in 1985; why should we believe it reflects today's technology?"
Methodology
Attacks how the data was actually collected.
Example:
"How many participants were actually in this study? Wasn't it fewer than twenty?"
Leading vs. Open Questions
Don't use Open:
"What do you think about the sample size of this study?"
Allows the opponent to explain away the flaw.
Use Leading:
"Isn't it true that only 12 people participated in this study?"
Forces a 'Yes' or 'No' and highlights the flaw directly.
PRO TIP
In Cross-Ex, you are not a student asking for help. You are a detective presenting a case through the opponent's mouth.
Analyzing the Masters
[Insert Courtroom Drama Clip: e.g., A Few Good Men "You can't handle the truth!"]
Watch for:
How the lawyer builds a "trap" with small, undeniable facts.
Listen for:
The shift from clarification to confrontation.
Question Forge Worksheet Question Forge
Strategic Questioning Workshop
Debater
The Situation
Your opponent has just finished their constructive speech. They cited a study that claims:The Problem: The "study" was actually a survey of 20 students conducted by the Vending Association of America.
TASK 1
Weapon Selection
Identify three distinct "flaws" in the opponent's evidence and categorize them.
Flaw #1 (Methodology)
Flaw #2 (Credibility)
Flaw #3 (Sample Size)
TASK 2
The Leading Question
Rewrite the following "weak" open questions into "strong" leading questions that force a specific admission.
Weak/Open:
"What do you think about the small amount of students they surveyed?"
Strong/Leading:
Weak/Open:
"Don't you think the funding source is a bit biased?"
Strong/Leading:
TASK 3
Setting the Trap
Write a 3-question sequence that builds to a final admission about the evidence's invalidity.
Goal: Your opponent must concede that the evidence is biased or unrepresentative.
Cross-Ex Stems Reference Cross-Ex Stems
A Teacher’s Guide to Question Structures
Use these question "stems" to model effective cross-examination for students. These stems are designed to force admissions rather than invite explanations.
Methodology Attacks
Focus: Sample Size
"Isn’t it true that this study only looked at [number] participants?"
Focus: Measurement
"To be clear, this study didn't actually measure [claim], it only measured [proxy]?"
Credibility Attacks
Logical Gaps
Focus: Causation
"The study shows these two things happened at the same time, but it doesn't prove one caused the other, correct?"
Focus: Generalization
"Is there any reason to believe that a study on [Group A] would apply equally to [Group B]?"
Strategic Pro-Tips
The "One Fact" Rule: Each question should contain exactly one piece of information.Don't Argue: If they give a long answer, simply say: "Thank you. Next question."The "Yes/No" Wall: Structure questions so that a 'Yes' or 'No' is the only logical response.
"In cross-examination, you don’t ask to learn; you ask to teach the judge through the opponent’s own admissions."
Hidden Link Slides THE HIDDEN LINK
Correlation vs. Causation in Evidence
Warning: Spurious Statistics
"When Ice Cream sales go up, Shark Attacks also go up."
Conclusion?
Eating ice cream makes sharks want to eat you. We must ban Ben & Jerry's to save our surfers!
The Third Variable
The "Link" is actually Summer. People eat more ice cream AND swim more when it's hot.
The Logical Trap
Correlation
Two things happen at the same time or follow the same trend.
Example: 100% of people who drink water eventually die.
Causation
One thing specifically causes the other to happen.
Example: Dropping a glass on concrete causes it to break.
Debate Spotlight
Opponent's Argument:
"Statistics show that cities with more police officers have higher crime rates. Therefore, increasing police presence actually increases crime!"
The Rebuttal Attack:
"The opponent is confusing correlation with causation. Cities with high crime rates hire more police in response to the crime. The crime caused the hiring, not the other way around."
The Hidden Link Test
1
Is there a Third Variable (the 'Summer' effect) at play?
2
Is the Direction of the link wrong? (Did B cause A?)
3
Is it just Random Coincidence (a Spurious Correlation)?
Spurious Links Activity Spurious Links
Data Interpretation Challenge
Analyst
"Just because two lines on a graph look the same doesn't mean they are related. Your goal is to identify the 'Hidden Variable' or logical error in each of the following claims."
Claim #1: The Library Link
"Studies show that households with more books have children who score higher on standardized tests. To fix our education system, the government should simply give every family 500 books."
Identify the Hidden Variable (The "Third" Factor):
Explain why the "Fix" might fail:
Claim #2: The Vitamin Trap
"Data indicates that people who take a daily multivitamin have lower rates of heart disease. Clearly, multivitamins are a medical breakthrough for cardiac health."
Is there a Correlation or Causation?
Pure Correlation
Likely Causation
What is the "Self-Selection Bias" here? (Who chooses to take vitamins?)
Claim #3: The Social Media Surge
"Since the rise of TikTok in 2018, the number of coffee shops in America has also increased by 20%. TikTok is driving our obsession with caffeine!"
Diagnosis: (Coincidence vs. Connection)
Write a cross-ex question to expose the absurdity of this link:
Ultimate Challenge
Select one of the claims above. Write a formal rebuttal statement (2-3 sentences) as if you were speaking to a debate judge. Use the phrase "confusing correlation with causation."
Evidence Showdown Slides SHOWDOWN
Live Evidence Cross-Examination
The Protocol
1
The Presentation (60s)
The "Witness" presents one piece of evidence from their card. They must sound confident!
2
The Cross-Ex (90s)
The "Interrogator" asks leading questions to find flaws in methodology, credibility, or logic.
3
The Verdict (30s)
The Jury (class) votes: Was the evidence "Sustained" or "Shattered"?
CRITICAL FOCUS
This is NOT about who is a better speaker. It is about who found the actual flaw in the research.
What does victory look like?
Precision
You didn't just say the study was "bad"; you identified exactly why (e.g., selection bias).
Control
You used leading questions to prevent the opponent from rambling or making excuses.
Admission
You forced the opponent to admit a weakness they were trying to hide.
"Questions are the picks and axes of the debater. Swing them with purpose."
Defense Strategies
For the "Witness":
Be brief. Don't volunteer extra information they can use against you.
Stay calm. If they find a flaw, defend the *logic* of the study, not the math.
If you don't know the answer, say: "I don't have that specific data point in front of me."
READY?
Grab your Battle Sheets. The showdown begins in 2 minutes.
Cross-Ex Battle Sheet Battle Sheet
Cross-Examination Simulation
Interrogator
Witness
Phase 1: Reconnaissance
Planning Time
The Evidence Under Review:
(Write the claim and key methodology points of your assigned evidence card here...)
Suspected Flaw #1:
Suspected Flaw #2:
Phase 2: The Interrogation
Execution
Draft your sequence of 3-4 leading questions designed to expose the flaws identified above.
Q1
Q2
Q3
Q4
Phase 3: Debrief
Self-Assessment
What admission did you force?
What would you change next time? Cross-Ex Evaluation Rubric Evaluation Rubric
Evidence Showdown Assessment
Criteria Mastery (4) Developing (2-3) Needs Focus (1) Flaw Identification Identifies a precise methodological flaw (e.g., selection bias, p-hacking) with 100% accuracy. Identifies a general flaw but may struggle to name the specific research failure. Struggles to identify any valid weakness in the research presented. Question Strategy Uses exclusively leading questions that force "Yes/No" answers and build a logical "trap." Uses some leading questions, but allows the opponent too much room for explanation. Relies on open-ended questions ("How do you feel about...?") that lose control of the session. Real-Time Analysis Actively listens and adjusts questions based on the opponent's responses to secure an admission. Sticks strictly to a script and misses opportunities to catch the opponent in a lie or error. Does not react to what the opponent says; lacks situational awareness. Impact Successfully forces an admission of the evidence's weakness or reveals a massive bias. Weakens the opponent's case slightly but doesn't "shatter" the evidence foundation. The evidence remains standing and largely unchallenged by the end of the session.
Instructor Feedback:
Total Score / Performance Tier:
PENDING
Proctoring Tip: During the session, watch for students who "badger" the witness. In debate, the judge values precision over volume. A quiet, clinical destruction of evidence is always more effective than a loud, emotional argument.
