Quantitative Research Designs and Methods

Forty thousand dollars, two questions

Willow Bay needs evidence about:

1. how often above-no-wake passages occur; and
2. how users explain compliance, access, and fairness.

Which question can numeric data answer well—and what could those numbers not tell us?

What you should be able to do

• Match a quantitative design to the inference it supports.
• Distinguish target population, sampling frame, sample, and respondent.
• Diagnose coverage, sampling, nonresponse, and measurement error.
• Bound conclusions from observation, questionnaires, and secondary data.

Quantitative describes with numbers

Quantitative research collects numeric data: counts, measurements, categories coded as numbers, and scales.

Its strength is structured comparison and, with a suitable design, inference beyond observed cases.

Numbers do not make a claim objective or causal by themselves.

Variable type constrains analysis

• Dichotomous: two categories—wake observed / not observed.
• Categorical: unordered or ordered categories—vessel type; support level.
• Continuous: measured along a numeric scale—minutes in the bay.

Do not treat labels as though distances between them are known.

Reliability and validity ask different questions

Reliability: Would the measurement or procedure produce consistent results?

Validity: Does the design and measurement accurately support the intended interpretation?

Two observers can agree perfectly on a flawed definition: reliable, not valid.

A target separates consistency from accuracy

Three quantitative design families

Experiments create a counterfactual

Random assignment makes intervention and control groups comparable on average.

If outcomes then differ, the intervention is a stronger explanation.

Many management questions cannot ethically or practically randomize people, places, or policies.

Quasi-experiments use imperfect comparisons

Before–after: Compare the same site before and after an action.

At Willow Bay, compare above-wake passages before and after a sign pilot. Season, weather, or other changes could explain a shift.

Treated–untreated: Compare a site receiving an action with a similar site that does not. Sites may differ before the action begins.

Description is not a failed experiment

Descriptive designs answer what, how much, where, when, and among whom.

They can guide decisions without identifying a cause.

Willow Bay passage observation can estimate sampled patterns; it cannot prove that wakes caused the decline in nests.

Why do a descriptive study?

• Establish a baseline or estimate how often and how much an issue occurs.
• Show where, when, and among whom a pattern appears.
• Monitor change when an experiment is not feasible or ethical.
• Identify questions for a later explanatory study.

At Willow Bay, a descriptive study can locate passage patterns and guide monitoring—even when it cannot explain why the pattern occurs.

Population, frame, sample, respondent

Target population → everyone the claim is about

Sampling frame → operational list used to reach that population

Sample → units selected from the frame

Respondents → selected people who actually provide data

These sets rarely match perfectly.

Four steps from population to respondents

Probability sampling supports population inference

Every unit has a known, non-zero chance of selection.

Random or systematic selection can reduce selection bias and quantify sampling uncertainty—if the frame and implementation are sound.

Probability selection does not guarantee response.

A census does not cure a bad frame

Contacting every registered boat owner is a census of that list.

It is not a census of Willow Bay users, because the list includes nonusers and omits many actual users.

Define the claim from the population—not from the convenience of the list.

Three common quantitative methods

• Questionnaires: structured self-reports.
• Behavioral observation: systematic records of actions or traces.
• Secondary data: records collected previously for another purpose.

The method should follow the question. Self-report and observed behavior are different evidence.

Observation needs a protocol

Time sampling records behavior during selected intervals; event sampling records a specified event and its context.

Willow Bay uses 36 randomly selected two-hour periods, stratified across days and times. The claim concerns those sampled temporal conditions—not every moment.

Secondary data inherit their history

Existing complaints and counters can be efficient and reveal patterns.

But categories, missing cases, collection purposes, quality, and timing may not fit the new question. Differences among records can reflect methods rather than real differences.

Survey research is more than a questionnaire

A survey integrates the research question, target population, frame, contact and response modes, questionnaire, sample, administration, and analysis.

A pre-test checks the instrument; a pilot test trials the larger process.

Four cornerstones organize survey error

1. Coverage: Does the frame include the target population?
2. Sampling: What uncertainty comes from observing a subset?
3. Nonresponse: Do respondents differ from nonrespondents?
4. Measurement: Do answers reflect the intended variable?

Coverage comes before sample size

Undercoverage omits eligible units; overcoverage includes ineligible ones.

A huge sample from registered owners cannot represent unlisted paddlers or visitors. Increasing the number selected does not add missing kinds of people.

Sampling error is expected

A random sample will differ from the population by chance.

Larger representative samples usually improve precision, expressed through a margin of sampling error and confidence level.

Those quantities do not measure coverage, nonresponse, or wording bias.

Nonresponse is about difference, not count

Unit nonresponse: a selected person provides no survey.

Item nonresponse: a respondent skips a question.

Error arises when nonrespondents differ systematically on what matters. A high number of responses does not make that threat disappear.

Measurement error can be designed in

“Do you support the reasonable quiet-water rule needed to save birds?”

This item signals the desired answer and embeds untested claims. It does not cleanly measure support.

Use neutral wording, one idea at a time, suitable response options, and a realistic recall period.

Diagnose the error source

Match each problem:

1. Paddlers absent from the contact list
2. Selected nonusers ignore the invitation
3. “Reasonable rule needed to save birds”
4. Different results from two random samples

Coverage, nonresponse, measurement, or sampling?

Check the diagnosis

1. Missing paddlers → coverage
2. Selected people do not answer and differ → nonresponse
3. Leading wording → measurement
4. Chance sample differences → sampling

Name the mechanism; “bias” alone is not a diagnosis.

Say what the observations can support

Ask: Of the passages observers actually recorded, how many met the study’s above-wake rule?

X = observed passages classified above-wake. Y = all observed passages.

Report: “During the sampled periods, X of Y observed passages met the above-wake rule.” This describes those periods; it does not explain the bird decline or measure public support.

Before Wednesday

Read Chapter 15 with one question in mind:

If observation shows when noncompliance occurs, how could we learn what those actions mean to different users—and make that interpretation trustworthy?