Counts to Effort: How the Estimate and its Error Are Built

This page explains what estimate_effort() computes from angler counts and why its standard error has the parts it has. It is for readers who know their survey and want to see the arithmetic behind the numbers. For the steps of an analysis, see vignette("tidycreel").

The chain has three steps, and each section below takes one:

one count  ->  one day's effort  ->  stratum total  ->  standard error
              (count x hours)       (N_h / n_h x sum)   (between days, within days)

library(tidycreel)

The sampling unit is the day

A creel season is divided into strata, groups of days expected to be alike, such as weekdays and weekend days. Within a stratum, some days are sampled and the rest are not. Each day belongs to exactly one stratum, and the day is the primary sampling unit (PSU): it is what was drawn at random, so it is what the standard error is computed over.

Write $N_h$ for the number of days in stratum $h$ and $n_h$ for the number sampled. Everything below happens within one stratum. Strata are estimated separately and added.

From a count to a day’s effort

An instantaneous count is the number of anglers fishing at one moment. It is a number of anglers, not an amount of effort. If the count was taken at a random moment in a fishing period of length $T$ hours, the count times $T$ is an unbiased estimate of the angler-hours in that period (Hoenig et al. 1993):

$\hat{e}_i = c_i \times T_i$

tidycreel makes this multiplication when you name the column holding $T$ with period_length_col, and reports the result in angler-hours. Without it, the estimate is the counts expanded but not multiplied, its unit is NA, and estimate_effort() warns that no period length was supplied. tidycreel never assumes a unit it was not given.

From sampled days to the stratum total

The stratum total is the number of days times the mean daily effort on the sampled days:

$\hat{E}_h = N_h \, \bar{e}_h = \frac{N_h}{n_h} \sum_{i=1}^{n_h} \hat{e}_i$

$N_h / n_h$ is the expansion factor: when one day in two was sampled, each sampled day stands for two.

Here is a stratum of 8 weekdays, 4 of them sampled, with one count per day on a 10-hour fishing day:

calendar <- data.frame(
  date     = as.Date("2024-06-03") + 0:7,
  day_type = "weekday"
)
counts <- data.frame(
  date      = as.Date("2024-06-03") + 0:3,
  day_type  = "weekday",
  n_anglers = c(15L, 25L, 35L, 45L),
  hours     = 10
)

design <- creel_design(calendar, date = date, strata = day_type)
design <- add_counts(design, counts, period_length_col = hours)
estimate_effort(design, target = "period_total")
#> 
#> ── Creel Survey Estimates ──────────────────────────────────────────────────────
#> Method: Total
#> Variance: Taylor linearization
#> Confidence level: 95%
#> Effort target: period_total
#> Unit: angler-hours
#> 
#> # A tibble: 1 × 7
#>   estimate    se se_between se_within ci_lower ci_upper     n
#>      <dbl> <dbl>      <dbl>     <dbl>    <dbl>    <dbl> <int>
#> 1     2400  516.       516.         0     757.    4043.     4

By hand: daily effort is 150, 250, 350 and 450 angler-hours, a mean of 300, and $\hat{E} = 8 \times 300 = 2{,}400$ angler-hours.

target = "period_total" asks for the total over all 8 days. The default, target = "sampled_days", is the total over the 4 days counted (1,200 here); vignette("tidycreel") shows the two side by side.

The standard error: variation between days

The estimate is uncertain because a different draw of days would have given a different mean. That uncertainty is estimated from how much the sampled days differ from each other. $s_b^2$ , the sample variance of the daily efforts $\hat{e}_i$ , measures it, and with one count per day the variance of the stratum total is

$\widehat{\text{Var}}_{\text{between}}(\hat{E}_h) = N_h^2 \, \frac{s_b^2}{n_h}$

For the stratum above, $s_b^2 = 16{,}667$ , so

$\widehat{\text{Var}} = 64 \times \frac{16{,}667}{4} = 266{,}667, \qquad \text{SE} = 516.4$

which is the se and se_between printed above.

Why there is no finite-population correction here

Days are drawn without replacement, and survey texts usually multiply this variance by the finite-population correction $1 - n_h/N_h$ : sample half the stratum and half the day-to-day uncertainty is gone; sample all of it and none is left. That reasoning covers only one of the two reasons a day’s estimate is uncertain. The other is when in the day the count fell: a count at another moment would have given a different number, and sampling more days does nothing about that. $s_b^2$ carries both. Multiplying it by $1 - n_h/N_h$ shrinks both, and with every day sampled it would report an SE of 0 although each count still fell at a random moment.

The two-stage variance below puts the within-day part back, but it needs at least two counts on a day to estimate it. So tidycreel applies the correction to a stratum only when every sampled day in it has two or more counts. With one count per day it leaves the correction out, which is conservative: the standard error is somewhat too large rather than too small (Cochran 1977, equation 10.42). Up to tidycreel 8.0.0 the correction was applied in every case, and single-count standard errors were too small (#505).

One count per day: `se_within` is 0, and what that means

With one count per day, se_within is reported as 0. That does not mean the number of anglers was steady through the day: $s_b^2$ contains the within-day variation, but with one count per day it cannot be separated out. The 0 is a placeholder for “not estimated”; #504 proposes reporting NA there instead.

Several counts per day: two-stage variance

When a day has $k$ counts at random times, the day’s effort is estimated from their mean, and the day is now itself a sample: of the moments in the day, $k$ were observed. That makes the design two-stage (days, then moments within days), and the variance has a term for each stage (Cochran 1977, chapter 10):

$\widehat{\text{Var}}(\hat{E}_h) = \underbrace{N_h^2 \left(1 - \frac{n_h}{N_h}\right) \frac{s_b^2}{n_h}}_{\text{between days}} \; + \; \underbrace{N_h \, \frac{s_w^2}{k}}_{\text{within days}}$

$s_b^2$ is the sample variance of the daily mean efforts.
$s_w^2 = \dfrac{\sum_i \sum_j (\hat{e}_{ij} - \bar{e}_i)^2}{n_h (k - 1)}$ is the within-day variance, pooled over the sampled days.

The within-day term is the second-stage term of Cochran’s two-stage variance, $f_1 (1 - f_2) s_2^2 / (nm)$ scaled to a total, with the second-stage sampling fraction $f_2$ taken as zero because a day contains effectively unlimited moments. Pope et al. (Chapter 17 draft) give the same two terms for creel counts. se_between and se_within are the square roots of the two terms, and se the square root of their sum.

What within-day variance measures. It is how much the number of anglers present changes over the day: a morning bite and an empty afternoon give a large $s_w^2$ . It is not observer error; an exact count at a different moment would still differ.

The same stratum, now with a morning and an afternoon count on each sampled day. The daily means are the same as before (150, 250, 350, 450 angler-hours), so the between-day term does not change:

counts_2 <- data.frame(
  date       = rep(as.Date("2024-06-03") + 0:3, each = 2),
  day_type   = "weekday",
  count_time = rep(c("am", "pm"), 4),
  n_anglers  = c(10L, 20L, 20L, 30L, 30L, 40L, 40L, 50L),
  hours      = 10
)

design_2 <- creel_design(calendar, date = date, strata = day_type)
design_2 <- add_counts(design_2, counts_2,
  count_time_col = count_time, period_length_col = hours
)
estimate_effort(design_2, target = "period_total")
#> 
#> ── Creel Survey Estimates ──────────────────────────────────────────────────────
#> Method: Total
#> Variance: Taylor linearization
#> Confidence level: 95%
#> Effort target: period_total
#> Unit: angler-hours
#> 
#> # A tibble: 1 × 7
#>   estimate    se se_between se_within ci_lower ci_upper     n
#>      <dbl> <dbl>      <dbl>     <dbl>    <dbl>    <dbl> <int>
#> 1     2400  392.       365.      141.    1154.    3646.     4

By hand: each day’s two efforts differ by 100 angler-hours, so each day’s sum of squares is $2 \times 50^2 = 5{,}000$ and $s_w^2 = 4 \times 5{,}000 / (4 \times 1) = 5{,}000$ . The within-day variance is $8 \times 5{,}000 / 2 = 20{,}000$ , an SE of 141.4. The total SE is $\sqrt{133{,}333 + 20{,}000} = 391.6$ .

The formula assumes every sampled day in a stratum has the same number of counts. Days with different numbers of counts are open as #405.

More days or more counts per day?

The two terms show where the uncertainty comes from, but not, on their own, what to change. Adding sampled days reduces all of the variance, the day-to-day part and the within-day part; adding counts per day reduces only the within-day part, and is usually cheaper than adding a day. The best split depends on both the variance components and the cost of a day versus the cost of a count; Cochran (1977, section 10.6, equation 10.26) gives the optimum number of counts per day. There is no fixed ratio of se_within to se beyond which one choice wins.

Progressive counts

A progressive count is a different field method. The observer travels the whole section once (a circuit, taking $\tau$ hours) and counts every angler passed. The day’s effort is

$\hat{e}_d = C_d \times T_d$

where $C_d$ is the circuit’s count and $T_d$ the hours the fishery is open that day.

Why $\tau$ does not appear. Follow Hoenig et al. (1993, p. 725). The circuit samples one $\tau$ -hour block of the day, so the count expands over its own duration first: $C_d \times \tau$ . The day is made of $K = T_d / \tau$ such blocks, and the circuit’s block was drawn at random from them, so the day’s effort is

$\hat{e}_d = C_d \times \tau \times \frac{T_d}{\tau} = C_d \times T_d$

$\tau$ enters twice, once expanding the block and once counting the blocks, and cancels. It has already done its work in defining the blocks the circuit was scheduled within, which is why generate_progressive_start() needs it and the final arithmetic does not. add_counts() still requires it: a fishing period shorter than one circuit ( $K < 1$ ) is an error, since no circuit could finish, and $\tau = T_d$ ( $K = 1$ ) is the legitimate all-day circuit of Robson (1961).

Six weekdays, three sampled, a 2-hour circuit on an 8-hour day:

calendar_p <- data.frame(
  date     = as.Date("2024-06-03") + 0:5,
  day_type = "weekday"
)
prog_counts <- data.frame(
  date        = as.Date("2024-06-03") + 0:2,
  day_type    = "weekday",
  n_anglers   = c(15L, 20L, 25L),
  shift_hours = 8
)

design_p <- creel_design(calendar_p, date = date, strata = day_type)
design_p <- add_counts(design_p, prog_counts,
  count_type = "progressive",
  circuit_time = 2,
  period_length_col = shift_hours
)

design_p$counts
#>         date day_type n_anglers
#> 1 2024-06-03  weekday       120
#> 2 2024-06-04  weekday       160
#> 3 2024-06-05  weekday       200
estimate_effort(design_p, target = "period_total")
#> 
#> ── Creel Survey Estimates ──────────────────────────────────────────────────────
#> Method: Total
#> Variance: Taylor linearization
#> Confidence level: 95%
#> Effort target: period_total
#> Unit: angler-hours
#> 
#> # A tibble: 1 × 7
#>   estimate    se se_between se_within ci_lower ci_upper     n
#>      <dbl> <dbl>      <dbl>     <dbl>    <dbl>    <dbl> <int>
#> 1      960  139.       139.         0     364.    1556.     3

add_counts() stores each day’s $C_d \times T_d$ : 120, 160 and 200 angler-hours. The total is $6 \times 160 = 960$ angler-hours. One circuit a day gives one estimate a day, so the variance is the between-day term alone, as for one instantaneous count a day, and for the same reason it carries no finite-population correction.

The estimate is unbiased only when the circuit’s start time, start point and direction were randomised and the observer moved faster than the anglers. None of that can be checked from the counts table; vignette("progressive-count-surveys") gives the field conditions.

Summary

Data	Day’s effort	Variance terms	`se_within`
One instantaneous count per day	$c_i T_i$	between days, no FPC	0: not estimated (#504)
$k$ instantaneous counts per day	mean of $c_{ij} T_i$	between days + within days	estimated
One progressive circuit per day	$C_d T_d$	between days, no FPC	0: not estimated (#504)

In every case the stratum total is $N_h$ times the mean daily effort and strata are added. The between-day term carries the finite-population correction only when every sampled day in the stratum has two or more counts, because only then is the within-day variation it would remove put back.

References

Cochran, W. G. (1977). Sampling Techniques, 3rd edition. Wiley, New York. Chapter 10, subsampling with units of equal size.
Hoenig, J. M., Robson, D. S., Jones, C. M., and Pollock, K. H. (1993). Scheduling counts in the instantaneous and progressive count methods for estimating sportfishing effort. North American Journal of Fisheries Management, 13, 723–736.
Pope, K. L., et al. Creel surveys. Chapter 17 (unpublished draft).
Robson, D. S. (1961). On the statistical theory of a roving creel census of fishermen. Biometrics, 17, 415–437.