Module 05: Summaries and Decisions

Slide Accessibility Note

• Images: none; diagrams are HTML text with labels.
• Speaker Notes: These slides follow the Module 05 follow-along script. They name the purpose, vocabulary, challenges, and checklist; the live coding happens in class/week_05_follow_along.R.

Today’s Workflow

A total you cannot check is a guess.

Before We Start

• Open this module’s repo (accepted in Classroom 50 before class).
• Open week_05.Rproj, then class/week_05_follow_along.R.
• Run here::here(): it should print your repo folder.
• Backup: the follow-along is also on this slide site.

Download the Module 05 follow-along script (.R)

Recap: Module 04

Answer in your script first:

1. After read_csv(), what do you run first on real data, and why?
2. A filter dropped 39 of 109 rows. What must you report with the answer?
3. What does 1987 %/% 10 * 10 give?

Learning Goals

By the end of today you can:

• summarize a table, and one row per group, with group_by() + summarize()
• use count() as the shortcut, including two-column counts
• handle NA in summaries: na.rm = TRUE, and report how many were missing
• make categories with if_else() (two) and case_when() (many)
• spot total rows mixed with detail rows, and summarize without double counting
• explain how the rows and the denominator change a percentage

The Data: Great Lakes Commercial Catch

• Great Lakes Fishery Commission commercial fish production
• 65,706 rows x 7 columns, 1867-2015, six lakes
• values = catch in thousands of pounds
• Used exactly as released. Via TidyTuesday (2021-06-08)

Vocabulary

Cycle 1: One Row Per Group

fishing <- read_csv(here("class", "data", "fishing.csv")) |>
  clean_names()

fishing |> count(lake)                        # you know this one
fishing |> summarize(total = sum(values))     # NA. Why?

fishing |>
  group_by(lake) |>
  summarize(
    rows      = n(),
    n_missing = sum(is.na(values)),           # report what you dropped
    total     = sum(values, na.rm = TRUE)
  ) |>
  arrange(desc(total))

Why Is The Sum NA?

• 21,916 of 65,706 values are NA (a third of the file)
• One unknown value makes the total unknown; R refuses to guess
• na.rm = TRUE drops them. That is your decision
• So report n_missing right beside the total, every time

A Number That Should Worry You

fishing |>
  filter(lake == "Michigan", species == "Lake Whitefish", year == 2000) |>
  summarize(total = sum(values, na.rm = TRUE))
#> 14381

Is that the U.S. commercial lake whitefish catch from Lake Michigan in 2000?

Challenge 1: Catch Audit (15 min)

Question: where did 14,381 come from?

• Base: show the rows behind the total (region, values). Compare with the U.S. Total row. How many times too big? Explain with distinct(lake, region)
• Extension: for every lake, all-rows sum vs U.S. total sum. Which lake is furthest off, and why is that ratio not only double counting?

The Same Catch, Counted Three Times

Lake Michigan lake whitefish, 2000 (thousand lb). All rows sum to 14,381.

Part 2 (Mon Oct 12): Where We Are

• Part 1: one row per group, and a total that counted the catch three times
• Today: decide categories, get the right totals, then start the assignment
• New file today: class/week_05_part2_follow_along.R
• Assignment 05 is now due Sun Oct 18

Get Part 2: Your First Pull

Pull = copy new changes from GitHub down to your computer (push goes up). I added one new file to your repo on GitHub. Your Part 1 file is not touched.

Check: week_05_part2_follow_along.R is in class/.

Stuck On The Pull?

Download the Part 2 follow-along script (.R)

Checkpoint: Same Start For Everyone

1. Session > Restart R
2. Open class/week_05_part2_follow_along.R
3. Run Section 0, the checkpoint block
4. nrow(fishing) should print 65706

Cycle 2: Categories You Decide

GOAL: label what kind of row each one is. WHY: summing every row counted the same fish three times.

fishing |>
  mutate(recorded = if_else(is.na(values), "missing", "recorded")) |>
  count(lake, recorded)

fishing <- fishing |>
  mutate(
    region_level = case_when(
      region %in% us_totals     ~ "US total",
      region %in% canada_totals ~ "Canada total",
      region %in% state_totals  ~ "state total",
      .default = "area"
    )
  )

fishing |> count(region_level, region)   # did every region land right?

How case_when() Reads

• Each line: if this ~ then that
• Lines are checked top to bottom; the first TRUE wins
• .default catches every row no line matched (forget it: NA)
• Always count() the result to check it

era = case_when(
  year < 2000 ~ "1950-1999",   # first: now 1867 lands here too
  year < 1900 ~ "1867-1899",   # never reached
  ...
)

Now The Totals Are Right

fishing |>
  filter(region_level == "US total") |>
  group_by(lake) |>
  summarize(n_missing = sum(is.na(values)),
            total = sum(values, na.rm = TRUE),
            .groups = "drop") |>
  arrange(desc(total))

• Michigan first: 3,072,965 thousand lb
• Saint Clair last, but 845 of its 1,171 rows are missing. Would you rank it with confidence?

Challenge 2: Species List (20 min)

Question: how many different fish are really in this file?

• Step 1 (everyone): fix the four misspellings listed in your script with case_when(), keep the rest with .default = species. CHECK: 47 labels
• Step 2 (Step 1 matches): find the other same-fish labels yourself. How many lakes report each species in the 2000s? Are "Drum" and "Freshwater Drum" the same fish?

Break (10 min)

Integrated Challenge (30 min)

Lake whitefish, Lake Michigan, U.S. catch: total catch by decade, with the number of years and missing years beside each total. Which decade peaked?

• Step 1 (everyone): fill in the pipeline in your script. Answer in pounds, not thousands. CHECK: 1990s, 69,967
• Step 2 (Step 1 matches): the 2010s look low. Compute catch per recorded year. Which decade “wins” now, and should you trust it?

Every Fix Raises The Next Question

• By total, the 1990s peak: 69,967 thousand lb (about 70 million lb)
• The 2010s look halved, but the data end in 2015: 6 years, not 10
• Per year, the 2010s dip only a little (4,265 vs 4,575)
• Per year, the 1870s “win”, on one recorded year

Assignment 05: Park Species Decisions

• Data: NPS species lists, 15 most-visited national parks (61k records)
• Decide: which records count as “in the park”; one category for two kinds of “unknown”
• Summarize: percent non-native per park. Watch the denominator
• Investigate: dplyr::across() to total several columns at once
• Due Sun Oct 18, 11:59 PM. Last push before then is graded

How It Is Graded (10 pts)

• Only assignment/ and README.md are checked. class/ never is.
• A red X is feedback, not a grade.

Assignment Workshop (35 min)

• Open assignment/assignment.qmd. Goal today: Tasks 1 and 2
• Task 2, decide which records count, is the same move as region_level today
• Run it chunk by chunk. The full render fails until later tasks have code: expected, not a problem
• Done with 1 and 2? Keep going
• Last 3 minutes: commit and push assignment/

Before You Leave

• Follow-along runs top to bottom after restarting R
• Commit and push class/ (required, not graded; participation). GitHub Desktop: summary, Commit to main, Push origin
• Push shows on GitHub
• Part 2: push your assignment/ work too
• You know where Assignment 05 lives: assignment/assignment.qmd (due Sun Oct 18)