AUDIT MATERIALS
Replication archive for the introductory evidence and Appendix A.3 of
"Beyond Isolated Headlines: A Framework for Coupling Scorekeeping and Explanatory
Modeling in Public Political Polls."
Revised 2026-09-12: the FiveThirtyEight files were consolidated with 2025 Internet Archive
captures and the sampled audit was re-run on the consolidated archive.

CONTENTS
  README.txt                     this file
  archived-approval-files/       the four archived poll files counted in the paper, the raw
                                 inputs to the consolidated favorability file, the consolidation
                                 script, and a README with provenance, capture URLs, and checksums
  record-count.py                recomputes the 22,085 record and 177 organization counts from the four files
  sponsor-count.py               recomputes the sponsor-attribution counts from the four files and crosswalks
  sponsor-crosswalk.csv          321 sponsor entities, each coded news organization yes/no
                                 (12 added 2026-09-12; 1 retained but no longer observed)
  fielding-org-crosswalk.csv     202 fielding organizations, each coded news-branded yes/no (9 added 2026-09-12)
  draw-sample.py                 draws the 400-record sample from the four files and writes an uncoded
                                 workbook (link-audit-400-draw.xlsx); it will not write to or replace
                                 the coded link-audit-400-v2.xlsx
  link-audit-400-v2.xlsx         the 400 sampled records, coded; plus the 51- and 177-organization sheets
                                 and the Second_coder_50 sheet
  nyt_link_audit.py              assessability pass over the tracker snapshot; writes the coding workbook
  nyt_prescreen.py               archives each URL, extracts text, and triages documents for human coding
  add-prescreen-columns.py       fills needs_visual and chars in the tracker workbook
  nyt_prescreen.csv              prescreening output: per-URL extraction and keyword counts
  tracker-audit-1104.xlsx        all 1,104 tracker records and 786 URLs, coded (unchanged from July 2026)
  Flynn.pdf                      archived copy of the one release coded as disclosing a model
  superseded-2026-07/            the July 2026 sampled audit (link-audit-400.xlsx) and a note on it

WHAT THE AUDITS REPORT
  Sampled link audit   400 records drawn from the 22,085-record consolidated archive; 304
                       assessable; none disclosed an explanatory model of the featured score.
                       Not assessable 96: 94 URLs did not resolve to the release, 2 records
                       carried no URL. By source file: historical approval 85 of 171, current
                       approval 5 of 89, favorability 6 of 122, tracker 0 of 18. The 400 came
                       from 51 organizations producing 20,627 of the 22,085 records (93%).
  Tracker audit        All 1,104 records of the frozen New York Times tracker; 1,083
                       assessable; one release, linked from 6 records, disclosed a model.
  Neither audit estimates organization-level prevalence, and the tracker records are a
  subset of the combined files, so the two audits are not independent.

SPONSOR ATTRIBUTION
The sponsor field names the entity to which a poll is publicly attributed as sponsor. The
archives do not record the underlying arrangement, which may involve payment, an exchange
in kind, or questions carried without charge in return for exposure. No distinction among
those arrangements is drawn, and none is recoverable from the files. A sponsor cell can name
several entities separated by a pipe; splitting on that delimiter yields 320 entities in the
consolidated archive. The news-organization code in each crosswalk is a human coding
decision, not a derived quantity. Academic institutions are not coded as news organizations;
where a university appears inside a joint fielding entity, that entity is news-branded on the
strength of its news partner. Of the twelve entities added on 2026-09-12, National Post
and The National are coded as news organizations; the other ten are not.

ORDER OF OPERATIONS
  1.  record-count.py      record and organization counts (four files in archived-approval-files/)
  1b. sponsor-count.py     sponsor-attribution counts (needs both crosswalks)
  2.  draw-sample.py       the 400-record sample, written to an uncoded link-audit-400-draw.xlsx;
                           compare its record_identifier column with link-audit-400-v2.xlsx
  3.  human coding         the sampled audit was coded by hand in a browser
  4.  nyt_link_audit.py    tracker: probe URLs, write the coding workbook
  5.  nyt_prescreen.py     tracker: archive documents, extract text, triage for coding
  6.  human coding         the explanatory-model verdict is never automated

HOW TO RUN
Run from this folder (audit-materials/) with Python 3 and pandas; draw-sample.py also
needs numpy and openpyxl. Expected values from the paper are noted beside each step.

  Step 1: record and organization counts (expected: 22,085 records; 177 organizations)

    python record-count.py \
      --ft-approval-historical archived-approval-files/president_approval_polls_historical.csv \
      --ft-approval-current archived-approval-files/president_approval_polls_current_capture-2025-03-06.csv \
      --ft-favorability archived-approval-files/favorability_polls_consolidated.csv \
      --nyt-approval archived-approval-files/nyt-approval-tracker-frozen-2026-07-11.csv

  Step 1b: sponsor attribution (expected: 10,591 records naming a sponsor; 320 sponsor
  entities; 8,075 + 438 = 8,513 records involving a news organization, 39%)

    python sponsor-count.py \
      --ft-approval-historical archived-approval-files/president_approval_polls_historical.csv \
      --ft-approval-current archived-approval-files/president_approval_polls_current_capture-2025-03-06.csv \
      --ft-favorability archived-approval-files/favorability_polls_consolidated.csv \
      --nyt-approval archived-approval-files/nyt-approval-tracker-frozen-2026-07-11.csv \
      --crosswalk sponsor-crosswalk.csv \
      --fielder-crosswalk fielding-org-crosswalk.csv

  Step 2: the 400-record sample (expected: 51 organizations producing 20,627 records)

    python draw-sample.py

  draw-sample.py reads the four files itself and writes link-audit-400-draw.xlsx. Its
  record_identifier column in the Sample_400 sheet matches link-audit-400-v2.xlsx row for row.

SAMPLED AUDIT, SEPTEMBER 2026
The four files were concatenated in the order documented in archived-approval-files/README.txt
and the sample drawn with RandomState(1963).choice(22085, 400, replace=False), preserving the
returned order as draw_order. The same method had been applied in July 2026 to the archive as
first assembled (19,725 records); 93 of the 400 September records carried a verdict from that
coding (2 identical records; 91 sharing a URL with a record coded then, in the sampled audit or
the tracker audit) and are marked as such in coding_notes. The remaining 307 were coded on
2026-09-12. One carried-over verdict, covering 5 records citing a Reuters/Ipsos polling page,
proved to be a July coding error (the same URL was coded Not available for 15 other July
records); it was caught at second coding and corrected. See superseded-2026-07/README.txt.

SCREENING IS A TRIAGE, NOT A CODE
The prescreening vocabularies route tracker documents to human coding; they do not decide the
explanatory-model question. Model terms and multiword weighting terms are matched as
literal substrings; the short weighting acronym ACS is matched only as a whole word to
avoid accidental matches inside unrelated longer words. The weighting-context rule can
only suppress a hit from the clean-hit count, never create one. In the tracker audit that rule
changed the classification of 7 of 726 screened documents. All seven remain weighting-
context documents under the corrected ACS rule because the model vocabulary appears
alongside other explicit weighting language (for example, sampling-frame, matched-case,
and propensity-score descriptions). A random 50 of the 708 screen-negative documents
were coded by hand as a validation check; none disclosed a model.

PROVENANCE
The archive_file values in tracker-audit-1104.xlsx are sha1(url).hexdigest()[:16] plus the
content-type extension, as computed by nyt_prescreen.py. That function reproduces the stem
for all 726 archived URLs, confirming that the deposited script produced the deposited
archive.

EXTRACTION COVERAGE
All 208 archived PDFs returned extractable text (min 722, median 33,042, max 223,797
characters); none was flagged for visual review. Two archived HTML documents returned zero
extractable text and were screened as containing no model vocabulary; they are cited by two
of the 1,104 records. The script's empty-extraction flag applies only to PDFs, so both were
then opened in a browser (2026-09-20) and coded individually; neither disclosed a model. Their
coding_route in tracker-audit-1104.xlsx is "zero-text HTML (2)", which brings the URLs coded
individually to 130, covering 188 records, and leaves 656 screen-negative documents unread.

SECOND CODING
A second coder independently applied the coding rule to a reproducible random subsample of
each audit, blind to the primary verdicts.
  Sampled audit   50 records drawn with RandomState(1963).choice(309, 50, replace=False)
                  over the 309 records marked assessable at the primary coding, in draw
                  order. Agreement on all 47 that the second coder could retrieve; 0
                  disagreements; 3 records citing one Reuters/Ipsos URL could not be
                  retrieved and are recorded as not assessable at second coding. That
                  failure exposed the carried-over error described above; after correction
                  304 records are assessable.
  Tracker audit   77 URLs drawn from the 768 assessable with
                  RandomState(1963).choice(768, 77, replace=False) in sheet order, plus the
                  single URL coded as disclosure, which fell outside that draw: 78 in total.
                  Agreement on all 77 that remained retrievable; 0 disagreements; 1 lapsed
                  link. The disclosure URL was independently returned as partial, matching
                  the primary coding of a disclosed model with a partially identified
                  specification. The second coder recorded a single verdict per URL:
                  "no model", or "partial" for a disclosed model whose specification is
                  only partly identified. "partial" therefore corresponds to the primary
                  coding's explanatory_model = yes with specification_identified =
                  partial; it is an agreement, as the second_coder_note on that row states.
Sampled-audit verdicts are recorded in the Second_coder_50 sheet and copied to the
second_coder_verdict column of Sample_400, where drawn records are flagged in the
second_coder column. Tracker verdicts are recorded in the
second_coder_verdict column of URLs_786 and in the Second_coder_tracker sheet.

SHA-256 CHECKSUMS
See ../MANIFEST-SHA256.txt for every file in this package, and
archived-approval-files/README.txt for the data files.
