SF Civic Digest — SKILL.md

Cold-start note: This file is designed to be self-sufficient. An agent reading only SKILL.md should be able to run reports, generate digests, and understand the full data stack. No other context required.

---

Quick Start

CODEBLOCK0

Note: sf_weekly_digest.py makes ~22 network calls sequentially and can take 2-3 minutes. For a quick test, try a single script first: INLINECODE1

District: Pass --district N (1–11) to any script. User preferences and neighborhood context live in USER.md — read it before writing a report. There is no civic_config.json or civic_config.example.json; those files have been removed.

⚠️ Path rule — critical for any agent (any agent, IDE, or human):

• - All scripts must be run with the absolute path to the scripts/ directory
• Find it: scripts/ is always a sibling of this SKILL.md file
• Example: if SKILL.md is at /home/alice/.openclaw/workspace/skills/sf-civic-digest/SKILL.md, then scripts are at INLINECODE8
• Never use bare relative paths like scripts/foo.py — they break when the working directory is anything other than the skill root
• When in doubt: realpath on any script path gives you the absolute path

---

Architecture

Scripts are data pipelines. The agent writes the narrative.

The scripts fetch, parse, dedup, and output JSON. The agent (you) reads that JSON and synthesizes it into a report following the style guide. There is no auto-generated report — the editorial judgment is yours.

CODEBLOCK1

For report generation, see Report Generation section below.

---

All Scripts

Canonical script location: skills/sf-civic-digest/scripts/

Script	What it does	Method	Notes
INLINECODE12	Master aggregator — calls all subscripts	—	Use --json for machine-readable output; pass to agent for synthesis
INLINECODE14

Board of Supervisors + all committees via Legistar | curl | --daily for delta only; --days N; state in sf_civic_state.json |
| sfmta_hearings.py | SFMTA Engineering Public Hearings | curl + PDF | Street changes, parking meters, signals. Every ~2 weeks. |
| sf_planning_notices.py | SF Planning project notices (Section 311) | curl | Earliest signal layer — before commission hearings. Needs Accept: text/html header. |
| sf_planning_commission.py | Planning Commission, HPC, ZA full agendas | curl + PDF | Server-rendered HTML (no browser). --body hpc for HPC, --body za for ZA. |
| sf_board_of_appeals.py | Board of Appeals | Wagtail API | sf.gov CMS API (api.sf.gov). 1st & 3rd Wednesdays, 5pm. No browser needed. |
| sf_building_permits.py | Building permits citywide | Socrata REST | Every permit filed. Classified HIGH/NOTABLE/noise. State-tracked. |
| sfgov_events.py | Supervisor office hours, district events | REST/JSON | Very sparse — supervisor events only. Low signal. |
| sf_311.py | 311 service requests by district | Socrata REST | Spike detection. Query two 7-day windows for trend comparison. |
| sf_ethics.py | Lobbyist contacts + campaign contributions | RSS + Socrata | Contacts: 5f5n-tdbf. Contributions: e6py-fg8b. Column names have NO underscores. |
| sf_rent_board.py | Rent Board commission meetings | Wagtail API | sf.gov CMS API (api.sf.gov). No browser needed. Rent increase: links to sf.gov (no hardcoded value). |
| sf_sfmta_board.py | SFMTA Board of Directors | curl | 1st & 3rd Tuesdays 1pm. Service changes, fare decisions, bike/ped safety. |
| sf_rec_park.py | Rec & Park Commission | curl (Granicus) | view_id=91. sfrecpark.org is unreachable from server — Granicus only. |
| sf_housing_pipeline.py | AB 2011 / SB 35 / density bonus housing pipeline | Socrata REST | Cross-refs parcels dataset for district. Watchlist tracks 400 Divisadero. State-tracked. |
| sf_journalism.py | News aggregator — 5 SF outlets | RSS | Mission Local, Streetsblog SF, SF YIMBY, 48 Hills, SF Standard. SF Examiner 404s. Archive: sf_journalism_archive.json. |
| sf_volunteer_cleanups.py | Community cleanup events | curl/JS bundle | Refuse Refuse SF + DPW Love Our City NBDs. GoDaddy SPA — data is in JS bundle, not HTML. |
| sf_bart_board.py | BART Board of Directors + advisory bodies | Legistar API | webapi.legistar.com/v1/bart/Events. Board = HIGH, advisory = MEDIUM. --next for next meeting. |
| sf_sfusd_board.py | SFUSD Board of Education meetings | Google Sheets + BoardDocs | 2nd & 4th Tuesdays 5pm. Falls back to projected schedule if sources unavailable. |
| sf_evictions.py | Eviction notices by district | Socrata REST | Dataset 5cei-gny5. Ellis Act + OMI = displacement signals. Trend comparison (two windows). --district N. |
| sf_sfcta.py | SF County Transportation Authority | curl (sfcta.org) | Congestion pricing, Muni Forward, bike/ped funding, Prop L sales tax. Board + CAC + committees. |
| sf_civic_actions.py | Protests, rallies, civic actions | Mobilize.us API + Indybay | No auth. Rallies/marches = HIGH, canvass/phone bank = MEDIUM. --type rally to filter. |
| sf_mission_local.py | Mission Local news only | RSS | Subset of sfjournalism.py. Use sfjournalism.py for multi-outlet. |
| config_loader.py | District config loader | — | All 11 districts pre-populated. Pass --district N to any script. |

No browser required

All scripts now work without a browser. sf_board_of_appeals.py and sf_rent_board.py use the sf.gov Wagtail CMS API (api.sf.gov/api/v2/pages/). sf_planning_commission.py uses curl against server-rendered Drupal HTML (the sfplanning.org calendar pages are not JS-dependent despite initial assumptions).

---

District Configuration

All scripts accept --district N (1–11). That's all you need for a basic run.

User preferences and neighborhood context (streets, addresses, people to flag) live in USER.md — a human-readable profile file. There is no civic_config.json; that file has been removed. Read USER.md to understand what the user cares about before writing any report.

District→neighborhood mappings: references/sf-districts.md
Build timeline benchmarks: references/sf-build-timelines.md — Read before writing about any housing project. Includes typical phase durations, financing tracking methods, and context for whether a project is on track or stalled.

---

Report Generation

Agent-driven (recommended for one-off reports)

CODEBLOCK2

Agent-driven reports (quarterly, annual, deep dives)

Give your agent the workspace path and let it orchestrate:

Read STYLE.md and USER.md. Run the scripts in scripts/ to collect data for District 5, then write a Q1 2026 civic digest. Save to reports/.

⚠️ Path gotcha: Agents default to wrong relative paths. Always resolve the absolute path to scripts/ from the location of this SKILL.md before running anything. Use full absolute paths — e.g. python3 /full/path/to/scripts/sf_housing_pipeline.py. Never just scripts/SCRIPTNAME.py.

Report style

Read STYLE.md before writing any report. Key rules:

• - TLDR (5-7 bullets) first — last bullet is always one action
• Officials box: YOUR elected officials only (supervisor, mayor, state reps)
• Short sections (3-5 sentences), scan-and-skip
• Connect threads across sources (lobbying + permits + 311 + journalism = one story)
• End each thread with a specific action (date, email address, what to say)
• Rates over counts. "Up 15%" beats "95 reports."
• Tone: pro-building, pro-tech, car-free lens. See STYLE.md for full framing guidance.

Reference: see STYLE.md for rules and examples.

---

Key Data Gotchas

These will burn you if you don't know them:

Socrata column names:

• - 311 (vw6y-z8j6): supervisor_district = '5.0' (float string), date field = INLINECODE72
• Building Permits (i98e-djp9): supervisor_district = '5'. Address split across street_number/street_name/street_suffix — no combined field. estimated_cost is a string, cast in Python.
• Lobbyist Contacts (5f5n-tdbf): NO underscores — officialname, clientname, INLINECODE83
• Lobbyist Contributions (e6py-fg8b): same — lobbyistname, candidatename, INLINECODE87
• Planning Records (qvu5-m3a2): pim_link returns {"url": "..."} dict, not string. No supervisor_district — cross-reference parcels (acdm-wktn) by block/lot.

Legistar: The API is broken for SF (returns "Agenda Draft Status" error). Use the RSS feed instead:
https://sfgov.legistar.com/Feed.ashx?M=C&ID=17442&GUID=EEE85B7C-1A1C-4A56-873E-355A0A0DE5C3&Mode=All&Format=rss
Returns ~100 meetings (~6 months). Parse meeting IDs from RSS, then fetch_meeting_detail() for agenda items.

Reddit: Direct curl blocked (403, datacenter IP). Use web_fetch tool — routes differently, works.

sfrecpark.org: Completely unreachable from this server. Granicus (view_id=91) is the only path.

311 trend pattern: Query two 7-day windows (current + prior week) and show week-over-week delta. "Encampments up 15%" is useful. "95 encampments" is not.

Journalism archive: sf_journalism.py persists articles to sf_journalism_archive.json (cap 2000). For quarterly history, use --archive --days 90.

---

Socrata Dataset Reference

Dataset	ID	Key fields
Building Permits	INLINECODE102	permitnumber, permittype, address (split), supervisordistrict ('5'), estimatedcost (string)
311 Cases

vw6y-z8j6 | servicename, requesteddatetime, supervisor_district ('5.0') |
| Planning Dept Records | qvu5-m3a2 | recordid, description, recordstatus, projectaddress, block, lot, numberofunitsnet, pim_link (dict) |
| SF Parcels | acdm-wktn | blocknum, lotnum, supervisordistrict, zoningcode |
| Lobbyist Contacts | 5f5n-tdbf | officialname, lobbyistname, clientname, outcomesought, subjectarea, filenumber |
| Lobbyist Contributions | e6py-fg8b | lobbyistname, candidatename, sourceoffunds, committeename |
| Assessor Parcels | fk72-cxc3 | propertylocation, parcelnumber, assessedlandvalue, zoning_code |
| Eviction Notices | 5cei-gny5 | evictionid, address, neighborhood, filedate, supervisordistrict, ellisactwithdrawal, ownermove_in |

Query pattern: https://data.sfgov.org/resource/{id}.json?$where=supervisor_district='5'&$order=date+DESC&$limit=50

---

Scraping Method Reference

Site	curl works?	Browser needed?
Legistar (sfgov.legistar.com)	✅	No
SFMTA (sfmta.com)

✅ | No |
| SF Planning notices (sfplanning.org/notices) | ✅ (needs Accept header) | No |
| SF Planning calendars (sfplanning.org/hearings-*) | ✅ (use -list views) | No |
| SF.gov meetings (api.sf.gov) | ✅ (Wagtail API) | No |
| Socrata (data.sfgov.org) | ✅ | No |
| SF Ethics (sfethics.org) | ✅ | No |
| Mission Local / journalism RSS | ✅ | No |
| Granicus (Rec & Park) | ✅ | No |
| Refuse Refuse SF (refuserefusesf.org) | ✅ (JS bundle) | No |
| sfrb.org | ✅ (Wagtail API via api.sf.gov) | No |

---

Scheduling

Weekly digest (Monday morning)

CODEBLOCK3

Daily delta (weekday noon check-in)

{
  "schedule": { "kind": "cron", "expr": "0 12 * * 1-5", "tz": "America/Los_Angeles" },
  "payload": { "kind": "systemEvent", "text": "Run SF civic daily delta and report any new agenda items" }
}

---

File Layout

CODEBLOCK5

---

Participation Guide

How to comment at SFMTA Engineering Hearings

• - Online: join at sfmta.com/EngHearing when your item is called, raise hand
• Phone: number + conference ID in hearing notice
• Written: email the contact listed per item with "Public Hearing" in subject
• Decisions posted the following Friday at INLINECODE113

How to comment at Planning Commission / HPC

• - Written comments accepted before each hearing via sfplanning.org
• HPC meets 1st and 3rd Wednesdays (low attendance, high impact)
• In-person: City Hall Room 400

How to reach your supervisor

• - Mahmood (D5): bilal.mahmood@sfgov.org
• Constituent meetings: request via supervisor's website
• Board public comment: first Tuesday of each month, City Hall Room 250