# Repository Completion Handoff — 2026-07-21 ## Purpose `osm-lead-source-service` is the standalone replacement for the legacy Overpass-based `osm-country-scraper`. The intended product is not merely a dashboard or a one-off OSM export. It is a complete monthly OpenStreetMap lead-source system that: 1. acquires country and regional Geofabrik PBF extracts; 2. generates normalized business candidates using versioned profiles; 3. reconciles them with existing NocoDB leads without creating duplicates; 4. adopts existing OSM-derived rows in place; 5. inserts new rows only through a guarded, capped, API-only writer after formal cutover approval; 6. tracks provenance and source presence over time; and 7. eventually handles safe stale lifecycle decisions using multiple snapshots and explicit approval. Windmill is not part of this service. The service is intended to run as a standalone Coolify application with its own scheduler, durable storage, and isolated PostgreSQL/PostGIS sidecar. ## Original architecture plan ### Source acquisition - Replace live Overpass scraping with checksum-verified Geofabrik `.osm.pbf` downloads. - Refresh configured markets monthly. - Cache source files by content hash and retain acquisition evidence. - Split large countries into the smallest available non-overlapping Geofabrik shards. - Isolate countries that share a regional extract with bounded `osmium extract` processing. ### Candidate generation - Parse nodes, ways, and relations with bounded streaming logic. - Apply versioned category profiles rather than hard-coded country scripts. - Normalize names, websites, phones, addresses, categories, and source identity. - Keep source presence separate from profile eligibility: an OSM object may still exist even when a profile no longer considers it a suitable lead. - Deduplicate by exact OSM identity first and then report possible business or website collisions for review. ### NocoDB compatibility - Keep the existing Noco lead-table schema compatible; do not redesign NocoDB around this service. - Use read-only PostgreSQL access only for scalable inspection of existing Noco rows. - Use the NocoDB API for every future mutation. - Never write directly to NocoDB's PostgreSQL database. - Do not treat absence of an exact `(osm_type, osm_id)` match as automatic permission to insert. ### Existing-lead adoption - Adopt existing OSM-derived Noco rows in place rather than deleting and recreating them. - Preserve existing IDs, campaign/workflow state, process marks, and non-OSM ownership evidence. - Fail closed when ownership or workflow state is unknown or protected. - Maintain sidecar mappings between source identities and adopted Noco records. ### Sidecar database The isolated PostgreSQL/PostGIS sidecar is intended to store: - source snapshots and region coverage; - source-presence history; - profile/version eligibility; - normalized candidates and collision evidence; - Noco mappings and adoption decisions; - provenance and reconciliation reports; - approvals, idempotency records, and writer leases; - missing counters, quarantine state, and stale evidence; and - audit and cutover evidence. It is not a replacement for NocoDB and must not share NocoDB's database. ### Controlled writer and cutover - The automatic source worker remains read-only. - Inserts use a separate explicit writer command and the NocoDB API. - Writer activation requires the exact approved report hash, destination table, insert cap, unexpired approval, final duplicate checks, lifecycle checks, single-writer lease, and explicit `--execute`. - The replacement writer must not operate concurrently with the legacy scraper. - Updates and deletes remain unavailable until separately implemented and approved. ### Stale lifecycle A missing object is not immediately stale. The original policy requires: - complete source snapshots rather than partial or failed runs; - at least two qualifying complete snapshots; - at least 60 days of evidence; - distinction between source deletion and profile exclusion; - quarantine before deletion; - ownership and workflow-protection checks; - rollback evidence; and - separate explicit deletion approval. No profile change may be treated as proof that the underlying business was removed from OSM. ### MCP scope Authenticated MCP integration was deliberately deferred until the core service is accepted. Any future MCP surface should expose bounded read/search, dry-run, and specifically approved actions only. It must not add arbitrary SQL, shell, or unrestricted write tools. ## Original phase model ### Phase 1 — Research and contracts Document the Noco compatibility contract, existing-lead adoption rules, stale safety policy, replacement architecture, and production constraints. ### Phase 2A — Read-only shadow implementation Build and validate: 1. foundation and immutable guardrails; 2. isolated sidecar schema and migrations; 3. PBF acquisition and parsing; 4. profiles, normalization, and deduplication; 5. Noco read adapter and adoption scanner; and 6. integrated deterministic shadow reconciliation reports. No production mutation is permitted in this phase. ### Phase 2B — Controlled insert-only writer Implement API-only inserts with final duplicate checks, lifecycle checks, idempotency, approvals, caps, and writer exclusivity. Do not activate it until read-only evidence and legacy cutover are accepted. ### Phase 2C — Monthly refresh and stale evidence Track source presence, missing counters, remap candidates, profile changes, and quarantine candidates without deleting anything. ### Phase 2D — Approved stale deletion Allow narrowly approved deletion only after the full two-snapshot/60-day policy, workflow protection, and rollback evidence are satisfied. ### Phase 2E — Optional authenticated MCP integration Add bounded operational tools only after the service itself is complete and accepted. ## Current state ### Completed - Release `0.4.0` is merged and deployed. - Production application: `osm-lead-source-service`. - Coolify UUID: `ny8uluv6fl1a76hgbgfvokjf`. - URL: `https://ny8uluv6fl1a76hgbgfvokjf.luxeillum.com`. - Production code commit recorded by deployment `kdp1rmmzrhlmwm4bh0fl8iq3`: `03a1ea1c1e5f85cab0468a30b119567b180bb7f9`. - Repository documentation may advance after that deployment; always verify the current `main` head before making changes. - The service is `running:healthy` in `shadow-read-only` mode. - The 58-country English-market registry and generic multi-market worker are implemented. - Large-country sharding and bounded shared-source extraction are implemented. - The read-only dashboard and JSON status endpoints are implemented. - Scheduler activation requires both request and approval flags. - Production runtime controls are explicitly normalized to: `SCHEDULER_APPROVED=false`, `RUN_ON_START=false`, and `COUNTRIES=all`. - The application was restarted after that reconciliation and returned to `running:healthy`; no worker or cycle log entry appeared after restart. - Writer, Noco mutation, stale deletion, and legacy cutover remain disabled. - Unit, integration, migration, package, and container validation were completed through isolated validators when GitHub Actions failed before runner allocation. - The ACT validation produced 980 active identities, 502 exact Noco OSM matches, and 478 identities without the same exact OSM key. No lead was written. ### Important clarification The repository is code-complete for a read-only multi-market generator, but the production system is not operationally complete. It is intentionally inert and has not yet run a durable production country cycle or written leads. The generic one-shot CLI is now implemented as `market-run-once`. It previews configuration by default and requires explicit `--execute` for Geofabrik work. It calls the same `WorkerConfig.from_env()` and `run_worker_once()` path as the scheduler, runs exactly once in the foreground, and never requires or changes scheduler approval. ## Actual next work Complete the repository in this order. ### 1. Durable runtime storage Attach persistent Coolify storage for: - `/var/lib/osm-lead-source/cache` - `/var/lib/osm-lead-source/reports` - `/var/lib/osm-lead-source/dashboard.json` or a persistent parent directory Update runtime paths accordingly. Confirm restart/redeploy persistence before approving any worker execution. Do not use container-local `/tmp` for a production cycle. ### 2. Generic one-shot read-only worker command — complete `market-run-once` is implemented and validated. It: - runs exactly one configured country selection and exits; - enforces the existing read-only worker refusals for writer, Noco write, and stale-deletion flags; - does not require or alter scheduler approval; - does not start the monthly scheduler or a background thread; - previews configuration without downloading by default; - requires explicit `--execute` before Geofabrik network/download work; - emits bounded machine-readable completion/failure evidence; - writes through the same cache, report, and dashboard paths as the scheduler; - has deterministic unit tests for one execution only, no scheduler start, fail-closed capabilities, masked unexpected errors, and identical worker configuration; and - adds no HTTP mutation endpoint, Windmill workflow, or MCP tool. Validation on commit `6074e3e3424e3da36b93d6cb3d714845b88b8b83` passed Ruff lint and formatting, strict mypy, all unit tests, package build, and the non-root runtime healthcheck in an isolated Coolify validator. ### 3. Controlled read-only subset Use the one-shot command to run a deliberately small representative subset before all 58 countries. A useful coverage set is: - `IE` — direct country extract; - `NZ` — direct country extract with different market metadata; and - `SG` — shared regional extract requiring bounded country isolation. Keep: - scheduler approval false; - writer disabled; - Noco writes disabled; - stale deletion disabled; and - run-on-start false. Verify reports, cache reuse, dashboard progress, restart persistence, bounded resource use, and no Noco mutation. ### 4. Full 58-country read-only cycle After subset acceptance: - set the country scope to `all`; - use the one-shot command for the first full cycle; - monitor disk, memory, duration, shard failures, and report bounds; - require a complete or explicitly reviewed partial result; - retain all source, report, and aggregate hashes; and - enable the monthly scheduler only after the one-shot full-cycle evidence is accepted. ### 5. Generic Noco comparison and adoption report Use the generated reports to perform a read-only comparison against the real Noco table: - exact OSM identity matches; - adoption candidates; - likely duplicate businesses or websites; - protected workflow rows; - non-OSM ownership conflicts; - ambiguous matches; and - genuinely new insert candidates. Do not equate unmatched OSM identity with a safe insert. ### 6. Production sidecar Provision a dedicated PostgreSQL/PostGIS database, run the existing migrations, and connect only the reconciliation/approval components. Verify backup, restore, migration rollback, and isolation from NocoDB before relying on it. ### 7. Evidence review and legacy cutover Review duplicate, adoption, provenance, workflow protection, lifecycle, and capacity evidence. Record formal proof that the legacy OSM writer is disabled before arming the replacement writer. ### 8. Small capped insert-only pilot Approve a deliberately small region/profile/report hash and insert cap. Run the writer once with final duplicate and lifecycle checks. Verify idempotent rerun behavior and Noco lineage before expanding. ### 9. Monthly refresh and stale lifecycle Only after insert operation is accepted, implement/activate Phase 2C presence history and quarantine. Keep deletion disabled until Phase 2D is explicitly approved after the two-snapshot/60-day rule is satisfied. ## Constraints for the next session - Keep the toolset minimal; do not add MCP tools or profiles merely for setup. - Do not introduce Windmill. - Do not add an HTTP mutation endpoint for operational runs. - Implement one-shot execution by reusing the existing worker, not by duplicating scheduler or reconciliation logic. - Do not change the existing Noco schema unless a proven incompatibility requires a separately approved migration. - Do not provision or connect the sidecar before durable storage and the controlled read-only subset are understood. - Do not enable Noco writes, writer execution, stale deletion, or legacy cutover while validating the generator. - Do not start a full 58-country cycle using ephemeral storage. - Preserve the legacy scraper until formal cutover evidence is accepted. - Remove every temporary validator, script, application, and test artifact after use. ## New-session instruction Use the following as the opening instruction in a new session: > Continue and complete `lutzkind/osm-lead-source-service` from > `docs/handoffs/2026-07-21-repository-completion.md`. First verify the current > GitHub `main` head and Coolify application `ny8uluv6fl1a76hgbgfvokjf`. Treat > the repository as a standalone monthly Geofabrik PBF lead generator and safe > Noco reconciliation/import service, not merely a dashboard. Keep the runtime > inert while working: scheduler approval false, run-on-start false, writer and > Noco writes disabled, stale deletion disabled, and legacy cutover unapproved. > Follow the documented order: persistent storage; a generic one-shot read-only > CLI command that reuses `run_worker_once`; a controlled `IE,NZ,SG` subset; the > first full 58-country one-shot cycle; generic Noco comparison/adoption; the > isolated sidecar; evidence review; formal legacy cutover; and a small capped > insert-only pilot. Do not add MCP tools, Windmill, or an HTTP mutation > endpoint. Update the handoff and project state after each accepted stage.