Metadata-Version: 2.4 Name: osm-lead-source-service Version: 0.4.0 Summary: Standalone OSM lead-source service with guarded reconciliation and writes Author: Lutz Kind License-Expression: LicenseRef-Proprietary Requires-Python: >=3.12 Description-Content-Type: text/markdown Requires-Dist: osmium<4.4,>=4.3 Requires-Dist: psycopg[binary]<4,>=3.2 Provides-Extra: dev Requires-Dist: alembic<2,>=1.14; extra == "dev" Requires-Dist: build<2,>=1.2; extra == "dev" Requires-Dist: coverage<8,>=7.6; extra == "dev" Requires-Dist: mypy<2,>=1.13; extra == "dev" Requires-Dist: psycopg[binary]<4,>=3.2; extra == "dev" Requires-Dist: pytest<9,>=8.3; extra == "dev" Requires-Dist: ruff<1,>=0.8; extra == "dev" Requires-Dist: SQLAlchemy<2.1,>=2.0; extra == "dev" # OSM Lead Source Service Standalone OpenStreetMap business-lead generation, reconciliation, and guarded NocoDB import service built around checksum-verified Geofabrik PBF extracts. It is the replacement for the legacy Overpass-based `osm-country-scraper`. Release `0.4.0` removes the ACT-only runtime boundary. The read-only worker can process all 58 configured sovereign English-language markets, automatically split large countries into the smallest available Geofabrik shards, isolate countries that share a regional extract, normalize matching businesses, and publish progress after every shard. ## Current production deployment - Application: `osm-lead-source-service` - Coolify UUID: `ny8uluv6fl1a76hgbgfvokjf` - URL: `https://ny8uluv6fl1a76hgbgfvokjf.luxeillum.com` - Repository access: dedicated read-only GitHub deploy key - Current deployed release: `0.4.0` - Current deployed source commit: `03a1ea1c1e5f85cab0468a30b119567b180bb7f9` - Current deployed scope: 58-country read-only worker - Service status: `running:healthy` - Runtime mode: `shadow-read-only` - Scheduler request: present - Scheduler approval: absent; effective scheduler state is disabled - Startup execution: disabled by default - Noco writer: disabled - Noco writes: disabled - Stale deletion: disabled - Legacy writer cutover: not approved Release `0.4.0` is deployed but intentionally inert. A market cycle cannot start until the independent scheduler-approval gate is explicitly enabled. ## Supported markets `OSM_LEAD_SOURCE_COUNTRIES=all` selects these 58 sovereign states: ```text AG AU BB BI BS BW BZ CA CM DM FJ FM GB GD GH GM GY IE IN JM KE KI KN LC LR LS MH MT MU MW NA NG NR NZ PG PH PK PW RW SB SC SD SG SL SS SZ TO TT TV TZ UG US VC VU WS ZA ZM ZW ``` `UK` is accepted as an alias for `GB`; `USA` is accepted as an alias for `US`. A comma-separated subset can be selected for a controlled run, for example: ```text OSM_LEAD_SOURCE_COUNTRIES=AU,CA,GB,IE,NZ,US ``` Large markets such as the United States, Canada, Australia, and the United Kingdom are resolved recursively through the live Geofabrik catalog and split into non-overlapping country shards. Singapore and the English-speaking Caribbean states without a dedicated Geofabrik file use an approved shared extract followed by a bounded `osmium extract` country crop. ## Automatic read-only worker The HTTP process starts the worker only when both independent gates are true: ```text OSM_LEAD_SOURCE_SCHEDULER_ENABLED=true OSM_LEAD_SOURCE_SCHEDULER_APPROVED=true ``` The first `0.4.0` deployment remains inert with the existing scheduler request present but approval withheld: ```text OSM_LEAD_SOURCE_ENVIRONMENT=production OSM_LEAD_SOURCE_READ_ONLY=true OSM_LEAD_SOURCE_NOCO_WRITE_ENABLED=false OSM_LEAD_SOURCE_WRITER_ENABLED=false OSM_LEAD_SOURCE_STALE_DELETION_ENABLED=false OSM_LEAD_SOURCE_LEGACY_WRITER_DISABLED=false OSM_LEAD_SOURCE_SCHEDULER_ENABLED=true OSM_LEAD_SOURCE_SCHEDULER_APPROVED=false OSM_LEAD_SOURCE_COUNTRIES=all OSM_LEAD_SOURCE_RUN_ON_START=false OSM_LEAD_SOURCE_DASHBOARD_SNAPSHOT_PATH=/tmp/osm-lead-source/dashboard.json OSM_LEAD_SOURCE_CACHE_DIR=/tmp/osm-lead-source/cache OSM_LEAD_SOURCE_REPORT_DIR=/tmp/osm-lead-source/reports OSM_LEAD_SOURCE_SCHEDULE_DAY=1 OSM_LEAD_SOURCE_SCHEDULE_HOUR_UTC=2 ``` `RUN_ON_START` defaults to `false`. After the container, dashboard, storage, and country selection are accepted, scheduler activation requires the separate `SCHEDULER_APPROVED=true` change. The scheduler request alone cannot start work. A cycle: 1. downloads and validates the live Geofabrik catalog; 2. resolves each configured country into the smallest available source shards; 3. downloads or reuses each checksum-validated PBF; 4. performs a bounded country crop when a source contains multiple countries; 5. streams nodes, ways, and relations through the versioned `luxeillum` profile; 6. normalizes and deduplicates businesses by country and OSM identity; 7. persists per-shard report and acquisition evidence locally; 8. publishes aggregate country/shard progress after every shard; and 9. continues after isolated shard failures, marking the cycle `partial`. The automatic worker has no Noco credential or mutation path. It cannot insert, update, adopt, relabel, or delete a lead. ## Controlled one-shot market cycle Use `market-run-once` for the first subset and full-market production evidence. It reuses the exact `WorkerConfig.from_env()` and `run_worker_once()` path used by the scheduler, but it runs in the foreground exactly once and does not require, start, or modify scheduler approval. Preview the resolved configuration without network access or file creation: ```bash OSM_LEAD_SOURCE_COUNTRIES=IE,NZ,SG \ osm-lead-source market-run-once ``` Execute one read-only cycle only after durable cache, report, and dashboard storage is attached: ```bash OSM_LEAD_SOURCE_COUNTRIES=IE,NZ,SG \ osm-lead-source market-run-once --execute ``` The command refuses to run when the writer, Noco writes, or stale deletion is enabled. Preview is the default; Geofabrik network/download work requires the explicit `--execute` flag. It does not expose an HTTP mutation route. ## Operator dashboard Open `/` for the read-only OpenStreetMap Lead Generator console. It shows: - current cycle state: running, partial, or completed; - configured, completed, and failed countries; - configured, completed, and failed shards; - active and review candidate totals; - category distribution; - aggregate source and report hashes; - Noco comparison status; and - production-readiness gates. Machine-readable endpoints: - `GET /healthz` - `GET /readyz` - `GET /status` - `GET /api/status` - `GET /api/dashboard` There is no HTTP mutation endpoint. Unknown routes and all POST, PUT, PATCH, and DELETE requests fail closed. ## Historical ACT validation ACT was the bounded validation region used to prove acquisition, parsing, normalization, profile evaluation, and exact Noco identity comparison. It is no longer the intended production scope. The accepted ACT run found: - 980 active business identities; - 502 exact existing Noco OSM identity matches; and - 478 identities without the same `(osm_type, osm_id)` in Noco. No ACT lead was written, updated, adopted, relabeled, or deleted. ## Implemented capabilities - 58-country English-market registry with country calling codes. - Machine-readable Geofabrik catalog validation. - Automatic smallest-available country sharding. - Bounded shared-extract country isolation through `osmium-tool`. - Checksum-verified acquisition and content-addressed caching. - Bounded streaming PBF parsing for nodes, ways, and relations. - Immutable profile evaluation, normalization, and shadow reports. - Cross-shard deduplication by country and OSM identity. - Read-only Noco schema, identity, duplicate, and adoption components. - Guarded API-only Noco insert implementation with final duplicate checks. - Sidecar approvals, action idempotency, and a global writer lease. - Non-root deployment image with dashboard and JSON status APIs. ## Safety boundaries - No Windmill dependency. - No direct writes to the Noco PostgreSQL database. - The automatic worker refuses to start when Noco writes, the writer, or stale deletion is enabled. - Inserts require an explicit writer flag, verified legacy cutover, an unexpired database approval, the exact report hash, the exact destination table, an insert cap, and `--execute`. - Update and delete execution remain unavailable. - Stale deletion requires two complete snapshots, at least 60 days of evidence, quarantine, workflow protection, and separate approval. - The dashboard never exposes credentials, tokens, database URLs, or private key material. ## Local validation Python 3.12 or newer is required. ```bash python -m venv .venv . .venv/bin/activate python -m pip install -e ".[dev]" python -m ruff check . python -m ruff format --check . python -m mypy src python -m pytest python -m build ``` ## Remaining production gates Release `0.4.0` is merged, deployed, and healthy. The remaining work is operational completion rather than another generator rewrite: 1. Attach durable cache, report, and dashboard storage; prove persistence across restart and redeploy. 2. Run and accept a controlled read-only subset covering a direct extract and a shared-source country crop. 3. Run and retain evidence for the complete 58-country read-only cycle. 4. Perform generic read-only Noco identity, duplicate, workflow-protection, and adoption comparison against the generated reports. 5. Provision and migrate the isolated PostgreSQL/PostGIS sidecar, with backup, restore, rollback, and NocoDB-isolation evidence. 6. Review provenance, duplicate, adoption, lifecycle, and capacity evidence. 7. Record formal proof that the legacy OSM writer is disabled. 8. Approve and execute one deliberately small, capped, insert-only pilot through the guarded NocoDB API writer. 9. Add monthly source-presence and quarantine evidence only after the insert pilot is accepted. 10. Keep updates and stale deletion disabled until separately implemented and approved under the two-snapshot/60-day policy. See `docs/handoffs/2026-07-21-repository-completion.md` for the full completion plan and new-session instruction. ## Documentation - [Project state](docs/PROJECT_STATE.md) - [Dashboard](docs/DASHBOARD.md) - [Operations](docs/OPERATIONS.md) - [Architecture](docs/ARCHITECTURE.md) - [Decisions](docs/DECISIONS.md) - [Roadmap](docs/ROADMAP.md) - [Security](docs/SECURITY.md) - [Sidecar schema](docs/SIDECAR_SCHEMA.md) - [PBF ingestion foundation](docs/PBF_INGESTION.md)