# Agent Instructions ## Project Purpose Build and maintain the standalone OSM lead-source replacement service. The service will use Geofabrik regional PBF extracts to ingest and reconcile OSM lead sources while preserving compatibility with the existing NocoDB lead table. ## Source Of Truth - GitHub is the authoritative source for code and documentation. - The shared ChatGPT directory is only a review handoff mechanism. - Do not rely on chat history as the sole project record. - Update `docs/PROJECT_STATE.md` and the relevant handoff after each accepted phase. ## Safety Boundaries - Do not modify the legacy scraper unless explicitly instructed in a separate task. - Do not modify the NocoDB lead-table schema. - Never write directly to the Noco PostgreSQL database. - Use the NocoDB API for future creates, updates, and deletes. - Do not introduce Windmill. - Do not enable production writes without an explicit phase approval. - Do not implement stale deletion before Phase 2D. - Treat unknown field ownership and workflow state as protected. - Do not expose secrets in reports, logs, fixtures, or shared directories. - Do not deploy as part of implementation tasks unless deployment is explicitly requested. ## Development Workflow Use this workflow: ```text ChatGPT architecture and implementation -> scoped GitHub branch and PR -> tests and CI -> shared review bundle -> independent adversarial review -> user approval -> merge -> separate deployment task ``` ## Shared-Directory Handoff After every substantial task, copy review artifacts into: ```text /root/mcp-shared/chatgpt/osm-lead-source-service// ``` The Shell MCP-visible equivalent is: ```text /mcp-shared/chatgpt/osm-lead-source-service// ``` Each handoff must include: - `HANDOFF.md` - `manifest.json` - Branch - Full commit SHA - PR number and state when applicable - Files changed - Commands executed - Exact test results - Known risks - Unresolved questions - Production modification status - SHA-256 checksums of copied artifacts Never copy credentials, `.env` files, databases, private keys, access tokens, archives containing secrets, or unrelated repository data. ## Phase Gates - Phase 1: investigation and compatibility design. - Phase 1.5: architecture correction and review. - Repository foundation: complete. - Phase 2A.1: accepted foundation and guardrails. - Phase 2A.2: accepted sidecar schema and migrations. - Phase 2A.3: PBF ingestion foundation, implementation in review. - Phase 2A.4: profiles, normalization, and deduplication. - Phase 2A.5: read-only Noco adapter and adoption scanner. - Phase 2A.6: integrated shadow reconciliation. - Phase 2B: insert-only controlled writer. - Phase 2B.1: checksum-verified read-only Geofabrik pilot acquisition. - Phase 2C: safe refresh and stale lifecycle. - Phase 2D: approved stale deletion. - Phase 2E: authenticated MCP integration. - Final controlled cutover. ## Phase 2A Offline Boundary All Phase 2A code remains read-only. A subphase may enable only the exact isolated read or loopback test path explicitly approved for that subphase. Do not add production writes, direct Noco PostgreSQL writes, stale deletion, scheduling, deployment, or unrelated network access. ## Local Validation Commands ```text python -m pip install -e ".[dev]" python -m ruff check . python -m ruff format --check . python -m mypy src python -m pytest python -m build ``` ## Phase 2A.2 test-database rules Use only this disposable test database for migration work: ```text docker compose -f compose.test.yml up -d --wait export OSM_LEAD_SOURCE_MIGRATION_ENV=test export OSM_LEAD_SOURCE_TEST_DATABASE_URL=postgresql+psycopg://test_user:test_password@127.0.0.1:55432/osm_lead_source_test python -m alembic upgrade head python -m alembic downgrade base python -m alembic upgrade head docker compose -f compose.test.yml down -v ``` Permanent database safety rules: - Never run migrations against an unapproved host. - Only `OSM_LEAD_SOURCE_TEST_DATABASE_URL` is accepted by the migration boundary. - Do not inspect or use production database variables, including `DATABASE_URL`, `POSTGRES_URL`, or `NOCODB_DATABASE_URL`. - Do not weaken offline CLI behavior; normal CLI commands must not import a database driver as a side effect or check database availability. - Do not add direct Noco PostgreSQL access or any NocoDB schema mutation. - `compose.test.yml` is disposable test infrastructure, not a production deployment. Its static credentials must never be reused. Phase 2A.2 integrity rules: - A complete snapshot requires hash, positive size, and ordered download, validation, parser, and reconciliation timestamps. - Snapshot attempts are append-only finalized evidence. - Stale confirmation requires two distinct complete missing snapshots, consistent observations, matched regions, immutable observation-backed timestamps, old last-seen evidence, and no boundary ambiguity. - Accepted mappings are immutable historical records with chronological mapping timestamps; supersession points only to a live active mapping for the same Noco row. Baseline chains reference the immediate prior superseded baseline. - Database tests may use only the exact host and port parsed from the validated test database URL. ## Phase 2A.3 PBF rules Phase 2A.3 is limited to synthetic, approved test inputs. Permanent rules: - Do not configure or contact a production Geofabrik URL. - Do not download a production or regional PBF. - Do not accept an unverified extract. - Do not create an unbounded in-memory list of parsed OSM objects. - Parse only an explicit `VerifiedPbfInput` whose SHA-256 and byte count match the local file. - Parse through transactional `PbfObjectSink`; stage detached callback data, call `finish()` only after a complete stable parse, and call `abort()` after any streaming or sink failure. - Support only nodes, ways, and relations. - Enforce file, Blob, actual zlib expansion, declared uncompressed Blob, object, tag, way-reference, and relation-member limits before retaining data. - Reject symbolic-link inputs and cache directories. - Use exact SHA-256 and byte-size validation. - Keep cached files content-addressed as `.osm.pbf`. - Do not silently replace an invalid cache entry. - Do not follow redirects, use environment proxies, or accept HTTP content encoding. - `ParsedExtractMetadata` is not a complete reconciliation snapshot and must never set or imply `reconciliation_completed_at`. - Generic variables such as `PBF_URL` and `GEOFABRIK_URL` are ignored. Only tests marked both `integration` and `pbf_network` may use the dynamically created loopback fixture server. They may connect only to its exact host and port. They may not create another listener, use UDP, resolve another host, or contact an external service. A test cannot combine `database` and `pbf_network` access. ## Phase 2B.1 Geofabrik pilot boundary Phase 2B.1 may contact only the exact checked-in Australian Capital Territory Geofabrik PBF URL and its companion `.md5` URL, and only through the explicit `geofabrik-acquire --execute` command. Preview mode must remain network-free. Permanent rules: - The only approved region is `au-act`. - Reject environment proxies, redirects, alternate hosts, URL credentials, query strings, fragments, and non-HTTPS URLs. - Require bounded `Content-Length` and identity HTTP content encoding. - Verify the companion MD5 evidence and compute SHA-256 locally. - Store PBFs only as `.osm.pbf` in a regular, non-symlink cache tree. - Never silently replace or trust an invalid cache entry. - This phase must not contact NocoDB, mutate the sidecar, schedule runs, enable the writer, or interact with the legacy scraper. - Production acquisition is not a complete reconciliation snapshot and must not imply Noco adoption, insert approval, stale evidence, or cutover readiness. Run focused validation with: ```text python -m pytest tests/unit/pbf tests/integration/pbf ``` The checked-in fixture is a Base64 transport representation of a small synthetic PBF. Decode it only into a temporary test directory. Never add production OSM extracts or PBF caches to Git.