# Phase 2A.3 PBF Ingestion Foundation Phase 2A.3 adds a bounded parser boundary and an explicitly test-only download path for synthetic Geofabrik-style `.osm.pbf` inputs. It does not enable a production extract URL, production cache, database writer, scheduler, or reconciliation run. ## Scope Implemented boundaries: - pyosmium-backed streaming parser for nodes, ways, and relations - explicit verified-input integrity evidence before parsing - detached immutable object records through a transactional sink - deterministic SHA-256 and byte-size evidence - parser-complete extract metadata - content-addressed PBF and metadata test cache - loopback-only HTTP test download - exact expected SHA-256 and size validation - redirect, proxy, content-encoding, content-type, and response metadata rejection - bounded PBF envelope, Blob count, cumulative expansion, object, tag, way-reference, and relation-member limits - explicit rollback-failure reporting - fail-closed test-network isolation `ParsedExtractMetadata` is parser-complete evidence only. It intentionally has no `reconciliation_completed_at` or `completeness_status` field and therefore cannot be treated as a complete sidecar `source_snapshot`. ## Synthetic fixture provenance The repository stores: ```text tests/fixtures/pbf/tiny.osm.pbf.b64 tests/fixtures/pbf/tiny.osm.pbf.sha256 tests/fixtures/pbf/README.md ``` The Base64 file is the transport form of a 438-byte synthetic PBF generated with pyosmium 4.3.1. It contains no production OSM data or personal data. Decoded SHA-256: ```text 1f181fa63b838de88c5c7d8ecff2a014ca52a9349c6470fd2656ad3c9e0ddb8c ``` Expected objects: | Type | ID | Evidence | |---|---:|---| | node | 1 | `Fixture Cafe`, `amenity=cafe` | | node | 2 | `Fixture Node` | | way | 10 | node refs 1 and 2, `amenity=restaurant` | | relation | 20 | way 10 as `outer`, `tourism=hotel` | Tests decode the fixture only into a pytest temporary directory. ## Parser boundary `OsmiumPbfParser` accepts only a `VerifiedPbfInput` containing an exact local path, SHA-256, byte count, and nonblank provenance. It re-hashes and re-counts the file before parsing, so a bare path or mismatched integrity evidence is rejected. `VerifiedPbfInput.provenance` is descriptive audit context supplied by the caller. It is not an authorization token and does not prove that a file was approved by an external authority. This phase has no production download or production input-authorization entrypoint. The parser streams detached `ParsedOsmObject` records to a transactional `PbfObjectSink`. Pyosmium callback objects are never retained. The sink stages objects during callbacks, receives `finish(summary)` only after parsing and file-stability checks succeed, and receives `abort()` after a streaming, validation, or sink failure. Callers must not make staged objects externally visible before `finish()`. If rollback itself fails, the parser raises `PbfTransactionError` with an `ExceptionGroup` cause containing both the original operation failure and the abort failure. Rollback failure is never silently discarded. Supported source types are exactly: ```text node way relation ``` Node coordinates are stored as deterministic integer E7 values. Ways retain bounded node-reference tuples. Relations retain bounded typed member evidence. The parser validates the file before invoking pyosmium: - regular file only - symbolic links rejected - positive bounded file size - SHA-256 calculated before parsing - device, inode, size, modification time, and change time stable through validation and parsing - valid PBF BlobHeader framing - duplicate BlobHeader type and datasize fields rejected immediately - `OSMHeader` before one or more `OSMData` blocks - only raw or zlib Blob payloads - duplicate Blob payload and `raw_size` fields rejected immediately - bounded Blob count - bounded compressed and per-Blob uncompressed sizes - bounded cumulative uncompressed bytes across the whole file - bounded zlib decompression with actual output required to equal `raw_size` Duplicate payload fields are rejected at the second field; the validator does not retain an attacker-controlled list of duplicate payloads. Default limits are deliberately fixture-scale and every configurable limit has a hard Phase 2A.3 ceiling. A compressed payload is decompressed only within the configured per-Blob and cumulative bounds; dishonest `raw_size` metadata, trailing compressed data, excessive block counts, and aggregate expansion beyond the total limit are rejected before pyosmium runs. ## Test-only download boundary Downloads require only the explicit variables below: ```text OSM_LEAD_SOURCE_PBF_DOWNLOAD_ENV=test OSM_LEAD_SOURCE_TEST_PBF_REGION_ID=fixture/test-region OSM_LEAD_SOURCE_TEST_PBF_URL=http://127.0.0.1:/tiny.osm.pbf OSM_LEAD_SOURCE_TEST_PBF_SHA256=<64 lowercase hex characters> OSM_LEAD_SOURCE_TEST_PBF_SIZE_BYTES= ``` Optional bounded variables: ```text OSM_LEAD_SOURCE_TEST_PBF_MAX_SIZE_BYTES OSM_LEAD_SOURCE_TEST_PBF_TIMEOUT_SECONDS ``` The URL must use plain HTTP to a literal loopback address with an explicit port and a path ending in `.osm.pbf`. Credentials, query parameters, fragments, redirects, proxies, compressed HTTP content encodings, nonapproved content types, missing or invalid `Content-Length`, changed response URLs, mismatched sizes, and mismatched checksums fail closed. Generic variables such as `PBF_URL`, `GEOFABRIK_URL`, `DATABASE_URL`, and `NOCODB_DATABASE_URL` are ignored. Validated bytes are stored as: ```text .osm.pbf .osm.pbf.metadata.json ``` The bounded metadata sidecar records exactly: - schema version - region ID - exact source URL - SHA-256 - byte count - UTC download timestamp - ETag - Last-Modified An existing cache entry is reusable only when both files exist and both pass full stable revalidation. Metadata uses a strict field set, bounded JSON size, bounded header values, exact region/URL/hash/size matching, and a timezone-aware download timestamp. Warm-cache results preserve the original ETag, Last-Modified value, source URL, region, hash, size, and download timestamp. Partial downloads are removed. New PBF and metadata files are published with create-if-absent hard links and rollback on publication failure, so a concurrent target is never overwritten. Incomplete, invalid, or conflicting cache pairs fail closed and are never silently replaced. ## Network policy The test suite remains offline by default. Only tests marked both: ```text integration pbf_network ``` may connect to the exact dynamically created loopback fixture server. Database tests retain their separate exact-endpoint allowance. A test cannot combine database and PBF network access. PBF tests cannot: - resolve or connect to another host or port - create another network listener - use UDP or connectionless sends - contact Geofabrik, NocoDB, GitHub, or any other external service ## Verification Corrected implementation gate before this documentation commit: ```text Head: f4562c59dc2a4e29eccfdfad1b4cd77c90694136 CI run: 29179985294 CI job: validate / 86615774810 Conclusion: success ``` Results: - Ruff lint passed. - Ruff formatting passed; 55 files were already formatted. - Strict mypy passed for 22 source files. - 116 unit tests passed. - 54 sidecar integration tests passed. - 6 PBF loopback integration tests passed. - 1 generic network-policy integration test passed. - 177 total tests passed. - Alembic upgrade, downgrade, and re-upgrade passed. - Package source distribution and wheel build passed. - Disposable PostgreSQL/PostGIS and CI network resources were cleaned up. ## Explicit exclusions Phase 2A.3 does not include: - a production Geofabrik region - production PBF download - production input authorization - unbounded regional ingestion - PostGIS writes - NocoDB reads or writes - profile classification - normalization or deduplication - adoption scanning - reconciliation execution - scheduling, deployment, FastAPI, or MCP - legacy scraper changes The initial production Geofabrik test region remains an unresolved decision for a separately approved phase.