# Architecture & consistency

How the Python pipeline, static assets, and browser scorer fit together.

LookAlike is built as two independent repositories joined by a versioned data contract: `LookAlike-data-pipeline` downloads and validates open sources and publishes immutable releases, while `LookAlike` contains the React application, TypeScript Worker API, and browser scoring engine.

## Overall structure

```
Input data (Census ACS, TIGER, OSM, Overture Maps)         ↓[Python pipeline: 8 stages]         ↓Static web assets (binaries, tiles, JSON)         ↓[React app: loads assets, scores live, re-weights interactively]         ↓Browser: choropleth map + detail panels + rankings
```

## The Python pipeline (8 stages)

Run `LOOKALIKE_RELEASE_ID=<id> python pipeline/run_all.py` from the data-pipeline repository. Each stage caches raw downloads and intermediate Parquet files in that repository. Here is what each produces:

1.  **fetch\_acs** → `acs_cells.parquet`
    
    -   US Census ACS 2023 5-year Summary File (keyless bulk `.dat` download).
    -   83,762 CONUS census tracts × 33 ACS cells (income, education, demographics, housing, employment, commute).
2.  **fetch\_geometry** → `tract_geo.parquet`, `tract_polys.parquet`
    
    -   US Census TIGER/Line 2024 tract shapefiles.
    -   83,777 tract centroids (lat/lon) and simplified polygons (for vector tiles).
    -   Includes land area per tract (for density calculations).
3.  **fetch\_stores** → `stores.parquet`
    
    -   Brand store locations from OpenStreetMap via the Overpass API (`brand:wikidata`, exact name match).
    -   31,702 store points across 11 brands after the 150 m grid dedup, before 1 km US containment.
4.  **fetch\_poi** → used by **build\_features**
    
    -   Overture Maps Places (public GeoParquet on S3, queried with DuckDB, release `2026-06-17.0`).
    -   ~15.4M US places; aggregated to per-km² densities for 5 POI categories: all POI, grocery stores, restaurants, cafés, shopping/apparel.
5.  **build\_features** → `tract_features.parquet`
    
    -   Derive the model feature table: 83,008 tracts × 21 model features (16 ACS-derived plus 5 Overture retail densities) + 3 context fields.
    -   The 16 ACS-derived features: median income, per-capita income, median home value, median rent, poverty rate, bachelor’s degree %, population density, households without a vehicle %, walk/bike/transit commute %, renter housing %, median age, average household size, households with children %, White non-Hispanic %, Hispanic/Latino %, Asian non-Hispanic %.
    -   The 5 Overture retail densities (added here from the POI aggregate): all points of interest, grocery stores, restaurants, cafés, and shopping per km².
    -   Context (shown in UI but not scored): total population, Black non-Hispanic %, unemployment rate.
6.  **build\_scores** → `scores.parquet`, `meta.json`, `candidates/*.json`
    
    -   Build per-brand archetypes (average tract features for each brand’s occupied tracts, one per tract).
    -   Compute similarity percentiles (0–100, within-brand rank) for all 83,008 tracts against each brand.
    -   Extract top-N candidate tracts per brand (full feature detail), written under `data/out` for the offline consistency-test fixture — these are not shipped in the web bundle; the browser rebuilds the ranked list live from the binaries.
    -   Calculate feature-level drivers (“why it matches”), the per-feature variance floor `sigma_eff = sqrt(sigma_brand² + (λ · nat_std)²)`, and coherence (k-fold footprint recovery with county-blocked folds).
    -   Write profiles, scaling metadata, per-brand occupied-tract distance quantiles (`dist_ref`), and the low-ACS-reliability flag.
7.  **export\_release** → `dist/releases/<release-id>/*`
    
    -   Package everything the application needs as one versioned release.
    -   (Detailed below.)
8.  **fetch\_cbsa** → `metros.json`, `cbsa.u16`
    
    -   US Census CBSA (Metropolitan Statistical Area) delineations, 2023.
    -   383 metros in the CONUS. Maps each tract to its CBSA code.
    -   Optional step: if it fails, the app still runs (metro filtering is a nice-to-have).

## Published release assets

The pipeline exports everything needed by the browser under one immutable R2 release prefix, and the hosted application reads it from there. The application repository retains a compatibility copy under `web/public/data/` for self-contained local development and as the consistency-test fixture; see `web/public/data/RELEASE.md` for which release it came from.

### JSON metadata

-   **`meta.json`**: brand profiles (name, id, store count, occupied tracts, archetype, coherence, per-brand occupied-tract distance quantiles `dist_ref`), feature group definitions, national feature min/max/mean.
-   **`manifest.json`**: array counts, dimensions, data types, endianness note (`little`\-endian for native x86/ARM).
-   **`metros.json`**: array of metro objects (CBSA code, name, tract count, bounding box, center latitude/longitude).

### Binary tract data (row-major, little-endian)

-   **`tracts/geoids.json`**: array of 83,008 GEOIDs (row order for all binaries).
-   **`tracts/labels.json`**: human-readable labels (parallel to geoids).
-   **`tracts/features.f32`**: Float32, 83,008 rows × 21 features (16 ACS-derived + 5 Overture retail densities). Raw model-feature values. The file is 83,008 × 21 × 4 = 6,972,672 bytes; a reader that assumes 16 features per row reads garbage after the first row.
-   **`tracts/context.f32`**: Float32, 83,008 rows × 3 context fields (population, Black %, unemployment).
-   **`tracts/centroids.f32`**: Float32, 83,008 rows × 2 (latitude, longitude).
-   **`tracts/sims.u8`**: Uint8, 83,008 rows × 11 brands. Pre-computed similarity 0–100; 255 = not applicable (e.g., brand has no presence).
-   **`tracts/has.u8`**: Uint8, 83,008 rows × 11 brands. Boolean: brand already has a store in this tract.
-   **`tracts/cbsa.u16`**: Uint16, 83,008 rows. CBSA code per tract (0 = not in a metro).
-   **`tracts/rel.u8`**: Uint8, 83,008 rows. Low-ACS-reliability flag (1 = small-population tract, missing income estimate, or a large income margin of error; 0 = normal).

### Stores

-   **`stores/<brand-id>.json`**: array of store points `[lat, lon]` (150 m grid-deduped, US-contained).

The per-brand top-N candidate JSON files are generated under `data/out` for the offline consistency test only; they are **not** shipped in the web bundle. The browser rebuilds the ranked candidate list live from the binaries above.

### Vector tiles

-   **`tracts.pmtiles`**: PMTiles archive (range-served, about 35 MiB in release `2026-07-11-01`). Contains tract polygons and stable GEOID identifiers. The current v2 map colors tracts from browser-computed feature state, so a brand-library rebuild can reuse the same tiles as long as tract geometry and GEOIDs do not change.

## The React/Vite front end

**Stack**: React 19, Vite, MapLibre GL (PMTiles), recharts, TypeScript.

**Key entry points**:

-   `web/src/App.tsx`: root component, state management (selected brand, filters, factor weights, custom model).
-   `web/src/lib/data.ts`: loader for all static assets; provides `TractData` interface.
-   `web/src/lib/scoring.ts`: re-implementation of the Python similarity formula (verified by consistency test).
-   `web/src/components/MapView.tsx`: MapLibre choropleth, feature-state coloring per brand.
-   `web/src/components/DetailPanel.tsx`: click any tract → full explanation (similarity %, radar, drivers, feature table, assessment, robustness, nearby chains).

**Live capabilities**:

-   Similarity re-scoring on every filter change and factor-weight adjustment.
-   Custom store uploads (CSV) → archetype computed in-browser, similarity re-scored.
-   Diverging compare-mode map (two brands, hue = which, brightness = how strongly).
-   Metro filtering and per-metro rankings with screening rollup.

## Python ↔ JavaScript consistency guarantee

The browser scorer must reproduce the Python pipeline’s numbers exactly—or users cannot trust the detail panel’s “why it matches” explanation.

**Verification**: `web/src/lib/consistency.test.ts` reads the real packaged binaries and recomputes similarity for all built-in brands:

-   Tests >50,000 tracts per brand.
-   Checks that 99.9%+ of tracts agree within 1 percentile point (rounding slack).
-   Maximum observed difference: ≤ 2 percentile points.

This test runs as part of `pnpm test` in the web directory. It is the authoritative proof that the map colors and the detail-panel numbers come from the same formula.

## File tree overview

```
LookAlike-data-pipeline/├── pipeline/                    ← source retrieval, features, scoring, packaging├── data/                        ← ignored source cache and intermediate outputs├── dist/releases/<id>/          ← ignored immutable release output└── scripts/                     ← publish and activate releases
LookAlike/├── web/│   ├── src/│   │   ├── App.tsx              ← root, state, orchestration│   │   ├── lib/│   │   │   ├── data.ts          ← asset loader│   │   │   ├── scoring.ts       ← JS formula (mirrors Python)│   │   │   └── consistency.test.ts  ← verification│   │   └── components/│   │       ├── MapView.tsx      ← MapLibre choropleth│   │       ├── DetailPanel.tsx  ← tract detail│   │       ├── CandidateList.tsx│   │       ├── ComparePanel.tsx ← diverging map│   │       └── ... (FactorWeights, UploadModal, ShortlistPanel, etc.)│   ││   └── public/data/             ← compatibility data for local development│       ├── meta.json│       ├── manifest.json│       ├── tracts.pmtiles│       ├── tracts/│       │   ├── geoids.json│       │   ├── labels.json│       │   ├── features.f32│       │   ├── context.f32│       │   ├── centroids.f32│       │   ├── sims.u8│       │   ├── has.u8│       │   ├── cbsa.u16│       │   └── rel.u8│       └── stores/│           ├── trader_joes.json│           └── ...└── docs-site/                   ← product and method documentation
```

(The pipeline’s diagnostic candidate files remain private build outputs and are not an application-repository dependency.)

## How it all fits together

1.  **User opens the app** → browser fetches `meta.json`, `manifest.json`, and the binary files (geoids, features, sims, centroids, cbsa).
2.  **Browser renders a brand** → loads that brand’s `stores/*.json`, computes the ranked candidate list from the binaries, and MapLibre visualizes tracts via `tracts.pmtiles`.
3.  **User clicks a tract** → the detail panel calls the JS scorer on that tract, fetches its feature values from the binaries, and renders drivers, radar, and assessment.
4.  **User adjusts factor weights** → the app re-scores all tracts live (no server call); MapLibre feature-state updates the colors.
5.  **User uploads a custom CSV** → the browser computes a new archetype and re-scores the entire country on the fly.

The Python pipeline runs on its own data-refresh schedule and publishes to R2 independently of application deployment. The Cloudflare Worker serves releases and product APIs; interactive national scoring still happens in the browser.

The scores are similarities—how closely tracts resemble a brand’s existing footprint—and are computed entirely from observable neighborhood features. They do **not** predict revenue, sales, foot traffic, or whether a location will succeed. See the [method](https://docs.lookalikeiq.com/method/how-it-works/) page for why that distinction matters.

Canonical URL: https://docs.lookalikeiq.com/method/architecture/