Skip to content
ATNoGPublic

Repository files navigation

inMotion

WiFi RSSI mobility classification: infer where a passenger is (inside the bus, at the stop) and which transition they are making, from signal-strength sequences captured by a fixed access point.

Part of the inMotion project (Integrated AI Platform for Urban Mobility and Transportation), a copromotion between Wavecom and Instituto de Telecomunicações.

Domain

Route labels — each row is one 10-second route, 10 RSSI readings at 1 Hz:

label meaning
AA stayed inside the bus
BB stayed at the stop
AB alighted (inside → outside)
BA boarded (outside → inside)

Each row also carries mac (device identifier) and noise (whether other devices were transmitting concurrently). See docs/eucnc/project_description.md for the collection protocol.

Schema

Bus networks and OD matrices

inmotion bus scrapes the two operators that cover the Aveiro region — BusWay (CIRA) and AveiroBus — and derives origin-destination matrices from their timetables. Neither publishes GTFS; BusWay renders every timetable as an HTML table of stops against trips, and AveiroBus embeds each timetable in its PDF as a single bitmap.

uv run inmotion bus fetch     # data/bus/raw + the checksum manifest
uv run inmotion bus build     # data/bus/{interim,processed} + build.json
uv run inmotion bus timeline  # re-slice the OD matrices by any window

bus build writes the OD matrix twice: aggregated over departure bands, and sliced into hourly intervals (od_timeline.csv, the dynamic view). bus timeline re-cuts the slices from the processed table, so a 30-minute window costs one command and no scrape.

bus build reads the AveiroBus timetables out of bitmaps, so it needs the tesseract binary with the Portuguese language pack (tesseract-ocr-por, or por.traineddata in the tessdata directory). Without it the build refuses and says so, rather than writing BusWay-only tables.

The network's coordinates come from OpenStreetMap, and the passenger flows are generated over the timetable:

uv run inmotion bus geocode   # stop_coords.csv, clusters, unresolved list
uv run inmotion bus demand    # data/bus/scenarios/<scenario>/ + index.csv
uv run inmotion bus demand -n 8 --seed 42                  # a batch to compare
uv run inmotion bus demand --sweep sweep.json              # an explicit grid

The page that comes out opens straight from disk. What it draws underneath the flows is chosen by --basemap, and the default auto picks the best thing the machine can actually render:

--basemap under the flows needs
auto (default) carto-dark when CARTO_API_KEY is set, otherwise network —
carto-dark, carto-light CARTO's basemaps CARTO_API_KEY in .env (gitignored)
network the timetable's own leg geometry nothing: no key, no network
osm OpenStreetMap's tiles the page served over HTTP
none nothing, the flows on the page's own surface nothing

Two tile servers were tried and rejected before this shape settled, and the reason is worth knowing because both failures look like something else. OpenStreetMap's volunteer tiles require an HTTP Referer, which a file:// page never sends, so they answer 403 with their "Access blocked" page. CARTO's CDN reads the key from a query parameter named key — with api_key it returns a perfectly ordinary tile that says "API KEY REQUIRED", the same tile a wrong key gives, so a wrong parameter looks like a broken key. With the key it returns a real basemap, and that is what auto uses when one is configured.

A CARTO key is a secret: it belongs in .env and is read from CARTO_API_KEY, never written into scenario.json. It is inside the generated page — that is how a client-side basemap key works — so a scenario built with one should not be committed; the credentials guard in tests/test_no_secrets.py knows the shape of a CARTO key for the files it scans.

bus geocode makes the one network call the generator needs — one Overpass request for every bus stop of the region, cached under data/bus/raw/osm — and then matches names locally. A stop that cannot be placed is left blank and listed in stops_unresolved.csv rather than guessed at.

Every scenario writes three pages, all self-contained and all openable by double-clicking: flows_map.html, the day on the ground, and flows_cube.html, the same day as a space-time cube — origin across, destination back, hours up, with entries and exits as the same rides read by the clock they stepped on and the clock they stepped off — and flows_replay.html, one bus route's day played back: buses move along the line on the timetable, riders waiting at a stop are a dot each, and a bus that calls shows who boarded, who got off and who was left standing there. The map filters by service day, operator, bus line, minimum riders, a stop search and an hour window, and plays the day through the hourly slider (or the sparkline) while the OD lines thin and thicken with the traffic. The cube filters by line and writes the same rides twice, so the two views answer different questions about one dataset.

A replay is only as busy as the day it was generated from: at the default dial a single route carries a few hundred riders over the day, so for a demonstration worth watching generate a denser scenario —

uv run inmotion bus demand --seed 42 --boardings-per-departure 40 --out data/bus/scenarios/demo-dense

bus demand turns supply into demand: a gravity model over the stop pairs the timetable actually serves, sampled into whole riders, spread over the departures that serve each pair, and then run through a queue at every stop, where a bus takes as many riders as it has free seats and the rest stay at the stop. What comes out is flows.csv (per trip and stop: arrivals, boarded, alighted, the load before and after, and who was left behind), stop_flow.csv (the queue per stop and hour), instants.csv (the same day in the AA/BB/BA/AB vocabulary of the WiFi captures, on the 10-second grid), and the passenger OD matrices — od_demand.csv before any bus could refuse a rider, and od_passengers.csv after. Every scenario carries the scenario.json that reproduces it and a report.json holding the counts and every invariant that was checked.

See AI_RESULT.md for what came out, how it was verified against the live sites, and what is known to be wrong. For the passenger-flow generator — every command and flag, the recipes, and how to share scenarios in a group — see docs/guides/passenger-flows.md.

Setup

uv sync --all-packages      # root project plus the demo workspace member
cp env.example .env         # credentials (never committed)
set -a && . ./.env && set +a

# optional: replace the portable wheel with the accelerator build
scripts/setup/gpu-rocm.sh   # AMD ROCm, gfx1151
scripts/setup/gpu-cuda.sh   # NVIDIA CUDA 12.x
export UV_NO_SYNC=1         # keep the accelerator torch; uv run would restore the lock wheel
UV_NO_SYNC=1 uv run --no-sync python scripts/gpu_check.py  # CPU vs GPU smoke test

Check the environment and the artifact store:

uv run inmotion doctor

Run a quantized artifact from Python (ONNX Runtime, CPU or ROCm):

uv run inmotion quantize run \
  --artifact results/edge/artifacts/lejepa-in-house-q4 \
  --compute q4 --device cpu \
  --input results/edge/windows/lejepa/windows.txt --json

The Rust engine exposes the same idea as flags:

rust/target/release/inmotion-infer bench \
  --artifact results/edge/artifacts/lejepa-in-house-q4 \
  --engine quantized --compute q8 --json

For a live MAC/RSSI stream, use the monitor (file or stdin; optional MQTT):

inmotion-infer monitor --artifact results/edge/artifacts/lejepa-in-house-q4 \
  --input stream.txt --engine quantized --compute q8 --json

For a new architecture, export ONNX and use the model-agnostic tract runner instead of adding a Rust graph:

uv run inmotion quantize export lejepa --out /tmp/art --format q4 \
  --onnx /tmp/lejepa.onnx
rust/target/release/inmotion-onnx bench --model /tmp/lejepa.onnx \
  --shape 1,10,4 --json

Running a model you trained

This is the part that used to require reading the training scripts.

uv run inmotion model list                 # every known model, and whether it is on disk
uv run inmotion model info ts_jepa         # architecture, parameter count, provenance
uv run inmotion model verify               # strict-load all of them
uv run inmotion model predict ts_jepa --input new_routes.csv

model predict needs to know how the inputs were scaled, because the original training code never saved its StandardScaler. Pass the CSV the model was trained on as --reference, or rely on the default for its feature set. Predictions built on reconstructed statistics say so.

A checkpoint that predates the sidecar convention can be adopted:

uv run inmotion model adopt path/to/checkpoint.pt --name my_model
uv run inmotion model inventory 20-aug        # scan a whole directory

Adoption rebuilds the architecture from the weights and refuses to guess when several architectures fit (TS-JEPA and CF-JEPA at equal width are genuinely indistinguishable). It reports the ambiguity instead.

Note: *_pretrain_best.pt files are pre-training bundles, not classifiers. See docs/reports/ARTIFACTS.md.

Datasets

Datasets are committed to git, so their history is the version history. data/DATASETS.json adds the parts git cannot express: checksums, row and class counts, lineage, and which versions are safe to train on.

uv run inmotion datasets list
uv run inmotion datasets show ds-augmented-v3
uv run inmotion datasets verify            # detect a file changed under git's feet

Current versions, newest lineage last:

id file rows notes
ds-pure dataset_only_pure.csv 160 early collection — labels reported incorrect
ds-noise dataset_only_noise.csv 1,196 early collection — labels reported incorrect
ds-augmented-v1 dataset_augmented.csv 10,113 augmentation config 1 — labels reported incorrect
ds-icaisf dataset.icaisf.csv 2,251 the ISAC paper's variant; labels unverified
ds-base-v2 dataset.csv 3,511 current base, 13 devices
ds-augmented-v2 dataset_augmented2.csv 20,275 augmentation config 2 (superseded)
ds-augmented-v3 dataset_augmented3.csv 35,839 canonical training set

ds-augmented-v3 is what the JEPA and SIGReg checkpoints were trained on, so it is the reference for reproducing their preprocessing. Full detail, including lineage and checksums, is in docs/reference/datasets.md.

Building datasets from raw captures:

uv run inmotion data export --input wavecom_files/routeAtoB.txt --output data/AB.csv \
  --group G1:<mac1>,<mac2> --group-label G1:AB
uv run inmotion data merge --input data --output dataset.csv
uv run inmotion data clean  -- --data dataset.csv
uv run inmotion data augment -- ...

Training and evaluation

Pipelines are invoked through one entry point; arguments pass through unchanged.

uv run inmotion run list
uv run inmotion train exotic --model ts_jepa --seed 42
uv run inmotion train dl     -- --seed 42
uv run inmotion evaluate mega-ensemble --data dataset_augmented3.csv

Cluster submission scripts are in scripts/slurm/, single-machine runs in scripts/local/; see scripts/README.md.

Tests

uv run pytest tests/                          # root suite
cd demo && uv run --project . pytest          # demo is a separate workspace member
uv run ruff check src dl ml_classification tests
uv run ruff format --check src dl ml_classification tests
uv run python scripts/docs/generate_reference.py --check

scripts/check.sh runs all of them at once:

./scripts/check.sh          # lint, format, docs, tests, rust checks
./scripts/check.sh --full   # also the demo suite and the Rust parity test

There is deliberately no CI: the checks run locally, on demand, so they cost nothing.

Model tests verify that every documented checkpoint loads with strict=True and matches its recorded parameter count, and that each model's output still matches a frozen fingerprint. They skip with a reason when the (gitignored) checkpoint store is absent.

Layout

src/inmotion/          the package
  cli.py               the `inmotion` entry point
  bus/                 Aveiro bus networks: scrape, parse into trips, build OD matrices
  models/              registry, factories, sidecars, adoption
  data/                tier paths, feature pipeline, export/merge/clean/augment
  pipelines/           training and evaluation pipelines + their registry
  evaluation/          checkpoint evaluation, SHAP, feature importance
  analysis/  viz/      multi-seed analysis, figure regeneration
data/
  raw/ interim/ processed/   the three data tiers
  bus/                 the bus-network scrape and its derived OD matrices
  DATASETS.json        checksums, lineage, label provenance
docs/                  papers (frozen), reference (generated), guides, audit
scripts/               slurm/ local/ archive/ oneoff/ docs/
tests/                 contract, adoption, CLI, docs, secrets guard
demo/                  FastAPI demo (separate workspace member, own image)
backup/  20-aug/       checkpoint stores (not tracked)
plots/  results/       generated output (not tracked)

Auditability

For a review, these are the entry points: docs/audit/AUDIT_CHECKLIST.md lists every claim and the command that verifies it, and docs/audit/MODEL_CARDS.md states each model's purpose and limitations. Known discrepancies — including three checkpoints whose parameter counts do not match the published table — are documented in docs/reproducibility.md rather than omitted.

Documentation

docs/README.md is the documentation index. Start there.

document contents
docs/architecture.md how data flows, the model families, the registry, sidecars
docs/reproducibility.md environment, seeds, split, and what is not reproducible
docs/guides/running-models.md list, inspect, verify, predict, adopt
docs/guides/training.md training each family, on a workstation or a cluster
docs/reference/ generated: CLI, models, modules, datasets
docs/audit/AUDIT_CHECKLIST.md what a reviewer can verify independently
docs/audit/MODEL_CARDS.md per-family model cards and limitations
docs/reports/ artifact rules, migration notes, refactor history

Citing

Dataset DOI: 10.21227/55nm-0r91

Releases

Packages

Contributors

Languages