The Ingestion Pipeline

What this chapter covers

This chapter follows a tracker message from the moment Flespi delivers it to the moment it becomes a durable positions row. Along the way it authenticates, gets validated field by field, is batched into the queue, and is processed by the engine — with three separate archives catching the traffic that cannot follow the happy path. The design goal is that no real telemetry is ever silently lost: a bad message is rejected loudly or set aside for investigation, never dropped on the floor.

The picture

Authority boundary

The queue carries normalized telemetry, not tenant authority. The engine resolves the registered device, vehicle, and tenant from current records before any tenant-scoped write.

The split is deliberate. The ingress side authenticates, validates, and normalizes transport data. It never decides which tenant owns a message. The engine side resolves the registered device, its vehicle, and its tenant from current database state before it writes anything, so a stale or forged queue payload can never attribute a truck's position to the wrong fleet.

The behavior

Authentication

Flespi posts each telemetry batch to Korido's ingest endpoint. Two checks gate it: the caller's IP must be on the allowlist, and the request must carry the shared secret token. The client IP is read through the same cross-runtime helper the whole platform uses, so the check behaves identically in local development and in production.

Validation, field by field

Every item in the batch is validated on its own against the telemetry schema. Two failure modes are handled very differently:

• A malformed request body — JSON that will not even parse — is rejected with a 400. This tells Flespi the delivery failed, so Flespi retries it. Whole-body rejection is reserved for transport corruption only.
• A single invalid item inside an otherwise-good batch is isolated: it is written to the invalid archive and the batch proceeds. A schema-poison item is a per-item concern, never a reason to reject the whole delivery.

Out-of-range values are nulled, never dropped

A handful of telemetry columns are narrower than any plausible wire value — GPS dilution figures carry sentinel values like 99.99 that mean "no usable reading," and a voltage glitch can briefly exceed a column's range. A value past a column's capacity would fail the database write and roll back the entire batch into a retry loop. Korido prevents that at normalization: an out-of-range field drops to null and the row — and every other row in the batch — survives.

Signal	Accepted range
External power voltage	0 … 999.9 V
GPS dilution (HDOP / PDOP)	0 … 99.9 (sentinels mean "no reading" → null)
Sensor battery current	−99.999 … 99.999 (signed: charge vs discharge)
CAN fuel rate	0 … 999.99
CAN fuel consumed	−99999.99 … 99999.99
Altitude, CAN speed / RPM / coolant, event codes	whole-number sensor range

Nulling one field is the correct outcome: it means "this one reading was unusable," and it costs the batch nothing.

Batching into the queue

Valid, normalized rows are grouped into queue messages under a strict byte budget, because the queue has a hard 128 KB per-message ceiling. The grouping serializes each row once and accounts for the JSON scaffolding, so every group provably fits. A single row that is itself larger than the budget — a bloated config dump, say — is flagged and sent alone in its own message, so it survives; the queue's own hard limit is the final gate.

If a group fails to send after earlier groups already went out, the response reports the true per-group count that made it through. Flespi replays the whole batch, and the duplicate protection below absorbs the resend cleanly.

Engine processing into positions

The engine receives queued batches, resolves every unique device against the registered-device records, groups the rows by device, sorts each device's stream by capture time, and runs them through the fleet state machine. The end product is positions rows — the durable, append-only record of everything a truck's tracker reported.

Fixes and status frames share one path

A status frame shares the exact same table and pipeline as a fix: it rides the same route and lands as a positions row with an empty position. It carries its full telemetry — ignition, battery, GSM, voltage, alarm, fuel — and it advances the device's heartbeat, feeds fuel detection, and drives ignition memory. It simply skips the geometry-dependent steps, because it has no geometry. A no-fix row keeps every signal despite being positionless.

SIM identity and configuration snapshots

On rare configuration frames a tracker includes a full settings dump — its alarm slots, geofences, defense mode, and the SIM's identity numbers. Korido parses that dump once at the door into structured fields, safely bounded so a corrupt or hostile dump degrades to a clipped snapshot rather than blowing the queue budget. The engine then reconciles the SIM identity against the device's known SIM: the first identity observation is recorded, and a later mismatch is a SIM-swap warning — a tamper signal. The full settings snapshot is stored against the device, advancing only when a newer dump arrives.

Observability

The pipeline is instrumented so "are my heartbeats being processed?" is answerable from a dashboard rather than a guess. Counters track status-frame volume per device family, oversized single items, messages from a registered device that has no vehicle link, and SIM-swap mismatches. Batch traces link the queue consumer back to the ingress that produced the message.

Edge cases

• Unparseable request body. Returns 400; Flespi retries the whole delivery. This is the only case where the entire body is rejected — it means the transport itself was corrupt.
• One poison item in a good batch. Archived to the invalid archive; the rest of the batch is normalized and queued. Archive failure is logged and counted but never blocks valid telemetry.
• A numeric that would overflow its column. The single field is nulled at normalization; the row and its batch survive.
• The same device reports the same capture instant twice. The row is stored once. The capture timestamp is kept at microsecond precision, so two genuinely distinct readings inside the same millisecond both survive; a true duplicate is absorbed. On an exact-instant tie, a fix-bearing message wins over a coordinate-less one.
• A single frame larger than the queue budget. It is sent alone in its own message; the 128 KB queue limit is the last line.
• Partial queue-send failure. The true per-group queued count is reported, Flespi replays the batch, and dedup absorbs the rows that already landed.
• An unknown device (unregistered IMEI). The message is written to the unregistered archive for investigation and acknowledged, which stops the retry loop. No position is written, because there is no vehicle or tenant to attach it to.
• A registered device with no vehicle link. Its heartbeat still advances — the device is alive and the connectivity view sees it — and the traffic is counted, but no position is written until it is linked to a vehicle.
• A message that keeps failing engine processing. After bounded retries it parks in the dead-letter archive, where it can be replayed once the underlying cause is fixed. Status frames replay through this path just like fixes.

Known limitations

The pipeline's guarantee is that no real telemetry is silently lost — not that every message arrives exactly once, and not that Korido sees traffic no device sent to Flespi.

• Delivery is at-least-once, so duplicates are structural. Flespi retries a failed delivery and a partial queue-send is replayed whole, which means the same reading can reach Korido more than once. Duplicate suppression on the capture timestamp is therefore a permanent, load-bearing part of the design, not an occasional safety net.
• Flespi is the single connectivity layer. Every tracker reaches Korido through Flespi's webhook. The platform reasons only about what Flespi delivers; a device that cannot reach Flespi is a silence Korido detects through liveness, not a message it can recover.
• Archived traffic waits for a human. An unparseable body, a poison item, an unregistered device, and a message that exhausts its retries are each preserved in an archive — but clearing them back into the record is a deliberate, investigated replay, not an automatic recovery.

How it connects

• Before: Trackers and what they report — the messages entering the pipeline.
• Next: Position quality — how the engine grades each fix it stores.
• Then: Liveness — how the stored rows become live vehicle state.