Building Reliable GNSS Telemetry Ingestion Pipelines

Start with an event contract

A valid coordinate is only one part of a telemetry event. A useful ingestion contract also states the authenticated device identity, the protocol message identity, event time, receive time, position quality and the version of the parser that produced the record. These fields answer different questions. Receive time describes the server; event time describes a device observation. Neither timestamp should silently replace the other.

This walkthrough describes an architecture pattern. It does not disclose or claim to demonstrate a private REI Track deployment. The public Rootcastle engineering portfolio and public protocol repositories provide separate entry points for inspecting available work.

Frame before parsing a business record

TCP is a byte stream. One socket read may contain half a frame, several frames or the end of one frame followed by another. Maintain a bounded receive buffer, extract complete protocol frames and validate their length and checksum before asking the domain parser to interpret coordinates. UDP preserves datagram boundaries, but a datagram still needs protocol validation.

Represent incomplete data separately from invalid data. An incomplete frame may become valid when more bytes arrive. An invalid frame needs a bounded resynchronization policy or rejection. A maximum frame length and an idle timeout keep a malformed connection from growing memory indefinitely. Use a synthetic frame corpus to test those paths without sending customer telemetry to a third-party service.

Make duplicate policy a domain decision

Retries and reconnects can repeat messages. A protocol may carry a sequence number, but counters can wrap or reset after a reboot. A deduplication key therefore needs documented scope: device, boot or session identity where available, message type and sequence. If the protocol lacks a stable message identity, describe the limitations of a time-and-payload fingerprint instead of promising exactly-once delivery.

The following toy example illustrates a device-scoped key. Its in-memory set has no durability or expiry and is not a production deduplication store.

seen = set()
def accept(event):
    key = (event['device'], event['session'], event['sequence'])
    if key in seen:
        return False
    seen.add(key)
    return True

a = {'device': 'synthetic-a', 'session': 'boot-1', 'sequence': 7}
b = {'device': 'synthetic-b', 'session': 'boot-1', 'sequence': 7}
assert accept(a)
assert not accept(a)
assert accept(b)

A real service must bound retention, handle concurrent workers and preserve the chosen identity across restarts. An acknowledgement should follow the protocol contract and the intended durability point. Acknowledging before durable acceptance can lose an event; acknowledging too late can provoke unnecessary retries.

Preserve time and quality instead of repairing silently

Keep event and receive timestamps, including an explicit UTC interpretation. Record invalid or absent GNSS fixes as quality states. Do not convert a missing fix into the coordinate zero, zero. Similarly, a late packet should not automatically move a live vehicle marker backward in time. Separate storage of late historical events from the policy that updates the latest-position view.

Validate latitude and longitude ranges, units, hemisphere signs, fix flags and speed semantics against the selected protocol. A parser version belongs with the normalized event so that an interpretation change can be traced later.

Replay is an acceptance test

A useful fixture set includes split frames, concatenated frames, invalid lengths, bad checksums, reconnect duplicates, sequence wrap, device reboot, late data and absent fixes. Save the expected normalized events and rejection reasons. Replay the same corpus through a parser change and compare semantic outputs, not only whether the process stayed alive.

Track bounded queue depth, validation failures, processing latency and duplicate rate without putting raw device identifiers or payloads into general application logs. Detailed evidence belongs in a permissioned, retention-controlled store.

Checklist and public starting points

• Define the event identity and the durability point before implementing acknowledgements.
• Test stream fragmentation independently from domain parsing.
• Keep event time, receive time and GNSS quality distinct.
• Bound buffers, retries, retention and diagnostic payload storage.
• Version fixtures and compare normalized outputs before rollout.

For inspectable protocol work, see the public JT808 parser repository. A repository's current README defines its actual scope; the presence of a parser does not establish a complete commercial fleet platform. For integration requirements, describe your system to Rootcastle.

Explore Rootcastle engineering tools → · Public profiles and publications

Updated: 09.11.2026