Sensor fusion and autonomy levels

Four sensors produce four partial, differently timed, partly contradictory accounts of the same road. The vehicle has to act on one. Everything in this topic is about how that reconciliation is done, and what it lets the vehicle be trusted to do.

The problem

Each sensor:

• sees part of the scene,
• reports at its own rate, so no two observations are quite simultaneous,
• has its own error characteristics, systematic as well as random,
• and will sometimes be wrong in ways it cannot report.

A controller cannot act on four disagreeing accounts. Fusion produces a single estimate — with an honest statement of how confident that estimate is, which turns out to matter as much as the estimate.

Early and late fusion

Late fusion. Each sensor produces its own object list — the radar says “something at 40 m closing at 8 m/s”, the camera says “a car, slightly left”. Those lists are then reconciled.

Simple, modular, and it degrades gracefully: lose a sensor and the rest still work. But every sensor has already discarded information to produce its list. A faint radar return and a weak camera detection might each fall below their own thresholds while together being conclusive — and late fusion has already thrown both away.

Early fusion. Raw measurements are combined before interpretation. Nothing is discarded, so exactly those weak-but-corroborating cases can be caught.

The price is heavy computation and tight synchronisation. At 100 km/h a vehicle moves nearly three metres in a tenth of a second, so combining raw data means knowing, to a millisecond, when each observation was taken.

Most production systems sit somewhere between the two.

Tracking, and the Kalman filter

Detecting something in one frame is not enough. The vehicle needs to know that this object is the same one it saw a moment ago, and where it is going.

The loop is: predict, associate, update, manage.

A Kalman filter carries two things — an estimate, and how uncertain that estimate is.

1. Predict. Where should this object be now, given where it was and how it was moving? The uncertainty grows, because prediction is guesswork.
2. Update. A measurement arrives. Combine it with the prediction in proportion to which is currently trusted more. A precise measurement pulls the estimate hard; a noisy one barely moves it. The uncertainty shrinks, because we now know more.

That weighting is the entire mechanism. It is not averaging — it is averaging with the weights set by relative confidence, recomputed at every step.

The extended Kalman filter exists because the plain one assumes linearity. A radar measures range and bearing while the track is held in x and y, and that conversion is trigonometric. The extended version linearises locally around the current estimate. It works well when the estimate is close and degrades when it is not.

Learned perception

Nobody can write down rules that reliably separate a pedestrian from a lamp post across every pose, every lighting condition and every partial occlusion. Hand-written rules are brittle in exactly the situations that matter.

A learned model derives those rules from labelled examples instead, and does the job far better.

Two consequences follow, and both are uncomfortable:

• Its competence is bounded by its training data, in ways not visible from outside. A model that has never seen a horse-drawn cart has no way to tell you that.
• Failures cannot be traced to a faulty line. There is no line. Diagnosis becomes a question about data and distributions rather than about code.

The SAE levels

Level	Who is driving	What it means
0	The driver	Warnings only, no sustained control
1	The driver	Steering or speed assisted, not both
2	The driver	Both assisted together — but the driver monitors continuously and is responsible
3	The system, when engaged	The system monitors and is responsible within its domain; the driver may disengage but must resume when asked
4	The system, when engaged	No expectation of a driver resuming — the system reaches a safe state itself
5	The system	Anywhere a competent human could drive. Does not exist

The step from 2 to 3 is the one that matters, and it is not about capability. A Level 2 system may steer and brake beautifully while the driver remains liable throughout. A Level 3 system accepts that responsibility. That is a legal and insurance threshold, which is why it is approached so cautiously by manufacturers whose engineering is already capable of more.

Operational design domain

Nothing is simply “Level 3”. It is Level 3 within a stated domain — perhaps divided motorways, below 60 km/h, daylight, clear weather, with lane markings present.

The domain is as much part of the specification as the level. And recognising that its edge is approaching — fog thickening, the motorway ending, markings disappearing under roadworks — is a substantial part of what the system must do.

Handover

A driver who has been legitimately permitted to disengage cannot resume instantly. Rebuilding awareness of surrounding traffic takes seconds, not moments.

So the system must:

• recognise the approaching limit early enough to give several seconds of warning,
• escalate that warning if there is no response,
• and, failing all of it, perform a minimal risk manoeuvre by itself — decelerating in lane or pulling onto the hard shoulder.

That last requirement is what makes Level 3 genuinely hard. The system must be able to finish the job alone, which means it must be nearly good enough for Level 4 in order to be honest about being Level 3.

What to take away

• Fusion turns several partial, differently timed, disagreeing accounts into one estimate with a stated confidence.
• Late fusion is modular and robust but discards information at each sensor; early fusion keeps it at the cost of computation and synchronisation.
• A Kalman filter weights prediction against measurement by relative confidence. That is all it is.
• Association — which measurement belongs to which track — is where tracking usually fails.
• Learned perception handles messy reality far better and is bounded by its training data in ways you cannot see.
• ISO 21448 covers the case where nothing is broken and the design was simply inadequate. That is the dominant risk.
• Levels 2 and 3 differ on responsibility, not capability, and every level is only meaningful alongside its operational design domain.

Next, what these systems actually do with the picture — active safety and driver assistance.