Evaluation¶

The evaluation is done using multiple stages.

Speaker: Car’s Position + Groundtruth¶

Create a simple interpretation of what’s happening.

State Machines¶

At the evaluation’s core are multiple single-purpose state machines that keep track of what’s happening:

Progress: Whether the car is at the beginning/middle/end of the road

Overtaking: Whether the car correctly overtakes obstacles

…

Example: OvertakingStateMachine¶

Graph of OvertakingStateMachine¶

Referee¶

The output of the state machines is monitored by a referee node that check’s if the

state_machines are in valid states -> Referee.DRIVING
car reaches the end of the road -> Referee.COMPLETED
car makes mistake -> Referee.FAILED

Example: Referee Output¶

The Complete Picture¶

$digraph EvaluationPipeline { node [style=dotted, shape=box]; groundtruth_services; car_state_topic; node [style=solid, shape=ellipse]; speaker_node; node [shape=box]; speaker_topics; broadcast_topic; node [shape=ellipse]; state_machine_node; node [shape=box]; state_topics; set_topics; node [shape=ellipse]; referee_node; groundtruth_services -> speaker_node [style=dotted, dir=both]; car_state_topic -> speaker_node [style=dotted]; speaker_node -> speaker_topics; speaker_node -> broadcast_topic; speaker_topics -> state_machine_node; broadcast_topic -> referee_node; state_machine_node -> state_topics; set_topics -> state_machine_node; state_topics -> referee_node; referee_node -> set_topics; subgraph speaker_topics { rank="same" speaker_topics broadcast_topic } subgraph referee_topics { rank="same" state_topics set_topics } }$

Schema of the Evaluation Pipeline¶

See simulation_evaluation for more details.