The evaluation is done using multiple stages.

Speaker: Car’s Position + Groundtruth

Create a simple interpretation of what’s happening.

State Machines

At the evaluation’s core are multiple single-purpose state machines that keep track of what’s happening:

  • Progress: Whether the car is at the beginning/middle/end of the road

  • Overtaking: Whether the car correctly overtakes obstacles

Example: OvertakingStateMachine

Graph of OvertakingStateMachine

Graph of OvertakingStateMachine


The output of the state machines is monitored by a referee node that check’s if the

  • state_machines are in valid states -> Referee.DRIVING

  • car reaches the end of the road -> Referee.COMPLETED

  • car makes mistake -> Referee.FAILED

Example: Referee Output

The Complete Picture

digraph EvaluationPipeline {

  node [style=dotted, shape=box]; groundtruth_services; car_state_topic;
  node [style=solid, shape=ellipse]; speaker_node;
  node [shape=box]; speaker_topics; broadcast_topic;
  node [shape=ellipse]; state_machine_node;
  node [shape=box]; state_topics; set_topics;
  node [shape=ellipse]; referee_node;

  groundtruth_services -> speaker_node [style=dotted, dir=both];
  car_state_topic -> speaker_node [style=dotted];

  speaker_node -> speaker_topics;
  speaker_node -> broadcast_topic;

  speaker_topics -> state_machine_node;
  broadcast_topic -> referee_node;

  state_machine_node -> state_topics;
  set_topics -> state_machine_node;

  state_topics -> referee_node;
  referee_node -> set_topics;

  subgraph speaker_topics {
  subgraph referee_topics {

Schema of the Evaluation Pipeline

See simulation_evaluation for more details.