Evaluation¶
The evaluation is done using multiple stages.
State Machines¶
At the evaluation’s core are multiple single-purpose state machines that keep track of what’s happening:
Progress: Whether the car is at the beginning/middle/end of the road
Overtaking: Whether the car correctly overtakes obstacles
…
Referee¶
The output of the state machines is monitored by a referee node that check’s if the
state_machines are in valid states -> Referee.DRIVING
car reaches the end of the road -> Referee.COMPLETED
car makes mistake -> Referee.FAILED
Example: Referee Output¶
The Complete Picture¶
![digraph EvaluationPipeline {
node [style=dotted, shape=box]; groundtruth_services; car_state_topic;
node [style=solid, shape=ellipse]; speaker_node;
node [shape=box]; speaker_topics; broadcast_topic;
node [shape=ellipse]; state_machine_node;
node [shape=box]; state_topics; set_topics;
node [shape=ellipse]; referee_node;
groundtruth_services -> speaker_node [style=dotted, dir=both];
car_state_topic -> speaker_node [style=dotted];
speaker_node -> speaker_topics;
speaker_node -> broadcast_topic;
speaker_topics -> state_machine_node;
broadcast_topic -> referee_node;
state_machine_node -> state_topics;
set_topics -> state_machine_node;
state_topics -> referee_node;
referee_node -> set_topics;
subgraph speaker_topics {
rank="same"
speaker_topics
broadcast_topic
}
subgraph referee_topics {
rank="same"
state_topics
set_topics
}
}](../../_images/graphviz-542ede879737a8b441049a2b18d47b57a7e682dd.png)
Schema of the Evaluation Pipeline¶
See simulation_evaluation for more details.