The evaluation is done using multiple stages.
Speaker: Car’s Position + Groundtruth¶
Create a simple interpretation of what’s happening.
At the evaluation’s core are multiple single-purpose state machines that keep track of what’s happening:
Progress: Whether the car is at the beginning/middle/end of the road
Overtaking: Whether the car correctly overtakes obstacles
The output of the state machines is monitored by a referee node that check’s if the
state_machines are in valid states -> Referee.DRIVING
car reaches the end of the road -> Referee.COMPLETED
car makes mistake -> Referee.FAILED
Example: Referee Output¶
The Complete Picture¶
See simulation_evaluation for more details.