The evaluation ROS package provides functionality to evaluate drives automatically.

├── launch
│   ├── test
│   │   ├── drive.test
│   │   ├── sign_evaluation_node.test
│   │   ├── speaker_node.test
│   │   └── state_machine_node.test
│   ├── drive_test.launch
│   ├── evaluation.launch
│   ├── evaluation_test.launch
│   ├── referee_node.launch
│   ├── sign_evaluation_node.launch
│   ├── speaker_node.launch
│   └── state_machine_node.launch
├── msg
│   ├── Broadcast.msg
│   ├── Referee.msg
│   ├── SignEvaluation.msg
│   ├── Speaker.msg
│   ├── State.msg
│   ├── TrafficSign.msg
│   └── TrafficSigns.msg
├── param
│   ├── drive
│   │   ├── paths
│   │   ├── default.yaml
│   │   └── topics.yaml
│   ├── referee
│   │   ├── default.yaml
│   │   └── topics.yaml
│   ├── sign_evaluation
│   │   ├── default.yaml
│   │   └── topics.yaml
│   ├── speaker
│   │   ├── default.yaml
│   │   └── topics.yaml
│   └── state_machine
│       └── topics.yaml
├── scripts
│   ├── drive_test_node
│   ├── evaluation_test_node
│   ├── referee_node
│   ├── sign_evaluation_node
│   ├── speaker_node
│   └── state_machine_node
├── src
│   ├── drive_test
│   │   ├──
│   │   └──
│   ├── evaluation_test
│   │   ├──
│   │   └──
│   ├── referee
│   │   ├──
│   │   ├──
│   │   └──
│   ├── sign_evaluation
│   │   ├──
│   │   ├── label_conversion.yaml
│   │   ├──
│   │   └──
│   ├── speaker
│   │   ├── speakers
│   │   ├──
│   │   └──
│   ├── state_machine
│   │   ├── docs
│   │   ├── state_machines
│   │   ├── states
│   │   ├── test
│   │   ├──
│   │   └──
│   └──
├── test
│   ├── drive_test
│   ├── test_sign_evaluation_node
│   ├── test_speaker_node
│   └── test_state_machine_node
├── CMakeLists.txt
├── package.xml

The simulation can be used to automatically detect errors in the car’s driving behavior. The simulation can even calculate a Carolo Cup score. The goal is to evaluate the car’s behavior in the same way a real referee would do.


Currently, the simulation cannot imitate manual interventions. If the car makes a mistake, the drive is considered a failure.

The evaluation pipeline consists of three main components: The Speaker Node, the State Machine Node, and the Referee Node; there’s more about these nodes in the following.

digraph EvaluationPipeline {

  node [style=dotted, shape=box]; groundtruth_services; car_state_topic;
  node [style=solid, shape=ellipse]; speaker_node;
  node [shape=box]; speaker_topics; broadcast_topic;
  node [shape=ellipse]; state_machine_node;
  node [shape=box]; state_topics; set_topics;
  node [shape=ellipse]; referee_node;

  groundtruth_services -> speaker_node [style=dotted, dir=both];
  car_state_topic -> speaker_node [style=dotted];

  speaker_node -> speaker_topics;
  speaker_node -> broadcast_topic;

  speaker_topics -> state_machine_node;
  broadcast_topic -> referee_node;

  state_machine_node -> state_topics;
  set_topics -> state_machine_node;

  state_topics -> referee_node;
  referee_node -> set_topics;

  subgraph speaker_topics {
  subgraph referee_topics {

Schema of the Evaluation Pipeline


The simulation.src.simulation_evaluation.src.speaker.node subscribes to the car’s state and accesses the groundtruth services. The SpeakerNode calls the groundtruth services for information about the simulated world. E.g. get the position of obstacles or if the car has to stop at an intersection. Additionally, the speaker node receives the car’s frame, i. e. the car’s dimensions, and it’s twist, i.e. linear and angular speed, through the car slate topic. Inside the node, there are multiple simulation.src.simulation_evaluation.src.speaker.speakers which use this information to complete different tasks:

The speakers’ results are used for two purposes.

  • The Zone,-Area, Event,-and Speedspeaker create Speaker messages that the StateMachineNode’s state machines take as input.

  • The BroadcastSpeaker creates a Broadcast message that the RefereeNode uses to calculate a score.

The Speaker message carries a type (and a name for debugging).


The SpeakerNode can be launched with:

roslaunch simulation_evaluation speaker_node.launch


The simulation.src.simulation_evaluation.src.state_machine.node.StateMachineNode contains multiple state machines that are used to automatically track the car’s behavior over time.

The current state of each state machine is published as a State message on a separate topic.

State Machines

There are five state machines which track the state of the drive.


This state machine keeps track of driving on the correct part of the road.

See simulation.src.simulation_evaluation.src.state_machine.state_machines.lane for implementation details.

Graph of LaneStateMachine

Graph of LaneStateMachine


This state machine keeps track of overtaking obstacles.

See simulation.src.simulation_evaluation.src.state_machine.state_machines.overtaking for implementation details.

Graph of OvertakingStateMachine

Graph of OvertakingStateMachine


This state machine keeps track of parking.

See simulation.src.simulation_evaluation.src.state_machine.state_machines.parking for implementation details.

Graph of ParkingStateMachine

Graph of ParkingStateMachine


This state machine keeps track of the car correctly stopping or halting in front of stop or halt lines.

See simulation.src.simulation_evaluation.src.state_machine.state_machines.priority for implementation details.

Graph of PriorityStateMachine

Graph of PriorityStateMachine


This state machines keeps track if the car has started, is driving or has finished the drive.

See simulation.src.simulation_evaluation.src.state_machine.state_machines.progress for implementation details.

Graph of ProgressStateMachine

Graph of ProgressStateMachine


The StateMachineNode can be launched with:

roslaunch simulation_evaluation state_machine_node.launch


This state machine keeps track of how fast the car is going and whether it is keeping to the speed limits.

See simulation.src.simulation_evaluation.src.state_machine.state_machines.speed for implementation details.

Graph of SpeedStateMachine

Graph of SpeedStateMachine


The simulation.src.simulation_evaluation.src.referee.node is used to evaluate the output of the state machines and calculate a score. The referee publishes a Referee message containing a state, a score and more information about the current drive.


The RefereeNode can be launched with:

roslaunch simulation_evaluation referee_node.launch


On start, the simulation.src.simulation_evaluation.src.drive_test.node runs through a series of steps to setup the simulation’s evaluation pipeline and subsequently start KITcar_brain. It then monitors the car’s progress and shuts down once the car has completed or failed the drive based on the referee_node ‘s output.


Start the DriveTestNode and the complete simulation with

roslaunch simulation_evaluation drive_test.launch

However, the drive.test yields a much more interesting usage. It is a ROS test, that can be used to retrieve a binary evaluation if the car broke any rules while driving on a given road in a given mission mode.

ROS Test

Test the car’s behavior using the DriveTestNode with:

rostest simulation_evaluation drive.test


The SpeakerNode’s dependence on groundtruth services and the car state topic means that a complete road (for the groundtruth) and Gazebo (for the car’s state) is necessary to test and debug the nodes in this package.

The simulation.src.simulation_groundtruth.src.groundtruth.test.mock_node can be used to create a groundtruth of simple predefined roads. However, the car state is still missing: The simulation.src.simulation_evaluation.src.evaluation_test.node can be used to create CarState messages as if the car is driving on a predefined path with a predefined speed.


RVIZ window with mocked groundtruth and blue car ‘driving’ on predefined path.

Available roads, paths, and other parameters are described in simulation_groundtruth/param/mock_groundtruth/default.yaml


The EvaluationTestNode can be launched with:

roslaunch simulation_evaluation evaluation_test_node.launch

By default this also launches RVIZ; appending rviz:=false prevents opening RVIZ.

The speakers base their interpretation on the groundtruth, queried from simulation.src.simulation_groundtruth.src.groundtruth topics, and the current position and speed of the car published by the simulation.src.gazebo_simulation.src.car_state.node.