Models and Sensors

Gazebo model and sensor plugins make for a great ROS integration. Model plugins enable interacting with models inside the simulated scene and sensor plugins can publish the output of sensors as ROS messages.

Dr. Drift

Dr. Drift’s model definition is a good place to understand how these two elements are utilized to make the simulation work. The complete definition of Dr. Drift in simulation/src/gazebo_simulation/param/car_specs/dr_drift/model.urdf is quite long and contains multiple cameras. However, only a view lines of the definition are sufficient to grasp what’s going on.

Warning

The following code snippets are not included from the actual model.urdf and details might differ. Nevertheless, the principle ideas stay the same.

The line

<plugin filename="libmodel_plugin_link.so" name="model_plugin_link"/>

includes the model plugin model_plugin_link. With it, the car’s pose and twist can be controlled through ROS topics. (See gazebo_simulation for more.)

The lines

<sensor name="front_camera" type="camera">
  <plugin name="camera_plugin" filename="libgazebo_ros_camera.so">
    <alwaysOn>1</alwaysOn>
    <updateRate>0</updateRate>
    <frameName>front_camera</frameName>
    <cameraName>front_camera</cameraName>
    <imageTopicName>/simulation/sensors/raw/camera</imageTopicName>
    <cameraInfoTopicName>/simulation/sensors/camera/info</cameraInfoTopicName>
  </plugin>
  <update_rate>60</update_rate>
  <camera>
    <horizontal_fov>2.064911321881509</horizontal_fov>
    <image>
      <width>1280</width>
      <height>1024</height>
      <format>L8</format>
    </image>
    <clip>
      <near>0.1</near>
      <far>4</far>
    </clip>
  </camera>
</sensor>

define the car’s front camera. Besides the actual sensor properties, a sensor plugin called “camera_plugin” is added. The sensor plugin publishes the camera image on the ROS topic "/simulation/sensors/raw/camera" on which it is then available to other nodes.

Note

The model.urdf is automatically generated using:

rosrun gazebo_simulation generate_dr_drift

Camera Image Augmentation

Additionally to Gazebo’s rendering engine there’s is a generative neural network, trained to translate the simulated camera into a real-looking image. The code for the network and surrounding scripts are located in simulation.utils.machine_learning.cycle_gan. Because training the network(s) is computationally heavy, pretrained weights of the network are stored in DVC and can be downloaded with

dvc pull simulation/utils/machine_learning/cycle_gan/checkpoints/dr_drift/latest_net_g_b.pth

. See Installation for instructions to set up DVC and make sure that the machine learning pip3 packages have been installed by selecting to do so when running the init/init.sh script. If everything is set up correctly, using the generative model is as easy as launching with apply_gan:=true:

Note

The camera image can be augmented using the cycle gan’s generative model by running:

roslaunch gazebo_simulation master.launch apply_gan:=true (control_sim_rate:=true evaluate:=true)

(The parameters control_sim_rate and evaluate are not necessary but ensure the camera image gets processed with 60 Hz.)