Put it into the CI!

We need

1. A Docker image that contains the simulation,

   • is easily maintainable and

   • accessible from the CI!

2. A new CI job in the kitcar-ros pipeline that runs the tests,

   • is integrable without many changes,

   • uses the simulation Docker image, and

3. good test cases

   • with reproducible results

   • that are clear and precise!

Keep the Image updated

# This is done everytime the master branch is updated
docker-images:
  stage: deploy
  image: docker:20.10.5
  needs: ["build-code"]
  rules:
  - if: '$CI_COMMIT_REF_NAME == "master"'
  - if: '$CI_PIPELINE_SOURCE == "merge_request_event"'
    when: never
  - when: manual
    allow_failure: true
  tags:
  - docker
  - server
  services:
  - docker:20.10.5-dind
  variables:
    # Ensure that lfs files are available in resulting images!
    DOCKER_HOST: tcp://docker:2375
    DOCKER_TLS_CERTDIR: ''
  before_script:
  - export IMAGE_URL=$CI_REGISTRY/kitcar/kitcar-gazebo-simulation
  - |
    if [ "$CI_COMMIT_REF_NAME" == "master" ]; then
      export CI_IMAGE_TAG=latest
      export CI_IMAGE_TAG_CML=cml
      export CI_IMAGE_TAG_ROS=latest
    else
      export CI_IMAGE_TAG=branch_${CI_COMMIT_REF_NAME}
      export CI_IMAGE_TAG_CML=branch_${CI_COMMIT_REF_NAME}_cml
      export CI_IMAGE_TAG_ROS=branch_${CI_COMMIT_REF_NAME}
    fi
  - apk add --no-cache docker-compose git
  - docker --version
  - docker info
  - docker login -u $CI_REGISTRY_USER -p $CI_REGISTRY_PASSWORD $CI_REGISTRY
  script:
    # Clone kitcar-rosbag
  - git clone https://gitlab-ci-token:${CI_JOB_TOKEN}@git.kitcar-team.de/kitcar/kitcar-rosbag.git

  - cd docker
    # Build Docker images
  - docker-compose build
    # Push docker images
  - docker-compose push

dvc-repro-images:
  image: git.kitcar-team.de:4567/kitcar/kitcar-gazebo-simulation/ci
  needs: ["build-code"]
  rules:
  - if: '$CI_COMMIT_REF_NAME == "master"'
  - if: '$CI_PIPELINE_SOURCE == "merge_request_event"'
    when: never
  - when: manual
  allow_failure: true
  stage: deploy
  parallel:
    matrix:
    - DATA_FOLDER: [labeled_images, simulated_images]
  tags:
  - workstation
  script:
  - ./gitlab-ci/setup_cml_bot_git.sh

  # Run ssh-agent (inside the build environment)
  - eval $(ssh-agent -s)
  # Add the SSH key stored in CML_BOT_SSH_PRIVATE_KEY variable to the agent store
  - ssh-add <(echo "$CML_BOT_SSH_PRIVATE_KEY")

  - export NEW_BRANCH_NAME=ci/${CI_COMMIT_BRANCH}_${CI_JOB_ID}
  - echo ${NEW_BRANCH_NAME}

  - mv /home/kitcar/kitcar-rosbag $KITCAR_REPO_PATH
  - ls $KITCAR_REPO_PATH
  - source $KITCAR_REPO_PATH/kitcar-rosbag/devel/setup.bash --extend
    # Actual script!
  - dvc pull simulation/utils/machine_learning/cycle_gan/checkpoints/dr_drift/latest_net_g_b_to_a.pth
  - cd data/${DATA_FOLDER}
  - dvc pull dvc.yaml -j 8 || true
  - dvc repro -s
  - |
    # Check if lock file has changed
    if [ $(./../../gitlab-ci/has_file_changed.sh dvc.lock) -eq 1 ]; then
      git add dvc.lock

      # Push to DVC Remote
      dvc push -j 8

      # Commit and Push to GitLab
      COMMIT_MSG="data: Reproduce ${DATA_FOLDER}"
      ./../../gitlab-ci/commit_and_push_to_branch.sh "${COMMIT_MSG}" "${NEW_BRANCH_NAME}"

      # Create Merge Request
      ENDPOINT_URL="${CI_API_V4_URL}/projects/${CI_PROJECT_ID}/merge_requests"
      BODY="{ \
        \"id\": ${CI_PROJECT_ID}, \
        \"source_branch\": \"${NEW_BRANCH_NAME}\", \
        \"target_branch\": \"${CI_COMMIT_BRANCH}\",\
        \"remove_source_branch\": true, \
        \"title\": \"${COMMIT_MSG}\", \
        \"labels\": \"topic::data\", \
        \"assignee_id\": 600 \
      }";
      curl \
          --request POST "${ENDPOINT_URL}" \
          --header "PRIVATE-TOKEN: ${AUTO_MERGE_REQUEST}" \
          --header "Content-Type: application/json" \
          --data "${BODY}";
    fi
  artifacts:
    paths:
    - data/labeled_images/dvc.lock
    - data/simulated_images/dvc.lock

Make it Accessible

Docker Images stored in the Gitlab Docker Registry

Easy Integration

Kitcar-ros Pipeline

simulation:
  stage: test
  image: git.kitcar-team.de:4567/kitcar/kitcar-gazebo-simulation/kitcar_ros_ci:focal
  variables:
    KITCAR_REPO_PATH: /builds/kitcar
    CAR_NAME: dr_drift
    DISPLAY: ":1.0"
  before_script:
    - ...
  script:
    - rostest simulation_evaluation drive.test road:=ci_roads/curves mission_mode:=1
    - rostest simulation_evaluation drive.test road:=ci_roads/obstacles mission_mode:=2
    - rostest simulation_evaluation drive.test road:=ci_roads/intersection mission_mode:=2
    - export INTERSECTION_TURN=1  # Left turn
    - rostest simulation_evaluation drive.test road:=ci_roads/intersection mission_mode:=2
    - export INTERSECTION_TURN=2  # Right turn
    - rostest simulation_evaluation drive.test road:=ci_roads/intersection mission_mode:=2

Results Should be Clear and Precise?

Well…

CI Output on Failure.

A good solution on how to deliver the results is still missing!

A good start would be to

1. record rosbags and deliver them as CI artifacts, and

2. clearly highlight the configuration of the simulation that errors can be reproduced locally.

Other ideas are very much appreciated…

Go back to Outlook: Possibilities.