Documentation/Nightly/Developers/Tutorials/DashboardSetup

From Slicer Wiki
Jump to: navigation, search
Home < Documentation < Nightly < Developers < Tutorials < DashboardSetup


For the latest Slicer documentation, visit the read-the-docs.


General Instruction

Prerequisites

  • Install recommended compiler for your system
  • Install/build prerequisites as described here:
    • CMake (use the latest CMake stable release)
    • git
    • svn
    • Qt
    • NSIS (only on Windows)
    • Docker (only on Linux to use SlicerBuildEnvironment docker image)
  • Disable detachedHead warnings:
git config --global advice.detachedHead false

Create dashboard client build directories

  • On Linux, create the following directories:
/home/username/Dashboards
/home/username/Dashboards/Logs
/home/username/Dashboards/DashboardScripts
  • On macOS, create a directory like
/Users/username/Dashboards
/Users/username/Dashboards/Logs
/Users/username/Dashboards/DashboardScripts
  • On windows, create a directory like
C:/D
C:/D/Logs
C:/D/DashboardScripts

Notes:

  • username should be changed to match your system.
  • On Windows, a short directory name is required to overcome path length limitation.
  • The Dashboards directory will be used to set the DASHBOARDS_DIR variable in the client configuration script.


Customize template dashboard client script

Download template scripts

Download the template script: http://svn.slicer.org/Slicer4/trunk/CMake/SlicerDashboardScript.TEMPLATE.cmake

On Linux and macOS: 

wget http://svn.slicer.org/Slicer4/trunk/CMake/SlicerDashboardScript.TEMPLATE.cmake
mv SlicerDashboardScript.TEMPLATE.cmake <MachineName>_<OperatingSystem>-slicer_preview_nightly.cmake

On windows, from a powershell terminal: 

wget http://svn.slicer.org/Slicer4/trunk/CMake/SlicerDashboardScript.TEMPLATE.cmake -OutFile <MachineName>_<OperatingSystem>-slicer_preview_nightly.cmake

Customize script

Edit the file <MachineName>_<OperatingSystem>-slicer_preview_nightly.cmake to match your machine settings.


The following directories will be created by the dashboard script: 

#   Source directory : <DASHBOARDS_DIR>/<Slicer_DASHBOARD_SUBDIR>/<Slicer_DIRECTORY_BASENAME>-<Slicer_DIRECTORY_IDENTIFIER>
#   Build directory  : <DASHBOARDS_DIR>/<Slicer_DASHBOARD_SUBDIR>/<Slicer_DIRECTORY_BASENAME>-<Slicer_DIRECTORY_IDENTIFIER>-build

To easily identify build options associated with source and build directories, the associated build options will be combined together to create an empty text file with the following name: 

<DASHBOARDS_DIR>/<Slicer_DASHBOARD_SUBDIR>/<Slicer_DIRECTORY_BASENAME>-<Slicer_DIRECTORY_IDENTIFIER>-build - ${CTEST_BUILD_NAME}-${CTEST_BUILD_CONFIGURATION}-${SCRIPT_MODE}.txt


Publish and version control the dashboard scripts

To backup your scripts and help the Slicer community setup additional dashboard clients by adapting your scripts, consider publishing your scripts in a GitHub repository.

Assuming your organization is named Awesome, you could create the Github project: Awesome/SlicerDashboardScripts.

For example, the official Slicer dashboard script are published here: https://github.com/Slicer/DashboardScripts


Scheduling on Unix-like

So that your dashboard can automatically start at 11pm daily, you could add an entry similar to the following one in your crontab. (You could run crontab -e to edit the current user cron list and crontab -l to dislay the current list.) 

0 23 * * * /path/to/cmake-3.13.4/bin/ctest -S /Users/username/DashboardScripts/<MachineName>_<OperatingSystem>-slicer_preview_nightly.cmake -VV -O /Users/username/Dashboards/Logs/<MachineName>_<OperatingSystem>-slicer_preview_nightly.cmake.log

Scheduling on Linux

Install SlicerBuildEnvironment docker image

Read more details at https://github.com/Slicer/SlicerBuildEnvironment#readme 

mdir /home/username/bin
env_name=qt5-centos7
script=/home/username/bin/slicer-buildenv-${env_name}
docker run --rm slicer/buildenv-${env_name} > $script
chmod +x $script


Create a machinename.sh file that performs the build and use SlicerBuildEnvironment docker image

To use SlicerBuildEnvironment docker images, create a wrapper script like the following updating username and machinename to match your system:

FileName: /home/username/Dashboards/DashboardScripts/machinename.sh 

export DISPLAY=:0.0

# Changing directory is required by "slicer-buildenv-qt5-centos7-latest" script
cd  /home/username/Dashboards

# Download up-to-date slicer/buildenv-qt5-centos7:latest image
/home/username/bin/slicer-buildenv-qt5-centos7-latest update

# Slicer dashboard settings
docker_args="-e run_ctest_with_disable_clean=${run_ctest_with_disable_clean-FALSE}"
docker_args+=" -e run_ctest_with_update=${run_ctest_with_update-TRUE}"
docker_args+=" -e run_ctest_with_test=${run_ctest_with_test-FALSE}" # XXX Re-enable testing after slicer/slicer-test images have been updated

# Slicer 'Preview' release
time /home/username/bin/slicer-buildenv-qt5-centos7-latest \
  --args "${docker_args}" \
ctest -S /work/DashboardScripts/machinename-slicer_preview_nightly.cmake -VV -O /work/Logs/machinename-slicer_preview_nightly.log


Scheduling on macOS

macOS is deprecating the use of cron in favor of launchd.

Create a file $HOME/Library/LaunchAgents/org.slicer.NightlyDashboard.plist

This is Jim's plist file which has extra EnvironmentVariables to get through his firewall. 

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
  <key>Label</key>
  <string>org.slicer.NightlyDashboard</string>

  <key>ProgramArguments</key>
  <array>
    <string>/usr/local/bin/ctest</string>
    <string>-S</string>
    <string>/Users/millerjv/Projects/Slicer4-Dashboard/SlicerDashboardScript.cmake</string>
  </array>

  <key>EnvironmentVariables</key>
  <dict>
        <key>ALL_PROXY</key>
        <string>proxy.research.ge.com:8080</string>
        <key>http_proxy</key>
        <string>proxy.research.ge.com:8080</string>
        <key>RSYNC_PROXY</key>
        <string>proxy.research.ge.com:8080</string>
  </dict>

  <key>Nice</key>
  <integer>1</integer>

  <key>StartCalendarInterval</key>
  <dict>
        <key>Hour</key>
        <integer>1</integer>
        <key>Minute</key>
        <integer>03</integer>
  </dict>

  <key>StandardErrorPath</key>
  <string>/tmp/org.slicer.NightlyDashboard.err</string>

  <key>StandardOutPath</key>
  <string>/tmp/org.slicer.NightlyDashboard.out</string>
</dict>
</plist>

You can manually load this agent using 

$ launchctl load $HOME/Library/LaunchAgents/org.slicer.NightlyDashboard.plist

and the agent will be reloaded automatically on a reboot.

It seems I have to be logged onto the console for the launch agent to get installed and run. What I do is log on, then select "Login Window..." so that other people can log onto the console. This is suboptimal as I need to do this after each reboot of the system.


Scheduling on Windows

Create a .bat file that performs the build

For example, a clean nightly build can be performed by a batch file like this: 


rmdir /S /Q "C:\D\P\Slicer-0-build"
"c:\Program Files (x86)\CMake\bin\ctest.exe" -S "C:\D\DashboardScripts\overload-vs2015-slicer_preview_nightly.cmake" -C Release -VV -O C:\D\Logs\overload-vs2015-slicer_preview_nightly.txt

Create a new scheduled task that runs the .bat file

  • Windows XP: Create a new scheduled task at Control Panel / Scheduled Tasks / Add Scheduled
  • Windows 7: Task scheduler can be found at Start Menu / All Programs / Accessories / System Tools / Task Scheduler.

Setting up a continuous dashboard

Similarly if you want to setup a continuous dashboard, 

$ cp <MachineName>_<OperatingSystem>-slicer_preview_nightly.cmake <MachineName>_<OperatingSystem>-slicer_preview_continuous.cmake

Edit <MachineName>_<OperatingSystem>-slicer_preview_continuous.cmake and set 

set(SCRIPT_MODE "*continuous*")

Then add the corresponding entry in crontab so that it starts everyday at 6am: 

0 6 * * * /path/to/cmake-3.13.4/bin/ctest -S /Users/JOE/DashboardScripts/<MachineName>_<OperatingSystem>-slicer_preview_continuous.cmake -VV -O /Users/JOE/Dashboards/Logs/<MachineName>_<OperatingSystem>-slicer_preview_nightly.log

The output of the build is contained in ctest_package_make_package_output.txt in your slicer build directory, you can check this file if something went wrong.

Factory machines setup

Tool used for task scheduling is described here: Documentation/Nightly/Developers/Factory

Dashboard scripts used on the factory are available here: https://github.com/Slicer/DashboardScripts

More details available here:

Remarks

Setting up a dashboard on a Windows machine

Disabling TortoiseSVN Icon Overlay

If you're using an interface for version control such as TortoiseSVN, make sure in the Icon Overlay settings that the status cache is set to None.

Make sure the status cache is set to "None."

Keep short path

Keep the paths to your Nightly, Continuous, and Experimental directories as terse as possible. A good standard is C:\D for dashboards, and C:\D\<N,C,E> subdirectories for Nightly, Continuous, and Experimental. Otherwise, CMake will choke on the paths.

How can I prevent the “… has stopped working” window

As documented here, set registry key HKEY_LOCAL_MACHINE\Software\Microsoft\.NETFramework\DbgJITDebugLaunchSetting value to 1.

Value 1 simply return control. This results in a stack dump, after which the process is terminated.

This has been documented while trying to prevent the dialog "Microsoft Visual Studio 8 has stopped working" from appearing when Slicer extensions dashboard were running. The details associated with the dialog were: 

Problem signature:
  Problem Event Name:	BEX
  Application Name:	devenv.exe
  Application Version:	9.0.21022.8
  Application Timestamp:	47317b3d
  Fault Module Name:	VCProjectEngine.dll_unloaded
  Fault Module Version:	0.0.0.0
  Fault Module Timestamp:	47318038
  Exception Offset:	72dbdce0
  Exception Code:	c0000005
  Exception Data:	00000008

Reference:

Lockhunter: Unlock directories or folders

Often on Windows, once a dashboard build is started, it is impossible to delete anything from the source or build directory. Windows will report that this file is currently in use by another program, or "locking" it. For this, we recommend installing Lockhunter.

Lockhunter screenshot
Retrieved from "https://www.slicer.org/w/index.php?title=Documentation/Nightly/Developers/Tutorials/DashboardSetup&oldid=61286"