Documentation/Nightly/Developers/Factory

For the latest Slicer documentation, visit the read-the-docs.

Overview

The Slicer factory machine is maintained by Kitware and produces dashboard builds of many software projects, including nightly, continuous, and packaged versions of Slicer and its approved extensions. See the Slicer4 dashboard.

These builds are generated every day on three different operating systems: Windows 7, Mac OS X, and Linux. Because it is not legal to run any version of Mac OS X on a virtual machine (unless its host also runs Mac OS X), the host OS is Mac OS X, and the Windows and Linux builds are run within this machine as virtual machines.

Dashboard build scripts are located on GitHub: https://github.com/Slicer/DashboardScripts

Required factory components

See dashboard prerequisites.

Host(s)

factory-south.kitware

Software

The factory machine runs Mac OS X 10.11.6 (15G31), El Capitan

• Tasks scheduled using cron

The compiler is the one provided by default after installing XCode:

Apple LLVM version 8.0.0 (clang-800.0.42.1)
Target: x86_64-apple-darwin15.6.0

Remote access

Remote access: Via SSH or VNC from Kitware internal network (10.171.2.166).

Hardware

Processor

• 2 x 2.4 GHz 6-Core Intel Xeon

Graphics Card

• Chipset Model: ATI Radeon HD 5770
• Type: GPU
• Bus PCIe
• VRAM (total): 1024MB
• EFI Driver Version: 01.00.436

Memory

• 8 x 8GB = 64 GB
• Type: DDR3 ECC
• Speed: 1333 MHz

overload.kitware and metroplex.kitware

Software

• overload: Windows 7 Professional 64-bit
• metroplex: Red Hat Enterprise Linux WS v7.0

Hardware

• Dell Precision R7910 Motherboard,BW

Processor

• Dual Intel® Xeon® Processor E5-2630 v3 (8C, 2.4GHz, Turbo, HT, 20M, 85W)

Graphics Card

• NVIDIA® Quadro® K620 2GB (DP, DL-DVI-I) (1 DP to SL-DVI adapter)

Memory

• 32GB (8x4GB) 2133MHz DDR4 RDIMM ECC

Storage

• 2 X Integrated Intel 4 port AHCI SATA chipset controllers (tot 8 X 6.0Gb/s) SW RAID 0,1,5,10
• 1 x 1TB Dell 4*Drive PCIe x16 M.2 Solid State Drive Card (boot)

overload.kitware

• Windows 7 Professional - Service Pack 1 (Build 7601)
• Compiler:
  • VS2013 Community Edition
  • VS2015 Pro Edition

Remote access

Remote access: Via RDP from Kitware internal network (192.168.113.173).

metroplex.kitware

• Red Hat Enterprise Linux Workstation

Product ID:     71
Version:        7.2
Arch:           x86_64

• Docker community edition

Remote access

Remote access: Via SSH from Kitware internal network (192.168.113.208).

Maintenance

CUDA Install notes

Linux

sudo yum -y install kernel-devel
sudo yum -y install https://dl.fedoraproject.org/pub/epel/epel-release-latest-7.noarch.rpm

wget https://developer.nvidia.com/compute/cuda/10.1/Prod/local_installers/cuda-repo-rhel7-10-1-local-10.1.168-418.67-1.0-1.x86_64.rpm
sudo rpm -i cuda-repo-rhel7-10-1-local-10.1.168-418.67-1.0-1.x86_64.rpm
sudo yum clean all
sudo yum -y install cuda

Refs:

• https://developer.nvidia.com/cuda-downloads?target_os=Linux&target_arch=x86_64&target_distro=CentOS&target_version=7&target_type=rpmlocal
• https://docs.nvidia.com/cuda/cuda-installation-guide-linux/index.html

Mac OS

Beware. Not tested. I don't have a mac.

wget https://developer.nvidia.com/compute/cuda/8.0/Prod2/local_installers/cuda_8.0.61_mac-dmg
wget https://developer.nvidia.com/compute/cuda/8.0/Prod2/patches/2/cuda_8.0.61.2_mac-dmg

dmgFilePath=cuda_8.0.61_mac-dmg
MOUNTDEV=$(hdiutil mount $dmgFilePath | awk '/dev.disk/{print$1}')
MOUNTDIR="$(mount | grep $MOUNTDEV | awk '{$1=$2="";sub(" [(].*","");sub("^  ","");print}')"
sudo installer -pkg "${MOUNTDIR}/"*.pkg -target /
hdiutil unmount "$MOUNTDIR"

dmgFilePath=cuda_8.0.61.2_mac-dmg
MOUNTDEV=$(hdiutil mount $dmgFilePath | awk '/dev.disk/{print$1}')
MOUNTDIR="$(mount | grep $MOUNTDEV | awk '{$1=$2="";sub(" [(].*","");sub("^  ","");print}')"
sudo installer -pkg "${MOUNTDIR}/"*.pkg -target /
hdiutil unmount "$MOUNTDIR"

Refs:

• https://developer.nvidia.com/cuda-80-ga2-download-archive
• https://docs.nvidia.com/cuda/archive/8.0/cuda-installation-guide-mac-os-x/index.html
• https://stackoverflow.com/questions/22934083/install-dmg-package-on-mac-os-from-terminal

Windows

According to the docs, the "-s" option should allow unattended install. However, I was unable to get this to work.

wget https://developer.nvidia.com/compute/cuda/10.1/Prod/local_installers/cuda_10.1.168_425.25_windows.exe
cuda_10.1.168_425.25_windows.exe -s

Refs:

• https://developer.nvidia.com/cuda-downloads?target_os=Windows&target_arch=x86_64&target_version=7&target_type=exelocal
• https://docs.nvidia.com/cuda/cuda-installation-guide-microsoft-windows/index.html

Maintenance Guides

See https://github.com/Slicer/DashboardScripts#maintenance-guides

Check Virtual Machines (factory-south-ubuntu)

factory-south-win7 VM is shutdown, build have been moved to overload.kitware

When

Every day (or as soon as issues are suspected)

What

• Try to:
  • SSH into factory-south-ubuntu

• If failure:
  • Connect to factory.south and check the state of the virtual machines.

• Are the virtual machine unresponsive ?

• No -> No further action required
• Yes -> Read below
  • Kill both VMs
  • Restart the host (factory-south)
  • Manually start the virtual machines
  • If needed, manually restarts the nightly build (See below)

Manually restart the night builds

If the factory machines were restarted after the regular Nightly startup time, the nightly scripts will have to be explicitly started.

• Linux: Run /home/kitware/Dashboards/DashboardScripts/factory-south-ubuntu.sh

• MacOSX: Run /Users/kitware/DashboardScripts/factory.sh

• Windows: Open "Task Scheduled" -> slicer -> right click on "Nightly" -> Start

Nightly Build startup times

• Linux: 11:10pm

• MacOSx: 11:10pm

• Windows: Midnight

Retired Hosts

factory.kitware (RETIRED)

Software

The factory machine runs Mac OS X v10.6.8, Snow Leopard.

• CMake version: 3.9.0 (before 2017/07/04 was 3.5.0-rc3 with this patch)
  • Since CMake release >= 3.6 is compiled only for >= 10.7, a custom build of cmake was done with -DCMAKE_INSTALL_PREFIX:PATH=/Users/kitware/Dashboards/Support/CMake-3.9.0.app/Contents
• Compiler llvm-3.1. Note that the default compiled shipped with XCode 4.2 is not used. See here for more details.
• Tasks scheduled using cron

• Nightly generation of HomebrewCasks: Installation aborted (newer system is required for homebrew/cask-repair).

Remote access: Via SSH or VNC from Kitware internal network (192.168.113.230).

Hardware

• "Mac Pro (Mid 2010)" which has the 64-bit kernel enabled by default. See here for details.

Processor

• 2 x 2.93 GHz 6-Core Intel Xeon

Graphics Card

• Chipset Model: ATI Radeon HD 5770
• Type: GPU
• Bus PCIe
• VRAM (total): 1024MB
• EFI Driver Version: 01.00.436

Memory

• 8 x 4GB = 32GB
• Type: DDR3 ECC
• Speed: 1333 MHz

factory-south-win7.kitware (RETIRED VM)

• Tasks are scheduled using the Windows Task Scheduler (Start->All Programs->Accessories->System Tools->Task Scheduler)
• On Windows, instead of bash, we use batch scripts comprised of commands to call ctest to run all the nightly scripts.

This VM is shutdown, build have been moved to overload.kitware

• Windows 7 Ultimate N - Service Pack 1 (Build 7601) on an x86 platform.
• CMake version: 3.5.0-rc3
• Compiler: VS2013 Community Edition
• Processors: 6
• Memory: 16GB

Also, keep in mind the Documentation/Nightly/Developers/Tutorials/DashboardSetup#Remarks remarks about building a dashboard on a Windows machine. These all apply here.

Remote access: Via Remote Desktop from Kitware internal network (10.171.2.245).

factory-south-ubuntu.kitware (RETIRED)

This VM is shutdown, build have been moved to metroplex.kitware

• Ubuntu 10.04
• CMake version: 3.9.0 (before 2017/07/04 was 3.5.0-rc3)
• GCC version: Ubuntu 4.4.3-4ubuntu5.1
• Tasks scheduled using cron
• Processors: 5
• Memory: 10GB

$ lsb_release -a
No LSB modules are available.
Distributor ID:	Ubuntu
Description:	Ubuntu 10.04.4 LTS
Release:	10.04
Codename:	lucid

These two compilers are available:

gcc/g++ 4.4.3

/usr/bin/cpp-4.4
/usr/bin/g++-4.4
/usr/bin/gcc-4.4
/usr/bin/gcov-4.4
/usr/bin/x86_64-linux-gnu-cpp-4.4
/usr/bin/x86_64-linux-gnu-g++-4.4
/usr/bin/x86_64-linux-gnu-gcc-4.4

• In the PATH:

$ gcc --version
gcc (Ubuntu 4.4.3-4ubuntu5.1) 4.4.3
$ g++ --version
g++ (Ubuntu 4.4.3-4ubuntu5.1) 4.4.3

gcc/g++ 4.6.4

/usr/bin/cpp-4.6
/usr/bin/g++-4.6
/usr/bin/gcc-4.6
/usr/bin/gcov-4.6
/usr/bin/x86_64-linux-gnu-cpp-4.6
/usr/bin/x86_64-linux-gnu-g++-4.6
/usr/bin/x86_64-linux-gnu-gcc-4.6

Installed following these instructions:

• https://launchpad.net/~ubuntu-toolchain-r/+archive/ubuntu/test?field.series_filter=lucid
• http://superuser.com/questions/310809/how-can-i-update-gcc-to-the-latest-version-in-ubuntu-10-04/394811#394811

sudo add-apt-repository ppa:ubuntu-toolchain-r/test
sudo apt-get update
sudo apt-get update
sudo apt-get install gcc-4.6
sudo apt-get install g++-4.6

Remote access

Remote access: Via SSH or VNC from Kitware internal network (10.171.2.113).