View source for Documentation/4.10/Developers/Factory

<noinclude>{{documentation/versioncheck}}</noinclude>
=Overview=

The Slicer factory machine is maintained by [http://www.kitware.com Kitware] and produces dashboard builds of many software projects, including nightly, continuous, and packaged versions of Slicer and its approved extensions.  See the [http://slicer.cdash.org/index.php?project=Slicer4 Slicer4 dashboard].

These builds are generated every day on three different operating systems: Windows 7, Mac OS X, and Linux.  Because it is not legal to run any version of Mac OS X on a virtual machine (unless its host also runs Mac OS X), the host OS is Mac OS X, and the Windows and Linux builds are run within this machine as virtual machines.

'''Dashboard build scripts are located on GitHub: https://github.com/Slicer/DashboardScripts'''

==Required factory components==

See [[Documentation/{{documentation/version}}/Developers/Tutorials/DashboardSetup|dashboard prerequisites]].

=Host(s)=

== factory.kitware (RETIRED) ==

=== Software ===
The factory machine runs Mac OS X v10.6.8, Snow Leopard.  
* CMake version: 3.9.0 (before 2017/07/04 was 3.5.0-rc3 with [https://github.com/Kitware/CMake/compare/master...jcfr:osx-framework this patch])
** Since CMake release >= 3.6 is compiled only for  >= 10.7, a custom build of cmake was done with <code>-DCMAKE_INSTALL_PREFIX:PATH=/Users/kitware/Dashboards/Support/CMake-3.9.0.app/Contents</code>  
* Compiler llvm-3.1. Note that the default compiled shipped with XCode 4.2 is not used. See [http://slicer-devel.65872.n3.nabble.com/MacOSX-factory-build-Transitioning-from-hybrid-quot-llvm-g-quot-to-quot-clang-quot-Update-to-Qt-4-8-6-tt4032198.html#a4032199 here] for more details.
* Tasks scheduled using [http://en.wikipedia.org/wiki/Cron cron]

* Nightly generation of HomebrewCasks: [[Documentation/Nightly/Developers/Factory/HomeBrewInstall|Installation aborted]] (newer system is required for <tt>homebrew/cask-repair</tt>).

Remote access: Via SSH or VNC from Kitware internal network (192.168.113.230).
<!--
The output of <pre>crontab -l</pre> on the factory machine is as follows:
<pre>
55 23 * * * cd /Users/kitware/DashboardScripts && /usr/bin/cvs -q up -dAP
0 0 * * * /Users/kitware/DashboardScripts/factory.sh > /Users/kitware/Dashboards/Logs/factory.log 2>&1
0 9 * * * /Applications/CMake\ 2.8-8.app/Contents/bin/ctest -V -VV -S /Users/kitware/DashboardScripts/factory-64bits_CTKAppLauncher_release_continuous.cmake > /Users/kitware/Dashboards/Logs/factory-64bits_CTKAppLauncher_release_continuous.log 2>&1
0 9 * * * /Applications/CMake\ 2.8-8.app/Contents/bin/ctest -V -VV -S /Users/kitware/DashboardScripts/factory-64bits_slicer4_release_continuous.cmake > /Users/kitware/Dashboards/Logs/factory-64bits_slicer4_release_continuous.log 2>&1
0 10 * * * /Applications/CMake\ 2.8-8.app/Contents/bin/ctest -V -VV -S /Users/kitware/DashboardScripts/factory-64bits_slicerextensions_release_continuous.cmake > /Users/kitware/Dashboards/Logs/factory-64bits_slicerextensions_release_continuous.log 2>&1
0 10 * * * /Applications/CMake\ 2.8-8.app/Contents/bin/ctest -V -VV -S /Users/kitware/DashboardScripts/factory-64bits_slicerextensions_411_release_continuous.cmake > /Users/kitware/Dashboards/Logs/factory-64bits_slicerextensions_411_release_continuous.log 2>&1
</pre>

The first tab is a command to update the cvs repository storing all of Kitware's dashboard scripts.  The second is a bash script which runs all of the nightly builds done by the Mac OS X factory.  The rest correspond to all the continuous builds.  An example of an entry in the bash script is as follows:
<pre>
open -a x11
export DISPLAY=:0.0 # just DISPLAY=:0.0 without export is not enough
export PATH=:/usr/local/git/bin:$PATH

CTEST=/Applications/CMake-3.5.0-rc3.app/Contents/bin/ctest
LOG_DIR=/Users/kitware/Dashboards/Logs
DASHBOARD_SCRIPTS_DIR=/Users/kitware/DashboardScripts

# Nightly build of CTKAppLauncher
"$CTEST" -S $DASHBOARD_SCRIPTS_DIR/factory-64bits_CTKAppLauncher_release_nightly.cmake -VV -O $LOG_DIR/factory-64bits_CTKAppLauncher_release_nightly.log

# Nightly build of slicer
"$CTEST" -S $DASHBOARD_SCRIPTS_DIR/factory-64bits_slicer4_release_nightly.cmake -VV -O $LOG_DIR/factory-64bits_slicer4_release_nightly.log

</pre>
-->

<!--
The virtual  machines are run using the Parallels tool.
-->

=== Hardware ===

* "Mac Pro (Mid 2010)" which has the 64-bit kernel enabled by default. See [http://support.apple.com/kb/HT3770 here] for details.

==== Processor ====
* 2 x 2.93 GHz 6-Core Intel Xeon

==== Graphics Card ====
* Chipset Model: ATI Radeon HD 5770
* Type: GPU
* Bus PCIe
* VRAM (total): 1024MB
* EFI Driver Version: 01.00.436

==== Memory ====
* 8 x 4GB = 32GB
* Type: DDR3 ECC
* Speed: 1333 MHz

== factory-south.kitware ==

=== Software ===

The factory machine runs Mac OS X 10.11.6 (15G31), El Capitan
* Tasks scheduled using [http://en.wikipedia.org/wiki/Cron cron]

The compiler is the following:
<pre>
Apple LLVM version 8.0.0 (clang-800.0.42.1)
Target: x86_64-apple-darwin15.6.0
</pre>

The virtual machines are run using the [http://www.parallels.com/ Parallels Desktop 11].
* Note that Parallels Desktop supports [http://kb.parallels.com/en/115487 OpenGL 2.1 only] (not higher)

==== Remote access ====

Remote access: Via SSH or VNC from Kitware internal network (10.171.2.166).

=== Hardware ===

==== Processor ====
* 2 x 2.4 GHz 6-Core Intel Xeon

==== Graphics Card ====
* Chipset Model: ATI Radeon HD 5770
* Type: GPU
* Bus PCIe
* VRAM (total): 1024MB
* EFI Driver Version: 01.00.436

==== Memory ====
* 8 x 8GB = 64 GB
* Type: DDR3 ECC
* Speed: 1333 MHz

== overload.kitware and metroplex.kitware ==

[[File:Kitware build servers metroplex-and-overload.jpg|600px]]


=== Software ===

* overload: Windows 7 Professional 64-bit
* metroplex: Red Hat Enterprise Linux WS v7.0

=== Hardware ===

* Dell Precision R7910 Motherboard,BW

==== Processor ====

* Dual Intel® Xeon® Processor E5-2630 v3 (8C, 2.4GHz, Turbo, HT, 20M, 85W)

==== Graphics Card ====

* NVIDIA® Quadro® K620 2GB (DP, DL-DVI-I) (1 DP to SL-DVI adapter)

==== Memory ====

* 32GB (8x4GB) 2133MHz DDR4 RDIMM ECC

==== Storage ====

* 2 X Integrated Intel 4 port AHCI SATA chipset controllers (tot 8 X 6.0Gb/s) SW RAID 0,1,5,10
* 1 x 1TB Dell 4*Drive PCIe x16 M.2 Solid State Drive Card (boot)

=== overload.kitware ===

* Windows 7 Professional - Service Pack 1 (Build 7601)
* Compiler:
** VS2013 Community Edition
** VS2015 Pro Edition

==== Remote access ====

Remote access: Via RDP from Kitware internal network (192.168.113.173).

=== metroplex.kitware ===

* Red Hat Enterprise Linux Workstation

<pre>
Product ID:     71
Version:        7.2
Arch:           x86_64
</pre>

* Docker community edition

==== Remote access ====

Remote access: Via SSH from Kitware internal network (192.168.113.208).

=VM: Linux=
<!--
== factory.kitware ==
* Ubuntu 11.04
* CMake version: 2.8.8
* GCC version: Ubuntu 4.4.3-4ubuntu5.1
* Tasks scheduled using [http://en.wikipedia.org/wiki/Cron cron]

The scripts on this VM are very similar to those on the Host.

=== Remote access ===
This machine is accessible through SSH and remote VNC.  It has been configured with its own static IP, and should be accessed by VNC through this IP instead of through a VNC connection on the Mac Host.  Do not try to connect through VNC to this machine on the Host.

===Known issues===
* X crash / GLX" missing on display. See [http://www.na-mic.org/Bug/view.php?id=2468 #2468]: '''FIXED'''
* Beside the fact Parallel has been updated, the X server still crash and auto-login does NOT succeed. See [http://www.na-mic.org/Bug/view.php?id=2624 #2624]: '''FIXED'''
-->


== factory-south-ubuntu.kitware (RETIRED) ==

<b>This VM is shutdown, build have been moved to metroplex.kitware</b>

* Ubuntu 10.04
* CMake version: 3.9.0 (before 2017/07/04 was 3.5.0-rc3)
* GCC version: Ubuntu 4.4.3-4ubuntu5.1
* Tasks scheduled using [http://en.wikipedia.org/wiki/Cron cron]
* Processors: 5
* Memory: 10GB

<pre>
$ lsb_release -a
No LSB modules are available.
Distributor ID:	Ubuntu
Description:	Ubuntu 10.04.4 LTS
Release:	10.04
Codename:	lucid
</pre>

These two compilers are available:

=== gcc/g++ 4.4.3 ===

<pre>
/usr/bin/cpp-4.4
/usr/bin/g++-4.4
/usr/bin/gcc-4.4
/usr/bin/gcov-4.4
/usr/bin/x86_64-linux-gnu-cpp-4.4
/usr/bin/x86_64-linux-gnu-g++-4.4
/usr/bin/x86_64-linux-gnu-gcc-4.4
</pre>

* In the PATH:

<pre>
$ gcc --version
gcc (Ubuntu 4.4.3-4ubuntu5.1) 4.4.3
$ g++ --version
g++ (Ubuntu 4.4.3-4ubuntu5.1) 4.4.3
</pre>

=== gcc/g++ 4.6.4 ===

<pre>
/usr/bin/cpp-4.6
/usr/bin/g++-4.6
/usr/bin/gcc-4.6
/usr/bin/gcov-4.6
/usr/bin/x86_64-linux-gnu-cpp-4.6
/usr/bin/x86_64-linux-gnu-g++-4.6
/usr/bin/x86_64-linux-gnu-gcc-4.6
</pre>

Installed following these instructions:
* https://launchpad.net/~ubuntu-toolchain-r/+archive/ubuntu/test?field.series_filter=lucid
* http://superuser.com/questions/310809/how-can-i-update-gcc-to-the-latest-version-in-ubuntu-10-04/394811#394811

<pre>
sudo add-apt-repository ppa:ubuntu-toolchain-r/test
sudo apt-get update
sudo apt-get update
sudo apt-get install gcc-4.6
sudo apt-get install g++-4.6
</pre>

=== Remote access ===

Remote access: Via SSH or VNC from Kitware internal network (10.171.2.113).

=VM: Windows 7=

* Tasks are scheduled using the Windows Task Scheduler (Start->All Programs->Accessories->System Tools->Task Scheduler)
* On Windows, instead of bash, we use batch scripts comprised of commands to call ctest to run all the nightly scripts.  

== factory-south-win7.kitware (RETIRED) ==

<b>This VM is shutdown, build have been moved to overload.kitware</b>

* Windows 7 Ultimate N - Service Pack 1 (Build 7601) on an x86 platform.
* CMake version: 3.5.0-rc3
* Compiler: VS2013 Community Edition
* Processors: 6
* Memory: 16GB



Also, keep in mind the [[Documentation/{{documentation/version}}/Developers/Tutorials/DashboardSetup#Remarks remarks]] about building a dashboard on a Windows machine.  These all apply here.

Remote access: Via Remote Desktop from Kitware internal network (10.171.2.245).

=Maintenance=

== Maintenance Guides ==

See https://github.com/Slicer/DashboardScripts#maintenance-guides

== Check Virtual Machines (factory-south-ubuntu) ==

<b>factory-south-win7 VM is shutdown, build have been moved to overload.kitware</b>

=== When ===

Every day (or as soon as issues are suspected)

=== What ===

* Try to:
** SSH into factory-south-ubuntu
<!--
** RDP into factory-south-win7
-->

* If failure:
** Connect to <code>factory.south</code> and check the state of the virtual machines. 

* Are the virtual machine unresponsive ?

* '''No''' -> No further action required
* '''Yes''' -> Read below
** Kill both VMs
** Restart the host (factory-south)
** Manually start the virtual machines
** If needed, manually restarts the nightly build (See below)

== Manually restart the night builds ==

If the factory machines were restarted after the regular Nightly startup time, the nightly scripts will have to be explicitly started.

* Linux: Run <code>/home/kitware/Dashboards/DashboardScripts/factory-south-ubuntu.sh</code>

* MacOSX: Run <code>/Users/kitware/DashboardScripts/factory.sh</code>

* Windows: Open "Task Scheduled" -> slicer -> right click on "Nightly" -> Start

== Nightly Build startup times ==

* Linux: 11:10pm

* MacOSx: 11:10pm

* Windows: Midnight