Difference between revisions of "Slicer3:Testing"

From Slicer Wiki
Jump to: navigation, search
 
(12 intermediate revisions by 5 users not shown)
Line 1: Line 1:
 
How to do testing in Slicer3.
 
How to do testing in Slicer3.
 +
 +
= On the Command Line =
 +
 +
Slicer developers may want to consider running the following handy command lines before checking in code:
 +
 +
  ./Slicer3 --exec exit
 +
 +
(this checks that all of slicer will start and exit cleanly)
 +
 +
  ./Slicer3 --no-modules --exec exit
 +
 +
(this will test just the base of slicer: no modules will be loaded or built).
 +
 +
  ./Slicer3 --ignore-module <ModuleName> --exec exit
 +
 +
(this will specifically leave out a module to help identify where a leak  or crash is coming from.  Note that the module's GUI class constructor will be called, but not the BuildGUI method).
  
 
= Writing Tests =
 
= Writing Tests =
 +
 +
See [http://wiki.na-mic.org/Wiki/index.php/Slicer3:Execution_Model_Testing  Execution Model Testing] for how to write tests for plug in modules.
  
 
== C++ ==
 
== C++ ==
  
 
== Tcl ==
 
== Tcl ==
 +
 +
The slicer3 tcl interpreter can be used to create tests that exercise practically the full API of the program.  You can run these tests using the --script command line argument to slicer. See, for example, the tests in Slicer3/Applications/GUI/Testing [http://viewvc.slicer.org/viewcvs.cgi/trunk/Applications/GUI/Testing/]
  
 
= Calling tests from CMake/CTest =
 
= Calling tests from CMake/CTest =
 +
 +
You can add references to tests in your CMakeLists.txt file.  Be sure to carefully construct the paths to work on multiple build platforms.  The best way to do this is to copy an example from a working test.
 +
 +
The following example, from Slicer3/Libs/ModuleDescriptionParser/CMakeLists.txt, shows how to use the pre-defined CMake variables to find the executable and launch it in such a way that the shared libraries will be correctly defined at run time:
 +
 +
<pre>
 +
set(EXE_PREFIX ${CMAKE_RUNTIME_OUTPUT_DIRECTORY} )
 +
if(WIN32)
 +
  set(EXE_PREFIX ${CMAKE_RUNTIME_OUTPUT_DIRECTORY}/${CMAKE_BUILD_TYPE})
 +
endif(WIN32)
 +
 +
set(TEST_DATA ${ModuleDescriptionParserTesting_SOURCE_DIR}/TestData)
 +
add_test(Parser1Test1 ${LAUNCH_EXE} ${EXE_PREFIX}/Parser1Test ${TEST_DATA}/ParserTest1.xml)
 +
</pre>
 +
 +
In particular, note:
 +
* CMAKE_BUILD_TYPE which will be something like Release or Debug on windows and will be an empty string on mac and linux
 +
* CMAKE_RUNTIME_OUTPUT_DIRECTORY which is where the executable will be stored in the build tree
 +
* LAUNCH_EXE which is a variable optionally set by the program that uses the library you are testing.  In the case of slicer3, this is set by the top-level CMakeLists.txt file to be the command line prefix to use the [[Slicer3:Launcher|Slicer3 launcher]] to start a process with the correct environment variables defined.
  
 
= Setting up automatic builds =
 
= Setting up automatic builds =
Line 47: Line 86:
 
</pre>
 
</pre>
  
−
== Unix ==
+
== Unix (linux) ==
  
 
Here's an example script that can be dropped into /etc/cron.daily
 
Here's an example script that can be dropped into /etc/cron.daily
  
 
<pre>
 
<pre>
−
#!/bin/tcsh
+
#!/bin/bash
  
 
cd /home/pieper/slicer3/nightly
 
cd /home/pieper/slicer3/nightly
 
rm -rf Slicer3 Slicer3-lib Slicer3-build
 
rm -rf Slicer3 Slicer3-lib Slicer3-build
  
−
su - pieper -c "cd /home/pieper/slicer3/nightly; svn co http://www.na-mic.org/svn/Slicer3/trunk Slicer3 |& tee nightly.log"
+
su - pieper -c "cd /home/pieper/slicer3/nightly; svn co http://svn.slicer.org/Slicer3/trunk Slicer3 2>&1 | tee nightly.log"
−
su - pieper -c "setenv DISPLAY :0; cd /home/pieper/slicer3/nightly; ./Slicer3/Scripts/getbuildtest.tcl -t Nightly |& tee -a nightly.log"
+
su - pieper -c "export DISPLAY=:0.0; cd /home/pieper/slicer3/nightly; ./Slicer3/Scripts/getbuildtest.tcl -t Nightly 2>&1 | tee -a nightly.log"
  
 
</pre>
 
</pre>
  
−
Note: this assumes your shell is csh-like.  If you use bash, replace the "|&" with "2>&1 |".
+
Note: this assumes your shell is bash.  If you use csh, replace the "2>&1 |" with "|&".
  
−
Note: the DISPLAY variable should be set to be the X display you want to run with.  Most systems will have :0 as the default display.
+
Note: the DISPLAY variable should be set to be the X display you want to run with.  Most systems will have :0.0 as the default display.
 +
 
 +
'''Be sure to always leave the X server running or tests will fail.'''
  
 
Here's a similar example for a continuous build
 
Here's a similar example for a continuous build
Line 74: Line 115:
 
date > continuous.log
 
date > continuous.log
 
export DISPLAY=:0
 
export DISPLAY=:0
−
./Scripts/getbuildtest2.tcl --update --test-type Continuous |& tee -a continuou\s.log
+
./Scripts/getbuildtest2.tcl --update --test-type Continuous |& tee -a continuous.log
 
date >> continuous.log
 
date >> continuous.log
 +
</pre>
 +
 +
You can set up the script to be called every hour by adding this line to your cron file (crontab -e)
 +
 +
<pre>
 +
0 * *  * * /PathToTheScriptShownJustAbove/TestSlicer3Continuous.sh
 +
</pre>
 +
 +
== Mac OSX ==
 +
 +
The scripts are basically the same as the linux versions, with the addition of a line to open the X11 application explicitly.  Also, the script doesn't run as superuser, so no su commands are needed.
 +
 +
<pre>
 +
#!/bin/bash
 +
 +
cd /extra/pieper/slicer3/nightly
 +
rm -rf Slicer3 Slicer3-lib Slicer3-build
 +
 +
cd /extra/pieper/slicer3/nightly
 +
svn co http://svn.slicer.org/Slicer3/trunk Slicer3 2>&1 > nightly.log
 +
open /Applications/Utilities/X11.app
 +
./Slicer3/Scripts/getbuildtest.tcl -t Nightly 2>&1 >> nightly.log
 +
 +
</pre>
 +
 +
On Mac, the launchctl program is used to manage nightly builds (see [http://surfer.nmr.mgh.harvard.edu/fswiki/MacOsLaunchd  Nick's excellent description from the FreeSurfer wiki]).
 +
 +
Here's the example .plist file for a slicer build that calls the script above every day at 4am. 
 +
 +
<pre>
 +
<?xml version="1.0" encoding="UTF-8"?>
 +
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
 +
<plist version="1.0">
 +
<dict>
 +
        <key>GroupName</key>
 +
        <string>slicer</string>
 +
        <key>Label</key>
 +
        <string>NightlyBuild</string>
 +
        <key>Program</key>
 +
        <string>/extra/pieper/slicer3/nightly/slicer.nightly</string>
 +
        <key>ProgramArguments</key>
 +
        <array>
 +
                <string>&amp;</string>
 +
        </array>
 +
        <key>ServiceDescription</key>
 +
        <string>nightly clean build of slicer trunk</string>
 +
        <key>StartCalendarInterval</key>
 +
        <dict>
 +
                <key>Hour</key>
 +
                <integer>4</integer>
 +
                <key>Minute</key>
 +
                <integer>0</integer>
 +
        </dict>
 +
        <key>UserName</key>
 +
        <string>pieper</string>
 +
</dict>
 +
</plist>
 +
 +
 +
</pre>
 +
 +
== Reporting errors during genlib stage ==
 +
 +
If you would like to report the errors during building the dependency libraries to be reported to the dashboard, you can modify the following script http://www.cdash.org/CDash/viewNotes.php?buildid=398460, modify it to fit your environment, and then launch it with
 +
<pre>
 +
ctest -S testscript.ctest
 
</pre>
 
</pre>

Latest revision as of 23:51, 28 December 2009

Home < Slicer3:Testing

How to do testing in Slicer3.

On the Command Line

Slicer developers may want to consider running the following handy command lines before checking in code: 

 ./Slicer3 --exec exit

(this checks that all of slicer will start and exit cleanly) 

 ./Slicer3 --no-modules --exec exit

(this will test just the base of slicer: no modules will be loaded or built). 

 ./Slicer3 --ignore-module <ModuleName> --exec exit

(this will specifically leave out a module to help identify where a leak or crash is coming from. Note that the module's GUI class constructor will be called, but not the BuildGUI method).

Writing Tests

See Execution Model Testing for how to write tests for plug in modules.

C++

Tcl

The slicer3 tcl interpreter can be used to create tests that exercise practically the full API of the program. You can run these tests using the --script command line argument to slicer. See, for example, the tests in Slicer3/Applications/GUI/Testing [1]

Calling tests from CMake/CTest

You can add references to tests in your CMakeLists.txt file. Be sure to carefully construct the paths to work on multiple build platforms. The best way to do this is to copy an example from a working test.

The following example, from Slicer3/Libs/ModuleDescriptionParser/CMakeLists.txt, shows how to use the pre-defined CMake variables to find the executable and launch it in such a way that the shared libraries will be correctly defined at run time: 

set(EXE_PREFIX ${CMAKE_RUNTIME_OUTPUT_DIRECTORY} )
if(WIN32)
  set(EXE_PREFIX ${CMAKE_RUNTIME_OUTPUT_DIRECTORY}/${CMAKE_BUILD_TYPE})
endif(WIN32)

set(TEST_DATA ${ModuleDescriptionParserTesting_SOURCE_DIR}/TestData)
add_test(Parser1Test1 ${LAUNCH_EXE} ${EXE_PREFIX}/Parser1Test ${TEST_DATA}/ParserTest1.xml)

In particular, note:

  • CMAKE_BUILD_TYPE which will be something like Release or Debug on windows and will be an empty string on mac and linux
  • CMAKE_RUNTIME_OUTPUT_DIRECTORY which is where the executable will be stored in the build tree
  • LAUNCH_EXE which is a variable optionally set by the program that uses the library you are testing. In the case of slicer3, this is set by the top-level CMakeLists.txt file to be the command line prefix to use the Slicer3 launcher to start a process with the correct environment variables defined.

Setting up automatic builds

Note: this can also be used for continuous builds using the --test-type Continuous flag.

Windows

For windows you need to use the Add Scheduled Task option in the Control Panel to create the build. This task runs in a 'dumb' dos shell, so you need to help it get jump started. The following works:

  • create a .bat file that that is called from the Scheduled Task
  • create a shell script that runs the build and call that from the .bat file

Here's an example for run.bat: 

echo off

C:
chdir C:\cygwin\bin

set CYGWIN=binmode tty ntsec

rxvt -sl 1000 -e c:/pieper/run.csh -l

this example uses rxvt, which is available from cygwin, but not installed by default (it's an xterm ported to cygwin).

And here's an example run.csh: 

#!/bin/csh

setenv PATH "/usr/local/bin:/usr/bin:/bin:/usr/X11R6/bin:/cygdrive/c/WINDOWS/system32:/cygdrive/c/WINDOWS:/cygdrive/c/WINDOWS/System32/Wbem"

c:/pieper/bwh/slicer3/latest/Slicer3/Scripts/getbuildtest.tcl --update --test-type Nightly |& c:/cygwin/bin/tee c:/pieper/bwh/slicer3/latest/Slicer3.build-log

c:/cygwin/bin/sleep 5

Unix (linux)

Here's an example script that can be dropped into /etc/cron.daily 

#!/bin/bash

cd /home/pieper/slicer3/nightly
rm -rf Slicer3 Slicer3-lib Slicer3-build

su - pieper -c "cd /home/pieper/slicer3/nightly; svn co http://svn.slicer.org/Slicer3/trunk Slicer3 2>&1 | tee nightly.log"
su - pieper -c "export DISPLAY=:0.0; cd /home/pieper/slicer3/nightly; ./Slicer3/Scripts/getbuildtest.tcl -t Nightly 2>&1 | tee -a nightly.log"

Note: this assumes your shell is bash. If you use csh, replace the "2>&1 |" with "|&".

Note: the DISPLAY variable should be set to be the X display you want to run with. Most systems will have :0.0 as the default display.

Be sure to always leave the X server running or tests will fail.

Here's a similar example for a continuous build 

#!/usr/bin/zsh

cd /playpen/davisb/Slicer3/continuous/Slicer3
date > continuous.log
export DISPLAY=:0
./Scripts/getbuildtest2.tcl --update --test-type Continuous |& tee -a continuous.log
date >> continuous.log

You can set up the script to be called every hour by adding this line to your cron file (crontab -e) 

0 * *  * * /PathToTheScriptShownJustAbove/TestSlicer3Continuous.sh

Mac OSX

The scripts are basically the same as the linux versions, with the addition of a line to open the X11 application explicitly. Also, the script doesn't run as superuser, so no su commands are needed. 

#!/bin/bash 

cd /extra/pieper/slicer3/nightly
rm -rf Slicer3 Slicer3-lib Slicer3-build

cd /extra/pieper/slicer3/nightly
svn co http://svn.slicer.org/Slicer3/trunk Slicer3 2>&1 > nightly.log
open /Applications/Utilities/X11.app
./Slicer3/Scripts/getbuildtest.tcl -t Nightly 2>&1 >> nightly.log

On Mac, the launchctl program is used to manage nightly builds (see Nick's excellent description from the FreeSurfer wiki).

Here's the example .plist file for a slicer build that calls the script above every day at 4am. 

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
        <key>GroupName</key>
        <string>slicer</string>
        <key>Label</key>
        <string>NightlyBuild</string>
        <key>Program</key>
        <string>/extra/pieper/slicer3/nightly/slicer.nightly</string>
        <key>ProgramArguments</key>
        <array>
                <string>&</string>
        </array>
        <key>ServiceDescription</key>
        <string>nightly clean build of slicer trunk</string>
        <key>StartCalendarInterval</key>
        <dict>
                <key>Hour</key>
                <integer>4</integer>
                <key>Minute</key>
                <integer>0</integer>
        </dict>
        <key>UserName</key>
        <string>pieper</string>
</dict>
</plist>

Reporting errors during genlib stage

If you would like to report the errors during building the dependency libraries to be reported to the dashboard, you can modify the following script http://www.cdash.org/CDash/viewNotes.php?buildid=398460, modify it to fit your environment, and then launch it with 

ctest -S testscript.ctest
Retrieved from "https://www.slicer.org/w/index.php?title=Slicer3:Testing&oldid=11702"