Difference between revisions of "Documentation/Nightly/Developers/ReleaseProcess"

From Slicer Wiki
Jump to: navigation, search
Line 475: Line 475:
  
  
 +
Source code available at https://github.com/mhalle/slicer4-download.
  
 
=== Explicit tagging of release packages ===
 
=== Explicit tagging of release packages ===

Revision as of 04:01, 19 October 2017

Home < Documentation < Nightly < Developers < ReleaseProcess

Release planning

Release

Since there all development occurs on master, each time version is updated, two commits will be required.

  1. Update the Slicer version information for the release:

    1. In CMakeLists.txt, update at least one these variables: Slicer_VERSION_MAJOR, Slicer_VERSION_MINOR, Slicer_VERSION_PATCH

    2. Re-run CMake with -DSlicer_RELEASE_TYPE:STRING=Stable. This is required to update Utilities/Scripts/SlicerWizard/__version__.py.

      Note If not doing a clean build, make sure to explicitly specify the version re-configuring with -DSlicer_VERSION_MAJOR:STRING=X -DSlicer_VERSION_MINOR:STRING=Y -DSlicer_VERSION_PATCH:STRING=Z .

    3. Commit the above changes using this message like:

      ENH: Slicer X.Y.Z
    4. Keep track of the SVN_REVISION and SVN_BRANCH for the next steps
  2. Update release scripts. See https://github.com/Slicer/DashboardScripts#rename-and-update-release-scripts
  3. Generate packages running Slicer package scripts and Slicer extensions scripts on each platform. (These are the scripts updated in the previous step)

  4. Tag the repository:

    SVN (Keep track of the branch revision. It will be referred below as BRANCHED_REVISION.): 

    cd ~/Projects
svn checkout http://svn.slicer.org/Slicer4/trunk Slicer-X.Y-SVN
svn copy -r<REVISION> http://svn.slicer.org/Slicer4/trunk  \
     http://svn.slicer.org/Slicer4/tags/Slicer-X-Y  \
    -m "ENH: Tag of X.Y.Z release based on r<REVISION>."

    GIT: 

    cd ~/Projects
git clone git@github.com:Slicer/Slicer Slicer-X.Y
git tag -s -m "ENH: Slicer X.Y.Z" vX.Y.Z origin/master
git push origin vX.Y.Z

  5. Create release branch:

    SVN: 

    cd ~/Projects/Slicer-X.Y-SVN
svn copy -r <REVISION> http://svn.slicer.org/Slicer4/trunk http://svn.slicer.org/Slicer4/branches/Slicer-X-Y \
    -m "ENH: Branching from trunk to Slicer-X-Y at r<REVISION>"

    svn checkout http://svn.slicer.org/Slicer4/branches/Slicer-X-Y

    GIT: 

     cd ~/Projects/Slicer-X.Y
 
 # Source: http://ivanz.com/2009/01/15/selective-import-of-svn-branches-into-a-gitgit-svn-repository/
 # and https://stackoverflow.com/questions/19712735/git-svn-cannot-setup-tracking-information-starting-point-is-not-a-branch
 
 git config --add svn-remote.svn-XY.url http://svn.slicer.org/Slicer4/branches/Slicer-X-Y
 git config --add svn-remote.svn-XY.fetch :refs/remotes/git-svn-XY
 
 git svn fetch svn-XY -r BRANCHED_REVISION
 
 git checkout -b master-XY remotes/git-svn-XY
 
 git svn rebase
 
 git push origin master-XY
  6. Update Release Details with release information

  7. Update the Slicer version information for the development:

    1. In CMakeLists.txt,

      • Update Slicer_VERSION_MAJOR and/or Slicer_VERSION_MINOR

    2. Re-run CMake with -DSlicer_RELEASE_TYPE:STRING=Experimental. This is required to update Utilities/Scripts/SlicerWizard/__version__.py.

      Note If not doing a clean build, make sure to explicitly specify the version re-configuring with -DSlicer_VERSION_MAJOR:STRING=X -DSlicer_VERSION_MINOR:STRING=Y -DSlicer_VERSION_PATCH:STRING=Z .

    3. Commit the above changes using this message like:

      cd ~/Projects/Slicer
git add -A
git commit -m "ENH: Begin X.Y.Z development"

Post release

Update Mantis

  • "Release" current target
  • Create new target
  • Check the "fixed in" fields

Update CDash

  1. Create new CDash groups for extension submissions associated with X.Y release:

    Extensions-X.Y-Nightly
Extensions-X.Y-Continuous

Update Slicer wiki

There are two steps involved:

  • copy of the pages associated with the Nightly namespace into the X.Y namespace.
  • update of permanent pages referencing the current Slicer version

As of November 2016, the process is fully automated and can be done using Base/Python/slicer/release/wiki.py script.

1) Install prerequisites:

  • Check with <email>jchris.fillionr@kitware.com</email> to get the credential associated with UpdateBot user.
  • Install mwdoc:
$ pip install mwdoc

2) Query current version information: 

$ python Base/Python/slicer/release/wiki.py query

3) Copy and update pages 

$ python Base/Python/slicer/release/wiki.py update 4.6
$ python Base/Python/slicer/release/wiki.py copy 4.6

The following pages have been updated:

The following pages with #REDIRECT have been updated:

Clean-up older nightly packages

This applies to the Midas instance associated with http://slicer.kitware.com

See https://gist.github.com/jcfr/ea9ef199bd5a3e071b8f

Manually sign packages

  1. Download package from http://slicer.cdash.org/index.php?project=Slicer4

  2. Sign the package following these instructions:

  3. Upload the signed package using upload-Slicer-to-slicer-kitware-com.cmake.
  4. Manually update the name of the item appending [signed]. Make sure to not update the bitstream name.

Tag release packages

This applies to the Midas instance associated with http://slicer.kitware.com


Since the Windows package was uploaded during the manual signing, it is already associated with the release tag. On the other hand, due to limitations of http://download.slicer.org, the Linux and macOS packages must be downloaded and re-uploaded.

1) Download packages

2) Compute md5 sums and check they correspond at the sum associated with download link posted on http://slicer.cdash.org.

2) Re-upload each packages using upload-Slicer-to-slicer-kitware-com.cmake script.

Confirm that packages are tagged

To confirm that all three packages are tagged. The following URL should return three packages:

http://slicer.kitware.com/midas3/api/json?method=midas.slicerpackages.package.list&release=4.8.0

Version NA-MIC data tree

This applies to the Midas instance associated with http://slicer.kitware.com

  1. If you do not have an account and/or if you do not belong to the DataManager group:

    • Create an account on the extension server and obtain an API Key. You will then use your midas login and api key to substitute <YOUR-MIDAS-LOGIN> and <YOUR-MIDAS-APIKEY> in the examples.

    • If not already done, go to NA-MIC community and click on Join community

    • Send an email on the developer list asking to be added to the DataManager group on NA-MIC community. That will grant you read/write permissions to the Data folder and sub-folders.

  2. Install prerequisites

    $ mkvirtualenv -p python2.7 version_namic_datatree
$ pip install pydas

  3. Identify id of the Data folder. For example 301

  4. Simulate creation of X.Y data folders based Nightly ones

    MIDAS_EMAIL=<YOUR-MIDAS-LOGIN>
MIDAS_API_KEY=<YOUR-MIDAS-APIKEY>
DEST_VERSION=X.Y

    $ cd /path/to/Slicer/Base/Python/slicer/release
$ python midasdata.py --url=http://slicer.kitware.com/midas3 --data_id=301 \
  --email=$MIDAS_EMAIL --apikey=$MIDAS_API_KEY --source_version=Nightly --dest_version=$DEST_VERSION --dry-run
Application ( folder_id: 302 )
'-Nightly ( folder_id: 831 )
'-'-Testing ( folder_id: 832 )
'-'-'-Baseline ( folder_id: 889 )
'-'-'-'-DiffusionTensorImagingTutorial.png ( item_id: 12067 )
'-'-'-'-NeurosurgicalPlanningTutorial.png ( item_id: 12066 )
'-'-'-'-SlicerTestingTest.png ( item_id: 17760 )
'-'-'-Input ( folder_id: 833 )
'-'-'-'-AtlasTests ( folder_id: 834 )
'-'-'-'-'-2012-10-26-BrainAtlas.mrb ( item_id: 10276 )
[...]
Module: VotingBinaryHoleFillingImageFilter ( folder_id: 1491 )
'-Nightly ( folder_id: 1491 )
'-'-Testing ( folder_id: 1492 )
'-'-'-Baseline ( folder_id: 1493 )
'-'-'-'-VotingBinaryHoleFillingImageFilterTest.nhdr ( item_id: 103418 )
'-'-'-'-VotingBinaryHoleFillingImageFilterTest.raw.gz ( item_id: 103419 )

  5. Create X.Y data tree based Nightly ones

    $ python midasdata.py --url=http://slicer.kitware.com/midas3 --data_id=301 \
  --email=$MIDAS_EMAIL --apikey=$MIDAS_API_KEY --source_version=Nightly --dest_version=$DEST_VERSION

Versioning of the NA-MIC Data tree for release X.Y...
Versioning Modules...
[...]
Versioning Modules...[DONE]
Versioning Application...
Creating folder X.Y under Application directory
Duplicating subfolders from Nightly to X.Y...
Duplicating subfolders from Nightly to X.Y...[DONE]
Versioning Application...[DONE]
Versioning of the NA-MIC Data tree for release X.Y...[DONE]

Update ExtensionsIndex

This applies to https://github.com/Slicer/ExtensionsIndex

1) Open a terminal

2) Create branch X.Y based of master 

DEST_VERSION=X.Y

cd /tmp
git clone git@github.com:Slicer/ExtensionsIndex 
cd ExtensionsIndex
git checkout master
git push origin master:${DEST_VERSION}

Update external website


Update User Survey (Not Needed Anymore)

Title of the survey should be updated. See https://docs.google.com/forms/d/1rggxoTV5KL_vt3gX9BNNAcKH4pnL5lkMqPnklOThINg

Instead, the Slicer welcome module has been updated to ask user to report feedback on the Slicer forum

Appendix

Sign packages

MacOSX

As discussed in issue #3965, the code signing command is of the form: 

codesign --deep --force --sign "Developer ID Application: <Name-of-certificate-in-keychain>" ./Slicer.app

where the argument to --sign is the name of the developer's certificate in Mac OS X Keychain.app .

Note: The signing of MacOSX package is not yet implemented in our nightly and release build workflows. See Documentation/Nightly/Developers/Mac_OS_X_Code_Signing for more information.

Windows

TBD - See #2697

Also see Documentation/Nightly/Developers/Windows_Code_Signing for more information.

Backport commit into release branch

The following steps will lead to the creation of new git-svn clone having two branches:

  • git-svn
  • git-svn-XY

Reference: http://www.dmo.ca/blog/20070608113513/

Step 1: Checkout sources

git clone git://github.com/Slicer/Slicer.git Slicer-X.Y
cd Slicer-X.Y
git svn init http://svn.slicer.org/Slicer4/trunk
git update-ref refs/remotes/git-svn refs/remotes/origin/master
git checkout master
git svn rebase

Step 2: Add release branch remote

Edit .git/config, and in addition to the existing 'git-svn' remote, add the following one: 

[svn-remote "svn-XY"]
   url = http://svn.slicer.org/Slicer4/branches/Slicer-X-Y
   fetch = :refs/remotes/git-svn-XY

Step 3: Pull associated SVN branch

git svn fetch svn-XY
git checkout -b master-XY git-svn-XY

Step 4: Backport

We can now cherry pick commit associated with master (trunk) into master-XY (Slicer-X-Y)


Script to upload Slicer packages

File: upload-Slicer-to-slicer-kitware-com.cmake

cmake_minimum_required(VERSION 3.9.0)

#
# To run this script:
#
#  cmake -P upload-Slicer-to-slicer-kitware-com.cmake
#

#
# Variables to update:
#
#   CMAKE_MODULE_PATH
#
#   release
#   source_revision
#   source_checkoutdate
#   MIDAS_PACKAGE_EMAIL
#   MIDAS_PACKAGE_API_KEY
#

list(APPEND CMAKE_MODULE_PATH
  D:/D/N/Slicer-1/CMake
  )
include(MIDASAPIUploadPackage)

set(operating_system    "win") # linux, macosx, win

set(release             "4.8.0")
set(source_revision     "26489")
set(source_checkoutdate "2017-10-18 05:11:33")
set(MIDAS_PACKAGE_EMAIL "<YOUR-MIDAS-LOGIN>")
set(MIDAS_PACKAGE_API_KEY "<YOUR-MIDAS-APIKEY>")

set(ext_linux  "tar.gz")
set(ext_macosx "dmg")
set(ext_win    "exe")

set(package_filepath    "C:/Users/dashboard/Downloads/Slicer-${release}-${operating_system}-amd64.${ext_${operating_system}}")

message(STATUS "Upload ${package_filepath}")

set(MIDAS_PACKAGE_URL http://slicer.kitware.com/midas3)                          
set(submission_type     "experimental")
set(architecture        "amd64")
set(package_type        "installer")

midas_api_upload_package(
  SERVER_URL ${MIDAS_PACKAGE_URL}
  SERVER_EMAIL ${MIDAS_PACKAGE_EMAIL}
  SERVER_APIKEY ${MIDAS_PACKAGE_API_KEY}
  SUBMISSION_TYPE ${submission_type}
  SOURCE_REVISION ${source_revision}
  SOURCE_CHECKOUTDATE ${source_checkoutdate}
  OPERATING_SYSTEM ${operating_system}
  ARCHITECTURE ${architecture}
  PACKAGE_FILEPATH ${package_filepath}
  PACKAGE_TYPE ${package_type}
  RELEASE ${release}
  RESULT_VARNAME output
  )
set(expected_output "ok")
if(NOT "${output}" STREQUAL "${expected_output}")
  message(FATAL_ERROR "Problem with midas_api_upload_package()\n"
                      "output:${output}\n"
                      "expected_output:${expected_output}")
endif()


Limitations of download.slicer.org

Considering that

(1) few minutes after a package is uploaded on http://slicer.kitware.com, it is indexed in the database associated with http://download.slicer.org, and

(2) the system indexing the package does NOT rewrite its database entries,

using the steps below will NOT result in an update of the packages reported on http://download.slicer.org.


Source code available at https://github.com/mhalle/slicer4-download.

Explicit tagging of release packages

Steps:

  • If needed, create a X.Y.Z folder in Slicer/Packages/Application/Release
    • Copy uploaded packages into the folder created above (Keep the default Create a reference to the existing item)
    • Identify the item_id associated with uploaded packages. For example: 316530, 316529
    • From Kitware internal network, SSH connect to jcfr@slicer.kitware.com (or jcfr@192.168.120.210)
    • Connect to mysql
      • password=$(cat /var/www/midas3/core/configs/database.local.ini | grep database.params.password | cut -d"=" -f2 | head -n1 | sed 's/"//g')
      • mysql -u midas -p$password -D midas
      • See file /var/www/midas3/core/configs/database.local.ini for password
      • Selected database is midas (use midas)
    • List packages associated with identified items and check they are the appropriate ones:
      • select * from slicerpackages_package as p , item as i where i.item_id = p.item_id and p.item_id in (316530, 316529);
    • Set release field
      • update slicerpackages_package set `release`="4.8.0" where item_id in (316530, 316529);
Retrieved from "https://www.slicer.org/w/index.php?title=Documentation/Nightly/Developers/ReleaseProcess&oldid=56671"