Merge pull request #4 from spack/validate-rev-aws20

Validate pipeline for aws20 container

UPDATING.md

@@ -12,11 +12,11 @@ There are two build rules (see this [section](./DOCKERHUB_SETUP.md#configure-bui
 for more info) governing when DockerHub will rebuild the container.  The
 first specifies that any time you push to the `master` branch on this repo,
 DockerHub will build an image and tag it with `latest`.  The second build rull
-specifies that any time you push a tag matching the regular expression
+specifies that any time you push a branch matching the regular expression
 `^rev-(.+)$`, DockerHub will build an image and use the matched group
-(everything in the tag following the `rev-`) to tag the image.  So if, for
-example, you want a container to be tagged `sc20`, you would tag this
-repository `rev-sc20` and push it.
+(everything in the branch name following the `rev-`) to tag the image.  So if,
+for example, you want a container to be tagged `sc20`, you would push a branch
+to this repository named `rev-sc20`.
 
 ## Steps to build an updated container
 
@@ -36,12 +36,15 @@ the helper script, run the following command from the root of the repository:
 
     $ ./keyhelp/export_pipeline_key.sh ./docker "name-of-key-to-export"
 
-1. Create a new mirror (this is required until the sync process between the
-mirror and the container build cache is controlled by/limited to the spack
-environment in which it runs).  The mirror url currently needs to be updated
+1. Either a) create a new mirror, or else b) clean out the existing one.  This
+is required until the sync process between the mirror and the container build
+cache is controlled by/limited to the spack environment in which it runs).  If
+you choose to create a new mirror, the mirror url currently needs to be updated
 in two locations: The `spack.yaml` and the DockerHub autobuild environment
 variables (see [here](./DOCKERHUB_SETUP.md#configure-builds) for more
-information).
+information).  If you choose to clean out the old mirror, just use the web
+interface to remove the entire "build_cache" directory, and then there is
+nothing else to change.
 
 1. Create a new buildgroup in CDash to track the builds from the version to be
 (done by editing the `spack.yaml` and changing the `build-group` property)
@@ -54,3 +57,28 @@ the packages build.
 
 1. Merge the PR into master and/or tag the result using the tagging scheme
 described in the section above.
+
+## Items that may require updates as OS/Spack version change
+
+Check what version of `gcc` gets installed by `apt-get` by default if you
+don't specify a version.  That should be the compiler the tutorial picks
+to install most packages.  Then an older `gcc` should be installed and used
+to install zlib with the "custom" system compiler.  If either of those
+compilers aren't in the image used to build the pipeline, they need to
+be bootstrapped in the `spack.yaml`.  Then a newer `gcc` should also be
+added to the `spack.yaml` and used for the demonstration of installing
+a `gcc` with spack (and building all the packages with it in the subsequent
+steps).  That compiler will need to be bootstrapped as well.
+
+Check if `tcl` is still a 1-dependency package, if it's not, it should
+be replaced with one.
+
+Check the hash of: `spack spec zlib @1.2.8 cppflags=-O3`, that hash should
+be used in the line that installs `tcl` with a specific `zlib` dependency.
+
+Check the version of `clang` we get with `apt-get install clang`.  If it is
+no longer clang-6.0.0, then update the version of clang we bootstrap
+in the `spack.yaml` and the packages that built against clang to match.
+
+Make sure to update the release branch of spack checked out in the
+tutorial script.

docker/Dockerfile

@@ -8,17 +8,18 @@ ARG AWS_SECRET_ACCESS_KEY="None"
 RUN apt-get update -y && \
     DEBIAN_FRONTEND=noninteractive apt-get install -y --no-install-recommends \
         autoconf \
+        build-essential \
         ca-certificates \
         curl \
         clang \
-        clang-3.7 \
         emacs \
+        file \
         g++ \
-        g++-4.8 \
+        g++-6 \
         gcc \
-        gcc-4.8 \
+        gcc-6 \
         gfortran \
-        gfortran-4.8 \
+        gfortran-6 \
         git \
         gnupg2 \
         iproute2 \
@@ -37,22 +38,20 @@ RUN apt-get update -y && \
     apt-get autoremove --purge && \
     apt-get clean
 
-RUN mkdir -p /mirror/build_cache && \
-    aws s3 sync ${REMOTE_BUILDCACHE_URL} /mirror/build_cache
-
 COPY /public.key /mirror/public.key
-# COPY /packages.yaml /etc/spack/packages.yaml
+COPY /packages.yaml /etc/spack/packages.yaml
 COPY /tutorial-test.sh /tutorial/.test/tutorial-test.sh
 
-### TODO: Find another way to set perms without increasing the image download size
 RUN useradd -ms /bin/bash spack && \
+    mkdir -p /mirror/build_cache && \
+    aws s3 sync ${REMOTE_BUILDCACHE_URL} /mirror && \
     chmod -R go+r /mirror && \
-#    chmod -R go+r /etc/spack && \
+    chmod -R go+r /etc/spack && \
     chmod go+rx /tutorial/.test/tutorial-test.sh
 
 USER spack
 
 WORKDIR /home/spack
 
-CMD ["bash"]i
+CMD ["bash"]
 

docker/packages.yaml

+packages:
+  all:
+    target: [x86_64]
+

docker/tutorial-test.sh

@@ -15,7 +15,7 @@ rm -rf ~/.spack/*
 # clone a new spack
 git clone https://github.com/spack/spack
 cd spack
-git checkout releases/v0.12
+git checkout releases/v0.15
 
 # shell init
 . share/spack/setup-env.sh
@@ -28,13 +28,13 @@ spack mirror add tutorial /mirror
 spack gpg trust /mirror/public.key
 spack install zlib %clang
 spack install zlib @1.2.8
-spack install zlib %gcc@4.8
+spack install zlib %gcc@6.5.0
 spack install zlib @1.2.8 cppflags=-O3
 spack find
 spack find -lf
 spack install tcl
 spack install tcl ^zlib @1.2.8 %clang
-spack install tcl ^/64mn 
+spack install tcl ^/h6i5
 spack find -ldf
 spack install hdf5
 spack install hdf5~mpi
@@ -51,10 +51,10 @@ spack find -lf zlib
 spack find ^mpich
 spack find cppflags=-O3
 spack find -px
-spack install gcc@7.2.0
+spack install gcc@8.3.0
 spack find -p gcc
-spack compiler add `spack location -i gcc@7.2.0`
-spack compiler remove gcc@7.2.0
+spack compiler add `spack location -i gcc@8.3.0`
+spack compiler remove gcc@8.3.0
 
 # Packagin
 spack install mpileaks
@@ -62,13 +62,13 @@ spack install mpileaks
 # Modules
 spack install lmod
 source `spack location -i lmod`/lmod/7.8/init/bash
-spack compiler add `spack location -i gcc@7.2.0`
-spack install netlib-scalapack ^openmpi ^openblas %gcc@7.2.0
-spack install netlib-scalapack ^openmpi ^netlib-lapack %gcc@7.2.0
-spack install netlib-scalapack ^mpich ^openblas %gcc@7.2.0
-spack install netlib-scalapack ^mpich ^netlib-lapack %gcc@7.2.0
-spack install py-scipy ^openblas %gcc@7.2.0
-spack compiler remove gcc@7.2.0
+spack compiler add `spack location -i gcc@8.3.0`
+spack install netlib-scalapack ^openmpi ^openblas %gcc@8.3.0
+spack install netlib-scalapack ^openmpi ^netlib-lapack %gcc@8.3.0
+spack install netlib-scalapack ^mpich ^openblas %gcc@8.3.0
+spack install netlib-scalapack ^mpich ^netlib-lapack %gcc@8.3.0
+spack install py-scipy ^openblas %gcc@8.3.0
+spack compiler remove gcc@8.3.0
 
 # Advanced packaging
 spack install netlib-lapack

spack.yaml

@@ -42,7 +42,7 @@ spack:
   - $gcc_spack_built_packages
 
   mirrors:
-    cloud_mirror: 's3://spack-tutorial-container/mirror-v0.15.1'
+    cloud_mirror: 's3://spack-tutorial-container/mirror'
 
   gitlab-ci:
     bootstrap:
@@ -88,8 +88,6 @@ spack:
   modules:
     enable: []
   packages:
-    suite-sparse:
-      version: [5.3.0, 5.4.0, 5.5.0, 5.6.0]
     all:
       target: [x86_64]
   repos: []