Unverified Commit 22d9fc1c authored by Greg Becker's avatar Greg Becker Committed by GitHub

Merge pull request #4 from spack/validate-rev-aws20

Validate pipeline for aws20 container
parents a7e5c96c fae46e0d
......@@ -12,11 +12,11 @@ There are two build rules (see this [section](./DOCKERHUB_SETUP.md#configure-bui
for more info) governing when DockerHub will rebuild the container. The
first specifies that any time you push to the `master` branch on this repo,
DockerHub will build an image and tag it with `latest`. The second build rull
specifies that any time you push a tag matching the regular expression
specifies that any time you push a branch matching the regular expression
`^rev-(.+)$`, DockerHub will build an image and use the matched group
(everything in the tag following the `rev-`) to tag the image. So if, for
example, you want a container to be tagged `sc20`, you would tag this
repository `rev-sc20` and push it.
(everything in the branch name following the `rev-`) to tag the image. So if,
for example, you want a container to be tagged `sc20`, you would push a branch
to this repository named `rev-sc20`.
## Steps to build an updated container
......@@ -36,12 +36,15 @@ the helper script, run the following command from the root of the repository:
$ ./keyhelp/export_pipeline_key.sh ./docker "name-of-key-to-export"
1. Create a new mirror (this is required until the sync process between the
mirror and the container build cache is controlled by/limited to the spack
environment in which it runs). The mirror url currently needs to be updated
1. Either a) create a new mirror, or else b) clean out the existing one. This
is required until the sync process between the mirror and the container build
cache is controlled by/limited to the spack environment in which it runs). If
you choose to create a new mirror, the mirror url currently needs to be updated
in two locations: The `spack.yaml` and the DockerHub autobuild environment
variables (see [here](./DOCKERHUB_SETUP.md#configure-builds) for more
information).
information). If you choose to clean out the old mirror, just use the web
interface to remove the entire "build_cache" directory, and then there is
nothing else to change.
1. Create a new buildgroup in CDash to track the builds from the version to be
(done by editing the `spack.yaml` and changing the `build-group` property)
......@@ -54,3 +57,28 @@ the packages build.
1. Merge the PR into master and/or tag the result using the tagging scheme
described in the section above.
## Items that may require updates as OS/Spack version change
Check what version of `gcc` gets installed by `apt-get` by default if you
don't specify a version. That should be the compiler the tutorial picks
to install most packages. Then an older `gcc` should be installed and used
to install zlib with the "custom" system compiler. If either of those
compilers aren't in the image used to build the pipeline, they need to
be bootstrapped in the `spack.yaml`. Then a newer `gcc` should also be
added to the `spack.yaml` and used for the demonstration of installing
a `gcc` with spack (and building all the packages with it in the subsequent
steps). That compiler will need to be bootstrapped as well.
Check if `tcl` is still a 1-dependency package, if it's not, it should
be replaced with one.
Check the hash of: `spack spec zlib @1.2.8 cppflags=-O3`, that hash should
be used in the line that installs `tcl` with a specific `zlib` dependency.
Check the version of `clang` we get with `apt-get install clang`. If it is
no longer clang-6.0.0, then update the version of clang we bootstrap
in the `spack.yaml` and the packages that built against clang to match.
Make sure to update the release branch of spack checked out in the
tutorial script.
......@@ -8,17 +8,18 @@ ARG AWS_SECRET_ACCESS_KEY="None"
RUN apt-get update -y && \
DEBIAN_FRONTEND=noninteractive apt-get install -y --no-install-recommends \
autoconf \
build-essential \
ca-certificates \
curl \
clang \
clang-3.7 \
emacs \
file \
g++ \
g++-4.8 \
g++-6 \
gcc \
gcc-4.8 \
gcc-6 \
gfortran \
gfortran-4.8 \
gfortran-6 \
git \
gnupg2 \
iproute2 \
......@@ -37,22 +38,20 @@ RUN apt-get update -y && \
apt-get autoremove --purge && \
apt-get clean
RUN mkdir -p /mirror/build_cache && \
aws s3 sync ${REMOTE_BUILDCACHE_URL} /mirror/build_cache
COPY /public.key /mirror/public.key
# COPY /packages.yaml /etc/spack/packages.yaml
COPY /packages.yaml /etc/spack/packages.yaml
COPY /tutorial-test.sh /tutorial/.test/tutorial-test.sh
### TODO: Find another way to set perms without increasing the image download size
RUN useradd -ms /bin/bash spack && \
mkdir -p /mirror/build_cache && \
aws s3 sync ${REMOTE_BUILDCACHE_URL} /mirror && \
chmod -R go+r /mirror && \
# chmod -R go+r /etc/spack && \
chmod -R go+r /etc/spack && \
chmod go+rx /tutorial/.test/tutorial-test.sh
USER spack
WORKDIR /home/spack
CMD ["bash"]i
CMD ["bash"]
packages:
all:
target: [x86_64]
......@@ -15,7 +15,7 @@ rm -rf ~/.spack/*
# clone a new spack
git clone https://github.com/spack/spack
cd spack
git checkout releases/v0.12
git checkout releases/v0.15
# shell init
. share/spack/setup-env.sh
......@@ -28,13 +28,13 @@ spack mirror add tutorial /mirror
spack gpg trust /mirror/public.key
spack install zlib %clang
spack install zlib @1.2.8
spack install zlib %gcc@4.8
spack install zlib %gcc@6.5.0
spack install zlib @1.2.8 cppflags=-O3
spack find
spack find -lf
spack install tcl
spack install tcl ^zlib @1.2.8 %clang
spack install tcl ^/64mn
spack install tcl ^/h6i5
spack find -ldf
spack install hdf5
spack install hdf5~mpi
......@@ -51,10 +51,10 @@ spack find -lf zlib
spack find ^mpich
spack find cppflags=-O3
spack find -px
spack install gcc@7.2.0
spack install gcc@8.3.0
spack find -p gcc
spack compiler add `spack location -i gcc@7.2.0`
spack compiler remove gcc@7.2.0
spack compiler add `spack location -i gcc@8.3.0`
spack compiler remove gcc@8.3.0
# Packagin
spack install mpileaks
......@@ -62,13 +62,13 @@ spack install mpileaks
# Modules
spack install lmod
source `spack location -i lmod`/lmod/7.8/init/bash
spack compiler add `spack location -i gcc@7.2.0`
spack install netlib-scalapack ^openmpi ^openblas %gcc@7.2.0
spack install netlib-scalapack ^openmpi ^netlib-lapack %gcc@7.2.0
spack install netlib-scalapack ^mpich ^openblas %gcc@7.2.0
spack install netlib-scalapack ^mpich ^netlib-lapack %gcc@7.2.0
spack install py-scipy ^openblas %gcc@7.2.0
spack compiler remove gcc@7.2.0
spack compiler add `spack location -i gcc@8.3.0`
spack install netlib-scalapack ^openmpi ^openblas %gcc@8.3.0
spack install netlib-scalapack ^openmpi ^netlib-lapack %gcc@8.3.0
spack install netlib-scalapack ^mpich ^openblas %gcc@8.3.0
spack install netlib-scalapack ^mpich ^netlib-lapack %gcc@8.3.0
spack install py-scipy ^openblas %gcc@8.3.0
spack compiler remove gcc@8.3.0
# Advanced packaging
spack install netlib-lapack
......
......@@ -42,7 +42,7 @@ spack:
- $gcc_spack_built_packages
mirrors:
cloud_mirror: 's3://spack-tutorial-container/mirror-v0.15.1'
cloud_mirror: 's3://spack-tutorial-container/mirror'
gitlab-ci:
bootstrap:
......@@ -88,8 +88,6 @@ spack:
modules:
enable: []
packages:
suite-sparse:
version: [5.3.0, 5.4.0, 5.5.0, 5.6.0]
all:
target: [x86_64]
repos: []
......
Markdown is supported
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment