Commit 17b2cb14 authored by Scott Wittenburg's avatar Scott Wittenburg Committed by Todd Gamblin
Browse files

Add more setup documentation

parent 60caea7b
......@@ -39,6 +39,8 @@ See the spack pipeline [documentation](
Because we want to use a custom spack (one other than the runners may already have available), we add the `SPACK_REPO` and `SPACK_REF` environment variables listed above. We then use those variables to clone spack in the pre-ci phase, as well as pass them along to the `spack ci generate` command so that custom spack will be cloned and activated as a part of the `before_script` of each generated pipeline job.
More detailed instructions on setting up the Gitlab CI/CD only repo can be found [here](./
## Spack mirror
This workflow makes use of a public mirror hosted on AWS S3. The automated pipeline pushes binaries to the mirror as packages are successfully built, and then the auto-build on DockerHub pulls those and copies them into the image for publishing. To authenticate with the S3 bucket, authentication credentials should be stored in CI variables as described in the section above.
# Setting up the Gitlab CI/CD Repo
This document describes the steps required to setup a Gitlab repo that
will run pipelines against an external GitHub repo whenever you make a
PR or push a commit on an open PR on that repo. Gitlab provides some
documentation on the process as well, you can read that
[here]( At a high
level, this document covers creating the repository, linking to the desired
GitHub repo, and how to set up necessary environement variables. It also
mentions some issues you may encounter as well as ways to work around them.
## Create the project
Start by navigating to your projects and clicking the green "New Project"
button near the upper right corner of the page.
![New Project](./images/new-project-top-only.png "New Project Page")
Next click on the "CI/CD for external repo" tab, also located near the top
right of the page. Then, under "Connect repositories from", click the
"GitHub" button.
![CI/CD Project](./images/ci-cd-external-repo.png "CI/CD for external repo")
For the next step you will need to have created a personal access token in
GitHub previously. Paste that token into the field marked "Personal Access
Token", and then click the green button labeled "List your GitHub repositories".
![Import Repositories](./images/import-repos-from-github-need-token.png "List GitHub repositories")
On the next page you should see a list of all the repositories under all
organizations to which you have access. Find the project you want Gitlab
to mirror and run pipelines for, and then click its "Connect" button in the
right column.
![Connect to Repository](./images/imported-repositories.png "Connect to repository of interest")
At this point, you can move on to configuring settings in the new Gitlab
CI/CD repository.
## Configure the project settings
This section guides you through configuring various project settings.
### Variables
The most important configuration requirement is to set up the necessary
CI/CD variables your automated pipelines will need. The spack [pipelines
describes the individual variables in more detail.
![Set CI Variables](./images/setup-ci-variables.png "Set CI Variables")
### General pipelines
To make sure your packages have enough time to build before Gitlab cancels
the job, you probably want to extend the job timeout. You may also want to
change how Gitlab gets your project (choose "clone" so the repo is in a
fresh state for every job). These settings can be found under the CI/CD
section of your project settings after you expand the "General Pipelines"
![Project Settings](./images/general-pipelines-settings.png "Project Settings")
### Auto DevOps
Since you'll be providing a valid CI configuration file, it is safe and
removes clutter from your pipelines to disable "Auto DevOps". To do this
(optional), navigate to the CI/CD section of your project settings and
expand the "Auto DevOps" section. Uncheck the box labeled "Default to
Auto DevOps pipeline", and click the "Save Changes" button.
![Disable Auto DevOps](./images/turn-off-autodevops.png "Disable Auto DevOps")
## Creating a PR on GitHub
At this point your Gitlab CI/CD project is set up for running pipelines
against your external GitHub repository. To actually have it run a
pipeline, you need to create a PR on the GitHub repo. Normally the
GitHub repo will contain a simple `.gitlab-ci.yml` file as well as a
`spack.yaml` (a spack environement) you want to build. Those files are
described in more detail in this
of the spack pipelines documentation.
Once Gitlab has noticed your PR (or pushed commit to a PR branch) and started
a pipeline for you, it will also create a status check on your GitHub PR.
![Pipeline Status Check](./images/status-check-github.png "Pipeline Status Check")
## Miscellaneous Issues
One issue you will encounter is related to how spack generates a pipeline
in a "pre-ci" phase. Until Gitlab provides the ability to dynamically
generate child pipelines, spack is relying on pushing a commit with the
generated workload to a `DOWNSTREAM_CI_REPO` (usually the same repo, but
it's up to you). So as soon as the "pre-ci" phase (job generation) completes
successfully, Gitlab will report success to your PR, and it will become
Sometimes it seems to take Gitlab awhile to notice your new PR or a commit
you've pushed to a PR branch. If this happens, you can try asking Gitlab
to refresh the mirrored repository manually. To do this, navigate to the
"Repository" section of your Gitlab project settings and expand the "Mirroring
repositories" section. At the bottom of that section, you should find your
linked external GitHub repo, and on the right is a button allowing you to
"Update now".
![Update mirrored repo](./images/refresh-mirrored-repo.png "Update mirrored repo")
Markdown is supported
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment