Troubleshooting#
What to do when a Preflight job fails or a gate behaves unexpectedly. Each entry names the cause, the fix, and how to check it. Most failures are a pin that moved or a setting that is missing. The checklist catches the rest.
The pipeline#
The include fails on another GitLab instance#
Components work only inside the GitLab instance that hosts this project, and they cannot be used across instances except by mirroring.
- Cause: the app lives on a different instance than the one that hosts Preflight.
- Fix: host the app on the same instance. For a package that is also mirrored to GitHub, use the reusable workflow at
.github/workflows/php-package.ymlin this repo instead of the components. - Check: the include path starts with
$CI_SERVER_FQDN/enterprise-apps/preflight/, and the app's project is on that same instance.
The components do not run on merge requests#
The components are ordinary jobs, so they run whenever the app's pipeline runs. Whether the pipeline runs on a merge request is decided by a workflow:rules block, and a component cannot set it for the project that includes it.
- Cause: the app's own
.gitlab-ci.ymlhas noworkflow:rulesblock with amerge_request_eventrule. - Fix: add the block to the app's own file:
workflow:
rules:
- if: $CI_PIPELINE_SOURCE == "merge_request_event"
- if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH
- Check: a merge request pipeline runs, and so does a push to the default branch.
A gate was skipped or weakened#
An included component is a set of ordinary jobs. The app's own .gitlab-ci.yml is merged over them, so an app can override a job's keys, skip it with rules, or delete the include: line. Preflight makes the safe pipeline the easy one. It does not lock it.
- Cause: a change in the app's own
.gitlab-ci.ymloverrides, skips or removes a component job. - Fix: review the diff of the app's
.gitlab-ci.ymlagainst the last known-good pipeline, and restore the job or theinclude:line. - Check: Rollcall is where skipped or weakened gates show up.
Two includes of the same component became one job#
Every component takes a job_name input, whose default is the job's usual name. Without it, two includes of the same component share a job name and merge into one.
- Cause: one project includes the same component more than once (for example in a monorepo with two apps) without setting
job_name. - Fix: set
job_nameon each include, for examplepint-appandpint-admin.job_nameships in 0.1.4, so pin 0.1.4 or a SHA at or after that release. - Check: the pipeline lists one job per include, with the names you set.
Pipeline creation fails on the unknown spec: component: key#
Every Preflight component declares spec: component:, and GitLab only accepts that key from 18.7 on. On an older instance the pipeline never starts: creation fails with "header:spec config contains unknown keys".
- Cause: the GitLab instance is older than 18.7.
- Fix: upgrade GitLab to 18.7 or later, or pin an older Preflight tag that does not declare
spec: component:. - Check: the instance banner shows 18.7 or later, and the pipeline creates and runs the component jobs.
A fork merge request ran with protected variables#
A fork's merge request pipeline runs the fork's configuration.
- Cause: the project lets fork pipelines see protected variables.
- Fix: review a fork's
.gitlab-ci.ymlbefore running its pipeline, and never give it protected variables. - Check: the fork pipeline's variable list does not include any protected variable.
Job failures#
A job misbehaves: which Preflight release ran?#
Every Preflight job prints a Preflight component <name> <version> (ref <reference>, sha <sha>) line before any other command. When a job fails or behaves unexpectedly, read that first line to see which Preflight release produced it: <name> is the component, <version> the pinned release tag, <reference> the ref the include points at, and <sha> the commit SHA of the template that ran. Compare it with the pin in the app's .gitlab-ci.yml; a mismatch means the include moved or the pipeline ran on an older pin.
- Cause: the job log does not say which Preflight release ran, so the failure cannot be tied to a pin.
- Fix: read the
Preflight componentline at the top of the job log. If the line is missing, the include is pinned to a Preflight release from before the line shipped; pin the current tag. - Check: the first line of the job log names the component, its version, reference and SHA, and they match the pin in the app's
.gitlab-ci.yml.
PROCTOR's preflight-pinned check warns#
A tag or branch can move. A SHA cannot. PROCTOR's preflight-pinned check warns when an include is not pinned to a commit SHA.
- Cause: the include is pinned to a tag or a branch instead of a full commit SHA.
- Fix: pin the full commit SHA and keep the tag in a trailing comment, as in the examples on the Components page.
- Check: PROCTOR's report no longer carries the
preflight-pinnedfinding.
composer-audit is green, then an advisory lands#
New advisories are published after code is merged, so a green pipeline today can be stale tomorrow.
- Cause: no pipeline has run since the advisory was published.
- Fix: create a weekly pipeline schedule on the default branch: project settings, Build, Pipeline schedules. The component needs nothing extra. It carries no
rules:block, so it runs in scheduled pipelines like any other. Someone with Maintainer rights on the app creates the schedule. - Check: the schedule appears under project settings, Build, Pipeline schedules, and its latest pipeline is green.
composer-audit fails on an abandoned package#
The job log names an abandoned package in the lock. An abandoned package is one its author no longer maintains, and composer audit fails the job on one by default.
- Cause: the lock holds a package marked abandoned, and the
abandonedinput is at its default,fail. - Fix: upgrade the package or move to its successor where the log names one. While that is not possible, set the
abandonedinput on the include:reportlists the abandoned packages without failing the job (right for an old app with abandoned transitive packages),ignoredoes not look at them at all. Advisories still fail the job in every mode. - Check: with
report, the job is green and its log lists the abandoned packages; withfail, the job fails until the lock no longer holds one.
npm-audit fails with "audit endpoint returned an error"#
The job log ends with npm-audit: the registry's audit endpoint could not be reached or refused the request, so nothing was audited.
- Cause: the runner cannot reach the npm registry over HTTPS, or the app's
.npmrcnames a registry that does not serve npm's audit endpoints (/-/npm/v1/security/advisories/bulk). - Fix: let the runner reach the registry the log names, or point the audit at one that serves the endpoints. The job never passes without an answer.
- Check: the job log shows npm's report (
found 0 vulnerabilitiesor a list) instead of the error.
npm-audit fails on a dev-only package#
- Cause:
omit_devisfalse, so the gate audits dev dependencies too. - Fix: upgrade the package, or leave
omit_devat its default,true: the gate then reads runtime dependencies only, and the full report after it still lists the dev finding without failing. - Check: the gate's lines say
failing on high or higher in runtime dependencies.
composer install stops: the lock needs another PHP or an extension#
The job log stops at composer install with your php version (8.4.x) does not satisfy that requirement, or with requires ext-gd * -> it is missing from your system (gd, oci8 and ldap are the usual ones). The php-test, pint and phpstan jobs install the app to run it, so they need what the app needs.
- Cause: the lock was built for an older PHP, or the app needs a PHP extension that the default image does not have. The default
php:*-cliandcomposerimages carry only the extensions PHP builds in. - Fix: for an old PHP, upgrade the app: that red job is the finding, not a fault in Preflight. For an extension, set the job's
imageinput to an image that has the same extensions as the app's Citadel image, for example one builtFROM php:8.4-cli-trixiewithdocker-php-ext-install gd. Theproctorandsbomjobs never run the app, so they install with--ignore-platform-reqsand need neither. - Check: the
composer installstep of the job passes, and the job fails, if at all, on its own check.
composer install cannot clone a private package#
The job log shows Failed to execute git clone --mirror for a repository on the app's own GitLab instance, or Could not connect to server.
- Cause: the lock installs a package straight from a private Git repository, and the job has no credential for it. A local
auth.jsonnever reaches CI. - Fix: give Composer a credential in the pipeline, for example a masked CI/CD variable
COMPOSER_AUTHholding anhttp-basicentry for the instance with a deploy token that can read the package's project. Prefer publishing the package to the GitLab package registry and listing only that project in the job token allowlist; see Keep the job token allowlist short. - Check: the
composer installstep passes in a pipeline on a branch.
eslint stops at npm ci#
The eslint job stops at npm ci with the usage text, before ESLint runs.
- Cause:
npm cineeds apackage-lock.json, and the app keeps only ayarn.lockor apnpm-lock.yaml. - Fix: commit a
package-lock.json(runnpm installonce), or leave theeslintcomponent out until the app moves to npm. - Check:
npm ci --ignore-scriptspasses locally from a clean checkout.
php-test-next is red#
A red php-test-next job is an early warning, not a stop. It is the second php-test include that tries the PHP release that is not supported yet, with allow_failure: true in the app's own file.
- Cause: the next PHP release still fails the app's tests, or only has release candidate images.
- Fix: leave
allow_failure: truein place while the release is unsupported. Once the release is supported (the supported versions followpolicy.json), drop theallow_failureand move the version into the mainphp_versionslist. - Check: the main
php-testmatrix is green. Only the namedphp-test-nextjob is red.
package-publish refuses the tag#
The job reads the registry before publishing. Republishing a version from the same commit is a no-op, and republishing it from a different commit fails instead of overwriting the version, because GitLab would otherwise overwrite it without a word.
- Cause: the tag's version already exists in the group's Composer registry, published from a different commit.
- Fix: publish a new version tag instead of moving the old one. Protect the release tags so only maintainers can create or move a tag.
- Check: the job log shows the version already on the manifest and the commit it was published from.
portability says an input is unknown#
The one-job portability component has new inputs, and buildah_image is now image (the old app-image input is gone).
- Cause: the include is pinned to 0.1.4, which carries the two-job form (
portability-build,portability-run, goss), or the include still setsbuildah_image. - Fix: pin a SHA at or after the one-job form and follow Upgrading:
buildah_imagebecomesimage, and the old appimageinput is gone because nothing is pushed. - Check: the pipeline lists one job,
portability.
portability fails before the build on the runner#
Buildah's chroot isolation opens a user namespace.
- Cause: the runner (or Docker's seccomp profile on a laptop) refuses user namespaces; the log shows
unshare(CLONE_NEWUSER): Operation not permitted. - Fix: run the job on a runner that permits user namespaces, which ordinary unprivileged Docker-executor runners do. Do not make the runner privileged for it. Locally, add
--security-opt seccomp=unconfinedtodocker run. - Check: the log reaches
buildah budand prints the build steps.
portability: health: FAIL on a Laravel app#
The probe asks for the health path with a bare IP address in Host (Host: <ip>:<port>), the way a platform's probe does.
- Cause: the trusted-hosts middleware, an HTTPS redirect, a session or an auth check runs before the health route.
- Fix: answer the health route before those, for example Laravel's
health: '/up'route inbootstrap/app.php, and exempt it from trusted hosts and HTTPS redirects. - Check:
portability: health: pass.
portability: sigterm: FAIL#
The main process must handle SIGTERM and exit by itself within stop_timeout, because PID 1 ignores a signal it has no handler for.
- Cause: the image
execs a server that has no SIGTERM handler (for examplephp -S), or a shell script starts the server without passing the signal on. - Fix: start the app under an init that forwards signals (
tini,dumb-init) or a server that handles SIGTERM itself (php-fpm, FrankenPHP); in a shell entrypoint, trap TERM and stop the child. - Check:
portability: sigterm: passwithhandled SIGTERM and exited by itself.
portability: an Apache image fails after the upgrade to 0.5.0#
Since 0.5.0 the job gives each replica stdio its uid owns and a session of its own, so the start_command shim older releases needed is in the way.
- Cause:
start_commandstill runssetsid(for exampleexec setsid bash -c 'exec > >(exec cat) 2>&1; exec squire-entrypoint'). Inside the job's launcher the shell already leads its own session, sosetsidforks the app away from the replica's main process, and the replica never answers:healthfails withlast answer: 000. - Fix: delete the
start_commandinput, so the job runs the image's ownENTRYPOINTandCMD. - Check:
portability: health: passandportability: sigterm: pass.
portability: session-hop: FAIL with "set session_login_path"#
session-hop needs a route that signs in a test user and one that answers 200 only for a signed-in user.
- Cause:
session_login_pathandsession_check_pathare empty, butsession-hopis inchecks. - Fix: add the two routes behind an environment variable only the job sets (see The portability component), or, for an app with no sign-in, leave
session-hopout ofchecks. - Check:
portability: session-hop: pass, orskippedwhen left out on purpose.
a11y fails on a page that looked fine#
The job checks each page in light and dark mode at 320 px and 1280 px, with reduced motion, so it sees layouts and palettes a quick look in one browser misses.
- Cause: a fault that shows in one mode only, and the failing test case names that mode, for example
docs/index.html dark 320px: a dark palette with weak contrast, a label hidden at phone width, or a fixed-width element that makes the page scroll sideways at 320 px (preflight-reflow, which names the element where the overflow starts). - Fix: open the page in that mode (dark color scheme, a 320 px window) and fix the element the report names; each axe-core rule links its help page. While a site works through its first findings, set
allow_failure: trueon the include and remove it once the job is green. - Check: the job's last line reads
a11y: clean, and the merge request's test summary shows every case passing.
a11y stops with a configuration error (exit 2)#
Exit 2 means the job never judged a page: an input or a tool is wrong.
- Cause: one of the messages below.
no file under site matches the page pattern ...: a glob inpagesmatched nothing, often because the docs folder moved. A literal page that is missing fails its check with HTTP 404 instead.site folder not found:site_dirdoes not exist in the job, for example because the site is built in another job and not passed as an artifact.cannot read the Playwright version from the image tag:imagewas overridden with an image whose tag does not carryvX.Y.Z.Chromium did not start: the playwright npm version does not match the browsers in the image.axe-core X ran, but axe_core_version is Y:axe_playwright_versionnames a bridge whose axe-core range does not includeaxe_core_version.
- Fix: correct the input the message names: fix the glob or
site_dir, setplaywright_versionto the Playwright release in the image, or moveaxe_core_versionandaxe_playwright_versiontogether (Renovate bumps them as one group). - Check: the job log's second line names the site or URL and the Playwright version, and the summary line names the axe-core version that ran.
squire-kit fails: the kit drifted (exit 1)#
The job installs the Squire release the kit records and runs squire init --check; the log lists every kit file with its state.
- Cause: one of the states below.
modified: someone, or a bot, edited a file Squire owns. Renovate's digest and tag bumps to.squire/files anddocker-compose.squire.ymlwere the usual culprit before the preset turned them off.outdated: the file is untouched but is not the template of the recorded release, usually because.squire.jsonwas bumped withoutsquire update.missing: a kit file is gone, or the release added one thatsquire updatedoes not write.untracked: a file sits in one of the kit's groups but is not recorded in.squire.json.
- Fix: for
outdatedandmissing, runvendor/bin/squire update --path=deploy, thenvendor/bin/squire init --path=deploy, review the diff and commit it. Formodified, put Squire's file back withvendor/bin/squire init --force --path=deployand send the change to Squire instead, so every kit gets it. While a repository works through its first findings, setallow_failure: trueon the include and remove it once the job is green. - Check:
vendor/bin/squire init --check --path=deployprintsRESULT: pass, and so does the job's last line.
squire-kit stops before checking (exit 2)#
Exit 2 means the job never compared a file.
- Cause: one of the messages below.
no .squire.json in ...:working_diris not the kit folder. The stack's repos keep the kit indeploy..squire.json records no released Squire version: the manifest is damaged or records a branch.could not install waynestate/squire X from Packagist: no such release, or the job cannot reach Packagist.
- Fix: set
working_dirto the folder holding.squire.json, setsquire_versionto the release the kit was written with, or let the job reach Packagist. - Check: the job log prints
Checking the kit in ... against Squire Xbefore the file list.
Renovate#
Renovate sees literal text instead of a token#
On GitLab 19.5 a self-reference such as RENOVATE_TOKEN: $RENOVATE_TOKEN in the job variables: arrives as literal text and shadows the masked project CI/CD variables. The component does not map either token on purpose. The job reads both straight from the project.
- Cause: the runner project, or the app's file, adds
RENOVATE_TOKEN: $RENOVATE_TOKENorGITHUB_COM_TOKEN: $GITHUB_COM_TOKENtovariables:. - Fix: delete the self-reference. Keep
RENOVATE_TOKEN(and optionallyGITHUB_COM_TOKEN) as masked CI variables in the private runner project. - Check: the job log no longer shows the literal text
$RENOVATE_TOKEN, and Renovate authenticates.
Renovate auto-merged a component pin bump#
In refs before 0.1.4 the component-rule manager name in renovate/default.json did not match repository deps, so a bump of a component: include version auto-merged instead of waiting for review.
- Cause: the shared preset comes from a ref before the 0.1.4 fix.
- Fix: point the repositories'
renovate.jsonat the shared preset from 0.1.4 or later. Component version bumps now carry theneeds-reviewlabel and never auto-merge. - Check: the merge request carries the
needs-reviewlabel.
A Squire kit merge request shows an artifact error#
The preset's Squire rule runs two commands after it bumps .squire.json: it installs the new Squire into /tmp and runs squire update on the kit folder. The merge request reports an artifact error when it could not.
- Cause: the runner does not allow the commands (the message says the command
has not been added to the allowed list in allowedCommands; the renovate component'ssquire_kit_tasksinput is off), orsquire updaterefused a kit file someone edited (the message names the file), or the release needs PHP or a network the runner lacks. - Fix: check out the branch, run
vendor/bin/squire update --path=deploy(andvendor/bin/squire init --path=deployfor a release that adds a file), review the diff and commit it to the branch. For an edited file, restore it first withvendor/bin/squire init --force --path=deploy. To let the runner do this itself, setsquire_kit_tasks: trueon its renovate include. - Check: the
squire-kitjob on the merge request is green.
Local checks#
The portability fixtures fail with "buildah is required"#
tests/check-portability.sh runs inside the component's Buildah image, not on the host.
- Cause: the script was run with
shon the host. - Fix: run it through
docker runin the Buildah image as Local checks shows, or runpwsh -File tests/check-portability.ps1on Windows. - Check: one result line per case, and the script exits 0.
composer ci fails on a stale docs site#
composer ci runs docs:check, which fails when the built docs site is stale or the generated command or environment reference has drifted from the CLI or source.
- Cause: a page under
docs/(or a generated reference) changed without rebuildingsite/docs/. - Fix: run
composer docsto rebuildsite/docs/fromdocs/, and commit the rebuilt site with the page. - Check:
composer docs:checkpasses.