Deploying lp-signing

The staging instance of lp-signing is deployed in PS5; the production instance runs on two bare-metal machines in the same datacentre, but is still deployed using the same mechanism. Each instance has its own management environment:

Instance	Management environment
production	prod-launchpad-signing@is-bastion-il3-not-ps5.internal
dogfood	stg-launchpad-signing@launchpad-bastion-ps5.internal

Both instances are deployed by developers using Mojo. The staging instance is managed by us directly, including the ability to juju ssh to units and inspect them. The production instance is managed by IS and we don’t have direct access to units (since they have business-critical uptime requirements and sensitive data), but we have an SSH trigger by which we can run ordinary upgrade operations.

These instructions are directed at developers who already have suitable access, which is controlled using Terraform.

The deployed machines are controlled by a combination of the lp-signing charm (built by a recipe on Launchpad) and configuration in the Juju bundle.

Deployable is used to track QA. The history of the Mojo spec serves as a log of changes.

Deploying to staging

Changes to the service payload made by new commits to master are automatically deployed to staging. However, sometimes you need to test changes to the charm, which are not automatically deployed.

1. Update the bundle.

   • Use juju info lp-signing to check what charm revisions are available, and set revision to the appropriate one (usually the latest).

   • Include any fixed bugs in your commit message.

   • If you’re only making changes of this form, then you can push them directly without needing a merge proposal.

2. As stg-launchpad-signing@launchpad-bastion-ps5.internal, run upgrade-staging.

3. Check the behaviour of http://staging.signing.lp.internal:8000/ as appropriate, perhaps via the dogfood publisher.

Deploying to production

1. Ensure that the same changes have been deployed to staging first, as above. If you need to deploy a change to the service payload, commit a change to the bundle to set build_label to the appropriate commit ID. (The automatic deployment mechanism on staging does the equivalent of this locally, but doesn’t commit it back to the bundle used for production deployments.)

2. Create a deployment request for the desired commit on deployable.

3. Announce the rollout in ~is-outage.

4. As stg-launchpad-signing@launchpad-bastion-ps5.internal, run upgrade-production. This should take on the order of five minutes.

5. Check production publisher logs as appropriate for any issues with the production signing service.

6. Change the status of any fixed bugs to “Fix Released”. (Note that fixing some bugs may be a multi-step process, so check whether they’re really done first.)

7. Mark the deployment request on deployable as deployed. (This has the effect of removing the deployed commits from the deployment queue.)

8. Ensure that bugs are filed for any deployment issues.