NERD PSDVOP2025 Exercise

Context

As a NERD DevOps engineer, you are asked by a fellow developer to help restore the behavior of a Continuous Integration Service that was building and testing a C++ project around early 2019.
The goal is to ensure the job could run again in similar conditions.

The service you will be investigating is a fresh Jenkins instance containing a job that was imported from a backup.

Unfortunately, when a fresh instance of Jenkins with the plugins list updated, the project's job that was imported from a backup of the original Jenkins instance could not be run out-of-the box...

The exercise consists of the following steps:

• build a virtual machine with the given instructions.
• launch the VM, access the mentioned service, observe and explore...
• and solve the questions!

Consider each of those steps has part of the exercise. Some steps have not been detailed extensively on purpose.

To avoid installing too many required components on your environment, the service is bundled inside a VM that you'll have to build and run yourself. This will take about 10min depending on your settings (you can also skip the VM part and provision the exercise locally)

Software Requirements

The following software requirements have been selected to ensure a minimal residual footprint on your system:

• Hashicorp packer
• One of these hypervisors:
  • Oracle VM Virtualbox
  • VMware Workstation Pro

This exercise was built on the following recommended configurations and validated (with exceptions in italic):

Debian Linux (sid/trixie 2025-02)	Microsoft Windows 11 23H2
Hashicorp Packer 1.11.2	Hashicorp Packer 1.11.2
Oracle VM VirtualBox 7.0.20	Oracle VM VirtualBox 7.1.6
VMware Workstation 17.6.2	VMWare was not tested but should work oob
n/a	Local provisionning on WLS2 Debian 12
QEMU was not tested	QEMU was not tested

Architecture

Here are some pieces of information you need to get the service built and running:

• you're given a Packer manifest and files that will build a ready-to-use VM (see below section Building The Virtual Machine)

  • the VM can be connected to with an SSH client
  • the VM exposes a Jenkins Server on its port 80
  • the VM is provisionned using a mix of Packer hcl2 directives and Puppet manifest

• the service running inside the VM consists of 3 Docker containers:

  • a proxy server based on Nginx with a pretty minimal configuration that runs oob on the port 80 of the VM.
  • a Jenkins server used to orchestrate the build and test of the project source code
    • you've been told that the generated test binary might not run perfectly.
  • a Jenkins agent used to build and run the project that connects automatically to the Jenkins Server.

Building The Virtual Machine

Depending on the hypervisor you selected (VirtualBox or VMWare, no need to build both), use the appropriate build source for your packer build command :

packer build -only="SOURCE_TYPE.SOURCE_NAME" NERD_Debian_Exercise.pkr.hcl

This will build a VM in the build directory that you will be able to import and run in your selected hypervisor.

Note about Packer and Firewall configuration

Packer spawns a HTTP server to be able to serve the preseed file to the Debian VM. Keep this in mind as Firewall rules on your Windows or Linux system might block this feature.

Note about Packer and VMware Workstation on Windows

Packer seems to have an issue with VMware network configuration detection leading to an early failure.
This looks like Packer requires the presence of the netmap.conf to proceed and potentially, your VMware Workstation installation folder may not contain it.
Regenerating it is not that hard as you just have to go to Edit -> Virtual Network Editor -> Change Settings and just click OK.
This appears to do the trick for Packer.

Details about the provisioning mechanism

Even if packer is provisoning components, the system is mainly provisioned using a Puppet manifest and modules that are deployed and called within the Packer build step.
If you are comfortable with the matter of provisioning, you'll find all the Puppet manifest code in the puppet directory.

Provision the exercise locally

DISCLAIMER: This approach alters the software/user environment of the guest running packer.

If you already have a dedicated testbed running a Debian-based Linux (chroot, wsl, aws, rancher, qemu...), you can skip the VM building part, install only Packer and only provision Packer using the following command line

packer build -only="null.local" NERD_Debian_Exercise.pkr.hcl
# or if you run in a WSL2 vanilla environment (Docker must not run on the host though)
packer build -only="null.wsl" NERD_Debian_Exercise.pkr.hcl

Keep in mind that the process will install some packages and will try and start a Docker daemon so if you are on WSL2 on Windows, this may interfere with your local Docker Daemon (managing the Docker service might need some tweaking in the puppet manifest as well).

Questions

1. When is the last time the test_app job ran ?

Give the full date

2. When building the job again, can you describe and explain what is going differently than the previous run ?

Explain in a few lines

3. Now that you have assessed the situation, make the necessary modifications to make sure the Job environment and/or Job executable behaves in the same manner as it was before.

Explain in a few lines the steps you took and provide a package (git repository / patch files) containing your modifications and a note explaining your changes.

4. Describe another solution to make the Job environment and/or Job executable behave in the same manner as it was before.

Explain in a few lines and compare it against your first solution.

5. Now that everything is running as expected, it is time to suggest a production rollout for this service. Please suggest changes to the service to ensure a secure, production-ready deployment to be accessed by a team of developers. The suggestions are not limited in scope, but should focus on security, performance and reliability.

Explain in a few lines.

Notes

• For questions 3 and 4, there is more than one way to get things done and what's important here is for you to make sure your decisions are explained and justified.
• Your goal is to provide the components that will ensure the service runs in the best conditions.
• Changes should be documented as commits in the provided git repository (in the puppet directory) and delivered as the repository itself or patches; providing a VM, docker layer or a link to prebuilt containers should not be necessary.
• Consider keeping the puppet modules archive, docker and stdlib as-is as these are commonly found modules but there should be no need to modify them (except for update if deemed necesssary)
• Puppet module nerd content can be changed to suit changes in the deployed software environment. This includes but is not limited to:
  • Dockerfile templates, compose files
  • source code files
• Note that the Jenkins server instance Dockerfile template was succesfully bumped beforehand and runs a more recent version than the original one. The nginx and agent Dockerfile templates were not modified since 2019 and started without any issue.
  • The Jenkins server configuration can also be modified to suit your recommendations; Don't forget to provide us the patches/changes.

[Back to top]