preloader
  • Bitbucket 'self-hosted' runner in Kubernetes environment
blog-thumb

Running self-hosted bitbucket runners on Kubernetes

Runners allow you to run pipelines from your repositories. Bitbucket allows you to host these runners in your own infrastructure, the so-called self-hosted runners.

In this article, we’ll walk you through how to create a workspace-level self-hosted runner and run it on Kubernetes.

Creating a runner in bitbucket

The first step is to create a new runner. Let’s create it at workspace level:

Navigate to your profile and click on the workspace you want.

bitbucket-image-01

Click on settings and select workspace runners.

bitbucket-image-02 bitbucket-image-03

Add runner.

bitbucket-image-04

Define the name for the runner and a label. The labels self.hosted and linux are defaults, we need to add a new label so we can identify in the pipeline which runner the build should be executed in, if there are other runners created.

Click on next. Output will be generated with the following docker command:

docker container run -it -v /tmp:/tmp -v /var/run/docker.sock:/var/run/docker.sock \
-v /var/lib/docker/containers:/var/lib/docker/containers:ro \
-e ACCOUNT_UUID={ACCOUNT_UUID}  \
-e RUNNER_UUID={RUNNER_UUID} \
-e RUNTIME_PREREQUISITES_ENABLED=true \
-e OAUTH_CLIENT_ID=OAUTH_CLIENT_ID \
-e OAUTH_CLIENT_SECRET=OAUTH_CLIENT_SECRET \
-e WORKING_DIRECTORY=/tmp \
--name runner-87c1a9ae-c216-5512-a92d-12fb95f77f0b \ docker-public.packages.atlassian.com/sox/atlassian/bitbucket-pipelines-runner:1

Copy this command, as we are going to use the values of the variables presented for the configuration of our self-hosted runner and click on finish.

Notice that the runner was created in the workspace, but its status is set to UNREGISTRED. After configuring the runner and running it in the Kubernetes environment, its status will change to ONLINE.

bitbucket-image-05

Kubernetes Configuration

The next step is to build the kubernetes files to host runner.

First let’s handle the variables given by the command:

-e ACCOUNT_UUID={ACCOUNT_UUID}  \
-e RUNNER_UUID={RUNNER_UUID} \
-e RUNTIME_PREREQUISITES_ENABLED=true \
-e OAUTH_CLIENT_ID=OAUTH_CLIENT_ID \
-e OAUTH_CLIENT_SECRET=OAUTH_CLIENT_SECRET \
-e WORKING_DIRECTORY=/tmp \ 
  • Copy the value of the variables ACCOUNT_UUID and RUNNER_UUID without the { }.
  • The variables OAUTH_CLIENT_ID and OAUTH_CLIENT_SECRET must be converted to base64.

echo -n OAUTH_CLIENT_ID | base64
echo -n OAUTH_CLIENT_SECRET | base64

Once this is done, we move on to the stage of building the Kubernetes files:

  • secret.yaml
  • job.yaml

Add variable values:

secret.yaml
apiVersion: v1
kind: Secret
metadata:
  name: runner-01-oauth-credentials
  labels:
    accountUuid: {ACCOUNT_UUID}
    runnerUuid: {RUNNER_UUID}
data:
  oauthClientId: {OAUTH_CLIENT_ID}
  oauthClientSecret: {OAUTH_CLIENT_SECRET}

job.yaml

Important: In the job.yaml file the accountUuid and runnerUuid variables, in the ‘container variables’ section, must be added between “ ” and { }, in this way “{ACCOUNT_UUID}”

apiVersion: batch/v1
kind: Job
metadata:
      name: runner-01
spec:
  template:
    metadata:
      labels:
         accountUuid: {ACCOUNT_UUID}
         runnerUuid: {RUNNER_UUID}
    spec:
      containers:
        - name: bitbucket-runner-01
          image: docker-public.packages.atlassian.com/sox/atlassian/bitbucket-pipelines-runner
        env:
          - name: ACCOUNT_UUID
            value: “{${ACCOUNT_UUID}}”
          - name: RUNNER_UUID
            value: “{${RUNNER_UUID}}”
          - name: RUNTIME_PREREQUISITES_ENABLED
            value: true
          - name: WORKING_DIRECTORY
            value: "/tmp"
          - name: OAUTH_CLIENT_ID
            valueFrom:
              secretKeyRef:
                name: runner-01-oauth-credentials
                key: oauthClientId
          -  name: OAUTH_CLIENT_SECRET
             valueFrom:
               secretKeyRef:
                 name: runner-01-oauth-credentials
                 key: oauthClientSecret
        volumeMounts:
          - name: tmp
            mountPath: /tmp
          - name: docker-containers
            mountPath: /var/lib/docker/containers
            readOnly: true
          - name: var-run
            mountPath: /var/run
      - name: docker-in-docker
        image: docker:20.10.7-dind
        securityContext:
          privileged: true
        volumeMounts:
          - name: tmp
            mountPath: /tmp
          - name: docker-containers
            mountPath: /var/lib/docker/containers
          - name: var-run
            mountPath: /var/run
      restartPolicy: OnFailure
      volumes:
        - name: tmp
        - name: docker-containers
        - name: var-run
  backoffLimit: 6
  completions: 1
  parallelism: 1

Now let’s apply the configurations built above.

kubectl -n namespace apply -f secrets.yaml
kubectl -n namespace apply -f job.yaml

Testing the new runner


pipelines:
  branches:
    'main':
      - step:
        runs-on: my.custom.label
        script:
          - echo "Hello world"

Your self-hosted runner is ready!

For more information about runners visit the bitbucket documentation runners.

Need help?

Access our page Contact and chat with us.

Success!

Additional reading:

Pipelines runners frequently asked questions