github continuous integration

1. Motivation
2. Contents
3. Progressive Implementation

1. Motivation

We want to use a custom (docker) container with github actions, and prepare it using nix.

Several advantages:

control over build software (e.g. gcc version).
reproducibility – nix-to-container pipeline guarantees perfect reproducibility
flexibility – container can include prepared software that isn't available on ubuntu
size – container with no excess baggage

2. Contents

scheme	builder	src-repo	action-repo
scheme 1	ubuntu	hello-example	(same)
scheme 2	ubuntu	docker-action-example	(same)
scheme 3	container	docker-nix-hello	docker-action-example2
scheme 4	container	docker-cpp-builder	docker-action-example3

3. Progressive Implementation

We'll present several progressive CI-with-github examples, that converge on goals above.

3.1. Scheme 1 - github build using base o/s platform

3.1.1. Preliminaries

source for Scheme 1 is on github here: https://github.com/Rconybea/hello-example

Directory structure

hello-example
+- .github
|  \- workflow
|     \- main.yml
+- CMakeLists.txt
\- hello.cpp

Source

// hello.cpp

#include <iostream>

using namespace std;

int
main(int argc, char * argv[]) {
    std::string subject = "World";

    if (argc > 1)
        subject = argv[1];

    cout << "Hello, " << subject << "!" << std::endl;
} /*main*/

Cmake build files

# CMakeLists.txt

cmake_minimum_required(VERSION 3.10)

project(hello-example VERSION 1.0)

enable_language(CXX)

# write compile_commands.json for LSP
set(CMAKE_EXPORT_COMPILE_COMMANDS ON CACHE INTERNAL "")

set(SELF_EXE hello-example)
set(SELF_SOURCE_FILES hello.cpp)

add_executable(${SELF_EXE} ${SELF_SOURCE_FILES})

install(TARGETS ${SELF_EXE} DESTINATION bin)

if(CMAKE_EXPORT_COMPILE_COMMANDS)
    # include otherwise-omitted system directories in compile_commands.json,
    # so LSP knows exactly what compiler is using
    set(CMAKE_CXX_STANDARD_INCLUDE_DIRECTORIES ${CMAKE_CXX_IMPLICIT_INCLUDE_DIRECTORIES})
endif()

Build + Run

$ cd hello-example
$ mkdir build
$ cmake ..
-- Configuring done
-- Generating done
-- Build files have been written to: /home/roland/proj/hello-example/build
$ make
[ 50%] Building CXX object CMakeFiles/hello-example.dir/hello.cpp.o
[100%] Linking CXX executable hello-example
[100%] Built target hello-example
$ ./hello-example
Hello, World!

Continuous Integration with Github

We need to do two things:

setup a github repo holding our sources:

$ cd hello-world
$ git remote -v
origin  git@github.com:rconybea/hello-example.git (fetch)
origin  git@github.com:rconybea/hello-example.git (push)

configure github actions for that repo We add one file, hello-world/.github/workflows/main.yml

name: c++ build with cmake

on:
  push:
    branches: ["main"]
  pull_request:
    branches: ["main"]

env:
  BUILD_TYPE: Release

jobs:
  build_job:
    runs-on: ubuntu-latest

    steps:
    - name: checkout self (hello-example)
      uses: actions/checkout@v3

    - name: configure self (hello-example)
      run: cmake -B ${{github.workspace}}/build -DCMAKE_INSTALL_PREFIX=${{github.workspace}}/local -DCMAKE_BUILD_TYPE=${{env.BUILD_TYPE}}

    - name: build self (hello-example)
      run: cmake --build ${{github.workspace}}/build --verbose

    - name: test self (hello-example)
      working-directory: ${{github.workspace}}/build
      run: ./hello-example

github will automatically trigger a build whenever new sources are pushed. build will run the contents of steps in order.

3.2. Scheme 2 - docker build using base o/s platform

Instead of building on github-provided base ubuntu platform, use github-built docker container.

Initially replacing c++ program with shell script, we'll restore it later.

3.2.1. Preliminaries

source for Scheme 2 is on github here: https://github.comt/Rconybea/docker-action-example

Build docker container (using resources from ubuntu base platform). Container:

invokes a shell script to say hello
can be used as a github action

Directory Structure

docker-action-example
+- .github
|  \- workflows
|     \- main.yml
+- action.yml
\- entrypoint.sh

Source

entrypoint.sh:

#!/usr/bin/env bash
echo "Hello $1"
time=$(date)
echo "time=$time" >> $GITHUB_OUTPUT

Docker Container

DockerFile:

# base container image
FROM ubuntu:22.04

# copy files from repo to container filesystem
COPY entrypoint.sh /entrypoint.sh

# default startup executable
ENTRYPOINT ["/entrypoint.sh"]

Use Container as github action

toplevel action.yml describes a custom github action that operates by invoking docker image (automagically built from enclosed DockerFile)

action.yml:

name: 'Hello World'

description: 'Greet someone + record the time'

inputs:
  who-to-greet:
    description: 'Who to greet'
    required: true
    default: "World"

outputs:
  time:
    description: "time when greeting made"

runs:
  using: 'docker'
  image: 'DockerFile'
  args:
    - ${{ inputs.who-to-greet }}

Continuous Integration with Github

Configure github actions for repo https://github.com/Rconybea/docker-action-example;

File .github/workflows/main.yml

on: [push]

jobs:
  hello_world_job:
    runs-on: ubuntu-latest
    name: a job to say hello, perhaps fiercely
    steps:
      - name: hello world action step
        id: hello
        uses: rconybea/docker-action-example@v1
        with:
          who-to-greet: 'Iffy!  What are you doing??'
          # do something with output
          - name: get output time
            run: echo "The time was ${{ steps.hello.outputs.time }}"

In main.yml here:

rconybea/docker-action-example is our github repo,
v1 is target github tag

3.3. Scheme 3 - custom docker container via nix (no c++)

3.3.1. Preliminaries

Source for Scheme 3 uses two repos:

Custom container build here: https://github.com/Rconybea/docker-nix-hello
Workflow using container here: https://github.com/Rconybea/docker-action-example2
See also: https://blog.bitsrc.io/using-github-container-registry-in-practice-295677c6f65e]]

Directory structure

docker-nix-hello
+- flake.nix
\- flake.lock    # automatically created by nix

docker-action-example2
\- .github
   \- workflows
      \- main.yml

Source

Prepare minimal custom docker container using a nix flake.

flake.nix:

{
  description = "hello world";

  # dependencies
  inputs = rec {
    nixpkgs.url = "github:nixos/nixpkgs/23.05";
  };

  outputs = { self, nixpkgs } :
    let
      system = "x86_64-linux";
      pkgs = import nixpkgs { inherit system; };

      hello_deriv = pkgs.writeShellScriptBin "entrypoint.sh" ''
        echo "Hello $1"
        time=$(date)
        echo "time=$time" >> $GITHUB_OUTPUT
      '';

      docker_hello_deriv =
        pkgs.dockerTools.buildLayeredImage {
          name = "docker-nix-hello";
          tag = "v1";
          contents = [ self.packages.${system}.hello
                       self.packages.${system}.bash
                       # for /bin/tail,  assumed by github actions when invoking a docker container
                       self.packages.${system}.coreutils ];

          config = {
            Cmd = [ "/bin/entrypoint.sh" ];
            WorkingDir = "/";
          };
        };

    in rec {
      packages.${system} = {
        default = docker_hello_deriv;

        docker_hello = docker_hello_deriv;
        hello = hello_deriv;

        bash = pkgs.bash;
        # for example,  github actions creates container with --entrypoint "tail",
        # so container must provide executable with that name in $PATH
        #
        coreutils = pkgs.coreutils;
      };
    };
}

Build Container + Upload to github
Container will be docker-nix-hello.

Steps:
1. get github personal access token, so docker can send images to ghcr.io. on https://github.com/Rconybea:
  - visit profile (upper rhs of github webpage)
  - developer settings (bottom of sidebar)
  - personal access token
  - tokens (classic)
  - generate personal access token with scopes: read:packages write:packages delete:packages
2. have docker login to github
```
export CR_PAT=${token}
echo $CR_PAT | docker login ghcr.io -u rconybea --password-stdin
```
3. nix builds custom image (using docker-nix-hello/flake.nix)
```
cd ~/proj/docker-nix-hello
nix build
```
4. load image into docker
```
docker load <~/proj/docker-nix-hello/result
```
5. tag image the way github expects: ghcr.io/${username}/${imagename}:${tag}
```
docker image tag docker-nix-hello:v1 ghcr.io/rconybea/docker-nix-hello:v1
```
6. push to github container registry (will show up at https://github.com/Rconybea?tab=packages)
```
docker image push ghcr.io/rconybea/docker-nix-hello:v1
```
7. make package public from package's settings link

Workflow Using Custom Container

in docker-action-example2/.github/workflows/main.yml:

on: [push]

jobs:
  hello_world_job:
    name: a job to say hello,  using separate docker container
    runs-on: ubuntu-latest
    container:
      image: ghcr.io/rconybea/docker-nix-hello:v1
      steps:
        - name: hello world action step
          id: hello
          run: /bin/entrypoint.sh 'Xioni!'
          - name: get output time
            run: echo "The time was ${{ steps.hello.outputs.time }}"

3.4. Scheme 4 - custom docker container via nix (with gcc)

Source for Scheme 4 in two repos:

Custom container build here: https://github.com/Rconybea/docker-cpp-builder
Workflow using container here: https://github.com/Rconybea/docker-action-example3

Directory structure

docker-cpp-builder
+- flake.nix
\- flake.lock    # automatically created by nix

docker-action-example3
+- .github
|  \- workflows
|     \- main.yml
+- Makefile
+- hello.cpp

Source

Prepare custom docker container to deliver build stack (gcc, cmake, …)

flake.nix:

{
  description = "docker c++ builder (using nix)";

  # dependencies
  inputs = rec {
    nixpkgs.url = "github:nixos/nixpkgs/23.05";
  };

  outputs = { self, nixpkgs } :
    let
      system = "x86_64-linux";
      pkgs = import nixpkgs { inherit system; };

      docker_builder_deriv =
        pkgs.dockerTools.buildLayeredImage {
          name = "docker-cpp-builder";
          tag = "v3";
          contents = [ self.packages.${system}.git
                       self.packages.${system}.cacert
                       self.packages.${system}.gnumake
                       self.packages.${system}.gcc
                       self.packages.${system}.binutils
                       self.packages.${system}.bash
                       # for /bin/tail,  assumed by github actions when invoking a docker contianer
                       self.packages.${system}.coreutils ];
        };

    in rec {
      packages.${system} = {
        default = docker_builder_deriv;

        docker_builder = docker_builder_deriv;

        git = pkgs.git;
        cacert = pkgs.cacert;
        gnumake = pkgs.gnumake;
        gcc = pkgs.gcc;
        binutils = pkgs.binutils;
        bash = pkgs.bash;
        coreutils = pkgs.coreutils;
      };
    };
}

Build Container + Upload to github
Instructions are the same as for Scheme 3, but using ~/proj/docker-cpp-builder instead of ~/proj/docker-nix-hello
```
cd ~/proj/docker-cpp-builder
git tag v1
nix build
docker load <~/proj/docker-cpp-builder/result
docker image tag docker-cpp-builder:v1 ghcr.io/rconybea/rconybea/docker-cpp-builder:v1
docker image push ghcr.io/rconybea/docker-cpp-builder:v1
```
After uploading package (docker image) for gcc builder appears here: https://github.com/Rconybea?tab=packages:

Workflow using Custom Container

in docker-action-example3/.github/workflows/main.yml:

on: [push]

env:
  # personal accesss token (using automatically-supplied GIT_TOKEN) with read access to public repos
  GIT_USER: rconybea

jobs:
  build_job:
    name: compile hello world, using prepared docker container
    runs-on: ubuntu-latest
    container:
      image: ghcr.io/rconybea/docker-cpp-builder:v3
    steps:
      - name: checkout
        run:
          # not using usual checkout action: bc complications from within container
          GIT_SSL_NO_VERIFY=true git clone https://${{env.GIT_USER}}:${{env.GIT_TOKEN}}@github.com/rconybea/docker-action-example3.git

      - name: compile
        run:
          # make,g++ will run in container..
          make -C docker-action-example3

      - name: hello
        run:
          ./docker-action-example3/hello Roland
          #${{github.workspace}}/hello Roland

Note: in non-container build we can checkout code with something like:

steps:
- name: checkout
- uses: rconybea/my-repo@my-git-tag

Unfortunately that doesn't work out-of-the-box when we use a container for build, because of a permissioning problem. Spent some time on various internet-advised workarounds, before settling on the solution above: including git in custom docker image, and checking source out from inside the container.

We need to set GIT_SSL_NO_VERIFY to prevent an obscure error (passed along from SSL) mentioning an `unrecognized scheme`. Presumably there's a problem authenticating github.com's certificate

github continuous integration

Table of Contents

1. Motivation

2. Contents

3. Progressive Implementation

3.1. Scheme 1 - github build using base o/s platform

3.1.1. Preliminaries

3.2. Scheme 2 - docker build using base o/s platform

3.2.1. Preliminaries

3.3. Scheme 3 - custom docker container via nix (no c++)

3.3.1. Preliminaries

3.4. Scheme 4 - custom docker container via nix (with gcc)