GitHub - VisionSystemsInc/docker_blueprints: An experiment using "blueprints" for repeated & complex docker tasks

A docker blueprint is a (usually complex) docker image that can be included in a multi-stage build so that you don't always have to find and repeat that "perfect set of Dockerfile lines to include software XYZ", such as GDAL, PDAL, etc.

How to use

Add blueprint services to your docker-compose.yml file, using project specific image names. For example:

services:

   gdal:
      build:
         context: "${VSI_COMMON_DIR}/docker/blueprints"
         dockerfile: blueprint_gdal.Dockerfile
         args:
            GDAL_VERSION: "3.3.3"
      image: &gdal_image
         example/project:gdal

   pdal:
      build:
         context: "${VSI_COMMON_DIR}/docker/blueprints"
         dockerfile: blueprint_pdal.Dockerfile
         args:
            GDAL_IMAGE: *gdal_image
            PDAL_VERSION: "2.3.0"
      image: &pdal_image
         example/project:pdal

   example:
      build:
         context: .
         dockerfile: example.Dockerfile
         args:
            GDAL_IMAGE: *gdal_image
            PDAL_IMAGE: *pdal_image
      image: example/project:example

The Dockerfile is then formulated as follows

# blueprints
ARG GDAL_IMAGE
ARG PDAL_IMAGE
FROM ${GDAL_IMAGE} AS gdal
FROM ${PDAL_IMAGE} AS pdal

# base image
FROM python:3.8

# copy from blueprints
COPY --from=gdal /usr/local /usr/local
COPY --from=pdal /usr/local /usr/local

# Only needs to be run once for all blueprints/recipes
RUN shopt -s nullglob; for patch in /usr/local/share/just/container_build_patch/*; do "${patch}"; done

Python Wheels

Docker blueprints may also build python wheels for a given tool. For example, GDAL python bindings are compiled against a user-specified PYTHON_VERSION and NUMPY_VERSION as follows:

services:

   gdal:
      build:
         context: "${VSI_COMMON_DIR}/docker/blueprints"
         dockerfile: blueprint_gdal.Dockerfile
         args:
            GDAL_VERSION: "3.3.3"
            PYTHON_VERSION: "3.9"
            NUMPY_VERSION: "1.22.3"
      image: example/project:gdal

Blueprints

GDAL

Name	GDAL
Output dir	`/usr/local`
Build Args	`GDAL_VERSION`	Version of GDAL to download
	`PYTHON_VERSION`	Build python bindings for this python version
	`NUMPY_VERSION`	Build python bindings for this numpy version

Compiles GDAL v3, including OPENJPEG 2.4, GEOS 3.11.0, libtiff 4.3, libgeotiff 1.7, PROJ v8

# global arguments
ARG PYTHON_VERSION

# blueprint input(s)
FROM example/project:gdal AS gdal

# base image
FROM python:$PYTHON_VERSION

# local args
ARG NUMPY_VERSION

# additional runtime dependencies
RUN apt-get update; \
   DEBIAN_FRONTEND=noninteractive apt-get install -y --no-install-recommends \
      libgeos-c1v5; \
   rm -r /var/lib/apt/lists/*

# add blueprint
COPY --from=gdal /usr/local /usr/local

# Patch all blueprints/recipes
RUN shopt -s nullglob; for patch in /usr/local/share/just/container_build_patch/*; do "${patch}"; done

# install numpy then GDAL python bindings
RUN pip install numpy==${NUMPY_VERSION}; \
    pip install /usr/local/share/just/wheels/GDAL*.whl

PDAL

Name	PDAL
Output dir	`/usr/local`
Build Args	`PDAL_VERSION`	Version of PDAL to download
	`PDAL_PYTHON_VERSION`	Version of PDAL python bindings to download
	`PYTHON_VERSION`	Build python bindings for this python version
	`NUMPY_VERSION`	Build python bindings for this numpy version

Compiles PDAL v2. Requires GDAL blueprint.

# global arguments
ARG PYTHON_VERSION

# blueprint input(s)
FROM example/project:gdal AS gdal
FROM example/project:pdal AS pdal

# base image
FROM python:$PYTHON_VERSION

# local args
ARG NUMPY_VERSION

# additional runtime dependencies
RUN apt-get update; \
   DEBIAN_FRONTEND=noninteractive apt-get install -y --no-install-recommends \
      libgeos-c1v5; \
   rm -r /var/lib/apt/lists/*

# add blueprint(s)
COPY --from=gdal /usr/local /usr/local
COPY --from-pdal /usr/local /usr/local

# Patch all blueprints/recipes
RUN shopt -s nullglob; for patch in /usr/local/share/just/container_build_patch/*; do "${patch}"; done

# install numpy then GDAL python bindings
RUN pip install numpy==${NUMPY_VERSION}; \
    pip install /usr/local/share/just/wheels/PDAL*.whl

SWIG bindings for S2Geometry

services:

   s2:
      build:
         context: "${VSI_COMMON_DIR}/docker/blueprints"
         dockerfile: blueprint_s2geometry.Dockerfile
         args:
            # S2GEOMETRY_VERSION: "v0.11.1"
            # https://github.com/google/s2geometry/tags
            # PYTHON_VERSION: "3.10.15"
            # https://hub.docker.com/_/python/tags
            # BASE_IMAGE: "python:3.10.15-bookworm"
            # https://hub.docker.com/_/python/tags
            # ABSEIL_VERSION: "v1.6.2"
            # https://github.com/abseil/abseil-cpp/tags
      image: &s2_image
         example/project:s2

   example:
      build:
         context: .
         dockerfile: example.Dockerfile
         args:
            S2_IMAGE: *s2_image
      image: example/project:example

Name	S2 Geometry
Output dir	`/usr/local`
Build Args	`BASE_IMAGE`	Base image to build the wheel in. Currently works in Debian instead of Alma
	`PYTHON_VERSION`	Build python bindings for this python version
	`ABSEIL_VERSION`	Abseil version to build from source
	`S2GEOMETRY_VERSION`	S2 Geometry version to build from source

Compiles S2 Geometry wheel for use in python.

# global arguments
ARG S2_IMAGE
FROM ${S2_IMAGE} AS s2

FROM some_image

...

COPY --from=s2 /usr/local /usr/local

RUN pip install /usr/local/share/just/wheels/*
# Or using pip-tools, add "--find-links /usr/local/share/just/wheels" to requirements.in

import s2Geometry as s2

ll = s2.S2LatLng.FromDegrees(51.5001525, -0.1262355)
print(s2.S2CellId(ll).ToToken())

pybind11 bindings for ng-log

services:

   nglog:
      build:
         context: "${VSI_COMMON_DIR}/docker/blueprints"
         dockerfile: blueprint_nglog.Dockerfile
         args:
            # NGLOG_VERSION: "v0.8.0"
            # https://github.com/ng-log/ng-log/tags
            # PYTHON_VERSION: "3.8.12"
            # https://github.com/pypa/manylinux/blob/main/docker/Dockerfile
            # BASE_IMAGE: "quay.io/pypa/manylinux_2_28_x86_64"
            # https://quay.io/repository/pypa/manylinux_2_28_x86_64?tab=tags&tag=latest
            # LIBUNWIND_VERSION: "v1.8.2"
            # https://github.com/libunwind/libunwind/tags
      image: &nglog_image
         example/project:nglog

   example:
      build:
         context: .
         dockerfile: example.Dockerfile
         args:
            NGLOG_IMAGE: *nglog_image
      image: example/project:example

Name	ng-log
Output dir	`/usr/local`
Build Args	`BASE_IMAGE`	Base image to build the wheel in. Default: quay.io/pypa/manylinux_2_28_x86_64
	`PYTHON_VERSION`	Build python bindings for this python version
	`LIBUNWIND_VERSION`	LibUnwind version to build from source
	`NGLOG_VERSION`	ng-log version to build from source

Compiles ng-log wheel for use in python. This is primarily to setup [Failure Signal Handlers](https://ng-log.github.io/ng-log/0.8.0/failures/).

# global arguments
ARG NGLOG_IMAGE
FROM ${NGLOG_IMAGE} AS nglog

FROM some_image

...

COPY --from=nglog /usr/local /usr/local

RUN pip install /usr/local/share/just/wheels/*
# Or using pip-tools, add "--find-links /usr/local/share/just/wheels" to requirements.in

import nglog

nglog.InitializeLogging("programName")
nglog.installFailureSignalHandler()

Python bindings for tiny-cuda-nn

services:

   tinycudann:
      build:
         context: "${VSI_COMMON_DIR}/docker/blueprints"
         dockerfile: blueprint_tinycudann.Dockerfile
         args:
            # CUDA_VERSION: "11.8.0"
            # https://hub.docker.com/r/nvidia/cuda/tags
            # PYTHON_VERSION: "3.10.18"
            # https://www.python.org/doc/versions/
            # TINYCUDANN_VERSION: "v2.0"
            # https://github.com/NVlabs/tiny-cuda-nn/tags
            # TCNN_CUDA_ARCHITECTURES: "70,86"
            # https://github.com/NVlabs/tiny-cuda-nn/blob/v2.0/bindings/torch/setup.py#L45-L47
            # TORCH_VERSION: "2.1.2+cu118"
            # https://download.pytorch.org/whl/torch/
            # VSI_RECIPE_REPO: "vsiri/recipe"
            # https://hub.docker.com/r/vsiri/recipe
      image: &tinycudann_image
         example/project:tinycudann

   example:
      build:
         context: .
         dockerfile: example.Dockerfile
         args:
            TINYCUDANN_IMAGE: *tinycudann_image
      image: example/project:example

Name	Tiny CUDA Neural Networks
Output dir	`/usr/local`
Build Args	`BASE_IMAGE`	Base image to build the wheel in. Defaults to latest `manylinux_2_28_x86_64`
	`CUDA_VERSION`	Build tiny-cuda-nn for this CUDA version
	`PYTHON_VERSION`	Build tiny-cuda-nn for this python version
	`TINYCUDANN_VERSION`	tiny-cuda-nn version to build from source
	`TCNN_CUDA_ARCHITECTURES`	Build tiny-cuda-nn for these CUDA architectures
	`TORCH_VERSION`	Build tiny-cuda-nn for this CUDA-enabled torch version
	`VSI_RECIPE_REPO`	VSI docker recipe repo

Compiles Tiny CUDA Neural Networks (tiny-cuda-nn) wheel for use in python. See [github repo](https://github.com/NVlabs/tiny-cuda-nn) for usage details.

A container using the tinu-cuda-nn wheel must have appropriate versions of pytorch and CUDA installed.

# global arguments
ARG TINYCUDANN_IMAGE
FROM ${TINYCUDANN_IMAGE} AS tinycudann

FROM some_image

...

COPY --from=tinycudann /usr/local /usr/local

RUN pip install /usr/local/share/just/wheels/*
# Or using pip-tools, add "--find-links /usr/local/share/just/wheels" to requirements.in

import torch
import tinycudann

Blueprint maintenance

To update build dependencies: docker compose run -f maintenance.yml--rm nglog-compile