Dependency Management

November 5, 2019

Who am I?

Scientific software developer since 15 years
PhD in theoretical chemistry
Linear scaling electron-electron correlation
Tailor-made force fields
BlasBooster
GROMACS-FDA, GROMACS-RAMD
PINK
HiFlow3

Outline

Introduction into dependency management
Docker development environment
Visual Studio Code
Python
- pip, virtualenv, setup.py, wheels
Conan.io
- install, create, upload

Introduction

Dependency management is a technique for declaring, resolving and using dependencies required by the project in an automated fashion.

(Source: Devopedia)

Dependencies are project related
Keep your dependencies up to date to reduce technical debts
Distinguish build-time and runtime requirements

Package vs Dependency

Package manager: Install applications, libraries and tools on a system
- apt (Linux)
- brew (MacOS)
Dependency manager: Handle project dependencies across environments
- pip, conda (Python)
- conan (C++)

What is a software dependency

Module A uses functionality of module B
Transitive dependency → Dependency graph

Dependency hell

Too many dependencies
Conflicting dependencies
Circular dependencies

Version numbering

major.minor[.patch]

Bugfixes in minor version
New interfaces and features in major version

Version specifier

Comparison

==: exact match
!=: exclusion
<=,>=: inclusive ordered
<,>: exclusive ordered

Compatibility (for stability)
```
~= 1.4.5
>= 1.4.5, == 1.4.*
```

Combination

~=3.1.0, != 3.1.3: version 3.1.0 or later,
                   but not version 3.1.3
                   and not version 3.2.0
                   or later

Docker

Virtualization of applications (lightweight VM)
Coded environment by Dockerfile
Important tool for software development
Images shared at DockerHub
Flexible deployment (Orchestration) via Kubernetes
Continuous Integration (Jenkins) by Jenkinsfile

Docker Development Environment

Visual Studio Code

Free and open source (not Microsoft Visual Studio)
Most popular development environment 2019
Language Server Protocol (LSP) as open standard for language specific features
- code completion and navigation
- refactoring, syntax highlighting, error markers
embedded git and GitHub support

Package installer for Python (PIP)

Python Package Index
Build recipe as code 'setup.py' from setuptools
Wheels for platform-specific C extensions (replace eggs, which was building everything from scratch)
Docker image 'manylinux' with old 'glibc' to support most Linux distributions

Exercise 1

Install TensorFlow in virtualenv

Install virtualenv
Create and activate virtualenv 'tensorflow'
Install tensorflow
Test tensorflow

Python packaging

setup.py

import setuptools

with open("README.md", "r") as fh:
    long_description = fh.read()

setuptools.setup(
    name="example-pkg-your-username",
    version="0.0.1",
    author="Example Author",
    author_email="author@example.com",
    description="A small example package",
    long_description=long_description,
    long_description_content_type="text/markdown",
    url="https://github.com/pypa/sampleproject",
    packages=setuptools.find_packages(),
    classifiers=[
        "Programming Language :: Python :: 3",
        "License :: OSI Approved :: MIT License",
        "Operating System :: OS Independent",
    ],
)

Docu / Example Tensorflow

Exercise 2

Install tqdm with setup.py

Clone tqdm from GitHub
Install via setup.py
Test tqdm

Exercise 3

Create package python-example

Clone python-example from GitHub
Create package
Upload to https://test.pypi.org

C++ dependency management with conan.io

Decentralized package manager
Client-server architecture similar to git push/pull
CMake integration with cmake-conan
Support all platforms (Linux, Apple, Windows, Android, embedded, …)
Support all build systems (CMake, Makefile, Visual Studio, …)

Conan repositories

conan-center: Official maintained by the Conan team (178 packages)
bincrafters: Group of OSS developers (370 packages)
braintwister: Personal repository at Bintray for OSS
Running conan_server for on-site repository

Installing dependencies

conanfile.txt

[requires]
Poco/1.9.0@pocoproject/stable

[generators]
cmake

name / version @ user / channel

Creating package

conanfile.py

from conans import ConanFile, CMake

class PackageConan(ConanFile):
    name = "<package name>"
    version = "0.1"
    license = "<Put the package license here>"
    url = "<Package recipe repository url>"
    description = "<Description of Hello here>"
    settings = "os", "compiler", "build_type", "arch"
    options = {"shared": [True, False]}
    default_options = {"shared": False}
    generators = "cmake"

    def source(self):
        self.run("git clone https://github.com/memsharded/hello.git")
        self.run("cd hello")

    def build(self):
        cmake = CMake(self)
        cmake.configure(source_folder="hello")
        cmake.build()

    def package(self):
        self.copy("*.h", dst="include", src="hello")
        self.copy("*.so", dst="lib", keep_path=False)

    def package_info(self):
        self.cpp_info.libs = ["hello"]

Exercise 4

Install range-v3 with conan

Clone conan-example from GitHub
Install with conanfile.txt
Compile and run

Exercise 5

Build package conan-example

Write conanfile.py
Create package
Upload to Bintray

Thank you

https://bernddoser.github.io/workshop-dependency-management