This post explores how to leverage the GitLab Registry to manage Python libraries within a Docker and Poetry setup. If you’re looking to streamline code sharing across projects in your organization, we’ll show you how to publish and use libraries stored in GitLab, building on the groundwork from our previous guide on library publication.
Adding a secondary source in Poetry
The first step is setting up a secondary source within Poetry. This allows it to look for a package name that it does not find on PyPi.
NOTE: Be careful with library naming as you should avoid shadowing any names in PyPi.
The name can be arbitrary but is used to distinguish multiple locations. The url is important as it actually defines the path where the libraries will be located. GitLab groups allow granular access control and can be used to group things together.
[[tool.poetry.source]]
name = "gitlab" # This name will be used in the configuration to retreive the proper credentials
# the group allows for a very specific and granular access
url = "https://gitlab.com/api/v4/groups/XXXXX/-/packages/pypi/simple" # URL used to download your packages from
secondary = true
GitLab Access Tokens
To be able to install these we need access to the specific groups. There are a few options we have, described nicely here too.
Personal Access Token: This token grants you the access your user has. It will become relevant for local installs, but should never be used for any other shared setups.
GitLab CI Token: This token is generated each CI run and grants access to the same group tree. But it will be different on each run, which migh mess with Docker caching. Otherwise it is the simplest way to go.
Deploy Token: Could be used when installing across various groups from other teams perhaps.
Local Installs
This is fairly simple and documented here already. But to recap. All you need is setup the repository location with a certain name and add the credentials.
NOTE: You can use a Personal Access Token with read_packages permissions from GitLab.
poetry source add --priority=supplemental foo https://pypi.example.org/simple/
poetry config http-basic.gitlab <your_username> <your_gitlab_pat>
Docker installs
The Dockerfile needs to include the repository location setup and add the CI Token as a configuration.
RUN poetry config repositories.gitlab https://gitlab.com/api/v4/groups/XXXX/-/packages/pypi/simple
ARG GITLAB_ACCESS_TOKEN
RUN poetry config http-basic.gitlab gl_ci_token $GITLAB_ACCESS_TOKEN
The build can then be run by injecting an argument to the docker builds.
docker build --build-arg="GITLAB_ACCESS_TOKEN=....."
GitLab CI direct installs
If you need to just run the install but use a basic Python Image you can install in the before script and rely on the CI_JOB_TOKEN.
test:
before_script:
# ensure you have poetry installed and python setup in the same version
# this take scare of a safe authentication
- poetry config repositories.gitlab https://gitlab.com/api/v4/groups/XXXX/-/packages/pypi/simple
- poetry config http-basic.gitlab gitlab-ci-token $CI_JOB_TOKEN
-
# run the subsequent installs
As you see the setup here is fairly similar as in the local setup. Only difference being its ephemeral so cannot be just done once.