Repour

= Repour - Archival Code Service

Repour archives source code from many untrusted origins into trusted git repositories. It supports https://git-scm.com/[git]. Tools like link:https://github.com/release-engineering/pom-manipulation-ext[PME] can optionally be applied as adjustments, and the results archived alongside the unmodified sources.

Why the name "Repour"? First, it contains the common short word for "repository" ("repo"); second, the action of "repouring" could be used to describe what the program does to the repositories.

== Ethos

Repour is not a source repository mirroring service, it archives source code for builds. This may sound the same but there is a big difference. Repour solves an entire class of problems surrounding the unreliability of "SCM coordinates" (a SCM url and commit/reference/revision), that a mirror is not able to solve.

=== Mirrors are flawed

A mirror can only be an exact copy of the upstream repository. Critically, any modifications to existing objects must be propagated, or the mirror ceases to be a mirror. If the upstream repository is not history protected, then neither the upstream repository or the mirror are suitable to build from. If the upstream repository is history protected, then there is no need for a mirror (except to ensure server availability). However, since virtually all upstreams are not history protected, or only weakly protected, a different solution is required.

=== The single requirement

How can the mutability conflict be handled if a mirror can't solve it? Strip everything back to the minimum required and go from there.

The only firm requirement of storing source code for builds is that the same coordinates, point to the same tree of files, forever. Everything else is a nice-to-have, including the upstream history.

== Repour as part of a build system

This section describes how Repour is intended to be employed within the collection of services that form a build system.

=== What Repour will do and not do

Repour does not mirror or in any way store more from the origins than the files that are required to build from. In exchange Repour provides these guarantees:

==== Coordinate Immutability

The git URL and tag Repour returns will always be valid, and will always point to the same file tree. This is an important part of ensuring builds can be reproduced.

==== File Tree Deduplication

For the same name and file tree, the same tag will be returned, as was returned the first time the tree was seen. This can be used to avoid rebuilding the same sources.

==== No Conflict Intervention

The overall design means there are no history conflicts, so manual intervention is never required to archive sources. This is an important part of a highly automated build system.

=== What the repository provider must do and not do

Repour needs a git repository provider that grants it the following permissions:

• Create repositories
• Clone repositories
• Push new tags
• Push commits to new branches
• Push fast-forward commits to existing branches

Also, the repository provider must not grant the following permissions to Repour and all other users:

• Delete repositories
• Delete tags or push existing tags (re-tagging)
• Push non-fast-forward commits (force push)

In this configuration, commits and tags are immutable in the repository, i.e. link:#coordinate-immutability[history is protected].

=== What the rest of the build system must do and not do

Since Repour is probably not exposed at the edge of the build system, other parts will have access to the raw SCM coordinates before Repour processes them. For the vast majority of cases, the rest of the build system should not attempt to interpret the raw SCM coordinates. The coordinates should be merely passed along, so there is no duplication of effort, or bad assumptions made about mutability.

Once Repour is called, it returns a set of git coordinates that can be used by the rest of the build system. See link:#what-repour-will-do-and-not-do[What Repour will do and not do].

==== Builder clients

When a builder host gets the sources, it won't communicate with Repour, but with the git repository provider where Repour has stored the sources.

The builder should not assume there is a default branch (master for example) in the repository, there won't be one there, as the repository is a collection of orphan branches.

Repour specifically returns a tag instead of a commit ID so the builder can perform a fast shallow clone. Cloning the full repository is of no benefit to the build, as it uses only the single file tree. Also, the full clone time will grow linearly with the number of unique file trees stored in the repository. This is the recommended git command to use:

[source,bash]

git clone --depth 1 --branch $tag $readonlyurl

Note that --branch is only usable with references (tags or branches), not commit IDs. The builder typically should not clone from a branch name as this is inherently mutable.

== Interface === Adjust

Checkout a git ref from the origin repo and push it to the target repo. If the ref is a branch, the branch will be pushed to the target repo. If the ref is a tag, the tag will be pushed to the target repo. If the ref is a SHA, then a branch will be created with name 'branch-<ref>' and pushed as a branch to the target repo. The latter is required to get a particular commit to the target repo.

You can specify an 'buildType' option to set if you want to run PME/GME/project-manipulator or not on it. You can also specify the 'adjustParameters'. The result of the PME run, together with the tag information is found in the response, under key 'adjust_result'. The value of 'adjust_result' is the same as for '/adjust'.

'buildType' can be 'MVN', 'GRADLE', and 'NODEJS'.

The response will contain the exact ref name as what was sent [cols="h,6a"] |=== |URL |/adjust

|Request |[cols="h,4a"] !=== !Method !POST

!Content-Type !application/json

!Body (Schema) ![source,python] { "ref": nonempty_str, Optional("adjustParameters"): All(dict), Optional("originRepoUrl"): Any(None, str), Optional("sync"): bool, Optional("callback"): { "url": Url(), Optional("method"): Any("PUT", "POST"), }, Optional("tempBuild"): bool, Optional("tempBuildTimestamp"): null_or_str, Optional("alignmentPreference"): null_or_str, Optional("taskId"): null_or_str, Optional("buildType"): nonempty_str, "internal_url": { "readwrite": Url(), "readonly": Url() } }

!Body (Example) ![source,javascript] { "ref": "pull-1436349331-root" "internal_url": { "readwrite": "git@github.com/test/me", "readonly": "https://github.com/test/me" } } !===

|Response (Success) |[cols="h,4a"] !=== !Status !200

!Content-Type !application/json

!Body (Schema) ![source,python] { "commit": str, "tag": str, "url": { "readwrite": Url(), "readonly": Url(), }, "adjustResultData": { "log": str, "VersioningState": { "executionRootModified": { "groupId": "value", "artifactId": "value", "version": "value" } }, "RemovedRepositories": [] } }

!Body (Example) ![source,javascript] { "commit": "abcd", "tag": "adjust-1439285354-root", "url": { "readwrite": "file:///tmp/repour-test-repos/example", "readonly": "file:///tmp/repour-test-repos/example" }, "adjustResultData": { "log": "help" "VersioningState": { "executionRootModified": { "groupId": "value", "artifactId": "value", "version": "value" } }, "RemovedRepositories": [] } } !===

|Response (Invalid request body) |[cols="h,4a"] !=== !Status !400

!Content-Type !application/json

!Body (Schema) ![source,python] [ { "error_message": str, "error_type": str, "path": [str], } ]

!Body (Example) ![source,javascript] [ { "error_message": "expected a URL", "error_type": "dictionary value", "path": ["url"] }, { "error_message": "expected str", "error_type": "dictionary value", "path": ["name"] } ]

!===

|Response (Processing error) |[cols="h,4a"] !=== !Status !400

!Content-Type !application/json

!Body (Schema) ![source,python] { "desc": str, "error_type": str, str: object, }

!Body (Example) ![source,javascript] { "desc": "Could not clone with git", "error_type": "PullCommandError", "cmd": [ "git", "clone", "--branch", "teiid-parent-8.11.0.Final", "--depth", "1", "--", "git@github.com:teiid/teiid.gitasd", "/tmp/tmppizdwfsigit" ], "exit_code": 128 } !=== |===

If the ref to align is a merge-request (Gitlab, with format '/merge-requests/<number>') or a pull-request (Github with format '/pull/<number>'), with sync switched on, the merge-request is checkout into a temporary branch, then the alignment tool is run, and the results are added in a commit and pushed in a tag that starts with 'Pull_Request-'. The latter is done to not create any confusing that the tag is from a merge-request.

With sync on for a merge-request, the checkout is not pushed into the downstream repository in a branch, unlike for a regular ref. The commits in the checkout are indirectly present via the 'Pull_Request-<tag>' only.

=== Clone

Checkout a git ref from the origin repo and force push it to the target repo. If ref is not a branch name, new branch named branch-{ref} pointing to the ref will be pushed instead. The response will contain the resulting ref name.

[cols="h,6a"] |=== |URL |/clone

|Request |[cols="h,4a"] !=== !Method !POST

!Content-Type !application/json

!Body (Schema) ![source,python] { "type": "git", # only git supported for n