`ci-diff-helper`¶

Diff Helper for Continuous Integration (CI) Services.

For an open source project, running unit tests, system tests, torture tests, fuzz tests, integration tests, code quality checks, etc. can quickly become a large task.

In order to limit the amount of time and resources that these jobs require, this tool provides a way to determine which files have changed and provides a Python API for these changes. In addition, this library provides the corresponding commit SHA (or other artifact) that is used as the diffbase.

The library supports (planned)

Continuous Integration Services
Verson Control Systems
- git
Project Hosting Sites
- GitHub

Note

When configuring your CI environment, it may be useful to set the GITHUB_OAUTH_TOKEN environment variable (GH_TOKEN). By authenticating in GitHub API requests, rate limiting can be avoided. Unauthenticated requests will be subject to rate limiting across the entire CI system.

To use this in your project, first install:

$ pip install --upgrade ci-diff-helper

Once you’ve done that, you can automatically detect your current environment and get a configuration object with information about your environment:

>>> import ci_diff_helper
>>> config = ci_diff_helper.get_config()
>>> config
<CircleCI (active=True)>

Common Configuration Properties¶

Long-lived configuration objects are provided as an interface for CI system information. These config objects cache the returned values for each property and use them to compute other useful values.

Each such configuration type (e.g. AppVeyor, CircleCI, Travis) has a common set of properties.

>>> config
<CircleCI (active=True)>
>>> config.active
True
>>> config.branch
'pull/808'
>>> config.tag is None
True

All configuration types can also be used to detect if a merge commit is currently being built:

>>> config.is_merge
False

`git` tools¶

The helpers git_root(), get_checked_in_files() and get_changed_files() are provided as tools for a git-based project.

The most relevant of these for finding diffs is get_changed_files(). For example, to find changed files between a current checkout and an upstream branch:

>>> ci_diff_helper.get_changed_files('HEAD', 'upstream/master')
['/path/to/your/git_checkout/project/_supporting.py',
 '/path/to/your/git_checkout/README.md']

In addition, being able to get the root of the current git checkout may be needed to collect files, execute scripts, etc. Getting all checked in files can be useful for things like test collection, file linting, etc.

>>> ci_diff_helper.git_root()
'/path/to/your/git_checkout'
>>> ci_diff_helper.get_checked_in_files()
['/path/to/your/git_checkout/setup.py',
 '/path/to/your/git_checkout/project/__init__.py',
 '/path/to/your/git_checkout/project/feature.py']

ci_diff_helper.get_config()[source]¶

Get configuration for the current environment.

Returns:	A configuration class for the current environment.
Return type:	`Union` [ `AppVeyor`, `CircleCI`, `Travis` ]
Raises:	`OSError` – If no (unique) environment is active.

ci_diff_helper.appveyor module¶

Set of utilities for dealing with AppVeyor CI.

This module provides a custom configuration type AppVeyor for the AppVeyor CI system.

This module uses a selection of environment variables to detect the state of AppVeyor configuration. See environment_vars for more details.

`AppVeyor` Configuration Type¶

When running in AppVeyor, you can automatically detect your current environment and get the configuration object:

>>> import ci_diff_helper
>>> config = ci_diff_helper.get_config()
>>> config
<AppVeyor (active=True)>

To use the AppVeyor configuration type directly:

>>> config = ci_diff_helper.AppVeyor()
>>> config
<AppVeyor (active=True)>
>>> config.branch
'master'
>>> config.provider
<AppVeyorRepoProvider.github: 'github'>

class ci_diff_helper.appveyor.AppVeyor[source]¶

Bases: ci_diff_helper._config_base.Config

Represent AppVeyor state and cache return values.

active¶: bool – Indicates if currently running in the target CI system.

branch¶

bool – Indicates the current branch in the target CI system.

This may indicate the active branch or the base branch of a pull request.

is_merge¶: bool – Indicates if the HEAD commit is a merge commit.

provider¶: str – The code hosting provider for the current AppVeyor build.

tag¶: str – The git tag of the current AppVeyor build.

Note

We only expect the APPVEYOR_REPO_TAG_NAME environment variable to be set when APPVEYOR_REPO_TAG=true indicates the build was started by a pushed tag. However, we don’t verify that we are in a build started by a tag before checking for the tag.

class ci_diff_helper.appveyor.AppVeyorRepoProvider[source]¶

Bases: enum.Enum

Enum representing all possible AppVeyor repo providers.

bitbucket = 'bitbucket'¶

github = 'github'¶

gitlab = 'gitlab'¶

kiln = 'kiln'¶

vso = 'vso'¶

ci_diff_helper.circle_ci module¶

Set of utilities for dealing with Circle CI.

This module provides a custom configuration type CircleCI for the CircleCI CI system.

This module uses a selection of environment variables to detect the state of Circle CI configuration. See environment_vars for more details.

`CircleCI` Configuration Type¶

When running in CircleCI, you can automatically detect your current environment and get the configuration object:

>>> import ci_diff_helper
>>> config = ci_diff_helper.get_config()
>>> config
<CircleCI (active=True)>

To use the CircleCI configuration type directly:

>>> config = ci_diff_helper.CircleCI()
>>> config
<CircleCI (active=True)>
>>> config.branch
'master'
>>> config.tag
'0.4.2'
>>> config.repo_url
'https://github.com/organization/repository'
>>> config.provider
<CircleCIRepoProvider.github: 'github'>
>>> config.slug
'organization/repository'

During a pull request build, we can determine information about the current PR being built:

>>> config = ci_diff_helper.CircleCI()
>>> config
<CircleCI (active=True)>
>>> config.in_pr
True
>>> config.pr
23
>>> config.branch
'pull/23'
>>> config.base
'7450ebe1a2133442098faa07f3c2c08b612d75f5'

class ci_diff_helper.circle_ci.CircleCI[source]¶

Bases: ci_diff_helper._config_base.Config

Represent CircleCI state and cache return values.

active¶: bool – Indicates if currently running in the target CI system.

base¶

str – The git object that current build is changed against.

The git object can be any of a branch name, tag, a commit SHA or a special reference.

Warning

This property will currently only work in a build for a pull request from a GitHub repository.

branch¶

bool – Indicates the current branch in the target CI system.

This may indicate the active branch or the base branch of a pull request.

in_pr¶

bool – Indicates if currently running in CircleCI pull request.

This uses the CIRCLE_PR_NUMBER environment variable to check if currently in a pull request.

is_merge¶: bool – Indicates if the HEAD commit is a merge commit.

pr¶

int – The current CircleCI pull request (if any).

If there is no active pull request, returns None.

provider¶: str – The code hosting provider for the current CircleCI build.

repo_url¶

str – The URL of the current repository being built.

For example: https://github.com/{organization}/{repository} or https://bitbucket.org/{user}/{repository}.

slug¶

str – The current slug in the CircleCI build.

Of the form {organization}/{repository}.

tag¶: str – The git tag of the current CI build.

class ci_diff_helper.circle_ci.CircleCIRepoProvider[source]¶

Bases: enum.Enum

Enum representing all possible CircleCI repo providers.

bitbucket = 'bitbucket'¶

github = 'github'¶

ci_diff_helper.environment_vars module¶

Comprehensive list of environment variables used in ci-diff-helper.

These environment variables are core to this library. They are used to detect the current environment.

For more details, see the Travis env docs, AppVeyor env docs and _CircleCI env docs.

ci_diff_helper.environment_vars.APPVEYOR_BRANCH = 'APPVEYOR_REPO_BRANCH'¶

Indicates the active AppVeyor branch.

In a “pull request” build it is the base branch the PR is merging into, otherwise it is the branch being built.

ci_diff_helper.environment_vars.APPVEYOR_REPO = 'APPVEYOR_REPO_PROVIDER'¶: The code hosting provided for the repository being tested in AppVeyor.

ci_diff_helper.environment_vars.APPVEYOR_TAG = 'APPVEYOR_REPO_TAG_NAME'¶

The tag of the current AppVeyor build.

This will only be valid when APPVEYOR_REPO_TAG, i.e. when the build was started by a pushed tag.

ci_diff_helper.environment_vars.CIRCLE_CI_BRANCH = 'CIRCLE_BRANCH'¶: Indicates the active git branch being tested on CircleCI.

ci_diff_helper.environment_vars.CIRCLE_CI_PR = 'CI_PULL_REQUEST'¶

Pull request containing the current change set.

If the current build is part of only one pull request, the URL of that PR will be populated here. If there was more than one pull request, this field contain one of the pull request URLs (picked randomly).

ci_diff_helper.environment_vars.CIRCLE_CI_PRS = 'CI_PULL_REQUESTS'¶: Comma-separated list of pull requests current build is a part of.

ci_diff_helper.environment_vars.CIRCLE_CI_PR_NUM = 'CIRCLE_PR_NUMBER'¶

The ID of the PR that started the current build.

We only expect this environment variable to be set during a build that is a part of a pull request from a fork.

ci_diff_helper.environment_vars.CIRCLE_CI_PR_OWNER = 'CIRCLE_PR_USERNAME'¶

The owner of the forked repository that started the current PR build.

We only expect this environment variable to be set during a build that is a part of a pull request from a fork.

ci_diff_helper.environment_vars.CIRCLE_CI_PR_REPO = 'CIRCLE_PR_REPONAME'¶

The name of the forked repository that started the current PR build.

We only expect this environment variable to be set during a build that is a part of a pull request from a fork.

ci_diff_helper.environment_vars.CIRCLE_CI_REPO_URL = 'CIRCLE_REPOSITORY_URL'¶: A link to the homepage for the current repository.

ci_diff_helper.environment_vars.CIRCLE_CI_TAG = 'CIRCLE_TAG'¶

The name of the git tag being tested

Only set if the build is running for a tag.

ci_diff_helper.environment_vars.GH_TOKEN = 'GITHUB_OAUTH_TOKEN'¶

GitHub OAuth 2.0 token.

This environment variable must be used to authenticate to the GitHub API. Making unauthenticated requests on a Continuous Integration server will typically be rate limited.

ci_diff_helper.environment_vars.IN_APPVEYOR = 'APPVEYOR'¶: Indicates if running in AppVeyor.

ci_diff_helper.environment_vars.IN_CIRCLE_CI = 'CIRCLECI'¶: Indicates if running in CircleCI.

ci_diff_helper.environment_vars.IN_TRAVIS = 'TRAVIS'¶: Indicates if running in Travis.

ci_diff_helper.environment_vars.TRAVIS_BRANCH = 'TRAVIS_BRANCH'¶

Indicates the active Travis branch.

In a “push” build, this is the branch that was pushed while in a “pull request” build it is the branch that a pull request is against.

ci_diff_helper.environment_vars.TRAVIS_EVENT_TYPE = 'TRAVIS_EVENT_TYPE'¶: Indicates the type of build that is occurring.

ci_diff_helper.environment_vars.TRAVIS_PR = 'TRAVIS_PULL_REQUEST'¶

Indicates which Travis pull request we are in.

Is an integer when in a pull request or “false” when not.

ci_diff_helper.environment_vars.TRAVIS_RANGE = 'TRAVIS_COMMIT_RANGE'¶

The range of commits changed in the current build.

This is not particularly useful in a PR build.

Note

This is empty for builds triggered by the initial commit of a new branch.

ci_diff_helper.environment_vars.TRAVIS_SLUG = 'TRAVIS_REPO_SLUG'¶

The GitHub repository slug for the current Travis build.

A slug is of the form {organization}/{repository}.

ci_diff_helper.environment_vars.TRAVIS_TAG = 'TRAVIS_TAG'¶

The tag of the current Travis build.

We only expect the TRAVIS_TAG environment variable to be set during a tag “push” build, but it can be set as the empty string in non-“push” builds.

ci_diff_helper.git_tools module¶

Helpers for interacting with git.

ci_diff_helper.git_tools.commit_subject(revision='HEAD')[source]¶

Gets the subject of a git commit.

Parameters:	revision (`Optional` [ `str` ]) – A `git` revision, any of a branch name, tag, a commit SHA or a special reference.
Returns:	The commit subject.
Return type:	str

ci_diff_helper.git_tools.get_changed_files(blob_name1, blob_name2)[source]¶

Gets a list of changed files between two git revisions.

A git object reference can be any of a branch name, tag, a commit SHA or a special reference.

Parameters:	blob_name1 (str) – A `git` object reference. blob_name2 (str) – A `git` object reference.
Returns:	List of all filenames changed.
Return type:	list

ci_diff_helper.git_tools.get_checked_in_files()[source]¶

Gets a list of files in the current git repository.

Effectively runs:

$ git ls-files ${GIT_ROOT}

and then finds the absolute path for each file returned.

Returns:	List of all filenames checked into the repository.
Return type:	list

ci_diff_helper.git_tools.git_root()[source]¶

Return the root directory of the current git checkout.

Returns:	Filesystem path to `git` checkout root.
Return type:	str

ci_diff_helper.git_tools.merge_commit(revision='HEAD')[source]¶

Checks if a git revision is a merge commit.

Parameters:	revision (`Optional` [ `str` ]) – A `git` revision, any of a branch name, tag, a commit SHA or a special reference.
Returns:	Flag indicating if the given revision.
Return type:	bool
Raises:	`NotImplementedError` – if the number of parents is not 1 or 2.

ci_diff_helper.travis module¶

Set of utilities for dealing with Travis CI.

This module provides a custom configuration type Travis for the Travis CI system.

Since Travis only works with GitHub, the commands in this module are GitHub and git centric.

This module uses a selection of environment variables to detect the state of Travis configuration. See environment_vars for more details.

`Travis` Configuration Type¶

When running in Travis, you can automatically detect your current environment and get the configuration object:

>>> import ci_diff_helper
>>> config = ci_diff_helper.get_config()
>>> config
<Travis (active=True)>

To use the Travis configuration type directly:

>>> config = ci_diff_helper.Travis()
>>> config
<Travis (active=True)>
>>> config.active
True
>>> config.in_pr
True
>>> config.branch
'master'

In addition this configuration provides extra features for determining a diffbase.

>>> config = ci_diff_helper.Travis()
>>> config.event_type
<TravisEventType.pull_request: 'pull_request'>
>>> config.slug
'organization/repository'
>>> config.repo_url
'https://github.com/organization/repository'
>>> config.pr
1234
>>> config.tag is None
True
>>> config.base
'master'

Not only is this object valuable during a pull request build, it can also be used to find relevant information in a “push” build:

>>> config = ci_diff_helper.Travis()
>>> config.event_type
<TravisEventType.push: 'push'>
>>> config.pr is None
True
>>> config.tag
'0.13.37'
>>> config.base
'4ad7349dc7223ebc02175a16dc577a013044a538'

Though the base property can be useful as a diffbase of a given commit, it may be inappropriate. In a “push” build, base will be computed from the TRAVIS_COMMIT_RANGE environment variable, and this value is not particularly reliable. Instead, merged_pr provides a way to determine the PR that was merged:

>>> config.is_merge
True
>>> config.merged_pr
1355

class ci_diff_helper.travis.Travis[source]¶

Bases: ci_diff_helper._config_base.Config

Represent Travis state and cache return values.

active¶: bool – Indicates if currently running in the target CI system.

base¶

str – The git object that current build is changed against.

The git object can be any of a branch name, tag, a commit SHA or a special reference.

This can be used in combination with get_changed_files() to determine files that need to be linted, tested or inspected in some other way:

>>> config
<Travis (active=True)>
>>> config.base
'master'
>>> ci_diff_helper.get_changed_files('HEAD', config.base)
['/path/to/your/git_checkout/project/_supporting.py']

Note

This will throw an OSError on the very first “push” build for a branch. This is because Travis leaves the value empty in builds triggered by the initial commit of a new branch.

Warning

This property is only meant to be used in a “pull request” or “push” build.

branch¶

bool – Indicates the current branch in the target CI system.

This may indicate the active branch or the base branch of a pull request.

event_type¶: bool – Indicates if currently running in Travis.

in_pr¶

bool – Indicates if currently running in Travis pull request.

This uses the TRAVIS_EVENT_TYPE environment variable to check if currently in a pull request. Though it doesn’t use the TRAVIS_PULL_REQUEST environment variable, checking that the value is set to an integer would be a perfectly valid approach.

is_merge¶: bool – Indicates if the HEAD commit is a merge commit.

merged_pr¶

int – The pull request corresponding to a merge commit at HEAD.

If not currently in a push build, returns None. If the HEAD commit is not a merge commit, returns None.

Note

This only uses the git checkout to determine the pull request ID. A more comprehensive check would involve veriying the ID by using the GitHub API.

Warning

This property is only meant to be used in a “pull request” or “push” build.

pr¶

int – The current Travis pull request (if any).

If there is no active pull request, returns None.

repo_url¶

str – The URL of the current repository being built.

Of the form https://github.com/{organization}/{repository}.

slug¶

str – The current slug in the Travis build.

Of the form {organization}/{repository}.

tag¶: str – The git tag of the current Travis build.

Note

We only expect the TRAVIS_TAG environment variable to be set during a tag “push” build, but we don’t verify that we are in a push build before checking for the tag.

class ci_diff_helper.travis.TravisEventType[source]¶

Bases: enum.Enum

Enum representing all possible Travis event types.

api = 'api'¶

cron = 'cron'¶

pull_request = 'pull_request'¶

push = 'push'¶

ci-diff-helper¶

Common Configuration Properties¶

git tools¶

ci_diff_helper.appveyor module¶

AppVeyor Configuration Type¶

ci_diff_helper.circle_ci module¶

CircleCI Configuration Type¶

ci_diff_helper.environment_vars module¶

ci_diff_helper.git_tools module¶

ci_diff_helper.travis module¶

Travis Configuration Type¶

`ci-diff-helper`¶

`git` tools¶

`AppVeyor` Configuration Type¶

`CircleCI` Configuration Type¶

`Travis` Configuration Type¶