ci-diff-helper

Diff Helper for Continuous Integration (CI) Services.

For an open source project, running unit tests, system tests, torture tests, fuzz tests, integration tests, code quality checks, etc. can quickly become a large task.

In order to limit the amount of time and resources that these jobs require, this tool provides a way to determine which files have changed and provides a Python API for these changes. In addition, this library provides the corresponding commit SHA (or other artifact) that is used as the diffbase.

The library supports (planned)

Note

When configuring your CI environment, it may be useful to set the GITHUB_OAUTH_TOKEN environment variable (GH_TOKEN). By authenticating in GitHub API requests, rate limiting can be avoided. Unauthenticated requests will be subject to rate limiting across the entire CI system.

To use this in your project, first install:

$ pip install --upgrade ci-diff-helper

Once you’ve done that, you can automatically detect your current environment and get a configuration object with information about your environment:

>>> import ci_diff_helper
>>> config = ci_diff_helper.get_config()
>>> config
<CircleCI (active=True)>

Common Configuration Properties

Long-lived configuration objects are provided as an interface for CI system information. These config objects cache the returned values for each property and use them to compute other useful values.

Each such configuration type (e.g. AppVeyor, CircleCI, Travis) has a common set of properties.

>>> config
<CircleCI (active=True)>
>>> config.active
True
>>> config.branch
'pull/808'
>>> config.tag is None
True

All configuration types can also be used to detect if a merge commit is currently being built:

>>> config.is_merge
False

git tools

The helpers git_root() and get_checked_in_files() are provided as tools for a git-based project. Being able to get the root of the current git checkout may be needed to collect files, execute scripts, etc. Getting all checked in files can be useful for things like test collection, file linting, etc.

>>> ci_diff_helper.git_root()
'/path/to/your/git_checkout'
>>> ci_diff_helper.get_checked_in_files()
['/path/to/your/git_checkout/setup.py',
 '/path/to/your/git_checkout/project/__init__.py',
 '/path/to/your/git_checkout/project/feature.py']
ci_diff_helper.get_config()[source]

Get configuration for the current environment.

Returns:A configuration class for the current environment.
Return type:Union [ AppVeyor, CircleCI, Travis ]
Raises:OSError – If no (unique) environment is active.

ci_diff_helper.appveyor module

Set of utilities for dealing with AppVeyor CI.

This module provides a custom configuration type AppVeyor for the AppVeyor CI system.

This module uses a selection of environment variables to detect the state of AppVeyor configuration. See environment_vars for more details.

AppVeyor Configuration Type

When running in AppVeyor, you can automatically detect your current environment and get the configuration object:

>>> import ci_diff_helper
>>> config = ci_diff_helper.get_config()
>>> config
<AppVeyor (active=True)>

To use the AppVeyor configuration type directly:

>>> config = ci_diff_helper.AppVeyor()
>>> config
<AppVeyor (active=True)>
>>> config.branch
'master'
>>> config.provider
<AppVeyorRepoProvider.github: 'github'>
class ci_diff_helper.appveyor.AppVeyor[source]

Bases: ci_diff_helper._config_base.Config

Represent AppVeyor state and cache return values.

__delattr__

x.__delattr__(‘name’) <==> del x.name

__format__()

default object formatter

__getattribute__

x.__getattribute__(‘name’) <==> x.name

__hash__
__reduce__()

helper for pickle

__reduce_ex__()

helper for pickle

__setattr__

x.__setattr__(‘name’, value) <==> x.name = value

__sizeof__() → int

size of object in memory, in bytes

__str__
active

bool: Indicates if currently running in the target CI system.

branch

bool: Indicates the current branch in the target CI system.

This may indicate the active branch or the base branch of a pull request.

is_merge

bool: Indicates if the HEAD commit is a merge commit.

provider

str: The code hosting provider for the current AppVeyor build.

tag

str: The git tag of the current AppVeyor build.

Note

We only expect the APPVEYOR_REPO_TAG_NAME environment variable to be set when APPVEYOR_REPO_TAG=true indicates the build was started by a pushed tag. However, we don’t verify that we are in a build started by a tag before checking for the tag.

class ci_diff_helper.appveyor.AppVeyorRepoProvider[source]

Bases: enum.Enum

Enum representing all possible AppVeyor repo providers.

bitbucket = <AppVeyorRepoProvider.bitbucket: 'bitbucket'>
github = <AppVeyorRepoProvider.github: 'github'>
gitlab = <AppVeyorRepoProvider.gitlab: 'gitlab'>
kiln = <AppVeyorRepoProvider.kiln: 'kiln'>
vso = <AppVeyorRepoProvider.vso: 'vso'>

ci_diff_helper.circle_ci module

Set of utilities for dealing with Circle CI.

This module provides a custom configuration type CircleCI for the CircleCI CI system.

This module uses a selection of environment variables to detect the state of Circle CI configuration. See environment_vars for more details.

CircleCI Configuration Type

When running in CircleCI, you can automatically detect your current environment and get the configuration object:

>>> import ci_diff_helper
>>> config = ci_diff_helper.get_config()
>>> config
<CircleCI (active=True)>

To use the CircleCI configuration type directly:

>>> config = ci_diff_helper.CircleCI()
>>> config
<CircleCI (active=True)>
>>> config.branch
'master'
>>> config.tag
'0.4.2'
>>> config.repo_url
'https://github.com/organization/repository'
>>> config.provider
<CircleCIRepoProvider.github: 'github'>
>>> config.slug
'organization/repository'

During a pull request build, we can determine information about the current PR being built:

>>> config = ci_diff_helper.CircleCI()
>>> config
<CircleCI (active=True)>
>>> config.in_pr
True
>>> config.pr
23
>>> config.branch
'pull/23'
>>> config.base
'7450ebe1a2133442098faa07f3c2c08b612d75f5'
class ci_diff_helper.circle_ci.CircleCI[source]

Bases: ci_diff_helper._config_base.Config

Represent CircleCI state and cache return values.

__delattr__

x.__delattr__(‘name’) <==> del x.name

__format__()

default object formatter

__getattribute__

x.__getattribute__(‘name’) <==> x.name

__hash__
__reduce__()

helper for pickle

__reduce_ex__()

helper for pickle

__setattr__

x.__setattr__(‘name’, value) <==> x.name = value

__sizeof__() → int

size of object in memory, in bytes

__str__
active

bool: Indicates if currently running in the target CI system.

base

str: The git object that current build is changed against.

The git object can be any of a branch name, tag, a commit SHA or a special reference.

Warning

This property will currently only work in a build for a pull request from a GitHub repository.

branch

bool: Indicates the current branch in the target CI system.

This may indicate the active branch or the base branch of a pull request.

in_pr

bool: Indicates if currently running in CircleCI pull request.

This uses the CIRCLE_PR_NUMBER environment variable to check if currently in a pull request.

is_merge

bool: Indicates if the HEAD commit is a merge commit.

pr

int: The current CircleCI pull request (if any).

If there is no active pull request, returns None.

provider

str: The code hosting provider for the current CircleCI build.

repo_url

str: The URL of the current repository being built.

For example: https://github.com/{organization}/{repository} or https://bitbucket.org/{user}/{repository}.

slug

str: The current slug in the CircleCI build.

Of the form {organization}/{repository}.

tag

str: The git tag of the current CI build.

class ci_diff_helper.circle_ci.CircleCIRepoProvider[source]

Bases: enum.Enum

Enum representing all possible CircleCI repo providers.

bitbucket = <CircleCIRepoProvider.bitbucket: 'bitbucket'>
github = <CircleCIRepoProvider.github: 'github'>

ci_diff_helper.environment_vars module

Comprehensive list of environment variables used in ci-diff-helper.

These environment variables are core to this library. They are used to detect the current environment.

For more details, see the Travis env docs, AppVeyor env docs and _CircleCI env docs.

ci_diff_helper.environment_vars.APPVEYOR_BRANCH = 'APPVEYOR_REPO_BRANCH'

Indicates the active AppVeyor branch.

In a “pull request” build it is the base branch the PR is merging into, otherwise it is the branch being built.

ci_diff_helper.environment_vars.APPVEYOR_REPO = 'APPVEYOR_REPO_PROVIDER'

The code hosting provided for the repository being tested in AppVeyor.

ci_diff_helper.environment_vars.APPVEYOR_TAG = 'APPVEYOR_REPO_TAG_NAME'

The tag of the current AppVeyor build.

This will only be valid when APPVEYOR_REPO_TAG, i.e. when the build was started by a pushed tag.

ci_diff_helper.environment_vars.CIRCLE_CI_BRANCH = 'CIRCLE_BRANCH'

Indicates the active git branch being tested on CircleCI.

ci_diff_helper.environment_vars.CIRCLE_CI_PR = 'CI_PULL_REQUEST'

Pull request containing the current change set.

If the current build is part of only one pull request, the URL of that PR will be populated here. If there was more than one pull request, this field contain one of the pull request URLs (picked randomly).

ci_diff_helper.environment_vars.CIRCLE_CI_PRS = 'CI_PULL_REQUESTS'

Comma-separated list of pull requests current build is a part of.

ci_diff_helper.environment_vars.CIRCLE_CI_PR_NUM = 'CIRCLE_PR_NUMBER'

The ID of the PR that started the current build.

We only expect this environment variable to be set during a build that is a part of a pull request from a fork.

ci_diff_helper.environment_vars.CIRCLE_CI_PR_OWNER = 'CIRCLE_PR_USERNAME'

The owner of the forked repository that started the current PR build.

We only expect this environment variable to be set during a build that is a part of a pull request from a fork.

ci_diff_helper.environment_vars.CIRCLE_CI_PR_REPO = 'CIRCLE_PR_REPONAME'

The name of the forked repository that started the current PR build.

We only expect this environment variable to be set during a build that is a part of a pull request from a fork.

ci_diff_helper.environment_vars.CIRCLE_CI_REPO_URL = 'CIRCLE_REPOSITORY_URL'

A link to the homepage for the current repository.

ci_diff_helper.environment_vars.CIRCLE_CI_TAG = 'CIRCLE_TAG'

The name of the git tag being tested

Only set if the build is running for a tag.

ci_diff_helper.environment_vars.GH_TOKEN = 'GITHUB_OAUTH_TOKEN'

GitHub OAuth 2.0 token.

This environment variable must be used to authenticate to the GitHub API. Making unauthenticated requests on a Continuous Integration server will typically be rate limited.

ci_diff_helper.environment_vars.IN_APPVEYOR = 'APPVEYOR'

Indicates if running in AppVeyor.

ci_diff_helper.environment_vars.IN_CIRCLE_CI = 'CIRCLECI'

Indicates if running in CircleCI.

ci_diff_helper.environment_vars.IN_TRAVIS = 'TRAVIS'

Indicates if running in Travis.

ci_diff_helper.environment_vars.TRAVIS_BRANCH = 'TRAVIS_BRANCH'

Indicates the active Travis branch.

In a “push” build, this is the branch that was pushed while in a “pull request” build it is the branch that a pull request is against.

ci_diff_helper.environment_vars.TRAVIS_EVENT_TYPE = 'TRAVIS_EVENT_TYPE'

Indicates the type of build that is occurring.

ci_diff_helper.environment_vars.TRAVIS_PR = 'TRAVIS_PULL_REQUEST'

Indicates which Travis pull request we are in.

Is an integer when in a pull request or “false” when not.

ci_diff_helper.environment_vars.TRAVIS_RANGE = 'TRAVIS_COMMIT_RANGE'

The range of commits changed in the current build.

This is not particularly useful in a PR build.

Note

This is empty for builds triggered by the initial commit of a new branch.

ci_diff_helper.environment_vars.TRAVIS_SLUG = 'TRAVIS_REPO_SLUG'

The GitHub repository slug for the current Travis build.

A slug is of the form {organization}/{repository}.

ci_diff_helper.environment_vars.TRAVIS_TAG = 'TRAVIS_TAG'

The tag of the current Travis build.

We only expect the TRAVIS_TAG environment variable to be set during a tag “push” build, but it can be set as the empty string in non-“push” builds.

ci_diff_helper.git_tools module

Helpers for interacting with git.

ci_diff_helper.git_tools.commit_subject(revision='HEAD')[source]

Gets the subject of a git commit.

Parameters:revision (Optional [ str ]) – A git revision, any of a branch name, tag, a commit SHA or a special reference.
Returns:The commit subject.
Return type:str
ci_diff_helper.git_tools.get_checked_in_files()[source]

Gets a list of files in the current git repository.

Effectively runs:

$ git ls-files ${GIT_ROOT}

and then finds the absolute path for each file returned.

Returns:List of all filenames checked into.
Return type:list
ci_diff_helper.git_tools.git_root()[source]

Return the root directory of the current git checkout.

Returns:Filesystem path to git checkout root.
Return type:str
ci_diff_helper.git_tools.merge_commit(revision='HEAD')[source]

Checks if a git revision is a merge commit.

Parameters:revision (Optional [ str ]) – A git revision, any of a branch name, tag, a commit SHA or a special reference.
Returns:Flag indicating if the given revision.
Return type:bool
Raises:NotImplementedError – if the number of parents is not 1 or 2.

ci_diff_helper.travis module

Set of utilities for dealing with Travis CI.

This module provides a custom configuration type Travis for the Travis CI system.

Since Travis only works with GitHub, the commands in this module are GitHub and git centric.

This module uses a selection of environment variables to detect the state of Travis configuration. See environment_vars for more details.

Travis Configuration Type

When running in Travis, you can automatically detect your current environment and get the configuration object:

>>> import ci_diff_helper
>>> config = ci_diff_helper.get_config()
>>> config
<Travis (active=True)>

To use the Travis configuration type directly:

>>> config = ci_diff_helper.Travis()
>>> config
<Travis (active=True)>
>>> config.active
True
>>> config.in_pr
True
>>> config.branch
'master'

In addition this configuration provides extra features for determining a diffbase.

>>> config = ci_diff_helper.Travis()
>>> config.event_type
<TravisEventType.pull_request: 'pull_request'>
>>> config.slug
'organization/repository'
>>> config.repo_url
'https://github.com/organization/repository'
>>> config.pr
1234
>>> config.tag is None
True
>>> config.base
'master'

Not only is this object valuable during a pull request build, it can also be used to find relevant information in a “push” build:

>>> config = ci_diff_helper.Travis()
>>> config.event_type
<TravisEventType.push: 'push'>
>>> config.pr is None
True
>>> config.tag
'0.13.37'
>>> config.base
'4ad7349dc7223ebc02175a16dc577a013044a538'

Though the base property can be useful as a diffbase of a given commit, it may be inappropriate. In a “push” build, base will be computed from the TRAVIS_COMMIT_RANGE environment variable, and this value is not particularly reliable. Instead, merged_pr provides a way to determine the PR that was merged:

>>> config.is_merge
True
>>> config.merged_pr
1355
class ci_diff_helper.travis.Travis[source]

Bases: ci_diff_helper._config_base.Config

Represent Travis state and cache return values.

__delattr__

x.__delattr__(‘name’) <==> del x.name

__format__()

default object formatter

__getattribute__

x.__getattribute__(‘name’) <==> x.name

__hash__
__reduce__()

helper for pickle

__reduce_ex__()

helper for pickle

__setattr__

x.__setattr__(‘name’, value) <==> x.name = value

__sizeof__() → int

size of object in memory, in bytes

__str__
active

bool: Indicates if currently running in the target CI system.

base

str: The git object that current build is changed against.

The git object can be any of a branch name, tag, a commit SHA or a special reference.

Note

This will throw an OSError on the very first “push” build for a branch. This is because Travis leaves the value empty in builds triggered by the initial commit of a new branch.

Warning

This property is only meant to be used in a “pull request” or “push” build.

branch

bool: Indicates the current branch in the target CI system.

This may indicate the active branch or the base branch of a pull request.

event_type

bool: Indicates if currently running in Travis.

in_pr

bool: Indicates if currently running in Travis pull request.

This uses the TRAVIS_EVENT_TYPE environment variable to check if currently in a pull request. Though it doesn’t use the TRAVIS_PULL_REQUEST environment variable, checking that the value is set to an integer would be a perfectly valid approach.

is_merge

bool: Indicates if the HEAD commit is a merge commit.

merged_pr

int: The pull request corresponding to a merge commit at HEAD.

If not currently in a push build, returns None. If the HEAD commit is not a merge commit, returns None.

Note

This only uses the git checkout to determine the pull request ID. A more comprehensive check would involve veriying the ID by using the GitHub API.

Warning

This property is only meant to be used in a “pull request” or “push” build.

pr

int: The current Travis pull request (if any).

If there is no active pull request, returns None.

repo_url

str: The URL of the current repository being built.

Of the form https://github.com/{organization}/{repository}.

slug

str: The current slug in the Travis build.

Of the form {organization}/{repository}.

tag

str: The git tag of the current Travis build.

Note

We only expect the TRAVIS_TAG environment variable to be set during a tag “push” build, but we don’t verify that we are in a push build before checking for the tag.

class ci_diff_helper.travis.TravisEventType[source]

Bases: enum.Enum

Enum representing all possible Travis event types.

api = <TravisEventType.api: 'api'>
cron = <TravisEventType.cron: 'cron'>
pull_request = <TravisEventType.pull_request: 'pull_request'>
push = <TravisEventType.push: 'push'>