Create a 'Python core' team

To manage permissions, a 'Python core' team will be created as part of the python organization [16]. Any repository that is moved will have the 'Python core' team added to it with write permissions [17]. Anyone who previously had rights to manage SSH keys on hg.python.org will become a team maintainer for the 'Python core' team.

Define commands to move a Mercurial repository to Git

Since moving to GitHub also entails moving to Git [4], we must decide what tools and commands we will run to translate a Mercurial repository to Git. The tools developed specifically for this migration are hosted at https://github.com/orsenthil/cpython-hg-to-git .

CLA enforcement

A key part of any open source project is making sure that its source code can be properly licensed. This requires making sure all people making contributions have signed a contributor license agreement (CLA) [18]. Up until now, enforcement of CLA signing of contributed code has been enforced by core developers checking whether someone had an * by their username on bugs.python.org [5]. With this migration, the plan is to start off with automated checking and enforcement of contributors signing the CLA.

Document steps to commit a pull request

During the process of choosing a new development workflow, it was decided that a linear history is desired. People preferred having a single commit representing a single change instead of having a set of unrelated commits lead to a merge commit that represented a single change. This means that the convenient "Merge" button in GitHub pull requests is undesirable, as it creates a merge commit along with all of the contributor's individual commits (this does not affect the other repositories where the desire for a linear history doesn't exist).

Luckily, Git [4] does not require GitHub's workflow and so one can be chosen which gives us a linear history by using Git's CLI. The expectation is that all pull requests will be fast-forwarded and rebased before being pushed to the master repository. This should give proper attribution to the pull request author in the Git history. This does have the consequence of losing some GitHub features such as automatic closing of pull requests, link generation, etc.

A second set of recommended commands will also be written for committing a contribution from a patch file uploaded to bugs.python.org [5]. This will obviously help keep the linear history, but it will need to be made to have attribution to the patch author.

The exact sequence of commands that will be given as guidelines to core developers is an open issue: Git CLI commands for committing a pull request to cpython.

Linking pull requests to issues

Historically, external contributions were attached to an issue on bugs.python.org [5] thanks to the fact that all external contributions were uploaded as a file. For changes committed by a core developer who committed a change directly, the specifying of an issue number in the commit message of the format Issue # at the start of the message led to a comment being posted to the issue linking to the commit.

Update linking service for mapping commit IDs to URLs

Currently you can use https://hg.python.org/lookup/ with a revision ID from either the Subversion or Mercurial copies of the cpython repo [15] to get redirected to the URL for that revision in the Mercurial repository. The URL rewriter will need to be updated to redirect to the Git repository and to support the new revision IDs created for the Git repository.

Deprecate sys._mercurial

Once Python is no longer kept in Mercurial, the sys._mercurial attribute will need to be changed to return ('CPython', '', ''). An equivalent sys._git attribute will be added which fulfills the same use-cases.

Update the devguide

The devguide will need to be updated with details of the new workflow. Mostly likely work will take place in a separate branch until the migration actually occurs.

Update PEP 101

The release process will need to be updated as necessary.

Handling Misc/NEWS

Traditionally the Misc/NEWS file [19] has been problematic for changes which spanned Python releases. Often times there will be merge conflicts when committing a change between e.g., 3.5 and 3.6 only in the Misc/NEWS file. It's so common, in fact, that the example instructions in the devguide explicitly mention how to resolve conflicts in the Misc/NEWS file [21]. As part of our tool modernization, working with the Misc/NEWS file will be simplified.

There are currently two competing approaches to solving the Misc/NEWS problem which are discussed in an open issue: How to handle the Misc/NEWS file.

Handling Misc/ACKS

Traditionally the Misc/ACKS file [20] has been managed by hand. But thanks to Git supporting an author value as well as a committer value per commit, authorship of a commit can be part of the history of the code itself.

As such, manual management of Misc/ACKS will become optional. A script will be written that will collect all author and committer names and merge them into Misc/ACKS with all of the names listed prior to the move to Git. Running this script will become part of the release process.

The script should also generate a list of all people who contributed since the last execution. This will allow having a list of those who contributed to a specific release so they can be explicitly thanked.

Create https://git.python.org

Just as hg.python.org [1] currently points to the Mercurial repository for Python, git.python.org should do the equivalent for the Git repository.

Backup of pull request data

Since GitHub [2] is going to be used for code hosting and code review, those two things need to be backed up. In the case of code hosting, the backup is implicit as all non-shallow Git [4] clones contain the full history of the repository, hence there will be many backups of the repository.

The code review history does not have the same implicit backup mechanism as the repository itself. That means a daily backup of code review history should be done so that it is not lost in case of any issues with GitHub. It also helps guarantee that a migration from GitHub to some other code review system is feasible were GitHub to disappear overnight.

Bot to handle pull request merging

As stated in the section entitled "Document steps to commit a pull request", the desire is to maintain a linear history for cpython. Unfortunately, Github's [2] web-based workflow does not support a linear history. Because of this, a bot should be written to substitute for GitHub's in-browser commit abilities.

To start, the bot should accept commands to commit a pull request against a list of branches. This allows for committing a pull request that fixes a bug in multiple versions of Python.

More advanced features such as a commit queue can come later. This would linearly apply accepted pull requests and verify that the commits did not interfere with each other by running the test suite and backing out commits if the test run failed. To help facilitate the speed of testing, all patches committed since the last test run can be applied and run in a single test run as the optimistic assumption is that the patches will work in tandem. Some mechanism to re-run the tests in case of test flakiness will be needed, whether it is from removing a "test failed" label, web interface for core developers to trigger another testing event, etc.

Inspiration or basis of the bot could be taken from pre-existig bots such as Homu [31] or Zuul [32].

The name given to this bot in order to give it commands is an open issue: Naming the bots.

Continuous integration per pull request

To help speed up pull request approvals, continuous integration testing should be used. This helps mitigate the need for a core developer to download a patch simply to run the test suite against the patch.

Which free CI service to use is an open issue: Choosing a CI service.

Test coverage report

Getting an up-to-date test coverage report for Python's standard library would be extremely beneficial as generating such a report can take quite a while to produce.

There are a couple pre-existing services that provide free test coverage for open source projects. Which option is best is an open issue: Choosing a test coverage service.

Notifying issues of pull request comments

The current development process does not include notifying an issue on bugs.python.org [5] when a review comment is left on Rietveld [6]. It would be nice to fix this so that people can subscribe only to comments at bugs.python.org and not GitHub [2] and yet still know when something occurs on GitHub in terms of review comments on relevant pull requests. Current thinking is to post a comment to bugs.python.org to the relevant issue when at least one review comment has been made over a certain period of time (e.g., 15 or 30 minutes). This keeps the email volume down for those that receive both GitHub and bugs.python.org email notifications while still making sure that those only following bugs.python.org know when there might be a review comment to address.

Allow bugs.python.org to use GitHub as a login provider

As of right now, bugs.python.org [5] allows people to log in using Google, Launchpad, or OpenID credentials. It would be good to expand this to GitHub credentials.

Web hooks for re-generating web content

The content at https://docs.python.org/, https://docs.python.org/devguide, and https://www.python.org/dev/peps/ are all derived from files kept in one of the repositories to be moved as part of this migration. As such, it would be nice to set up appropriate webhooks to trigger rebuilding the appropriate web content when the files they are based on change instead of having to wait for, e.g., a cronjob to trigger.

Link web content back to files that it is generated from

It would be helpful for people who find issues with any of the documentation that is generated from a file to have a link on each page which points back to the file on GitHub [2] that stores the content of the page. That would allow for quick pull requests to fix simple things such as spelling mistakes.

Splitting out parts of the documentation into their own repositories

While certain parts of the documentation at https://docs.python.org change with the code, other parts are fairly static and are not tightly bound to the CPython code itself. The following sections of the documentation fit this category of slow-changing, loosely-coupled:

• Tutorial
• Python Setup and Usage
• HOWTOs
• Installing Python Modules
• Distributing Python Modules
• Extending and Embedding
• FAQs

These parts of the documentation could be broken out into their own repositories to simplify their maintenance and to expand who has commit rights to them to ease in their maintenance.

It has also been suggested to split out the What's New documents. That would require deciding whether a workflow could be developed where it would be difficult to forget to update What's New (potentially through a label added to PRs, like "What's New needed").

Backup of Git repositories

While not necessary, it would be good to have official backups of the various Git repositories for disaster protection. It will be up to the PSF infrastructure committee to decide if this is worthwhile or unnecessary.

Identify potential new core developers

The Python development team has long-standing guidelines for selecting new core developers. The key part of the guidelines is that a person needs to have contributed multiple patches which have been accepted and are high enough quality and size to demonstrate an understanding of Python's development process. A bot could be written which tracks patch acceptance rates and generates a report to help identify contributors who warrant consideration for becoming core developers. This work doesn't even necessarily require GitHub integration as long as the committer field in all git commits is filled in properly.