Using mass-driver

Installation

Install the package:

pip install mass-driver

We recommend you install CLIs via pipx, for dependency isolation:

pipx install mass-driver

If you want to install from a git branch rather than Pypi:

pipx install https://github.com/OverkillGuy/mass-driver

See pipx docs: https://pypa.github.io/pipx/#running-from-source-control

Command help

Use the help menu to start with:

mass-driver --help

usage: mass-driver [-h]
                   {drivers,driver,forges,forge,sources,source,run,scanners,view-pr}
                   ...

Send bulk repo change requests

options:
  -h, --help            show this help message and exit

Commands:
  {drivers,driver,forges,forge,sources,source,run,scanners,view-pr}
    drivers (driver)    Inspect drivers (Plugins)
    forges (forge)      Inspect forges (Plugins)
    sources (source)    Inspect sources (Plugins)
    run                 Run mass-driver, migration/forge activity across repos
    scanners            List available scanners
    view-pr             Check status of given pull requests

Inspecting drivers (forges and scanners have similar menus) via mass-driver drivers --help.

And about running the actual mass driver run-command mass-driver run --help:

usage: mass-driver run [-h] [--json-outfile JSON_OUTFILE] [--no-pause]
                       [--parallel] [--no-cache] [--repo-path [REPO_PATH ...]
                       | --repo-filelist REPO_FILELIST]
                       activity_file

positional arguments:
  activity_file         Filepath of activity to apply (TOML file)

options:
  -h, --help            show this help message and exit
  --json-outfile JSON_OUTFILE
                        If set, store the output to JSON file with this name
  --no-pause            Disable the interactive pause between Migration and
                        Forge
  --parallel            Run the processing of repos in parallel, via up to 8
                        threads
  --no-cache            Disable any repo caching
  --repo-path [REPO_PATH ...]
                        One or more Repositories to patch. If not local paths,
                        will git clone them
  --repo-filelist REPO_FILELIST
                        File with list of Repositories to Patch. If not local
                        paths, will git clone them

Preparing a change

Let’s prepare for doing a change over dozens of repositories. We’ll need to find a PatchDriver that suits our needs, and configure it accordingly.

List available PatchDrivers via:

mass-driver drivers --list
# The docs for a single driver:
mass-driver driver --info counter

Remember, PatchDrivers are exposed via a python plugin system, which means anyone can package their own!

Once you’ve got a driver, you should create a Migration file, in TOML:

# Saved as "fix_teamname.toml"
[mass-driver.migration]
# As seen in 'git log':
commit_message = """Change team name

Team name XYZ is wrong, we should be called ABC instead.
See JIRA-123[1].

[1]: https://example.com/tickets/JIRA-123
"""

# Override the local git commit author
commit_author_name = "John Smith"
commit_author_email = "smith@example.com"

branch_name = "fix-team-name"

# PatchDriver class to use.
# Selected via plugin name, from "massdriver.drivers" entrypoint
driver_name = "teamname-changer"

# Config given to the PatchDriver instance
driver_config = { filename = "catalog.yaml", team_name = "Core Team" }

# Note: No "forge" section = no forge activity to pursue (no PR will be created)

With this file named fix_teamname.toml in hand, we can apply the change locally, either against a local repo we’ve already cloned:

mass-driver run fix_teamname.toml --repo-path ~/workspace/my-repo/

Or against a repo being cloned from URL:

mass-driver run fix_teamname.toml --repo-path 'git@github.com:OverkillGuy/sphinx-needs-test.git'

The cloned repo will be under .mass_driver/repos/USER/REPONAME/. We should expect a branch named fix-team-name with a single commit.

To apply the change over a list of repositories, create a file with relevant repos:

cat <<EOF > repos.txt
git@github.com:OverkillGuy/sphinx-needs-test.git
git@github.com:OverkillGuy/speeders.git
EOF

mass-driver run fix_teamname.toml --repo-filelist repos.txt

Creating PRs

Once the commits are done locally, let’s send them up as PR a second step. For this, we’ll be creating a second activity file containing a Forge definition.

Similarly, forges can be listed and detailed:

mass-driver forges --list
# The docs for a single forge:
mass-driver forge --info counter

Consider using the forge_name = "github". Create a new Activity with a Forge:

# An Activity made up of just a forge
[mass-driver.forge]
forge_name = "github"

base_branch = "main"

head_branch = "fix-teamname"
draft_pr = true
pr_title = "[JIRA-123] Bump counter.txt to 1"
pr_body = """Change team name

Team name XYZ is wrong, we should be called ABC instead.
See JIRA-123[1].

[1]: https://example.com/tickets/JIRA-123
"""

# Do you need to git push the branch before PR?
git_push_first = true

Now run mass-driver, remembering to set the FORGE_TOKEN envvar for a Github/other auth token.

export FORGE_TOKEN="ghp_supersecrettoken"
mass-driver run fix_teamname_forge.toml --repo-filelist repos.txt

Combining migration then forge

Sometimes, we wish to expedite both the committing and the PR creation in a single move.

The Activity file can contain both sections:

# An activity made up of first a Migration, then a Forge
[mass-driver.migration]
# As seen in 'git log':
commit_message = """Change team name

Team name XYZ is wrong, we should be called ABC instead.
See JIRA-123[1].

[1]: https://example.com/tickets/JIRA-123
"""

# Override the local git commit author
commit_author_name = "John Smith"
commit_author_email = "smith@example.com"

branch_name = "fix-team-name"

# PatchDriver class to use.
# Selected via plugin name, from "massdriver.drivers" entrypoint
driver_name = "teamname-changer"

# Config given to the PatchDriver instance
driver_config = { filename = "catalog.yaml", team_name = "Core Team" }

# And a forge = PR creation after Migration
[mass-driver.forge]
forge_name = "github"

base_branch = "main"

head_branch = "fix-teamname"
draft_pr = true
pr_title = "[JIRA-123] Bump counter.txt to 1"
pr_body = """Change team name

Team name XYZ is wrong, we should be called ABC instead.
See JIRA-123[1].

[1]: https://example.com/tickets/JIRA-123
"""

# Do you need to git push the branch before PR?
git_push_first = true

Discovering repos using a Source

Sometimes, the repos we want to apply patches to is a dynamic thing, coming from tooling, like a Github repository search, some compliance tool report, service catalogue, etc.

To address this, mass-driver can use a Source plugin to discover repositories to apply activities to.

# An Activity file with a file-list Source
[mass-driver.source]
source_name = "repo-list"
# Source 'repo-list' takes a 'repos' list of cloneable URLs:
[mass-driver.source.source_config]
repos = [
  "git@github.com:OverkillGuy/mass-driver.git",
  "git@github.com:OverkillGuy/mass-driver-plugins.git",
]

Because we included a Source, we can omit the CLI flags --repo-path or --repo-filelist, to instead rely on the activity’s config to discover the repos.

mass-driver run activity.toml

Smarter Sources can use more elaborate parameters, maybe even secret parameters like API tokens.

Note that to pass secrets safely at runtime, config parameters passed via source_config in file format can be passed as envvar, using prefix SOURCE_. So we could have avoided the repos entry in file, by providing a SOURCE_REPOS envvar instead. This feature works because the Source class derives from Pydantic.BaseSettings.

As a Source developer, though, you should really look into usage of Pydantic.SecretStr to avoid leaking the secret when config or result is stored. See Pydantic docs on Secret fields.

Using the scanners

Before doing any actual migration, we might want to explore existing repositories to see what kind of change is required.

Mass-driver provides for this usecase via the scanners plugin system, enabling a simple python function to be run against many repos, with the purpose of gathering information.

Let’s define an Activity file specifying a list of scanners to run:

# An Activity file for scanning
[mass-driver.scan]
scanner_names = ["root-files", "dockerfile-from"]

This can be run just like a migration:

mass-driver run scan.toml --repo-filelist repos.txt

Reviewing bulk PR status

Have a look at the view-pr subcommand for reviewing the status of many PRs at once.

It requires specifying a forge like github, along with setting any required tokens, such as via FORGE_TOKEN envvar for github forge.

export FORGE_TOKEN=xyz
mass-driver view-pr github \
    --pr \
    https://github.com/OverkillGuy/mass-driver/pull/1 https://github.com/OverkillGuy/mass-driver/pull/2
# Can specify multiple PRs as a args list

Equivalently via a file full of newline-delimited PR URLs

export FORGE_TOKEN=xyz
mass-driver view-pr github --pr-filelist prs.txt

With sample result:

> Pull request review mode!
[001/004] Fetching PR status...
[002/004] Fetching PR status...
[003/004] Fetching PR status...
[004/004] Fetching PR status...

Merged:
https://github.com/OverkillGuy/mass-driver/pull/1
https://github.com/OverkillGuy/sphinx-needs-test/pull/1

Closed (but not merged):
https://github.com/OverkillGuy/mass-driver/pull/2
https://github.com/OverkillGuy/sphinx-needs-test/pull/2

In summary: 4 unique PRs, of which...
- 002 (50.0%) merged
- 002 (50.0%) closed (but not merged)