Plugins

How plugins work

GitProxy supports extensibility in the form of plugins. These plugins are specified via configuration as NPM packages or JavaScript/TypeScript code on disk. For each plugin configured, GitProxy will attempt to load each package or file as a standard Node module. Plugin authors will create instances of the extension classes exposed by GitProxy and use these objects to implement custom functionality.

For each loaded "plugin object", it is inserted into GitProxy's chain of actions which are triggered on a given Git action received by GitProxy such as git push or git fetch. Action chains are divided into different phases, according to the available data and completed checks.

caution

Before v2.3, plugins executed before GitProxy's built-in steps by default. Now, you can configure when to run each plugin by specifying a PullPhase or PushPhase as explained below. Plugins will execute immediately after the phase begins, and in the order they are configured in proxy.config.json. If you wish to use a combination of features, ensure that your custom plugins do not conflict or interfere with later steps in the processing chain.

GitProxy uses the load-plugin package to provide the Node module resolution.

TypeScript Support

Plugins can be written in TypeScript, as shown in our push secret scanner sample.

Limitations

Plugins are only supported on the Git HTTP proxy server. There is no similar extensibility today for the dashboard UI or its backing API.

Using plugins

The primary goals of the plugin system is to:

Allow users deploying GitProxy to do so via the distributed binary package. ie. npx -- @finos/git-proxy
Allow users to consume plugins that packaged using standard tools (npm) and optionally distributed via npm themselves.
Reuse as much native Node functionality as possible for calling 3rd-party code (plugins) from the main application (git-proxy).

The below instructions are using the @finos/git-proxy-plugin-samples package as an demonstrative set of plugins.

via npm packages

Steps

Install both @finos/git-proxy and the @finos/git-proxy-plugin-samples package to the local node_modules/ directory using the following command:

$ npm install -g @finos/git-proxy @finos/git-proxy-plugin-samples

Alternatively, you can create local packages from source and install those instead.

$ git clone https://github.com/finos/git-proxy
$ cd git-proxy
$ npm pack
$ (cd plugins/git-proxy-plugin-samples && npm pack)
$ npm install -g \
  ./finos-git-proxy-2.1.0.tgz \
  ./plugins/git-proxy-plugin-samples/finos-git-proxy-plugin-samples-0.1.2.tgz

Create or edit an existing proxy.config.json file to configure the plugin(s) to load. You must include the full import path that would typically be used in a import {} (ESM) or require() (CJS) statement:

{
  "plugins": [
    "@finos/git-proxy-plugin-samples",
    "@finos/git-proxy-plugin-samples/example.cjs",
    "@finos/git-proxy-plugin-samples/customSecretScanner",
    "@finos/git-proxy-plugin-samples/extractPullData",
    "@finos/git-proxy-plugin-samples/pullSecretScanner"
  ]
}

Run git-proxy.

$ git-proxy
Service Listening on 8080
HTTP Proxy Listening on 8000
HTTPS Proxy Listening on 8443
Found 5 plugin modules
Loaded plugin: RunOnPullPlugin
Loaded plugin: HelloPlugin
Loaded plugin: CustomSecretScanner
Loaded plugin: ExtractPullDataPlugin
Loaded plugin: PullSecretScanner

via JavaScript/TypeScript file(s)

caution

This section is considered highly experimental and not recommended for general use. Even when authoring local plugins that are not distributed, it is best to use node_modules/ and not rely on bespoke system setup or layout of files. Use npm pack and npm install path/to/plugin.tgz if you must use local plugin code to ensure GitProxy's plugin manager can properly load your plugins.

Plugins written as standalone JavaScript or TypeScript files are used similarly to npm packages. The main difference is that file-based module loading requires additional steps to ensure that the given files (written as Node CommonJS or ES modules) have all the necessary dependencies to be imported. Since GitProxy plugin system relies on class-based inheritence, the module will require at least a dependency to @finos/git-proxy to be able to import the required classes. Plugins do not have to be distributed by NPM to be used in this fashion which may be advantageous in certain environments.

To use a plugin that is written as a standalone JavaScript file, ensure that the JS code has all its necessary dependencies:

$ cd path/to/plugindir
$ cat package.json
{
  "name": "foo-plugin",
  ...
  "dependencies": {
    "@finos/git-proxy": "^2.1.0"
  }
}
# Alternatively, add git-proxy that is cloned locally as a file-based dependency
$ cat package.json
{
  "name": "foo-plugin",
  "dependencies": {
    "@finos/git-proxy": "file:/path/to/checked/out/finos/git-proxy"
  }
}
$ npm install

Create or edit an existing proxy.config.json file to configure the plugin(s) to load:

{
  "plugins": ["path/to/plugin/index.js"]
}

Run git-proxy

$ git-proxy
Service Listening on 8080
HTTP Proxy Listening on 8000
HTTPS Proxy Listening on 8443
Found 1 plugin modules
Loaded plugin: FooPlugin

Developing new plugins

To develop a new plugin, you must add @finos/git-proxy as a peer dependency. The plugin & proxy classes can then be imported from @finos/git-proxy via the following extension points:

import { PushActionPlugin } from '@finos/git-proxy/plugin';

Use this class to execute as an action in the proxy chain during a git push

import { PullActionPlugin } from '@finos/git-proxy/plugin';

Use this class to execute as an action in the proxy chain during a git fetch

import { Step, Action } from '@finos/git-proxy/proxy/actions';

These are internal classes which act as carriers for git state during proxying. Plugins should modify the passed in Action for affecting any global state of the git operation and add its own custom Step object to capture the plugin's own internal state (logs, errored/blocked status, etc).

GitProxy will load your plugin only if it extends one of the two plugin classes above.

Please see the sample plugin package included in the repo for details on how to structure your plugin package.

If your plugin relies on custom state, it is recommended to create subclasses in the following manner:

import { PushActionPlugin } from '@finos/git-proxy/plugin';

class FooPlugin extends PushActionPlugin {
  constructor() {
    super(); // don't pass any function to the parent class - let the parent's this.exec function be undefined
    this.someProperty = 'Hello from the FooPlugin';
    // overwrite the parent's exec function which is executed as part of GitProxy's action chain.
    // use an arrow function if you require access to the instances's state (properties, methods, etc.)
    this.exec = async (req, action) => {
      console.log(this.someProperty);
      this.utilMethod();
      // do something
      return action;
    };
  }

  utilMethod() {
    // misc operations specific to FooPlugin
  }
}

Reacting to successful actions with `onChainSuccess`

Plugins may optionally define an onChainSuccess(req, action) method. GitProxy calls it once the whole action chain has finished without being rejected/blocked by intermediate steps or the plugin itself. Any errors thrown inside onChainSuccess will be logged and won't affect the result of the git operation.

This is useful for storing state that requires the whole action to execute successfully before running the plugin. For example, extractPullData caches the wants to prevent re-fetching, but only for pulls that weren't blocked:

class ExtractPullDataPlugin extends PullActionPlugin {
  constructor() {
    super(exec, { phase: PullPhase.AFTER_AUTHORISATION });
  }

  onChainSuccess(req, action) {
    rememberRecentFetch(action);
  }
}

Instructions for creating a new plugin package

Create a new directory for your plugin package and initialize a new npm package:

$ npm init
# ...
$ npm install --save-peer @finos/git-proxy

Create a new JavaScript file for your plugin. The file should export an instance of PushActionPlugin or PullActionPlugin:

import { PushActionPlugin } from '@finos/git-proxy/plugin';
import { Step } from '@finos/git-proxy/proxy/actions';

//Note: Only use a default export if you do not rely on any state. Otherwise, create a sub-class of [Push/Pull]ActionPlugin
export default new PushActionPlugin(function (req, action) {
  // the action parameter holds all the details about a push.
  // see https://github.com/finos/git-proxy/blob/main/src/proxy/actions/Action.js

  // create a new Step to track any side effects of this plugin function (errors, blocking pushes, logs, etc)
  // see https://github.com/finos/git-proxy/blob/main/src/proxy/actions/Step.js
  const step = new Step('bar-plugin');
  action.addStep(step);
  // add custom logic here.
  // ...
  return action; // always return the action - even in the case of errors
});

In your GitProxy installation, install your plugin package & update the proxy.config.json file to include the path to your plugin module(s):

$ npm install @finos/git-proxy /path/to/your/plugin/package
$ cat proxy.config.json
{
  "plugins": [
    "/path/to/your/plugin/package/index.js"
  ]
}
$ git-proxy
Service Listening on 8080
HTTP Proxy Listening on 8000
HTTPS Proxy Listening on 8443
Found 1 plugin modules
Loaded plugin: PushActionPlugin

Plugin configuration

Plugins accept options (PushPluginOptions and PullPluginOptions to fine-grain their behaviour:

Chain Phases

Action chains are divided into phases. In each phase of the chain, certain conditions will be guaranteed and certain attributes will be available.

For example, branch pushes include an AFTER_PERMISSIONS phase, which guarantees that the repository is in the authorised list before running the plugin. By default, plugins are set to phase: AFTER_PERMISSIONS, which tells GitProxy to execute the plugin immediately after we've finished checking that the repository can actually be pushed to.

Push chain phases

Currently, there are 4 different PushPhases to run your plugins. Note that the phases executed will depend on the type of object being pushed (tags vs pushes).

AFTER_PERMISSIONS: The push is guaranteed to be in the authorised list, and identity of the user pushing is also present. The following Action attributes will be populated: action.user, action.userEmail, action.commitData, etc.
AFTER_CHECKOUT: Commit messages, author emails and the user's push permission have finished validating. The actual remote is already pulled but the diff hasn't been processed. The following Action attributes will be populated: action.newIdxFiles.
AFTER_DIFF: The push will now contain the diff data, accessible via action.diff (obtained from getDiff). Note that tag pushes do not contain any diff, thus the AFTER_DIFF PushPhase would be invalid for tag pushes.
BEFORE_APPROVAL: The push has gone through the remaining checks (GitLeaks, built-in diff scanning) in the chain, and is about to request approval.

Depending on what your plugin does, you'll want to pick a phase where the data you need is available. A custom diff scanning plugin must be set to AFTER_DIFF or BEFORE_APPROVAL, otherwise it will not have the action.diff readily available.

Pull chain phases

The pull chain currently executes checkRepoInAuthorisedList followed by two PullPhases:

AFTER_AUTHORISATION: The repository is guaranteed to be in the authorisedList. By default, plugins run in this phase. This is because it's unsafe to run plugins before validating that the repository can actually be pulled from.
AFTER_CHECKOUT: Runs right after AFTER_AUTHORISATION. Core GitProxy does not parse pulls or fetch their data, so this phase is meant to be used after a plugin such as extractPullData has checked out the repo and populated the pull data. It's meant for policy plugins that inspect the content of commits being fetched.

Scanning pulls

GitProxy doesn't parse or fetch pulls by default, it merely checks that an admin has enabled to pull from that repo. Pull plugins only see basic Action fields (action.url, action.repo, etc.). In order to inspect actual content being fetched, you need two types of plugins (both of these available in @finos/git-proxy-plugin-samples):

A data plugin that parses the raw git request and stores information about the pull into the Action. It doesn't enforce any policies by itself, but may reject the pull if it fails to extract the data. extractPullData is the sample data plugin.
A policy plugin that reads the data and decides whether the pull is allowed. pullSecretScanner is a sample policy plugin.

These two sample plugins can be used as follows:

{
  "plugins": [
    "@finos/git-proxy-plugin-samples/extractPullData",
    "@finos/git-proxy-plugin-samples/pullSecretScanner"
  ]
}

If you look at the plugin code, you'll notice that they don't import each other. They work by sharing the Action fields described below. Their order in plugins doesn't matter either, since they run in their defined Phase.

Data provided by `extractPullData`

action.pullData: The parsed upload-pack request object which includes: command, wants, haves, wantRefs, refPrefixes, shallow, capabilities, etc. action.actionType is set to fetch or ls-refs.
action.proxyGitPath: A temporary bare repository containing a shallow, blobless fetch of the wanted commits (pullData.fetchedWants). It gets deleted once the chain has finished.
pullData.base / action.commitFrom and action.commitTo: The pull's diff range. The base is the first of the client's haves that exists upstream (all zeros if none, meaning a full-tree scan). commitTo is the first want.
pullData.wantedRefs, action.branch and action.tags: The ref names the wanted commits resolve to.

The plugin runs git through simple-git. For HTTPS upstreams with self-signed certificates, configure git with the right CA (or set GIT_SSL_NO_VERIFY).

Writing your own policy plugin

Set phase: PullPhase.AFTER_CHECKOUT, then:

Check that action.pullData is present. If not, the data plugin isn't configured correctly and the pull must be rejected.
If action.proxyGitPath isn't set, there is nothing to inspect. This could happen with an ls-refs request, or pullData.repeatedRound when the same commits were alreadry accepted.
Otherwise, run git against action.proxyGitPath. Blobs are fetched from the upstream repository on demand.

The sample pullSecretScanner provided follows these rules. It scans the content based on the pullData.fetchedWants and blocks the fetch if a secret is found or git fails. Note how it scans the net diff rather than each commit.

`isCollectible`

Errors in a given action or plugin can be collected at the end of the chain. When isCollectible: true, the chain will continue running even if the plugin errors out. If set to false, plugin failure will immediately stop the chain and return the error to the user.

Setting isCollectible to true allows for a better user experience, as users can see all the errors and fix them in one go. However, actions that perform critical tasks, such as parsing the push (parsePush) or obtaining the diff (getDiff) must fail fast as the missing attributes would affect subsequent actions and plugins.

`chains`

Currently, chains are only available for PushActionPlugin. By default, plugins run for both branch pushes and tag pushes. However, you can fine-tune which chains to execute on, by passing in a PushChainName[] as follows: chains: ['tag', 'branch'].

The phase must exist on the chain to be valid. For example, a diff scanner plugin would have to set the phase to AFTER_DIFF and chains: ['branch'], because tags don't have a diff or an AFTER_DIFF phase.

`displayName`

This is the user-facing plugin name. It will be shown on the push database, as well as during the push itself as a sideband message:

$ git push

Enumerating objects: 5, done.
Counting objects: 100% (5/5), done.
Delta compression using up to 14 threads
Compressing objects: 100% (3/3), done.
Writing objects: 100% (3/3), 367 bytes | 367.00 KiB/s, done.
Total 3 (delta 1), reused 0 (delta 0), pack-reused 0
remote: Resolving user from token...
remote: Checking for empty branch...
remote: Checking repository is authorised...
remote: Checking commit messages...
remote: Checking author emails...
remote: Checking push permissions...
remote: Fetching remote repository...
remote: Writing pack data...
remote: Checking for hidden commits...
remote: Checking approval status...
remote: Running pre-receive hook...
remote: Computing diff...
remote: Running CustomSecretScanner...       <- displayName shown
remote: Scanning for secrets...
remote: Scanning diff contents...
remote: Requesting approval...
remote: 
remote: 
remote: 
remote: GitProxy has received your push ✅

How plugins work​

TypeScript Support​

Limitations​

Using plugins​

via npm packages​

Steps​

via JavaScript/TypeScript file(s)​

Developing new plugins​

Reacting to successful actions with onChainSuccess​

Instructions for creating a new plugin package​

Plugin configuration​

Chain Phases​

Push chain phases​

Pull chain phases​

Scanning pulls​

Data provided by extractPullData​

Writing your own policy plugin​

isCollectible​

chains​

displayName​