Orchestrate your builds using Dynamic Pipelines (2024)

This tutorial describes how to create a Forge app implementing bitbucket:dynamicPipelinesProvidermodule and install it in a Bitbucket workspace. The Dynamic Pipelines provider app responds torequests from Bitbucket Pipelines when user wants to see which pipelines they can run for a branch,or when a particular pipeline is requested to run by a user, on schedule, or by a commit pushedevent in the repository. The app is expected to respond with a valid Pipelines configuration, orwith an error in case the pipeline run should be aborted instead.

Before you begin

To complete this tutorial, you need the latest version of Forge CLI. To update your CLI version, runnpm install -g @forge/cli@latest on the command line.

We recommend that you complete all the steps in Getting startedso that you’re familiar with the Forge development process.

For Bitbucket apps you need to join or create a shared Bitbucket team workspace (as Forge apps are not supported on personal workspaces).If you don't have a Bitbucket workspace, see the references below for related instructions:

Creating a Bitbucket Cloud account.
Join or create a workspace.

A free Bitbucket team space can have up to 5 users.

Step 1: Create your app

Create an app based on the Dynamic Pipelines provider template.

Navigate to the directory where you want to create the app.
Create your app by running:
```
12
forge create
```
Enter a name for the app. For example, dynamic-pipelines-provider-app.
Select the Triggers and Validators category.
Select Bitbucket product.
Select the bitbucket-dynamic-pipelines-provider template from the list.
Open the app directory to see the app files.

Step 2: Define app permissions

Should your app need to retrieve more information about the pipeline context such as commit orbranch information from Bitbucket Cloud, it has to define the Forge app scopes to use when callingthe Bitbucket Cloud REST API.

In this section, we'll add the read:repository:bitbucket scope (required to access the commitsin the repository).

Open the manifest.yml file.

Add the following section:

12
permissions: scopes: - 'read:repository:bitbucket'

See Scopes for detailed information about the available scopes.

The full manifest file should now look like:

12
modules: bitbucket:dynamicPipelinesProvider: - key: dynamic-pipelines-provider function: main name: Example Dynamic Pipelines provider description: A Dynamic Pipelines provider example function: - key: main handler: index.mainpermissions: scopes: - 'read:repository:bitbucket'app: id: <your-app-id-here>

Step 3: Review & deploy the template app

The dynamic pipeline app is executed when the Forge runtime calls the function declared in themanifest file. This will happen in one of two scenarios:

When an event occurs that requires the generation of a pipeline configuration in order to actuallyexecute a pipeline. e.g. A new commit being pushed to a Pull Request.
When a user completes an action that requires the generation of a list of pipeline configurationsthat could be run against a particular branch or tag. e.g. Opening the “Run Pipeline” dialogon the Pipelines tab of a repository in Bitbucket Cloud.

For this tutorial, when the app is executed the main function in the src/index.js file will be called.

12
export const main = (request, context) => { console.log(`Handling Dynamic Pipelines request: ${JSON.stringify(request, null, 4)}`); return { pipelines_configuration: request.pipelines_configuration };};

The implementation of the main function logs the request payload and returns the unchangedconfiguration back to Bitbucket. This would result in a detailed log being written to the Forge applogs, and the Pipelines system executing the exact configuration that was contained in the .ymlfile of the originating repository.

In your app's top-level directory, deploy your app by running:

12
forge deploy
See Also
Job Opening: Machinist IV at My Next Move for Veterans

Then run the install command:
```
12
forge install
```

Then select Bitbucket from the product list:

12
Select a product: Jira Confluence Compass❯ Bitbucket

Enter the URL for your Bitbucket workspace:

12
Enter the workspace URL: bitbucket.org/{your-workspace-id}

Agree to the scopes that the app will have access to:

12
Your app will be installed with the following scopes:- read:repository:bitbucket? Do you want to continue? (y/N) y

The app will then install into your workspace.

12
✔ Install complete!Your app in the development environment is now installed in Bitbucket on bitbucket.org

Step 4: Configuring your repository to use the dynamic pipeline app

Now that the app is installed into the workspace, you need to configure it to run for a particularrepository when pipelines are executed.

Navigate to the settings for a repository in the installed workspace where you want to test thedynamic pipeline app and open the “Dynamic Pipelines” tab.
From that tab, select the dynamic pipelines provider you just installed from the “Repositorydynamic pipeline provider” list, and then press “Save”:

Step 5: Use the Forge tunnel to test the app

Next up, we’re going to test the Dynamic Pipeline to make sure it is working correctly. To do this,we will use the Forge tunnel utility that you would have set up in theGetting started guide.

Start a tunnel by running:
```
12
forge tunnel
```
This enables you to test your app without having to deploy it after each change.

The tunnel works for changes to the code, not for configuration changes to the manifest.ymlfile. If you change the app configuration in the manifest.yml, you need to redeploy the app(forge deploy), and then restart the tunnel (forge tunnel).
Run a pipeline in the repository you previously configured with your dynamic pipeline by eithercreating a commit in that repo, or manually via UI.

Verify Handling Dynamic Pipelines request plus the printed request object appears in yourtunnel CLI output.

Note: The request object you see will be different to that shown here, the main thing toverify is that a request object with a similar shape is printed.

12
INFO 23:55:21.092 b8a065db98924683b85b09b383c6dc9c Handling Dynamic Pipelines request: { "target": { "commit": { "hash": "<commit-hash>" }, "type": "pipeline_commit_target" }, "trigger": "definitions", "pipelines_configuration": { "pipelines": { "default": [ { "step": { "name": "Scan for secrets", "script": [ { "pipe": "atlassian/git-secrets-scan:1.5.0", "variables": { "ANNOTATION_SUMMARY": "Secret found in code.", "FILES": "*" } } ] } } ] }, "image": { "name": "node:18" } }, "creator": "<user-uuid>", "repository": "<repo-uuid>", "workspace": "<workspace-uuid>", "context": { "cloudId": "<random-uuid>", "moduleKey": "my-bitbucket-dynamic-pipelines-provider" }, "contextToken": "a-really-long-string"}

Step 6: Transform the pipeline configuration

In Step 5, we demonstrated how requesting a pipeline run invoked the main method in src/index.js.

In this step, you’ll extend the app to modify the pipeline definition and add an extra step to it incases where the default pipeline from the static YML configuration was selected.

Create a new addStepIfDefaultPipeline function:

12
const addStepIfDefaultPipeline = (request) => { // If the pipeline does not contain a 'default' definition, just return what was sent in the request. if (!request.pipelines_configuration?.pipelines?.default) { return { pipelines_configuration: request.pipelines_configuration }; } // Otherwise prepend a step to the 'default' pipeline request.pipelines_configuration.pipelines.default = [ { step: { name: "Dynamic step", script: ["echo 'This step was added dynamically'"], }, }, ...request.pipelines_configuration.pipelines?.default, ]; // and return it. return request;};

This function prepends a step to the pipeline definition in situations where the defaultpipeline was selected from the static bitbucket-pipelines.yml file.

Modify the main function to call the addStepIfDefaultPipeline function when a pipeline run isrequested, log the outcome from that function call, and return the outcome in the response:

12
export const main = (request, context) => { const updatedPipelineRequest = addStepIfDefaultPipeline(request); console.log(`Updated request: ${JSON.stringify(updatedPipelineRequest, null, 4)}`); return { pipelines_configuration: updatedPipelineRequest.pipelines_configuration };};

Start your tunnel if it’s not still running from Step 5.
Run the default pipeline in a repo in your Bitbucket workspace, and verify the following:
- Updated request appears in your tunnel and the logged output contains the new step.
- The triggered pipeline has an extra step which was generated by the app, and that stepcorrectly logged the input specified.
- The link to the dynamic configuration appears in the sidebar.
- And you can open the dynamically generated configuration and view it from the UI.
  Note: The configuration rendered in the UI is translated back into the equivalent YAMLrepresentation, with the dynamically generated changes includes. We do this as most users arefamiliar with pipeline configurations in their YAML representation, not JSON.
Run some other pipeline in the same repo (not the default pipeline), and verify the following:
- Updated request appears in your tunnel.
- The triggered pipeline does not have steps generated dynamically.

Step 7: Use the Bitbucket REST API to modify the response

In Step 6, we dynamically applied some changes to the pipeline configuration. In this section,you'll update your app to call the Bitbucket REST API. Using theProduct Fetch API from the @forge/api package,you'll get the latest commit stats and print the number of changed files directly to the pipelinestep logs so that they’re visible from the pipeline details view in Bitbucket Cloud.

In the app’s top-level directory, install the npm package dependency for the Forge API by running:
```
12
npm install @forge/api --save
```
Start your tunnel (or restart it in case you still have it running from Step 6) to use the newNPM modules by running:
```
12
forge tunnel
```

In the src/index.js, import the @forge/api package by adding the following to the top of the file:

12
import api, { route } from "@forge/api";

Declare a new getCommitDiffStat function:

12
const getCommitDiffStat = async (workspaceId, repositoryId, commitHash) => { const requestUrl = route`/2.0/repositories/${workspaceId}/${repositoryId}/diffstat/${commitHash}`; // Use the Forge Runtime API to fetch data from Bitbucket API using your app credential let response = await api.asApp().requestBitbucket(requestUrl); if (response.status !== 200) { console.log(response.status); const responseBody = await response.json(); throw `Unable to get diffstat for commit ${commitHash}, status: ${ response.status }, response: ${JSON.stringify(responseBody, null, 4)}`; } return response.json();};

Update your addStepIfDefaultPipeline function to use async/await now that you are makinganother API call from inside it:

12
const addStepIfDefaultPipeline = async (request) => { // code from earlier};

From inside the addStepIfDefaultPipeline function, call the new getCommitDiffState functionand pass in the workspace ID, repo ID, & commit hash, remembering to await the response:

12
const addStepIfDefaultPipeline = async (request) => { // If not the 'default' pipeline definition matched from the static YML configuration, return unmodified configuration if (!request.pipelines_configuration.pipelines?.default) { return { pipelines_configuration: request.pipelines_configuration }; } // All the values you need are part of the "request" object. const commitDiffStat = await getCommitDiffStat( request.workspace, request.repository, request.target.commit.hash, ); // Otherwise prepend a step to the 'default' pipeline ...};

From the returned commitDiffStat, extract the size property, which indicates the number offiles changed, and include that in the log line you print from the injected step:

12
const addStepIfDefaultPipeline = async (request) => { // If the pipeline does not contain a default definition, just return what was sent in the request. if (!request.pipelines_configuration?.pipelines?.default) return request; // get the diff state for the commit. const commitDiffStat = await getCommitDiffStat( request.workspace, request.repository, request.target.commit.hash, ); // get the number of files changed. const changedFilesNumber = commitDiffStat.size; // add a second line to the script section, injecting the number of files changed for the comit. request.pipelines_configuration.pipelines.default = [ { step: { name: "Dynamic step", script: [ "echo 'This step was added dynamically'", `echo 'The change for this request contained ${changedFilesNumber} files.'` ], }, }, ...request.pipelines_configuration.pipelines?.default, ]; // and return the whole request. return request;};

Finally, adjust the main function to be async and ensure you await theaddStepIfDefaultPipeline() request:

12
export const main = async (request, context) => { const updatedPipelineRequest = await addStepIfDefaultPipeline(request); console.log(`Updated request: ${JSON.stringify(updatedPipelineRequest, null, 4)}`); return { pipelines_configuration: updatedPipelineRequest.pipelines_configuration };};

Run the default pipeline in a repo in your Bitbucket workspace, and verify that the triggeredpipeline now has an extra step that prints the number of changed files in the commit the pipelineruns against.

Step 8: Deploy your app

In the previous step, you made changes to your app using the Forge Runtime API and then tested thechanges using a tunnel. Now that your app is working, deploy the changes so the app continues towork when the tunnel is closed.

Close the tunnel by pressing Ctrl+C.
Deploy the app by running:
```
12
forge deploy
```
Run a pipeline in a repo in the workspace the app is installed in, and confirm that it runs theexpected steps.

Great work! You’ve created a Forge app that:

Detects when a pipeline is requested to run in the workspace where it is installed.
Responds by looking up the commit the pipeline was requested to run against, identifying how manyfiles were changed in that commit, and dynamically adjusting the configuration of that pipelinebased on what you found.

What’s next?

You could check which files were changed, and adjust which pipeline steps you run based on the outcome.
You could apply some validation to your pipeline, to ensure it meets the processes and rules setby your team.
You could configure this dynamic pipeline to run at the Workspace level, and see how it worksacross every repository in the workspace.
And many many more things in addition to just those.

If you’d like to learn more, share ideas for different capabilities, ask questions, and see what’scoming soon, make sure you check out the Forge for Bitbucket Cloud community groupand get involved!