(Github-Workflows)(DevEx) Wordpress Playground for Gutenberg PRs

[fullofcaffeine]

What?

Post a comment in PRs with a WordPress Playground URL allowing devs to use it to test the changeset in the PR.

Inspired by the work done for Woocommerce by @samueljseay.

Why?

To provide a quick way to test the changesets in a fully client-side-based disposable WordPress instance that can be brought up with a single click. No need to build the plugin locally and spin up wp-env if you don't want to :). Wordpress+Gutenberg manual testing bliss <3

How?

The Playground is a novel way to quickly create disposable and fast WordPress instances in the browser. It also has supporting logic that allows its configuration to be orchestrated via the JS or URL params (blueprints). We can pass a JSON in the URL, and the Playground app will use it as a definition for setting up the clientside WordPress instance with the plugins defined. In our case, we use the proxy provided by it, which exposes an interface to install plugins to the instance already.

❌ I tried different approaches (see the commit history), including one with a separate workflow that polled the build-plugin-zip.yml every x seconds. This worked and gave additional flexibility in the sense that we could add additional event criteria, like ignoring some files in the PR. See the implementation here: 44badf8. However, it later became clear that it became a little too complex + it could cause throttling if there were a lot of PRs triggering it at the same time. it's a good example of how to use polling and the GH API to make a separate workflow depend on one another. Clunky but works.

✅ I ended up scrapping that and decided to implement it as a new job called create-playground-link inside the build-plugin-zip.yml workflow. It depends on the build job and only runs if the trigger event is pull_request and if build is successful. This means that the workflow won't try to run this job if it was started via the workflow_dispatch event, so it won't affect the release flow in any way. The job checks- out Gutenberg and runs a JS script that builds the JSON blueprint and encodes it in the playground URL. Finally, it posts the blueprint in a comment to the PR (or updates it if it already exists).

💡 I borrowed from the work done in WooCommerce and adapted it to the Gutenberg workflows. Kudos to @samueljseay for the initial work there! 🙇🏻

Testing Instructions

Testing the workflow - opening a PR:

1. Fork my Gutenberg repo in https://github.com/fullofcaffeine/gutenberg;
2. Create a new branch from trunk and push it, create a new PR;
3. Go to the Actions tab and find the "Build Gutenberg Plugin Zip` action on the left sidebar, click on it and click on the pending one - you'll need to wait a bit here until an agent picks it up, it can take a while;
4. It should start running and skip quite a few jobs, running only the "Build Release Artifact" job (because it was started by a pull request event). Wait for it to finish (brew some coffee or tea, as it can take up to 10m).
5. Once it finishes, it should then run the "Create WordPress Playground Link" job which should run pretty quickly - and once this one finishes, the comment should then appear in your PR:

Editing a PR (pushing updates):

1. Push any changes to the PR;
2. You should not see any duplicate comments.

Testing that the link works

After you have access to the link in the message from your test repo/PR, if you click it, you'll notice an error like this:

This is because the blueprint JSON refers to a PR that the playground proxy can't find. You'll need to edit the JSON in the URL to point to a PR that actually exists in the official Gutenberg repo, the repo that the proxy from playground.wordpress.net is set up to look into (unless you install playground somewhere or locally and proxy it!). This won't happen in production as all URLs will be generated from PRs in the official GB repo.

So, you can do the following:

1. Copy the link in your test PR (or use this PR: Foobar fullofcaffeine/gutenberg#25);
2. Paste it in a text editor, find the "pr=x" part where x is the number of your PR;
3. Change it to an ID of any existing PR in Gutenberg, e.g 62391;
4. Now paste it into a new tab in your browser, and it should work fine (see screenshot below). You'll notice the loading label ("Downloading Gutenberg PR xx) will still refer to the old Gutenberg number (see screencast below), but that shouldn't affect anything - if it makes you feel better, you can also edit the number there in the blueprint JSON :).

Screenshots or screencast

Screen.Recording.2024-06-06.at.8.27.26.p.m.mov

[github-actions]

The following accounts have interacted with this PR and/or linked issues. I will continue to update these lists as activity occurs. You can also manually ask me to refresh this list by adding the props-bot label.

If you're merging code through a pull request on GitHub, copy and paste the following into the bottom of the merge commit message.

Co-authored-by: fullofcaffeine <fullofcaffeine@git.wordpress.org>
Co-authored-by: DaniGuardiola <daniguardiola@git.wordpress.org>
Co-authored-by: WunderBart <bartkalisz@git.wordpress.org>
Co-authored-by: youknowriad <youknowriad@git.wordpress.org>
Co-authored-by: ramonjd <ramonopoly@git.wordpress.org>
Co-authored-by: kevin940726 <kevin940726@git.wordpress.org>
Co-authored-by: noisysocks <noisysocks@git.wordpress.org>

To understand the WordPress project's expectations around crediting contributors, please review the Contributor Attribution page in the Core Handbook.

[DaniGuardiola]

LGTM and makes a lot of sense. I only have a few suggestions and nits you can ignore if you want.

[youknowriad]

With this, it's going to be the sixth bot comment on PRs on Gutenberg. All of the comments are useful but their number makes it so that it's very noisy and distract a lot from the PR.

We have:

• A comment to welcome new contributors
• A comment to tell new contributors that they're missing the connection in WP profiles (yes, I think there are two comments for new contributors)
• A comment from the props bot
• A comment about the bundle size
• A comment about flaky tests
• A comment about modified php files
• A comment about the previews on Playground
• And we also discussed adding a comment about the "performance job results" (link to summary)

I like all of these comments and I actually have ideas to add more comments but this is not sustainable IMO for PR authors and reviewers. We should explore merging all of these comments into:

• One pre-merge comment.
• One post-merge comment.

And smartly use things like "details" to hide/show sections.

(I don't consider this comment a blocker for this PR, but I think we should explore that soonish)

[fullofcaffeine]

(I don't consider this comment a blocker for this PR, but I think we should explore that soonish)
We should explore merging all of these comments into:

That's an interesting idea. I think the simplest way would be an action that would look for comments and aggregate/merge them. This way, we wouldn't need to change the existing integrations to call a different intermediate API, for example. The other challenge is how to make the merged comment look good - it'd be quite a long/tall one I think, so we'd have to experiment and compare the approach to see if it's really worth it UX-wise.

[youknowriad]

so we'd have to experiment and compare the approach to see if it's really worth it UX-wise.

What's really clear to me is that the current approach (multiple comments) is not great in terms of DevX to be honest. You're so overwhelmed that you ignore all of them.

[fullofcaffeine]

I also tested by using the URL, pushing an update to the PR (using this test PR), waiting for the new zip to be available, and using that URL again, and it does load the new version with the changes 👍🏻

[fullofcaffeine]

This should be ready to go, but there's an error in the build-plugin-zip when running from this PR: https://github.com/WordPress/gutenberg/actions/runs/9425690036/job/25967789409?pr=62393:

(...)
    data: {
      message: 'Resource not accessible by integration',
      documentation_url: 'https://docs.github.com/rest/issues/comments#create-an-issue-comment',
      status: '403'
    }
(...)

It's weird, as we should have permission to create the comment (we have other integrations/actions doing the same). It does test well from my fork: fullofcaffeine#27 (comment). I'm not sure if it has to do with the fact it's running from a branch in another repo. I'm past-EOD now so I'll refrain from merging now and will double-check this early next week.

[ramonjd]

Thanks for working on this @fullofcaffeine! It's something I tried out ages ago but ran into some permissions funk for forked PRs.

With this, it's going to be the sixth bot comment on PRs on Gutenberg. All of the comments are useful but their number makes it so that it's very noisy and distract a lot from the PR.

I agree, but I understand Github does not provide many alternatives.

One thing I tried (and failed) was to have a placeholder in Gutenberg PR description templates.

So, after a PR is created, the placeholder would be used as a hook to embed the Playground content into the description itself.

Another thing I didn't try was to hook onto another event that's only triggered after PR creation, e.g., the build plugin action, and then append a Playground link to the PR description (or not if the link already exists).

It seems like others would like this feature out of the box too!

• https://github.com/orgs/community/discussions/36086

I didn't test the workaround on the above discussion though.

🍺

[kevin940726]

One thing I tried (and failed) was to have a placeholder in Gutenberg PR description templates.

One thing that I discovered way too late after implementing flaky-test-reporter is that GH seems to keep the markdown intact. Since GH-flavored markdown is essentially just a subset of HTML, we can theoretically insert a dummy <div> and use it to position or store any content.

<div id="playground-link">

Playground markdown here...

</div>

Then we can just parse the content of the markdown and replace the content more easily.

The advantage of this approach over using HTML comments is that the markdown can be easier to read and write to. And we can use a proper parser instead of relying on regexp, which is harder to maintain. For instance, using <data> tags to store machine-readable data alongside the text.

<data id="build-date" value="1718087147767">6/11/2024, 2:25:47 PM</data>

I think it's possible to group all comments from bots into one comment and just update them through this method. Not sure about the interactions between different third-party apps though 🤔.

[noisysocks]

What's really clear to me is that the current approach (multiple comments) is not great in terms of DevX to be honest. You're so overwhelmed that you ignore all of them.

Huge +1, it's already overwhelming.

Instead of having an action that aggregate comments, can't we combine all of the existing actions into a Gutenberg Pre-Commit action and a Gutenberg Post-Commit action? Each one can be compromised of separate .mjs files for maintainability.

Sorry to chime in on this PR, it's not a blocker for this work. Maybe best to open a separate issue about reducing the action spam.

[DaniGuardiola]

Taking over.