Job logs now have the addition of light theme on build summaries – ensuring those that find lower contrast easy-to-read have an alternative option.

Unblocking engineers - and your work – is the core theme of how we think of everything here at Buildkite, including usability. The new light theme was inspired by Ethan Schoonover's Solarized theme. Now, when you open a job log, you’ll see a new “Theme” control – enabling you to switch between our current dark mode theme and the new light theme. Your preference will be saved for future views, with the option for you to switch views.

As always, we'd love your feedback. Drop into our Slack community, or send us an email: hello@buildkite.com 👋

Today we’re introducing the ability to configure multiple GitHub Enterprise Servers.

Some of our bigger teams with different requirements for source control have multiple GitHub Enterprise Server installations. We’ve only supported one per Buildkite organization — until now. If you’re on a compatible plan, you can now add as many different servers as you like from the repository providers page in your organization settings.

We love feedback! If you use this feature and love it, or it doesn’t quite do what you need, drop into our Slack community, or send us an email: hello@buildkite.com 👋

Failed builds now jump straight to a new Issues tab on the build page 🗂

Builds with hundreds of jobs can hide what really matters. Instead of scrolling through the noise, Issues shows you only the failed annotations and jobs in a build. This helps you get your code shipping faster. And the rest of your jobs are only a tab click away.

Check out the blog post to learn more about this change.

As always, we'd love your feedback. Drop into our Slack community, or say hi: hello@buildkite.com 👋

Update, Sep 2023: We are no longer offering per pipeline build retention. For more information on retention per plan, see check out the docs.

Today, we're introducing build retention controls for pipelines so you can automatically remove old builds ⏰🧨

You can now configure pipelines to keep builds for a certain period of time. There's also an option to always keep a minimum number of the latest builds, so you don't lose context on pipelines that don't move quite so fast. These are available today — find any pipeline, then go to Settings, Builds, and look for the Build Retention section.

This change is currently optional, but we'll be announcing a broader rollout for build retention soon.

We'd love to hear from you if you find these controls useful, or if they don't quite fit your needs first. Drop into our Slack community, or send us an email: hello@buildkite.com 👋

We've released a new schema browser for our GraphQL API 🎉

You can now easily browse the entire structure of the API right there in the docs, without needing to log in first

You can now specify default timeouts for command steps ⏰💥

Timeouts can be specified on command steps already, but it's a pain to have to do it every time, and easy to forget. Setting default timeouts means you won't have command steps slip through the cracks and accidentally run forever, or consume too many job minutes.

Default and maximum timeouts can be specified at an organization and pipeline level. Defaults apply in order of specificity — the step's timeout will be used if supplied, otherwise it defaults to the pipeline's default if supplied, or finally to the organization. Maximums are always enforced, when supplied — the smallest value will be used.

These timeouts can be configured from your organization's Pipeline Settings page, or on a pipeline's Builds settings page.

We'd love to hear from you if you find these timeouts useful, or if they don't quite fit your needs first. Drop into our Slack community, or send us an email: hello@buildkite.com 👋

Today we're introducing a new "failing" build state for faster feedback 🏃‍♂️💨

We know it's frustrating when your build fails. But when it does, you want that information as soon as possible so you can get it green.

Now, as soon as a single job fails, the build can be marked as "failing". You'll see a GitHub Commit Status, or a Slack notification — whatever your team uses — as soon as we know your build is going to fail. And on the dashboard your build will show as failing. If you're using annotations or Test Analytics then your build can also tell you what to do to fix the problem while it's still finishing.

But fast feedback is only half of the equation. There's nothing worse than a build which takes forever just to tell you it's broken. If you're in a big team running lots of builds, running the rest of a failing build might be a waste of time. So we're also introducing "cancel on build failing."

steps:
  - command: bundle && bin/rails spec
    parallelism: 100
    cancel_on_build_failing: true

Adding the cancel_on_build_failing: true attribute to your command steps will cause those jobs to be canceled as soon as a build becomes failing. That way your agents can move on to the next piece of work. But we'll still send failing notifications before cancelation is complete.

We realise that some folks might need to update API clients or adjust pipeline and workflows, so this new feature is being introduced as opt-in for a smooth migration path. But you can turn on failing builds right now by going to your pipeline on the Buildkite dashboard, then going to the Settings page, Builds, and enabling Fail Fast:

You can also turn this on for your whole organization on your org's Pipeline Settings page.

The buttons will go away and the feature will be enabled for everybody on Monday, 5th September, 2022.

If this does exactly what you need, or you have a suggestion, we'd love to hear from you in our community Slack channel, or drop us an email to support@buildkite.com.

For customers triggering builds via GitHub, we've added a new environment variable to track the draft status of a pull request:

BUILDKITE_PULL_REQUEST_DRAFT="true"

If a pull request triggers a build while in a draft state, this variable will be present in all the build's command jobs. The variable will be absent otherwise.

This variable allows customizing your uploaded pipelines or running jobs at the agent level based on a pull request's draft status. For example, you might run a faster subset of your tests while a pull request is still in a draft state.

Check out our docs on environment variables for more information.

If you have any feedback, we'd love to hear from you in our community Slack channel, or drop us an email to support@buildkite.com.

We know how frustrating finding and fixing flaky tests can be. And how costly your test suite performance degradations can become over time. We’ve been working to solve these problems for good, with our newest product — Buildkite Test Analytics.

As of today, Test Analytics is out of beta and available to everyone 🎉

• Automatically identify flaky tests, see what is causing the most disruption for your team, and get a head-start on fixing them for good.
• Get deep visibility and tracing for your test suite, integrated with your programming language and test framework.
• Set targets and identify problems as soon as they occur with configurable speed and reliability monitors.

Test Analytics is included in all Buildkite plans and is free to get started. You can even use it with any CI system, including GitHub Actions, CircleCI, and Jenkins ⚡️

Read more on the website, in our documentation — and if you're super excited, retweet!

We've added Build Matrix so you can create multiple jobs from a single, multi-variant step definition. 🧮

Build Matrices can also include multiple dimensions, different combinations with the adjustments key, and matrix jobs can be grouped together to clean up build pages. ⚡️

Read more about it in the blog, documentation, or on Twitter.

We've added a new group pipeline step type so you can group steps together, set dependencies between them, reduce YAML repetition, and clean up your build pages 🏗✨

You can read more about it in the blog post, documentation, or on Twitter.

For teams using Datadog, we've recently made it easier to send information about your Buildkite pipelines to Datadog’s Continuous Integration Visibility. This is a simple integration that any organization using both Datadog and Buildkite can enable to get insights into their pipeline’s performance over time. 📈

For more details on the integration check out the documentation 📚

For teams that run Buildkite Pipelines and build Xcode based software projects for macOS, iOS, iPadOS, tvOS, and watchOS apps, you can now run your Buildkite Builds on AWS EC2 Mac instances using a CloudFormation template. 🎉

This experimental template creates an Auto Scaling group, launch template, and host resource group to launch a pool of EC2 Mac instances that run the Buildkite Agent.

📣 Big shoutout to Buildkite customer Oliver Koo for his early input into this feature 🙏

📚 For details on how to prepare and deploy this template to your AWS Account, checkout the Auto Scaling EC2 Mac documentation, or jump straight into Elastic CI Stack for EC2 Mac.

Identify, track, and fix problematic tests with ✨Test Analytics✨, now in private beta.

🔍 Identify flaky tests: See which tests are most disruptive with automatic flaky test identification and analysis over time.

👀 Monitor speed & reliability: Get alerts when slow tests are introduced.

📈 Deeply analyze performance: First-class framework integrations provide in-depth performance analysis, helping you find external dependency bottlenecks and more.

🏎 View real-time results: Watch your test suite run in real-time and get immediate performance insights.

Learn more and join our waitlist at https://buildkite.com/test-analytics.

For teams running a lot of jobs who want more control over what order their jobs run in, we've introduced the Job Priority attribute to specify its priority within its queue. 🙋🏻‍♀️

It's a new attribute defined on a step in the pipeline configuration that changes how the dispatcher assigns work to agents. The higher the value you set for priority, the sooner a job will be prioritised. 🔝

Our recently released Eager Concurrency feature can also be used in conjunction with Job Priority.

Enterprise teams using Okta or Azure AD for SSO with their Buildkite organization can now optionally enable SCIM to automatically deprovision users ⚡️

Okta and Azure AD SSO with SAML is still available to all users.

A step-by-step guide to enabling SCIM support is available in our Single Sign-On with Okta and Azure AD documentation 📚

Traditionally, concurrency groups enforce strict ordering of the jobs within them. Concurrent jobs will be limited to the number set for the group, and will only start executing in the order they were created. 📝

However, sometimes you only need the limit, and it's fine for eligible jobs to run in whatever order their dependencies allow. 🏃🏻‍♀️

Which is why we've added a concurrency_method step attribute you can set to 'eager'. This will allow any job, up to the concurrency limit, to start as soon as it's eligible, regardless of creation order. 🔀

As an example:

steps:
  - command: echo "Using a limited resource, only 10 at a time, but we don't care about order"
    concurrency_group: saucelabs
    concurrency: 10
    concurrency_method: eager

For more information, check out our guide to Controlling Concurrency.

For teams running their own compute scheduler or those needing single-use agents, we've added the --acquire-job flag to the Buildkite Agent 🧰

With this flag, agents are mapped 1:1 with jobs. This makes them perfect for folks using a system like Kubernetes, Nomad, or ECS, or anyone who wants a clean-room environment.

To get started, pass the --acquire-job flag with the job ID to the buildkite agent start command. When you start an agent with this flag it will run the job then exit rather than polling for more work.

You can use acquire-job with agents v3.17.0 and above, anywhere that the agent can run 🏃🏻✨

For details on how to use the flag, see the agent cli documentation on Running a single job.

For teams wanting to add an extra layer of security to their webhooks, we have just added the option to include HMAC signatures with your Webhook Notification services 🔐

You'll find the HMAC signature option under the Token section when editing or creating Webhook Notifications.

For more information on verifying webhook signatures and defending against replay attacks, see our Webhook API documentation.

For complete control over when to trigger builds, use conditional expressions to filter incoming webhooks 📥

Build filtering with conditionals is available with any repository provider. You can add a conditional on your Pipeline Settings page or via the REST API 🔀

For more information on supported conditionals, check out our guide to Using Conditionals.