Skip to main content
Version: 1.50

Configuring BuildKit for High Performance

A high-performance BuildKit setup speeds up image builds, tests, and the deploy commands that run with Remote Execution.

1. Dedicated Node Pool for BuildKit​

BuildKit is resource-intensive. We recommend running BuildKit in a dedicated node pool, for the best performance and to keep other workloads from interfering with it.

To run BuildKit in a dedicated node pool, add taints and tolerations to the BuildKit node pool in Kubernetes, and add the following settings to your Okteto Helm configuration:

buildkit:
nodeSelectors:
okteto-node-pool: build
tolerations:
- effect: NoSchedule
key: okteto-node-pool
operator: Equal
value: build

2. Vertical Scaling for Faster Builds​

BuildKit performance depends heavily on the CPU and memory it gets. Start with 2 nodes for higher availability, with these recommended instance types (4 CPUs and 16 GB of RAM):

  • Amazon Web Services: m6a.xlarge
  • Google Cloud Platform: t2d-standard-4
  • Microsoft Azure: Standard_D4as_v5

Set replicaCount to the number of nodes in the BuildKit node pool:

buildkit:
replicaCount: 2

Monitor performance and adjust node resources as needed. For example, upgrading your BuildKit nodes to 8 CPUs and 32 GB of memory each can roughly double their performance.

3. SSD Storage​

BuildKit is I/O-intensive, especially when it pulls, extracts, and pushes container images. An SSD storage class can significantly improve BuildKit performance:

buildkit:
persistence:
enabled: true
class: <<your-ssd-storage-class>>

You can also increase the size of the BuildKit cache (100 GB by default) to get more cache hits:

buildkit:
persistence:
enabled: true
size: 200Gi

4. HPA for Performance and Costs​

The Horizontal Pod Autoscaler (HPA) adjusts the number of BuildKit pods to the active builds. You get enough capacity at peak times, and lower costs when builds are idle.

Build Queue System and HPA

The Build Queue System distributes builds across BuildKit instances and queues builds when instances exceed resource thresholds. This keeps build performance consistent, but the Build Queue System doesn't scale BuildKit pods.

Without HPA, you must choose between running fewer BuildKit instances (which queues more builds during peak usage, increasing wait times) or running many replicas at all times (which wastes resources during idle periods). HPA complements the Build Queue System by automatically adding and removing BuildKit pods based on demand — providing enough build capacity at peak times while reducing costs when demand is low.

Okteto supports the following metrics for HPA scaling:

  • okteto_build_active_builds: number of active builds (recommended)
  • okteto_build_cpu_pressure: CPU pressure in 10-second intervals
  • okteto_build_memory_usage: memory usage, as a percentage
  • okteto_build_io_pressure: I/O pressure in 10-second intervals

By default, HPA scales on the okteto_build_active_builds metric, with the following settings:

buildkit:
hpa:
enabled: true
min: 1
max: 5
metrics:
- type: Pods
pods:
metric:
name: okteto_build_active_builds
target:
type: AverageValue
averageValue: "0.99"
behavior:
scaleUp:
policies:
- type: Pods
value: 1
periodSeconds: 30
scaleDown:
stabilizationWindowSeconds: 600
policies:
- type: Pods
value: 1
periodSeconds: 150

This configuration:

  • Scales up quickly when build pressure increases: it adds up to one pod every 30 seconds
  • Scales down conservatively to avoid thrashing: it waits 10 minutes after the load drops, then removes one pod every 150 seconds
  • Maintains between 1 and 5 BuildKit pods based on active builds
Metrics Adapter

Okteto includes a metrics adapter that exposes BuildKit metrics through the v1beta2.custom.metrics.k8s.io API. Okteto deploys it automatically when HPA is enabled.

If you already run your own custom metrics adapter, such as Prometheus Adapter, disable Okteto's adapter by setting buildkit.hpa.adapter.enabled: false in your Helm configuration. Your adapter can then scrape the metrics directly from the BuildKit pods on port 8080. All metrics have the okteto_build prefix.

5. Best Practices for Dockerfiles and Remote Execution​

A fast BuildKit setup doesn't make up for unoptimized Dockerfiles or inefficient Remote Execution configurations. To get the most out of BuildKit, we recommend that you: