Configuring BuildKit for High Performance
A high-performance BuildKit setup speeds up image builds, tests, and the deploy commands that run with Remote Execution.
1. Dedicated Node Pool for BuildKit
BuildKit is resource-intensive. We recommend running BuildKit in a dedicated node pool, for the best performance and to keep other workloads from interfering with it.
To run BuildKit in a dedicated node pool, add taints and tolerations to the BuildKit node pool in Kubernetes, and add the following settings to your Okteto Helm configuration:
buildkit:
nodeSelectors:
okteto-node-pool: build
tolerations:
- effect: NoSchedule
key: okteto-node-pool
operator: Equal
value: build
2. Vertical Scaling for Faster Builds
BuildKit performance depends heavily on the CPU and memory it gets. Start with 2 nodes for higher availability, with these recommended instance types (4 CPUs and 16 GB of RAM):
- Amazon Web Services:
m6a.xlarge - Google Cloud Platform:
t2d-standard-4 - Microsoft Azure:
Standard_D4as_v5
Set replicaCount to the number of nodes in the BuildKit node pool:
buildkit:
replicaCount: 2
Monitor performance and adjust node resources as needed. For example, upgrading your BuildKit nodes to 8 CPUs and 32 GB of memory each can roughly double their performance.
3. SSD Storage
BuildKit is I/O-intensive, especially when it pulls, extracts, and pushes container images. An SSD storage class can significantly improve BuildKit performance:
buildkit:
persistence:
enabled: true
class: <<your-ssd-storage-class>>
You can also increase the size of the BuildKit cache (100 GB by default) to get more cache hits:
buildkit:
persistence:
enabled: true
size: 200Gi
4. HPA for Performance and Costs
The Horizontal Pod Autoscaler (HPA) adjusts the number of BuildKit pods to the active builds. You get enough capacity at peak times, and lower costs when builds are idle.
The Build Queue System distributes builds across BuildKit instances and queues builds when instances exceed resource thresholds. This keeps build performance consistent, but the Build Queue System doesn't scale BuildKit pods.
Without HPA, you must choose between running fewer BuildKit instances (which queues more builds during peak usage, increasing wait times) or running many replicas at all times (which wastes resources during idle periods). HPA complements the Build Queue System by automatically adding and removing BuildKit pods based on demand — providing enough build capacity at peak times while reducing costs when demand is low.
Okteto supports the following metrics for HPA scaling:
okteto_build_active_builds: number of active builds (recommended)okteto_build_cpu_pressure: CPU pressure in 10-second intervalsokteto_build_memory_usage: memory usage, as a percentageokteto_build_io_pressure: I/O pressure in 10-second intervals
By default, HPA scales on the okteto_build_active_builds metric, with the following settings:
buildkit:
hpa:
enabled: true
min: 1
max: 5
metrics:
- type: Pods
pods:
metric:
name: okteto_build_active_builds
target:
type: AverageValue
averageValue: "0.99"
behavior:
scaleUp:
policies:
- type: Pods
value: 1
periodSeconds: 30
scaleDown:
stabilizationWindowSeconds: 600
policies:
- type: Pods
value: 1
periodSeconds: 150
This configuration:
- Scales up quickly when build pressure increases: it adds up to one pod every 30 seconds
- Scales down conservatively to avoid thrashing: it waits 10 minutes after the load drops, then removes one pod every 150 seconds
- Maintains between 1 and 5 BuildKit pods based on active builds
Okteto includes a metrics adapter that exposes BuildKit metrics through the v1beta2.custom.metrics.k8s.io API. Okteto deploys it automatically when HPA is enabled.
If you already run your own custom metrics adapter, such as Prometheus Adapter, disable Okteto's adapter by setting buildkit.hpa.adapter.enabled: false in your Helm configuration. Your adapter can then scrape the metrics directly from the BuildKit pods on port 8080. All metrics have the okteto_build prefix.
5. Best Practices for Dockerfiles and Remote Execution
A fast BuildKit setup doesn't make up for unoptimized Dockerfiles or inefficient Remote Execution configurations. To get the most out of BuildKit, we recommend that you:
- Follow the Dockerfile best practices.
- Reduce file transfer times to BuildKit with .dockerignore and .oktetoignore files.
- Use BuildKit cache mounts to persist cache folders between image builds. Caching dependencies and build artifacts can significantly speed up builds.
- Use Okteto Test caches to persist cache folders between test executions. Test dependencies stay cached, so later test runs take less time.