Configuring BuildKit for High Performance
Optimizing BuildKit performance is essential for a smooth development experience with Okteto. A high-performance BuildKit setup accelerates image builds, test executions, and deploy commands when using Remote Execution.
1. Dedicated Node Pool for BuildKit
BuildKit is resource-intensive. We recommend running BuildKit in a dedicated node pool, for the best performance and to keep other workloads from interfering with it.
To run BuildKit in a dedicated node pool, add taints and tolerations to the BuildKit node pool in Kubernetes, and add the following settings to your Okteto Helm configuration:
buildkit:
nodeSelectors:
okteto-node-pool: build
tolerations:
- effect: NoSchedule
key: okteto-node-pool
operator: Equal
value: build
2. Vertical Scaling for Faster Builds
BuildKit performance depends heavily on the CPU and memory it gets. Start with 2 nodes for higher availability, with these recommended instance types (4 CPUs and 16 GB of RAM):
- Amazon Web Services:
m6a.xlarge - Google Cloud Platform:
t2d-standard-4 - Microsoft Azure:
Standard_D4as_v5
Set replicaCount to the number of nodes in the BuildKit node pool:
buildkit:
replicaCount: 2
Monitor performance and adjust node resources as needed. For example, upgrading your BuildKit nodes to 8 CPUs and 32 GB of memory each can roughly double their performance.
3. SSD Storage
BuildKit is I/O-intensive, especially when it pulls, extracts, and pushes container images. An SSD storage class can significantly improve BuildKit performance:
buildkit:
persistence:
enabled: true
class: <<your-ssd-storage-class>>
You can also increase the size of the BuildKit cache (100 GB by default) to get more cache hits:
buildkit:
persistence:
enabled: true
size: 200Gi
4. HPA for Performance and Costs
The Horizontal Pod Autoscaler (HPA) adjusts the number of BuildKit pods to the active builds. You get enough capacity at peak times, and lower costs when builds are idle.
Okteto supports the following metrics for HPA scaling:
okteto_build_active_builds: number of active builds (recommended)okteto_build_cpu_pressure: CPU pressure in 10-second intervalsokteto_build_memory_usage: memory usage, as a percentageokteto_build_io_pressure: I/O pressure in 10-second intervals
By default, HPA scales on the okteto_build_active_builds metric, with the following settings:
buildkit:
hpa:
enabled: true
min: 1
max: 5
metrics:
- type: Pods
pods:
metric:
name: okteto_build_active_builds
target:
type: AverageValue
averageValue: "0.99"
behavior:
scaleUp:
policies:
- type: Pods
value: 1
periodSeconds: 30
scaleDown:
stabilizationWindowSeconds: 600
policies:
- type: Pods
value: 1
periodSeconds: 150
This configuration:
- Scales up quickly when build pressure increases: it adds up to one pod every 30 seconds
- Scales down conservatively to avoid thrashing: it waits 10 minutes after the load drops, then removes one pod every 150 seconds
- Maintains between 1 and 5 BuildKit pods based on active builds
Okteto includes a metrics adapter that exposes BuildKit metrics through the v1beta2.custom.metrics.k8s.io API. Okteto deploys it automatically when HPA is enabled.
If you already run your own custom metrics adapter, such as Prometheus Adapter, disable Okteto's adapter by setting buildkit.hpa.adapter.enabled: false in your Helm configuration. Your adapter can then scrape the metrics directly from the BuildKit pods on port 8080. All metrics have the okteto_build prefix.
5. Best Practices for Dockerfiles and Remote Execution
A fast BuildKit setup doesn't make up for unoptimized Dockerfiles or inefficient Remote Execution configurations. To get the most out of BuildKit, we recommend that you:
- Adhere to Dockerfile best practices.
- Reduce file transfer times to BuildKit by using .dockerignore and .oktetoignore files.
- Use BuildKit cache mounts to persist cache folders between image builds. This can significantly speed up build times by caching dependencies and build artifacts.
- Use Test Container caches to persist cache folders between test executions. This ensures that test dependencies are cached, reducing the time taken for subsequent test runs.