Skip to main content
Version: 1.51

Queue Splitting

Queue splitting is making sure each message is consumed by the namespace it belongs to. Divert routes HTTP and gRPC requests, but it doesn't intercept message brokers like Kafka, RabbitMQ, or SQS. Without a strategy, a consumer in your Overlay Namespace and a consumer in the Shared Namespace compete for the same messages, and each message is processed by whichever gets it first.

Service-Owned Queues​

The recommended approach is that each subsystem owns its queues: only its own services publish to them and consume from them. Other subsystems send work to it through its gRPC or HTTP API, and the subsystem publishes to its own queue.

In the following example, the API gateway sends work to the rentals and catalog subsystems through their APIs. Each subsystem has its own Kafka, and only its own worker consumes from it:

The API gateway calls the rent API and the catalog API. The rentals subsystem has the rent API, its own Kafka, the rent worker, and a PostgreSQL database used by the API and the worker. The catalog subsystem has the catalog API, its own Kafka, the catalog worker, and a MongoDB database used by the API and the worker

With this pattern, Divert splits queues without changes to your producers or consumers. In the following example, Cindy deploys only the rentals subsystem in her Overlay Namespace:

Requests with baggage okteto-divert=cindy go from the API gateway of staging to Cindy's rent API. Cindy's Overlay Namespace runs the whole rentals subsystem: the rent API, its own Kafka, the rent worker, and PostgreSQL

  1. In your Overlay Namespace, deploy the whole subsystem: its API, its broker (or its own vhost), its workers, and its databases.
  2. A request with your divert header reaches your API, which publishes to your broker. Your service resolves the broker by a short name, such as rabbitmq:5672, so it gets the one in your Overlay Namespace.
  3. Your workers consume from your broker. Messages for the Shared Namespace never reach them, and yours never reach the shared workers.

Two rules make the pattern work:

  • Deploy the API and the workers together. If you deploy only the API, it publishes to a broker that nobody consumes from.
  • Workers propagate the divert header. If a worker calls other services while processing a message, it forwards the baggage header it received, as described in Header Propagation.

Sending Messages to Other Subsystems​

If a service needs to send messages to another subsystem, the recommended approach is that the other subsystem receives them through an HTTP or gRPC API, and publishes them to its own queue. Your service calls that API by its service name:

  • If the other subsystem is also deployed in your Overlay Namespace, the call reaches your copy, which publishes to your broker.
  • If it isn't, the call reaches the API of the Shared Namespace, which publishes to the shared broker.

Divert resolves this for you, so queues work with Divert without changes to your producers or consumers.

Publishing Directly to Another Subsystem's Broker​

If an API in front of the other subsystem isn't possible, and your service publishes directly to its broker, your application decides which broker to use. Configure the address of the broker of each subsystem that you publish to:

  • When the subsystem that owns the queue is deployed in your Overlay Namespace, use the broker of your Overlay Namespace, such as catalog-kafka:9092.
  • When it isn't, use the broker of the Shared Namespace, with its fully qualified domain name, such as catalog-kafka.staging.svc.cluster.local:9092.
environment:
  CATALOG_BROKER: catalog-kafka.staging.svc.cluster.local:9092 # catalog isn't deployed in your Overlay Namespace
  RENTALS_BROKER: rentals-kafka:9092                           # rentals is deployed in your Overlay Namespace

Message-Level Routing​

Message-level routing is sharing the broker of the Shared Namespace, and making each consumer process only the messages of its namespace. You can deploy only the consumer you work on, without a broker, but every producer and consumer needs changes.

In the following example, Cindy deploys only the rent worker. Both rent workers receive every message, and each one skips the messages of the other:

The rent API of staging publishes to the shared broker with the baggage in the message headers. The rent worker of staging skips the messages with okteto-divert=cindy. Cindy's rent worker processes only the messages with okteto-divert=cindy

  1. Producers copy the divert header into the message. A service that publishes a message while handling a request copies the baggage header of the request to the message headers.
  2. Each Overlay Namespace creates its own subscription. Give your consumer its own subscription, such as rent-worker-cindy, starting from the latest message, so it receives every new message.
  3. Consumers filter by the divert header. Your consumer processes only the messages with the okteto-divert entry of your environment. The shared consumer processes the rest.
  4. Consumers propagate the divert header. A consumer forwards the baggage header of the message on any request or message it sends while processing it.

Delete your subscription when you destroy your Overlay Namespace.