E2E Queue Count Testing: Proving Eventual Consistency Without Flaky Tests

A queue count looks like the smallest possible product feature: place work on a queue, show the number somewhere in the application, and verify that the number changes.

In an end-to-end test, it becomes a distributed-systems exercise.

The count is rarely owned by one component. A producer publishes a message. A broker accepts it. A consumer may pick it up before the test can inspect the queue. A database projection may be updated asynchronously. A cache may briefly hold an earlier value. Finally, an API and browser render a number that represents a moment which has already passed.

That is why a naive E2E queue count test often fails for the wrong reasons. It asks the system to be synchronous when the architecture intentionally is not.

The goal is not to make the test wait longer. The goal is to define precisely what the count means, control the parts of the system that make it unstable, and wait for an observable business outcome rather than an arbitrary delay.

First, define what is being counted#

Before writing a test, name the counter's contract.

A label reading Queued: 3 might mean any of the following:

• Three messages currently exist in RabbitMQ and have not been delivered to consumers.
• Three jobs have been accepted by the application but are not terminal yet.
• Three jobs are waiting specifically for a worker in one processing stage.
• Three jobs are visible to the current tenant or user.
• Three jobs were queued when a background projection last refreshed.

These definitions are not interchangeable.

Broker depth is operational state. It can change immediately when a consumer receives a message, even though that job remains far from complete. A business-level pending-job count is application state. It is usually more useful in a UI, but it requires durable job records and an explicit lifecycle.

For most product-facing E2E tests, the best contract is not “the broker has N messages.” It is something closer to:

After a user submits N jobs, the application eventually displays N jobs in the pending state, until processing is allowed to continue.

That wording contains the essential test design: submission is observable, pending is a domain state, and processing is deliberately held back.

The race hidden inside queue depth#

Consider this straightforward sequence:

1. The test creates a job through an HTTP endpoint.
2. The application publishes a message to RabbitMQ.
3. The test opens the dashboard.
4. The dashboard should show one queued job.

A fast worker can make step four fail even when everything is healthy. The worker consumes the message between steps two and three. The queue depth becomes zero. The job may be processing, completed, or waiting for another downstream service.

This is not test flakiness caused by a bad selector. It is a race between the assertion and a legitimate consumer.

The most effective fix is to make the test own the consumer boundary. In an E2E environment, provide a controlled worker mode:

• Start the application and message broker normally.
• Allow jobs to be published normally.
• Disable the real worker, or configure it to pause before acknowledgement.
• Create test jobs through the same public path used by the product.
• Assert the pending count through the UI or public API.
• Resume processing only when the test needs to verify the next lifecycle state.

A paused worker turns an unpredictable timing window into a stable system state.

Prefer a job lifecycle over broker inspection#

RabbitMQ management APIs are useful diagnostics, but they should not be the primary assertion for a user-visible queue count. They expose broker mechanics, not necessarily product semantics.

A durable job table gives the application a stronger model. For example, a .NET service might persist a record before publishing an event:

The dashboard count can then be based on an explicit query:

This distinction matters operationally too. A message can be redelivered. A consumer can crash after receiving it. Multiple queues can participate in one workflow. A job lifecycle is where those details are translated into a product-level answer.

The database does not need to be PostgreSQL specifically, but the state transition needs transactional discipline. A common pattern is the transactional outbox: persist the job and an outbound event in the same database transaction, then publish the event asynchronously. The UI count reads the durable job state, while the outbox ensures that a committed job eventually reaches the queue.

That architecture gives the E2E test meaningful checkpoints:

1. Job creation was accepted.
2. The job persisted as Pending.
3. An outbox message was created.
4. The message was published.
5. A worker eventually transitions the job to Processing and then Completed.

Not every test must prove all five. But the test should be clear about which boundary it is proving.

Wait on a condition, never on hope#

The most recognizable anti-pattern in asynchronous tests is a fixed delay:

It may pass locally and fail under CI load. Increasing the delay only trades flaky failures for slower flaky failures.

Instead, poll an observable condition until a deliberate timeout. At the API level, that could look like this:

The helper should record useful diagnostics on timeout: the last observed count, relevant job IDs, recent worker logs, and perhaps broker metrics. A timeout without context creates a second debugging problem after the first failure.

For browser-level tests, use the same principle through the browser framework's built-in retrying assertions. The UI test should wait until the displayed count becomes correct, not sleep and then read the page once.

A stable test identifier is important here. Avoid tying a business assertion to a layout class, an icon, or translated surrounding text. The count is the behavior under test; give it a durable locator.

Build the test as a small story#

A good queue count E2E test reads like a short, controlled narrative:

1. Create an isolated tenant or correlation ID.
2. Ensure the worker is paused.
3. Submit three jobs using the production API.
4. Wait until the pending projection reports three jobs.
5. Open the dashboard as that tenant.
6. Assert that the queue count is three.
7. Resume the worker.
8. Wait until all three jobs reach terminal states.
9. Assert that the pending count eventually returns to zero.

The correlation ID is especially valuable. Shared test environments accumulate noise: old messages, retries, parallel builds, and jobs from other suites. A count scoped to a unique tenant, batch, or test run avoids treating unrelated activity as a failure.

If global queue depth itself is the requirement, isolate infrastructure instead. Use a dedicated queue per test run or a uniquely named virtual host. Do not run assertions against a shared queue and expect determinism.

Make cleanup part of correctness#

Queue tests can leave behind exactly the kind of state that causes later tests to become mysterious. Cleanup should cover persisted jobs, outbox rows, test-created queues where appropriate, and paused-worker controls.

The most important cleanup action is resuming processing even after an assertion fails. Put it in a finally block or framework teardown hook. A failed test that leaves the worker paused can corrupt an entire CI run.

In CI/CD, collect artifacts when the test fails:

• browser screenshots and traces;
• application and worker logs;
• RabbitMQ queue statistics;
• database rows for the test correlation ID;
• the final API response used by the assertion.

These artifacts turn “expected 3, received 0” into an actionable diagnosis. Was publication delayed? Did the consumer run despite being paused? Was the UI querying the wrong tenant? Did a projection fail?

The real assertion is trust#

An E2E queue count test is not fundamentally about a numeral in a badge. It verifies that asynchronous work remains intelligible after it leaves the request-response path.

Users need to know whether their work was accepted. Operators need to distinguish backlog from failure. Developers need a test that understands eventual consistency without surrendering to randomness.

By modeling job state explicitly, controlling consumers in the test environment, isolating data, and waiting for observable conditions, the queue count becomes testable. More importantly, it becomes honest: a number with a defined meaning, backed by a system whose asynchronous behavior is no longer hidden behind a fragile sleep.