You’re trying to buy tickets for a massive concert. You click “buy” and suddenly find yourself stuck in a digital line. As a developer, your first thought might be to use a Message Queue like Kafka. It’s literally called a queue, and it’s designed to handle requests in order-perfect, right?

Well, not quite. When you’re dealing with large-scale, first-come-first-served events, a Message Queue alone often falls short. The missing piece? User Experience.

The Visibility Problem

Message Queues are great at processing tasks behind the scenes, but from the user’s perspective, they offer zero visibility. Once your request enters the queue, the system knows it’s there, but you have no idea where you are in line or how many people are ahead of you.

This lack of transparency creates three major problems:

  1. You can’t show users their position in real-time. Telling someone “You’re 5,342nd in line” is extremely difficult with just a Message Queue.
  2. The refresh frenzy. When users don’t see any feedback, they start frantically refreshing the page. In a traditional Message Queue, each refresh can create a duplicate request, turning your orderly queue into chaos.
  3. Removing users is complicated. If someone closes their browser and gives up, finding and removing that specific request from the middle of a massive queue is incredibly complex.

Redis Sorted Sets: Your Real-Time Leaderboard

This is where Redis Sorted Sets shine. Instead of a black box, Redis gives you a transparent, real-time leaderboard that users can actually see.

Here’s what makes it work:

  • Instant position updates: Redis can tell users exactly where they stand in line, right now.
  • Refresh-proof: Redis Sorted Sets handle duplicate requests gracefully. Thanks to idempotency, even if someone refreshes 100 times, the system treats it as a single entry. Their original place in line stays safe, and the queue doesn’t get cluttered.
  • Easy to manage: If someone decides to leave, removing them from Redis is just a simple, one-line command.

Here’s how you might implement this in practice:

Go Example

package main

import (
    "context"
    "fmt"
    "time"

    "github.com/redis/go-redis/v9"
)

type QueueService struct {
    client *redis.Client
    queueKey string
}

func NewQueueService(redisAddr string) *QueueService {
    rdb := redis.NewClient(&redis.Options{
        Addr: redisAddr,
    })
    return &QueueService{
        client: rdb,
        queueKey: "waiting_queue",
    }
}

// JoinQueue adds a user to the queue (idempotent - same user ID = same position)
func (q *QueueService) JoinQueue(ctx context.Context, userID string) error {
    // Use timestamp as score for first-come-first-served ordering
    // ZADD with NX ensures idempotency - if user already exists, score doesn't change
    score := float64(time.Now().UnixNano())
    return q.client.ZAddNX(ctx, q.queueKey, redis.Z{
        Score:  score,
        Member: userID,
    }).Err()
}

// GetPosition returns the user's position in queue (0-indexed)
func (q *QueueService) GetPosition(ctx context.Context, userID string) (int, int, error) {
    // Get user's rank (0-indexed)
    rank, err := q.client.ZRank(ctx, q.queueKey, userID).Result()
    if err == redis.Nil {
        return -1, 0, fmt.Errorf("user not in queue")
    }
    if err != nil {
        return -1, 0, err
    }

    // Get total queue size
    total, err := q.client.ZCard(ctx, q.queueKey).Result()
    if err != nil {
        return -1, 0, err
    }

    return int(rank), int(total), nil
}

// RemoveFromQueue removes a user from the queue
func (q *QueueService) RemoveFromQueue(ctx context.Context, userID string) error {
    return q.client.ZRem(ctx, q.queueKey, userID).Err()
}

// GetNextInLine returns the user at the front of the queue and removes them
func (q *QueueService) GetNextInLine(ctx context.Context) (string, error) {
    // Get the user with the lowest score (oldest timestamp)
    result, err := q.client.ZRangeWithScores(ctx, q.queueKey, 0, 0).Result()
    if err != nil {
        return "", err
    }
    if len(result) == 0 {
        return "", fmt.Errorf("queue is empty")
    }

    userID := result[0].Member.(string)
    
    // Remove from Redis queue
    if err := q.client.ZRem(ctx, q.queueKey, userID).Err(); err != nil {
        return "", err
    }

    return userID, nil
}

// Example usage
func main() {
    ctx := context.Background()
    queue := NewQueueService("localhost:6379")

    // User joins queue
    if err := queue.JoinQueue(ctx, "user123"); err != nil {
        panic(err)
    }

    // User refreshes - same position maintained (idempotent)
    queue.JoinQueue(ctx, "user123")

    // Check position
    position, total, err := queue.GetPosition(ctx, "user123")
    if err != nil {
        panic(err)
    }
    fmt.Printf("Position: %d/%d\n", position+1, total)

    // When user's turn comes, move to Kafka
    userID, err := queue.GetNextInLine(ctx)
    if err != nil {
        panic(err)
    }
    // Send to Kafka for actual processing
    fmt.Printf("Processing user: %s\n", userID)
}

Rust Example

use redis::{Commands, Connection};
use std::time::{SystemTime, UNIX_EPOCH};

struct QueueService {
    conn: Connection,
    queue_key: String,
}

impl QueueService {
    fn new(redis_url: &str) -> Result<Self, redis::RedisError> {
        let client = redis::Client::open(redis_url)?;
        let conn = client.get_connection()?;
        Ok(QueueService {
            conn,
            queue_key: "waiting_queue".to_string(),
        })
    }

    // JoinQueue adds a user to the queue (idempotent - same user ID = same position)
    fn join_queue(&mut self, user_id: &str) -> Result<(), redis::RedisError> {
        // Use timestamp as score for first-come-first-served ordering
        // ZADD with NX ensures idempotency - if user already exists, score doesn't change
        let score = SystemTime::now()
            .duration_since(UNIX_EPOCH)
            .unwrap()
            .as_nanos() as f64;
        
        self.conn.zadd_nx(&self.queue_key, user_id, score)
    }

    // GetPosition returns the user's position in queue (0-indexed)
    fn get_position(&mut self, user_id: &str) -> Result<(usize, usize), redis::RedisError> {
        // Get user's rank (0-indexed)
        let rank: Option<usize> = self.conn.zrank(&self.queue_key, user_id)?;
        let rank = rank.ok_or_else(|| {
            redis::RedisError::from((
                redis::ErrorKind::TypeError,
                "User not in queue",
            ))
        })?;

        // Get total queue size
        let total: usize = self.conn.zcard(&self.queue_key)?;

        Ok((rank, total))
    }

    // RemoveFromQueue removes a user from the queue
    fn remove_from_queue(&mut self, user_id: &str) -> Result<(), redis::RedisError> {
        self.conn.zrem(&self.queue_key, user_id)
    }

    // GetNextInLine returns the user at the front of the queue and removes them
    fn get_next_in_line(&mut self) -> Result<String, redis::RedisError> {
        // Get the user with the lowest score (oldest timestamp)
        let result: Vec<(String, f64)> = self.conn.zrange_withscores(&self.queue_key, 0, 0)?;
        
        if result.is_empty() {
            return Err(redis::RedisError::from((
                redis::ErrorKind::TypeError,
                "Queue is empty",
            )));
        }

        let user_id = result[0].0.clone();
        
        // Remove from Redis queue
        self.conn.zrem(&self.queue_key, &user_id)?;

        Ok(user_id)
    }
}

// Example usage
fn main() -> Result<(), Box<dyn std::error::Error>> {
    let mut queue = QueueService::new("redis://localhost:6379/")?;

    // User joins queue
    queue.join_queue("user123")?;

    // User refreshes - same position maintained (idempotent)
    queue.join_queue("user123")?;

    // Check position
    let (position, total) = queue.get_position("user123")?;
    println!("Position: {}/{}", position + 1, total);

    // When user's turn comes, move to Kafka
    let user_id = queue.get_next_in_line()?;
    // Send to Kafka for actual processing
    println!("Processing user: {}", user_id);

    Ok(())
}

The key Redis commands used here:

  • ZADD NX: Adds a member to the sorted set only if it doesn’t exist (idempotency)
  • ZRANK: Gets the rank (position) of a member in the sorted set
  • ZCARD: Gets the total number of members in the sorted set
  • ZRANGE: Gets members by rank (used to get the front of the queue)
  • ZREM: Removes a member from the sorted set

The Hybrid Approach: Best of Both Worlds

So should we throw away Message Queues entirely? Absolutely not. The best system designs don’t choose between Redis or Kafka. They use both together.

Here’s how it works:

  • Redis handles the waiting room. It manages all those high-frequency status checks and maintains the real-time order that users can see.
  • Message Queues handle the actual processing. Once someone reaches the front of the line in Redis, they get passed to Kafka (or your MQ of choice) to safely and reliably complete their transaction.

Each tool does what it’s best at.

How It All Fits Together

Here’s a simplified view of how this hybrid “waiting room” system works:

[ USER ] 
    |
    | (1) Enters Queue / Refreshes Page
    v
+---------------------------------------+
|          REDIS (Sorted Set)           |
|  "The Transparent Waiting Room"       |
|---------------------------------------|
| * Shows user their real-time rank     |
| * Filters out duplicate refresh       |
|   requests                            |
| * Handles users leaving the line      |
+---------------------------------------+
    |
    | (2) Turn Reached (Front of Line)
    v
+---------------------------------------+
|      MESSAGE QUEUE (e.g., Kafka)      |
|     "The Processing Engine"           |
|---------------------------------------|
| * Receives validated requests         |
|   (no duplicates, users already       |
|    filtered by Redis)                 |
| * Processes actual orders/purchases   |
| * Ensures stable, sequential tasks    |
+---------------------------------------+
    |
    v
[ TRANSACTION COMPLETE ]

The Takeaway

The key insight here is shifting your perspective from “how do we process this?” to “how does the user experience this?”

Redis gives users a smooth, transparent experience while they wait. Message Queues ensure the final transaction is handled with rock-solid reliability. It’s the classic “right tool for the right job” approach, and it’s what turns a frustrating wait into a seamless experience.