Dynamic Batching: Implement a Request Batcher That Groups Re

Interview Experience

Problem

Build a DynamicBatcher that collects incoming requests and flushes them as a batch either when the batch reaches max_size items or after max_wait_ms milliseconds since the first request in the batch arrived — whichever comes first.

python
class DynamicBatcher:
    def __init__(self, max_size: int, max_wait_ms: int,
                 process_batch: Callable[[list], list]): ...

    def submit(self, request: dict) -> Future:
        """

**Returns** a Future that resolves when the batch containing
        this request has been processed.
        """

Example behavior:

batcher = DynamicBatcher(max_size=10, max_wait_ms=50, process_batch=db_bulk_insert)
f1 = batcher.submit({"id": 1, "data": "..."})
f2 = batcher.submit({"id": 2, "data": "..."})
# If 8 more arrive within 50ms -> all 10 flushed together
# If timeout hits first -> flush whatever is pending

Follow-ups

How do you associate each request's Future with its position in the batch result list?
What threading model do you use — a background flusher thread, asyncio, or something else?
How do you handle partial batch failures where some items succeed and others fail?
If the process_batch function is slow and requests pile up, how do you add backpressure?

Full Details

Problem

Build a DynamicBatcher that collects incoming requests and flushes them as a batch either when the batch reaches max_size items or after max_wait_ms milliseconds since the first request in the batch arrived — whichever comes first.

python
class DynamicBatcher:
    def __init__(self, max_size: int, max_wait_ms: int,
                 process_batch: Callable[[list], list]): ...

    def submit(self, request: dict) -> Future:
        """

**Returns** a Future that resolves when the batch containing
        this request has been processed.
        """

Example behavior:

batcher = DynamicBatcher(max_size=10, max_wait_ms=50, process_batch=db_bulk_insert)
f1 = batcher.submit({"id": 1, "data": "..."})
f2 = batcher.submit({"id": 2, "data": "..."})
# If 8 more arrive within 50ms -> all 10 flushed together
# If timeout hits first -> flush whatever is pending

Follow-ups

How do you associate each request's Future with its position in the batch result list?
What threading model do you use — a background flusher thread, asyncio, or something else?
How do you handle partial batch failures where some items succeed and others fail?
If the process_batch function is slow and requests pile up, how do you add backpressure?

Free preview. Unlock all Xai questions →

About This Question

This is a candidate experience report from a xai interview during the onsite round.

It covers the following topics: Coding, Onsite .

Topics

Coding Onsite

Dynamic Batching: Implement a Request Batcher That Groups Requests for Efficient Processing

Interview Experience

Problem

Full Details

Problem

About This Question

Topics

More Xai Interview Questions