Fall back to a second model on timeout · Python practice

Fall back to a second model on timeout

Medium~20 min

Write async def complete(prompt, providers, timeout). providers is a list of async functions provider(prompt) -> str, in order of preference. Call them one at a time:

If a provider returns within timeout seconds, return {"text": ..., "provider": index}.
If it times out or raises any exception, move on to the next provider.
If all fail, return {"text": None, "provider": None}.

The harness builds fake providers with fixed delays and failures.

Function

Implement complete. Input: prompt: str, providers: list of async callables, timeout: float. Returns: dict.

Examples

>>> complete('hi', [{'name': 'primary', 'delay': 0.2}, {'name': 'backup', 'delay': 0.001}], 0.05)

{'text': 'backup: hi', 'provider': 1}

Provided environment

This code runs before yours (fake clients, clocks and the test harness). You can use its names; don't edit them.

import asyncio

def _make(spec):
    async def provider(prompt):
        await asyncio.sleep(spec["delay"])
        if spec.get("error"):
            raise RuntimeError(spec["error"])
        return f"{spec['name']}: {prompt}"
    return provider

async def __harness__(fn, prompt, specs, timeout):
    return await fn(prompt, [_make(s) for s in specs], timeout)

Details

Level: API integration against fakes, data processing, error handling, decorators and asyncio. Several cooperating parts.
Roles: Generative AI Engineer, AI Engineer, Solutions Architect
Topics: asyncio, Resilience patterns, LLM integration
Tests: 1 public (run with Run), 4 hidden (graded on Submit)
Grading: Exact match of the return value