Fetch concurrently with a limit · Python practice

Fetch concurrently with a limit

Medium~20 min

Write async def fetch_all(urls, limit) that fetches every URL with the provided coroutine fetch(url) (it takes a few milliseconds), running at most limit fetches at the same time but more than one when limit > 1. Return the results in the same order as urls.

fetch is available as a global (the harness defines it and measures concurrency).

Function

Implement fetch_all. Input: urls: list[str], limit: int. Returns: list[str] (results in input order).

Examples

>>> fetch_all(['a', 'b', 'c', 'd', 'e'], 2)

{'results': ['A', 'B', 'C', 'D', 'E'], 'concurrent': True, 'within_limit': True}

Provided environment

This code runs before yours (fake clients, clocks and the test harness). You can use its names; don't edit them.

import asyncio

_state = {"active": 0, "peak": 0}

async def fetch(url):
    _state["active"] += 1
    _state["peak"] = max(_state["peak"], _state["active"])
    try:
        await asyncio.sleep(0.005)
        return url.upper()
    finally:
        _state["active"] -= 1

async def __harness__(fn, urls, limit):
    _state.update(active=0, peak=0)
    results = await fn(urls, limit)
    return {"results": results, "within_limit": _state["peak"] <= limit, "concurrent": _state["peak"] > 1 or limit == 1 or len(urls) < 2}

Details

Level: API integration against fakes, data processing, error handling, decorators and asyncio. Several cooperating parts.
Roles: AI Engineer, Backend Engineer, Generative AI Engineer, Python Developer
Topics: asyncio, Threads and concurrency
Tests: 1 public (run with Run), 4 hidden (graded on Submit)
Grading: Exact match of the return value