Modal in Python: Serverless GPU and Batch Compute for Data Science and ML (2026)

Updated: September 10, 2026

Modal is a serverless Python platform that runs ordinary functions on GPUs and CPUs in the cloud with per-second billing, no Dockerfiles, and scale-to-zero. You decorate a Python function with @app.function(gpu="H100"), run modal deploy, and Modal builds an image, provisions a container, and gives you an HTTPS endpoint or a callable Python object. In 2026, after Modal 1.0 (May 2025) and the Series C at a $4.65B valuation (May 2026), it's become the default choice for Python teams that want GPU inference and batch jobs without operating Kubernetes. I use it from the FastAPI side of my stack constantly, and honestly, this guide is the tour I wish I'd had a year ago.

• Modal 1.5.3 (July 2026) requires module-mode CLI (modal deploy -m pkg.app) and dropped Python 3.9; 3.10–3.14 are supported.
• 2026 GPU pricing is per-second: H100 SXM5 at $3.95/hr, H200 at $4.54/hr, B200 at $6.25/hr, L4 at $0.80/hr, plus $30/mo free credits on the Starter plan.
• @app.cls with @modal.enter is how you cache weights once per container; pair it with a modal.Volume so downloads survive scale-down.
• Cold starts are ~3–4s for slim images and 10–20s for multi-GB PyTorch images. Use min_containers=1 plus scaledown_window to trade dollars for latency.
• Modal wins under <12 hrs/day of GPU utilization; above that break-even, dedicated H100s from Lambda Labs or RunPod pods are cheaper.
• Deprecations to watch: @modal.web_endpoint was replaced by @modal.fastapi_endpoint; MODAL_SANDBOX_V2 becomes the default in SDK 1.6.

What is Modal serverless Python used for?

Modal is used to run Python code (single functions, classes, or FastAPI apps) on managed compute that autoscales GPU and CPU containers from zero, without you writing Dockerfiles, YAML, or Terraform. The mental model that clicked for me, coming from FastAPI, is this: Modal is what you'd get if @app.get secretly meant “also provision me an H100 in Oregon, cache my weights on a persistent volume, and only bill me while a request is in flight.” The developer surface is a Python decorator; the substrate is a bespoke container scheduler that Modal built to keep cold starts in the low single-digit seconds.

The typical workloads I run on it: (1) batch inference over hundreds of thousands of rows fanned out across L4s, (2) low-QPS but latency-sensitive HTTP endpoints that need an H100 warm, (3) fine-tuning jobs that must not lose their checkpoint volume when the container dies, (4) sandboxed code execution for LLM agents, and (5) scheduled ETL that transcodes video or pushes embeddings into a vector store. If your workload is high-QPS at 24/7 utilization, dedicated GPUs are cheaper, and we'll do the math in the pricing section. If it's spiky or intermittent, Modal is almost always the right call because you pay only while the container runs, per second, with a scale-to-zero floor of $0.

The 2026 product surface goes well beyond FaaS. Modal now ships browser-based Notebooks with GPU memory snapshots (~10× faster startup than Colab), Sandboxes for agent code execution, Batch for large embarrassingly parallel jobs, Training for multi-node clustered GPUs over RDMA, and Inference endpoints as first-class objects. The catch: everything is proprietary, there's no BYOC or self-host, and if you leave, you're rewriting decorators as Kubernetes manifests.

How does Modal.com pricing work for GPU workloads?

Modal charges per-second, per-container, with separate meters for GPU, CPU cores, memory, and volume storage. There are no reserved instances and no monthly GPU commitments. When your function isn't running, the meter is off. Below are the 2026 on-demand GPU rates (from the official Modal pricing page), rounded from the per-second numbers to per-hour so you can compare with Lambda Labs or SageMaker.

GPU	$/second	~$/hour	VRAM
B300	0.001972	$7.10	288 GB
B200	0.001736	$6.25	192 GB
H200 SXM	0.001261	$4.54	141 GB
H100 SXM5	0.001097	$3.95	80 GB
RTX PRO 6000	0.000842	$3.03	96 GB
A100 80GB	0.000694	$2.50	80 GB
L40S	0.000542	$1.95	48 GB
A10	0.000306	$1.10	24 GB
L4	0.000222	$0.80	24 GB
T4	0.000164	$0.59	16 GB

CPU is metered at $0.0000131 per core per second (minimum 0.125 cores per container), memory at $0.00000222 per GiB per second, and Volume storage at $0.09 per GiB-month with the first 1 TiB free. The Starter plan is $0/month with $30 of monthly compute credits, 3 seats, and a cap of 100 concurrent containers with 10 concurrent GPUs, which is enough for real experimentation. The Team plan is $250/month plus $100 credits, raises those caps to 5,000 containers and 50 concurrent GPUs, and adds 30-day log retention, custom domains, static IP proxy, and blue-green rollbacks.

There are two multipliers worth calling out because they will surprise you at billing time: region pinning is 1.15–1.75× depending on the region, and non-preemptible (guaranteed-not-to-be-restarted) compute is 3×. In practice I only pin regions when latency is user-facing, and I only mark jobs non-preemptible for training checkpoints where a preemption in the middle costs more than the 3× premium.

Your first Modal function: @app.function to modal deploy

The smallest useful Modal program is about ten lines. Install the SDK with uv for Python package management (uv pip install modal), then authenticate with modal setup. Save the following as hello.py:

import modal

app = modal.App("hello-modal")
image = modal.Image.debian_slim().uv_pip_install("requests")

@app.function(image=image, cpu=0.5, memory=512)
def fetch_status(url: str) -> int:
    import requests
    return requests.get(url, timeout=10).status_code

@app.local_entrypoint()
def main():
    urls = ["https://example.com", "https://modal.com", "https://python.org"]
    for url, status in zip(urls, fetch_status.map(urls)):
        print(url, status)

Now the two commands you'll type ten times a day. modal run -m hello spins up an ephemeral container, runs main(), streams logs to your terminal, and tears everything down when it returns. That's your inner development loop. modal deploy -m hello registers the app so its functions become callable from anywhere via modal.Function.from_name("hello-modal", "fetch_status"), and its web endpoints get permanent HTTPS URLs. Modal 1.0 made module-mode (the -m flag) mandatory; older script-mode invocations from tutorials will fail on the current SDK.

Notice the .map() call. That's Modal's fan-out primitive: it sends one container per input (up to your concurrency limit), runs them in parallel, and streams results back in order. On a laptop, three requests.get calls take ~600ms serialized; on Modal, they take ~600ms plus a one-time cold start because they run concurrently. The Python code is unchanged, and that's the whole selling point.

Serving ML models with @app.cls, Volumes, and FastAPI endpoints

Functions are fine for stateless calls, but ML inference has to load model weights once and reuse them across requests. That's what @app.cls is for. A class decorated with @app.cls becomes a container-scoped object: the container starts, @modal.enter methods run (typical place to load weights into GPU memory), and subsequent requests reuse that same process until the container scales down. If you've built production ML serving with BentoML or Triton, this is the same pattern with less ceremony.

import modal

app = modal.App("sentiment-api")

image = (
    modal.Image.debian_slim(python_version="3.12")
    .uv_pip_install("transformers==4.56.0", "torch==2.5.1", "fastapi==0.115.0")
)
volume = modal.Volume.from_name("hf-hub-cache", create_if_missing=True)

@app.cls(
    image=image,
    gpu="L4",
    volumes={"/root/.cache/huggingface": volume},
    scaledown_window=300,       # keep idle container 5 min
    min_containers=1,           # always one warm instance
    max_containers=20,
    timeout=120,
)
@modal.concurrent(max_inputs=8)   # 8 concurrent requests per container
class SentimentModel:
    @modal.enter()
    def load(self):
        from transformers import pipeline
        self.pipe = pipeline(
            "sentiment-analysis",
            model="distilbert-base-uncased-finetuned-sst-2-english",
            device=0,
        )

    @modal.method()
    def score(self, texts: list[str]) -> list[dict]:
        return self.pipe(texts, truncation=True, max_length=512)

    @modal.fastapi_endpoint(method="POST", label="score")
    def http_score(self, payload: dict):
        return {"results": self.pipe(payload["texts"], truncation=True)}

Run modal deploy -m sentiment_api and Modal prints an HTTPS URL you can curl. Three things to notice. First, the Volume is mounted at /root/.cache/huggingface, which is where transformers writes downloaded weights, so the first invocation pays the download cost and every subsequent cold start reads from the cached volume in seconds. Second, min_containers=1 keeps one L4 warm 24/7, which costs about $576/month at $0.80/hr but means end users never see a cold start. Drop it to 0 if latency spikes are acceptable. Third, @modal.concurrent(max_inputs=8) lets one container handle eight parallel requests, which matters because the L4 GPU is barely loaded by a single distilbert call.

For anything beyond a single route, replace @modal.fastapi_endpoint with @modal.asgi_app and return a real FastAPI application (full routing, CORS middleware, background tasks, streaming responses). If you're already streaming tokens with FastAPI streaming for SSE and LLM APIs, that same code drops in unchanged. Modal deprecated the older @modal.web_endpoint, so check the Modal Python SDK changelog if you're upgrading from a 2024 codebase.

Fan-out batch jobs with spawn_map and .map()

The batch inference workload that finally sold me on Modal was scoring 800k product descriptions with an embedding model in ~14 minutes. The trick is spawn_map: it queues inputs asynchronously and returns immediately with a list of function calls you can poll or await, instead of blocking your driver process while every container finishes.

import modal
from datasets import load_dataset

app = modal.App("batch-embeddings")

image = (
    modal.Image.debian_slim()
    .uv_pip_install("sentence-transformers==3.4.1", "pyarrow==18.0.0")
)
vol = modal.Volume.from_name("embeddings-out", create_if_missing=True)

@app.cls(
    image=image,
    gpu="L4",
    volumes={"/data": vol},
    max_containers=100,
    scaledown_window=60,
)
class Embedder:
    @modal.enter()
    def load(self):
        from sentence_transformers import SentenceTransformer
        self.model = SentenceTransformer("BAAI/bge-small-en-v1.5", device="cuda")

    @modal.method()
    def embed_batch(self, shard_id: int, texts: list[str]) -> str:
        import pyarrow as pa
        import pyarrow.parquet as pq
        vecs = self.model.encode(texts, batch_size=64, convert_to_numpy=True)
        path = f"/data/shard_{shard_id:05d}.parquet"
        pq.write_table(pa.table({"vec": vecs.tolist()}), path)
        return path

@app.local_entrypoint()
def main(shard_size: int = 4096):
    texts = load_dataset("Open-Orca/OpenOrca", split="train[:800000]")["question"]
    shards = [(i, texts[i:i + shard_size]) for i in range(0, len(texts), shard_size)]
    embedder = Embedder()
    # spawn_map: async fan-out, no blocking wait
    call = embedder.embed_batch.spawn_map(
        [s[0] for s in shards],
        [s[1] for s in shards],
    )
    for path in call.get_gen():   # stream results as they finish
        print("wrote", path)

At 100 concurrent L4s, that job costs roughly 100 × $0.80/hr × 0.23 hrs ≈ $18.40. That's less than a takeaway pizza, and I don't have to argue with Kubernetes namespaces. Compare that to standing up a Ray cluster, waiting for GPU quota approval, then dismantling everything after the job. When batch fan-out is the shape of your problem, this is where Modal earns its rent.

The two related primitives worth knowing: .map() is synchronous, so the driver blocks until all inputs finish. .spawn() submits one call and returns a modal.FunctionCall handle you can poll later, even from a different Python process using modal.FunctionCall.from_id(call_id). This last capability is what lets you build async workflows where a FastAPI endpoint kicks off a Modal job, stores the call ID in Postgres, and lets a webhook poll it from a totally different process.

How long are Modal cold starts and how do you keep containers warm?

Cold start on Modal in 2026 is dominated by image size, not scheduler latency. A slim image with a few pure-Python dependencies cold-starts in about 2–4 seconds. A PyTorch + CUDA image with 5–8 GB of weights loaded from a Volume can take 10–20 seconds on the first hit and drop to 3–6 seconds on subsequent hits as the container filesystem caches. GPU memory snapshotting (the same technology that made Notebooks 10× faster) is starting to roll out for @app.cls workloads. When your class's @modal.enter is expensive, a snapshot lets Modal resume the container from a checkpoint of GPU memory instead of re-running load().

The three knobs I reach for, in that order: min_containers, scaledown_window, and buffer_containers. Setting min_containers=1 pins one container permanently warm and eliminates the first-request cold start entirely. It costs money every second, so use it on the endpoint that actually serves user traffic, not on the batch job. scaledown_window=300 keeps a container alive for 5 minutes after its last request, which absorbs bursts. buffer_containers=2 pre-warms two extras when the queue depth grows, which is useful for spiky traffic where you want autoscaling to overshoot rather than lag.

Everything else is image hygiene: use uv_pip_install instead of pip_install where possible (uv resolves and downloads faster), pin exact versions so Image caching doesn't invalidate on every deploy, and put slow-changing dependencies (torch, transformers) in an earlier layer than fast-changing ones (your own code). If you need to bake weights or preprocessed data into the image itself, use Image.run_function(), but read the Modal Images guide first, because the cache-invalidation rules for run_function are subtle.

Modal vs Beam vs RunPod vs AWS Lambda: what actually differs

Modal is not the only serverless GPU platform in 2026, and honest positioning matters if you're picking one for the next 12 months. Here's the H100/hr price grid I keep in a note next to my desk.

Platform	H100 $/hr	Cold start	Model
Modal	$3.95	3–4s	Fully-managed serverless, per-second billing, no BYOC
Beam (beam.cloud)	$1.74–$3.20	~2s	Serverless + AGPL runtime, self-host possible
RunPod Serverless	$2.39–$2.69	5–8s	Serverless + pod rental hybrid
Baseten	$6.50	varies	Managed inference (Truss framework)
Lambda Labs (reserved)	$2.49–$3.29	N/A (dedicated)	Reserved GPU, always-on
AWS Lambda	N/A	N/A	No GPU support at all

The AWS Lambda comparison keeps coming up in searches, but it's a category error: AWS Lambda does not support GPU workloads in 2026. Your options on AWS are SageMaker (from $1.006/hr baseline for a much heavier control plane), ECS with EC2 GPU instances, or Bedrock for hosted foundation models. If someone hands you a “Modal vs Lambda” requirements doc, the honest answer is “Lambda isn't in this bracket.”

The real trade-off is Modal vs Beam vs RunPod. Beam is meaningfully cheaper on paper, has open-source runtime you can self-host under AGPL, and matches or beats Modal on cold-start latency. Its DX and community are smaller. RunPod is the price leader with the biggest GPU catalog (including B200 pods), but its serverless is younger and cold starts are longer. Modal wins on Python DX, documentation depth, and reliability, and you pay a premium for it. If your bill is under $2k/month and you're a small team, that premium is worth it. If your bill is $30k/month and half of that is GPU compute, you should be pricing Beam and RunPod side by side.

For inference-server internals (how vLLM's continuous batching, PagedAttention, or SGLang's radix attention compare), see the vLLM vs TGI vs SGLang comparison. Modal is the substrate; those are the frameworks you'd run on top of it inside a @app.cls.

Production pitfalls and gotchas

After a year of production Modal use, these are the pitfalls that cost me time and money. So, let's walk through them.

Volume writes silently disappear

The single most common Modal bug in my Slack DMs: someone writes to a mounted modal.Volume, doesn't call volume.commit(), and is confused when the file isn't there on the next call. Writes are staged locally until commit. Reads see the last committed snapshot, not your local writes from earlier in the same request. Wrap volume writes in a helper that always commits.

Killed function still costs money

If your function hits its timeout= and gets killed, you still pay for every second it ran. The default timeout is a few minutes; push it explicitly for long jobs (max 24 hours) and make sure you have a heartbeat check before deploying anything runaway-capable.

GPU quota queues silently

If you request an H100 and Modal's H100 pool is temporarily saturated, your call queues rather than falling back to a smaller GPU. There's no automatic downgrade. For latency-sensitive endpoints on hot GPUs, pin min_containers so you're never at the mercy of pool depth.

Image rebuild rules for run_function

Image.run_function() only invalidates the layer when the function's own source, kwargs, and referenced globals change. Editing a helper function called by run_function does NOT invalidate the image, so your “fixed” setup silently reuses the broken build. If you're not sure, force a rebuild with modal deploy --force-build.

Cls.with_options doesn't fully override

Cls.with_options(gpu=None) does not unset a GPU already configured on the class, so you can't turn a GPU class into a CPU-only variant this way. Also, when you pass volumes= or secrets= to with_options, they replace rather than merge, so pass the union explicitly.

Modal 1.0 CLI break

Script-mode invocation (modal run hello.py) is gone. Everything is module mode now (modal run -m hello), which means your app must be importable as a proper Python module or package. Copy the migration checklist from the Modal 1.0 announcement before upgrading a large codebase.

No BYOC, no multi-node training

You can't run Modal on your own AWS account, and until the 2025 clustered-GPU launch there was no way to do multi-node training. Clustered training over RDMA exists now, but it's behind a request-access flag. If regulatory or data-residency constraints require compute in a specific tenant, Modal is not your platform.

Frequently Asked Questions

Is Modal cheaper than AWS Lambda for AI inference?

The comparison isn't apples-to-apples because AWS Lambda does not support GPU workloads in 2026. For CPU-only inference, Lambda can be cheaper at very low volume; for anything that needs a GPU, the real comparison is Modal against SageMaker Serverless Inference or a Modal alternative like Beam or RunPod, and Modal sits roughly mid-pack on price with the best Python DX.

Can Modal run FastAPI apps?

Yes. Use @modal.asgi_app to expose an entire FastAPI (or Starlette) application including middleware, routers, and streaming, or @modal.fastapi_endpoint for a single route. Both give you a permanent HTTPS URL after modal deploy and inherit the container-level GPU and scaling config.

Is there a free tier for Modal?

The Starter plan is $0/month and includes $30 of free compute credits every month, 3 seats, and a cap of 100 concurrent containers with 10 concurrent GPUs. That's enough for real development and a small hobby project. Most tutorials in this guide will consume less than a dollar of credit.

What is the difference between modal run and modal deploy?

modal run spins up an ephemeral, one-shot invocation and streams logs to your terminal, which is ideal for development and CI jobs. modal deploy registers the app persistently so its functions become callable from other processes and its web endpoints have permanent URLs, which is what you want for production.

How do Modal Volumes work versus local disk?

Local container disk is ephemeral. Anything written there disappears when the container scales down. A modal.Volume is a distributed persistent filesystem mounted at a path you choose; writes must be committed with volume.commit() to persist, and reads see the last committed snapshot. Use it for model weights, dataset caches, and job checkpoints.

How do you keep a Modal container warm?

Set min_containers=1 (or higher) on the function or class to pin at least one container always running, and use scaledown_window=300 to keep idle containers alive for five minutes after their last request. Combine both for user-facing endpoints where cold starts would hurt the p95.