Runpod Python
**Define GPU functions in Python. Run them in the cloud with one decorator.**
[](https://badge.fury.io/py/runpod)
[](https://pepy.tech/project/runpod)
[](https://github.com/runpod/runpod-python/actions/workflows/CI-pytests.yml)
[](LICENSE)
[Documentation](https://docs.runpod.io) • [Examples](examples/apps) • [Discord](https://discord.gg/pJ3P2DbUUq)
**Using an AI agent?** Point it to https://docs.runpod.io/agent-setup.md to install the Runpod skills and MCP server.
## Installation
```bash
pip install runpod # or: uv add runpod
rp login # authenticate once
```
Requires Python 3.10+. Installing the package also installs the `rp` CLI.
## Example
```python
import runpod
from runpod import App, Model, NetworkVolume, Secret
app = App("inference")
models = NetworkVolume("models", size=100)
llama = Model("meta-llama/Llama-3.1-8B-Instruct")
# an autoscaling job queue on cloud H100s: weights pre-cached,
# dependencies vendored at deploy time, scale-to-zero when idle
@app.queue(
gpu="H100",
workers=(0, 3),
dependencies=["vllm"],
mounts={"/runpod-volume": models},
model=llama,
env={"HF_TOKEN": Secret("hf-token")},
)
def chat(prompt: str):
import vllm
llm = vllm.LLM(model=str(llama.path)) # weights already on disk
return llm.generate(prompt)
# one ephemeral pod per call: provisions, runs to completion, terminates
@app.task(gpu="H100", gpu_count=2, mounts={"/models": models})
def finetune(steps: int = 1000):
...
return {"loss": final_loss}
@runpod.local_entrypoint
def main():
print(chat.remote("why is the sky blue?")) # blocks for the result
job = finetune.spawn(steps=500) # fire and forget -> Job
```
```bash
rp flash dev main.py # live dev session: edit, re-run, logs stream back
rp flash deploy # deploy production endpoints
```
## Contributing
Pull requests and issues are welcome — see the [contributing guide](CONTRIBUTING.md) to get started.
```bash
git clone https://github.com/runpod/runpod-python.git
cd runpod-python
make setup
make test
```
## ⚡ | Serverless Worker (SDK)
This python package can also be used to create a serverless worker that can be deployed to Runpod as a custom endpoint API.
### Quick Start
Create a python script in your project that contains your model definition and the Runpod worker start code. Run this python code as your default container start command:
```python
# my_worker.py
import runpod
def is_even(job):
job_input = job["input"]
the_number = job_input["number"]
if not isinstance(the_number, int):
return {"error": "Silly human, you need to pass an integer."}
if the_number % 2 == 0:
return True
return False
runpod.serverless.start({"handler": is_even})
```
Make sure that this file is ran when your container starts. This can be accomplished by calling it in the docker command when you set up a template at [console.runpod.io/serverless/user/templates](https://console.runpod.io/serverless/user/templates) or by setting it as the default command in your Dockerfile.
See our [blog post](https://www.runpod.io/blog/build-basic-serverless-api) for creating a basic Serverless API, or view the [details docs](https://docs.runpod.io/serverless-ai/custom-apis) for more information.
### Local Test Worker
You can also test your worker locally before deploying it to Runpod. This is useful for debugging and testing.
```bash
python my_worker.py --rp_serve_api
```
## 📁 | Directory
```BASH
.
├── docs # Documentation
├── examples # Examples
├── runpod # Package source code
│ ├── api # rest api v2 wrapper
│ ├── cli # Command Line Interface Functions
│ ├── endpoint # Language library - Endpoints
│ └── serverless # SDK - Serverless Worker
└── tests # Package tests
```