Sub-second transcription, multi-speaker diarization, and 99+ language detection. Deploy on serverless GPU containers or host on-premise with zero data retention.
Multi-speaker diarization enabled • Flash Attention v2 CUDA Kernel
"Whisper Large-v3 accelerated with Flash Attention 2 delivers up to 5x higher throughput compared to traditional CUDA pipelines with half the VRAM requirement."
"By utilizing Modal's serverless GPU infrastructure, we scale from zero to 80 concurrent transcription workers in under 400 milliseconds."
Stop paying per-minute markup fees to black-box APIs. Voxly gives you raw GPU speed, full privacy control, and sub-second transcription.
Powered by OpenAI Whisper large-v3 with native Flash Attention 2 CUDA kernels. Voxly an hour of audio in under 12 seconds.
Seamlessly identify who spoke when. Distinguish up to 10 distinct speakers with timestamp boundaries and color tagging.
Run serverless GPU workers on Modal or deploy on-premise Docker containers on your local RTX/A100 cluster.
Automatic language identification, accents parsing, and optional English translation pipeline for global audio content.
Export instantly to JSON, SRT, VTT, or raw TXT. Fully compatible with Premiere Pro, DaVinci Resolve, and REST APIs.
By combining PyTorch 2.0 SDPA, Flash Attention v2 CUDA kernels, and PyTorch FP16 quantization, Voxly destroys traditional speech recognition overhead.
Tested on NVIDIA A10G 24GB Tensor Core GPU
Get complete source code, serverless autoscaling, and raw inference speed in 3 simple shell commands.
from modal import Image, Stub, method, NetworkFileSystem, asgi_app
from fastapi import Request, FastAPI
import tempfile
import time
MODEL_DIR = "/model"
web_app = FastAPI()
def download_model():
from huggingface_hub import snapshot_download
snapshot_download("openai/whisper-large-v3", local_dir=MODEL_DIR)
image = (
Image.from_registry("nvidia/cuda:12.1.0-cudnn8-devel-ubuntu22.04", add_python="3.9")
.apt_install("git","ffmpeg")
.pip_install(
"transformers", "ninja", "packaging", "wheel", "torch", "hf-transfer~=0.1", "ffmpeg-python"
).run_commands("python -m pip install flash-attn --no-build-isolation", gpu="A10G")
.env({"HF_HUB_ENABLE_HF_TRANSFER": "1"})
.run_function(download_model)
)
stub = Stub("whisp-x", image=image)
@stub.cls(gpu="A10G", allow_concurrent_inputs=80, container_idle_timeout=40)
class WhisperV3:
def __enter__(self):
import torch
from transformers import AutoModelForSpeechSeq2Seq, AutoProcessor, pipeline
self.device = "cuda:0" if torch.cuda.is_available() else "cpu"
model = AutoModelForSpeechSeq2Seq.from_pretrained(
MODEL_DIR, torch_dtype=torch.float16, use_safetensors=True, use_flash_attention_2=True
).to(self.device)
processor = AutoProcessor.from_pretrained(MODEL_DIR)
self.pipe = pipeline(
"automatic-speech-recognition",
model=model,
tokenizer=processor.tokenizer,
feature_extractor=processor.feature_extractor,
chunk_length_s=30,
batch_size=24,
return_timestamps=True,
device=0,
)
@method()
def generate(self, audio: bytes):
fp = tempfile.NamedTemporaryFile(delete=False, suffix=".mp3")
fp.write(audio)
fp.close()
start = time.time()
output = self.pipe(fp.name, chunk_length_s=30, batch_size=24, return_timestamps=True)
return output, time.time() - start
@stub.function()
@web_app.post("/")
async def transcribe(request: Request):
form = await request.form()
audio = await form["audio"].read()
output, elapsed = WhisperV3().generate.remote(audio)
return {"transcript": output, "elapsed_seconds": elapsed}modal deploy modal_app.py curl -X POST -F "audio=@sample.mp3" https://your-modal-app.modal.runChoose between self-hosting on your hardware or utilizing our serverless cloud API.
Ideal for hackers, researchers, and teams with their own GPU servers.
On-demand GPU transcription powered by Modal serverless infrastructure.
Dedicated GPU clusters deployed inside your AWS / GCP Virtual Private Cloud.
Start transcribing audio with OpenAI Whisper-v3 and Flash Attention 2 in less than 2 minutes. Try the interactive playground or deploy serverless on Modal.