Our product lets companies upload their own recordings, videos, documents and images, then ask an LLM to analyse them and produce reports. Some tenants upload a few gigabytes; the largest has over 6 TB of call recordings and training videos. Three requirements drove the whole architecture: ingest any common file type at that scale without blocking the API, guarantee that one tenant's data never reaches another tenant's prompts, and bill each tenant for what they actually consumed, which is mostl...
Multi-tenant LLM platform design: terabyte ingestion, isolation and usage billing
Tenants upload terabytes of audio, video, PDFs and images, and expect an LLM to answer questions over all of it, with their data isolated and a bill that matches what they used. Here is the AWS design we shipped: ingestion pipeline, isolation layers and metering.
This is a paid story
Subscribe to read the full content, access all paid stories, and support the authors.
Add a label
Advertisement
Comments
0No comments yet. Be the first to comment!
Sign In to post a comment.