Best way to deploy open-source LLM models

Originally published on Medium on May 20, 2024. This article reflects that comparison, with wording clarified for republication. Provider examples and pricing comparisons are historical.

Deploying open-source models can be challenging, especially when considering privacy, security, and cost. In May 2024, I researched pricing, speed, privacy, and control to understand the different ways to deploy an LLM.

I divided the options into two categories: self-managed deployment and hosted APIs.

Self-managed deployment

Running a model yourself gives you control over its infrastructure and configuration. Depending on the model's license and your resources, you can also modify or fine-tune it. The tradeoff is that you take responsibility for getting the deployment working and keeping it running.

How it works

Typically, you provision a GPU instance, such as an EC2 instance or Azure VM. You install the runtime and dependencies, download compatible model weights from Hugging Face, configure the inference server, and set up network access, authentication, and routing.

You then expose an endpoint your application can use. Maintaining that endpoint, including monitoring, updates, capacity, and failures, becomes your responsibility.

Advantages

1. Control. You choose the infrastructure, serving configuration, and model version. You have more flexibility to customize the deployment.
2. Application-specific tuning. Understanding the model's strengths and limitations lets you tune the system around your application's requirements.

Tradeoffs

1. Cost and latency. You pay for the infrastructure you provision. Longer prompts and outputs consume more inference resources, while the hardware, serving stack, and traffic pattern affect latency. An idle GPU can be expensive; a well-utilized one can change the economics.
2. Operational work. Models require memory, storage, and compute, along with the engineering time to operate them. This can be a substantial commitment for a small team.

The original article includes the pricing comparison image. It is a May 2024 snapshot, not a current price list.

Hosted APIs for open-source models

A hosted API lets another provider operate the inference infrastructure. Your application sends requests to its endpoint rather than running the model server itself.

In the original comparison, examples included Together AI, Replicate, DeepInfra, Perplexity, and AWS Bedrock. Model availability and API compatibility vary by provider and change over time.

Many providers let you switch between supported models through a model parameter. That can make experimentation easier, although switching models may still require changes to prompts, parameters, or response handling.

Advantages

1. Ease of use. You can focus on your application instead of provisioning and maintaining model infrastructure.
2. Flexibility. APIs can be integrated into different languages and application stacks, often with a choice of supported models.
3. Lower initial overhead. Hosted inference can be more economical for small or uneven workloads because you do not have to operate a dedicated server. The right comparison still depends on usage and provider pricing.

Tradeoffs

1. Less control. The provider determines which models, configurations, and customization options are available.
2. Provider dependency. Outages, rate limits, policy changes, and model retirements can affect your application.

Which approach fits your project?

My starting recommendation in 2024 was to use hosted APIs for projects where speed of development and a limited operating budget mattered most, then consider self-managed deployment when control requirements justified the additional work.

When I would consider a hosted API

• Personal projects and prototypes.
• Bootstrapped startups without a dedicated infrastructure team.
• Consumer applications where the provider meets the application's cost and latency requirements.
• Company projects whose data handling requirements are compatible with the provider's terms and controls.

When I would consider self-managed deployment

• Financial or banking workloads with specific infrastructure and data handling requirements.
• Applications using internal documents where the company needs direct control over where processing happens.
• Projects that need deployment-level customization unavailable through a hosted API.

Self-managed deployment does not automatically make a system private or secure, and hosted services are not automatically unsuitable for sensitive work. The decision needs to account for access controls, retention, data residency, contracts, and the team's ability to operate the system.

The useful question is which approach meets your requirements at a cost your team can sustain, including the cost of maintaining it.

You can find more of my work on Instagram or email me.