Top Open Source LLMs: Families, Licenses and How to Run Them

Open-weight language models are now a genuine deployment option rather than a research curiosity. The strongest families — Llama, Mistral, Qwen, DeepSeek, Gemma and a growing set of challengers — cover everything from 1B models that run on a phone to sparse mixture-of-experts systems that need a server rack. The hard part is no longer capability. It is choosing a license you can actually live with and sizing the hardware honestly.

What is the difference between open source and open weights?

Open weights means the trained parameters are downloadable, runnable on your own hardware and fine-tunable. Open source, in the traditional sense, additionally prohibits field-of-use restrictions and expects enough information about training data and code for a competent third party to rebuild the system. Most models marketed as open source are accurately described as open weights.

Almost every model in this article is distributed as open weights: the trained parameters are downloadable, you can run them on your own hardware, and you can fine-tune them. That is a meaningful freedom and it is what most teams actually need.

It is not the same as open source in the traditional sense. The Open Source Initiative’s definition of open source software prohibits field-of-use restrictions, and its later definition of open source AI additionally expects sufficient information about training data, along with the training and inference code, for a competent third party to rebuild the system.

Judged against that bar, the picture changes. Models released under Apache 2.0 or MIT have genuinely open licenses but almost never publish their training data. Models with custom licenses — Llama, Gemma, some Qwen releases — are not open source at all, regardless of how they are marketed.

A small number of projects clear both bars. Allen Institute for AI’s OLMo family publishes weights, training code, intermediate checkpoints and the pretraining corpus under permissive terms. EleutherAI’s older Pythia suite did the same. These matter for research reproducibility even when they are not the strongest models available.

The practical takeaway: read the license file, not the blog post. “Open” in a launch announcement is a marketing word.

Which open-weight families matter most?

Six cover most production use: Llama for ecosystem and tooling depth, Mistral for quality per parameter, Qwen for breadth across sizes and modalities, DeepSeek for reasoning and cost efficiency at scale, Gemma for quality per gigabyte on a single GPU, and gpt-oss for reasoning under permissive licensing. Behind them sits a long tail worth knowing about.

Llama (Meta)

Llama did more than any other release to make local inference mainstream, and the ecosystem effect is still its biggest advantage. Nearly every quantization tool, fine-tuning framework and serving engine treats Llama as the reference architecture, so tooling support is the deepest of any family.

Recent generations span small dense models suitable for edge deployment through to large mixture-of-experts variants aimed at data center serving. Multimodal and long-context variants are part of the line.

The license is the caveat. The Llama Community License permits broad commercial use but adds conditions no standard open source license contains: an attribution requirement, a rule that derivative model names carry the Llama prefix, a binding acceptable use policy, and a monthly-active-user threshold above which you must negotiate separate terms with Meta. That threshold is high enough to be irrelevant to almost everyone, but it is a field-of-use restriction and it disqualifies the license from being open source.

Mistral

Mistral AI built its reputation on small models that punch above their parameter count, and that remains the family’s character. The smaller dense models and the Mixtral sparse models are released under Apache 2.0 — genuinely permissive, no user thresholds, no naming rules.

The company also ships models under the Mistral Research License and the Mistral Non-Production License, which restrict use to research and evaluation. Its largest and most specialized models often land in this category, with commercial deployment requiring a paid agreement. Two Mistral models can therefore have completely different commercial implications, and the model card is where you find out which is which.

Mistral’s code-oriented and reasoning-oriented releases are worth attention if you are building developer tooling. See our roundup of the best AI coding assistants for how open-weight code models compare with hosted alternatives.

Qwen (Alibaba)

Qwen is arguably the most complete open-weight family available. It spans sub-1B models to very large mixture-of-experts systems, with strong dedicated variants for code, vision-language, audio and embeddings, and unusually good multilingual coverage.

Recent Qwen generations have shipped under Apache 2.0 across the whole size range, which is a significant improvement over earlier releases where the largest and smallest checkpoints carried a custom Qwen license with its own user threshold. Check the specific checkpoint rather than assuming the family is uniform.

The breadth is the real story. If you need a small embedding model, a mid-size instruct model and a large reasoning model that all share a tokenizer and prompt format, Qwen is usually the shortest path.

DeepSeek

DeepSeek’s contribution was demonstrating that frontier-adjacent reasoning could be trained and released openly, and released under a genuinely permissive license. Its flagship mixture-of-experts models and reasoning models have been published under MIT terms, with no use restrictions attached to the weights.

DeepSeek also popularized distillation as a distribution strategy: reasoning traces from a large model used to fine-tune much smaller Llama and Qwen bases, producing compact models with disproportionately good step-by-step reasoning. Those distilled variants inherit the license of their base model, which is a licensing trap worth knowing about.

The architectural work — multi-head latent attention, aggressive sparsity, efficient training recipes — has been influential well beyond the models themselves.

Gemma (Google)

Gemma models are derived from the same research lineage as Gemini and are optimized for the small-to-mid range where you care about quality per gigabyte. Recent generations added multimodal input and substantially longer context at sizes that still fit on a single consumer GPU.

The Gemma Terms of Use permit commercial use and redistribution but impose a prohibited use policy that must be passed through to anyone you distribute the model or derivatives to. Like Llama, it is a usable license rather than an open source one.

gpt-oss (OpenAI)

OpenAI’s open-weight release put two Apache 2.0 reasoning models into the ecosystem, sized so that the larger targets a single high-memory data center GPU and the smaller runs comfortably on consumer hardware. They shipped with low-precision quantization as the native format rather than as an afterthought, which is a meaningful signal about where deployment is heading.

The rest of the field

Microsoft’s Phi family pursues small models trained on heavily curated and synthetic data, generally under MIT terms, and is a reasonable default when you need something that runs on a laptop. Nvidia’s Nemotron line takes existing open bases and applies additional alignment and efficiency work. IBM’s Granite models target enterprise use with Apache 2.0 licensing and unusually thorough documentation of training data provenance.

Several Chinese labs beyond Alibaba and DeepSeek now publish large models under MIT or modified-MIT terms, including sparse systems in the trillion-total-parameter class. Falcon from TII continues to explore hybrid state-space and attention architectures. Cohere publishes weights for some models under non-commercial terms only, which makes them useful for research and useless for products.

Comparison table

Capability rankings move every few weeks, so this table deliberately avoids scores. Treat license character and size range as the durable columns.

How much VRAM do you need to run a model?

Multiply parameter count by bytes per parameter, then add roughly 20 to 30 percent for the KV cache, activations and framework overhead. That puts a 7B model at about 14 GB in 16-bit and roughly 3.5 GB at 4-bit, and a 70B model at over 140 GB in 16-bit and roughly 40 GB at 4-bit. The arithmetic is simple enough to do in your head.

• 16-bit: 2 bytes per parameter. A 7B model needs about 14 GB before overhead.
• 8-bit: 1 byte per parameter. That same 7B model needs about 7 GB.
• 4-bit: roughly 0.5 bytes per parameter. About 3.5 GB.

Scaled up, a 70B model needs over 140 GB at 16-bit, around 70 GB at 8-bit, and roughly 40 GB at 4-bit. That last number is why 48 GB workstation cards and dual 24 GB consumer setups are the sweet spot for serious local inference.

What does that formula leave out?

Two corrections to the naive formula matter. First, long context is not free: the KV cache grows linearly with sequence length and with batch size, and at 100k tokens it can rival the weights in size. Models using grouped-query or latent attention reduce this substantially.

Second, mixture-of-experts models break the intuition that active parameters determine cost. Compute scales with active parameters, so a sparse model generates tokens quickly. Memory scales with total parameters, because every expert must be resident. A sparse model with a few tens of billions of active parameters can still require hundreds of gigabytes of memory.

Apple silicon is a genuine third option because unified memory means the GPU can address system RAM. A machine with 64 GB or more of unified memory runs models that would need multiple discrete GPUs, at lower token throughput but with far less complexity.

Which quantization format should you use?

Prefer 8-bit or FP8 if you have the memory, use 4-bit K-quants or AWQ when you do not, and avoid anything below 4-bit except for experimentation. Quantization reduces the numeric precision of weights, and sometimes activations, to shrink memory and increase throughput. The formats you will encounter:

• GGUF is the format used by llama.cpp and everything built on it. The K-quant variants — Q4_K_M is the usual default — balance size and quality well, and GGUF runs on CPU, Metal and GPU.
• AWQ and GPTQ are GPU-oriented post-training quantization methods that use calibration data to protect the weights that matter most. Both are well supported by production serving engines.
• FP8 is hardware-native on recent Nvidia data center GPUs and gives near-lossless quality with a clean 2x memory reduction against 16-bit. Newer 4-bit hardware formats extend the same idea.
• bitsandbytes NF4 is the standard choice for QLoRA fine-tuning rather than for serving.

The practical rule: prefer 8-bit or FP8 if you have the memory, use 4-bit K-quants or AWQ when you do not, and avoid anything below 4-bit except for experimentation. Always evaluate the quantized model on your own task. Quality loss from quantization shows up first in exactly the places that are hardest to spot in casual testing — long-context recall, structured output validity and non-English fluency.

Should you use Ollama or vLLM?

Ollama for local development, single-user workloads and quick experiments; vLLM for production serving, where continuous batching and paged attention give far higher throughput per GPU. Below both sits llama.cpp for unusual hardware and CPU-only deployment, and beside them a handful of specialists — SGLang, TensorRT-LLM and MLX.

Ollama is the fastest path from nothing to a working local model. It manages downloads, applies sensible defaults, and exposes an OpenAI-compatible HTTP endpoint, which means code written against the patterns in our OpenAI API tutorial usually works against a local model with only a base URL change.

# Pull and run a quantized model locally
ollama pull qwen3:8b
ollama run qwen3:8b "Summarize the tradeoffs of 4-bit quantization."

# Same model, served on an OpenAI-compatible endpoint
curl http://localhost:11434/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{"model":"qwen3:8b","messages":[{"role":"user","content":"Hello"}]}'

llama.cpp is the layer Ollama sits on and the right choice when you need control or unusual hardware. It runs on CPU, on Apple Metal, on Vulkan and on CUDA, and it is the only realistic option for CPU-only deployment.

vLLM is the production default. Paged attention and continuous batching give it several times the throughput of naive serving under concurrent load, and it supports tensor and pipeline parallelism across GPUs, structured output constraints, and prefix caching.

# Production serving with an OpenAI-compatible API
vllm serve mistralai/Mistral-Small-Instruct \
  --quantization awq \
  --max-model-len 32768 \
  --gpu-memory-utilization 0.90

SGLang competes with vLLM and tends to win on workloads with heavy prompt reuse thanks to its radix-tree prefix cache. TensorRT-LLM extracts the most performance from Nvidia hardware at the cost of a compilation step and much less flexibility. MLX is the Apple silicon native option.

For any agent or retrieval workload, the serving layer matters more than people expect, because those systems issue many short calls with heavily shared prefixes. Prefix caching alone can halve cost. Our guide to retrieval-augmented generation covers why that pattern dominates production traffic.

Which open-weight model should you choose?

Start from constraints, not leaderboards. Choose Apache 2.0 and MIT releases if you need permissive licensing with nothing to audit, Llama if you need the deepest fine-tuning and tooling ecosystem, Qwen if you need one family spanning many modalities and sizes, Gemma or Phi if you are memory-constrained on a single consumer GPU, and DeepSeek if you need step-by-step reasoning on a budget.

If you need permissive licensing with no conditions to audit, restrict yourself to Apache 2.0 and MIT releases: Mistral’s permissive checkpoints, recent Qwen, DeepSeek flagships, gpt-oss, Phi, Granite and OLMo. This removes most of the legal review burden.

If you need the deepest fine-tuning and tooling ecosystem, Llama remains the safest technical bet, provided your legal team accepts a custom license.

If you need one model family covering many modalities and sizes with a consistent interface, Qwen is the most complete.

If you are memory-constrained on a single consumer GPU, Gemma and Phi are designed for exactly that envelope.

If you need step-by-step reasoning on a budget, look at DeepSeek’s models and the distilled variants, remembering that the distills carry their base model’s license.

Then run your own evaluation. Public leaderboards measure aggregate ability on tasks that are almost certainly not yours, and contamination makes the gaps between similar models unreliable. A hundred examples from your actual workload, scored consistently, will tell you more than any benchmark table — the same principle we apply in our prompt engineering guide.

The bottom line

Open-weight models have closed most of the capability gap for mainstream tasks, and the remaining decision is about governance and infrastructure rather than raw quality. Pick a license you can defend in a review, size your hardware with the parameter-times-bytes formula rather than optimism, quantize to 4-bit or 8-bit and verify the loss on your own data, and serve with vLLM once you leave the laptop.

The field moves fast enough that any specific model recommendation ages within a quarter. The license structures, the memory arithmetic and the serving stack do not. Build against those.

Browse more coverage in the open source LLMs category.