> ## Documentation Index
> Fetch the complete documentation index at: https://portkey-docs-feat-rerank-documentation.mintlify.site/llms.txt
> Use this file to discover all available pages before exploring further.
# Ollama
> Integrate Ollama-hosted models with Portkey for local LLM deployment with full observability.
Portkey provides a robust gateway to facilitate the integration of your **locally hosted models through Ollama**.
First, install the Gateway locally:
```bash npx theme={"system"}
npx @portkey-ai/gateway
```
```bash Docker theme={"system"}
docker run -d -p 8787:8787 portkeyai/gateway:latest
```
Then, connect to your local Ollama instance:
```python Python theme={"system"}
from portkey_ai import Portkey
portkey = Portkey(
base_url="http://localhost:8787", # Your local Gateway
provider="ollama",
custom_host="http://localhost:11434" # Your Ollama instance
)
response = portkey.chat.completions.create(
model="llama3",
messages=[{"role": "user", "content": "Hello!"}]
)
```
## Integration Steps
Expose your Ollama API using a tunneling service like [ngrok](https://ngrok.com/) or make it publicly accessible. Skip this if you're self-hosting the Gateway.
For using Ollama with ngrok, here's a [useful guide](https://github.com/ollama/ollama/blob/main/docs/faq.md#how-can-i-use-ollama-with-ngrok):
```sh theme={"system"}
ngrok http 11434 --host-header="localhost:11434"
```
1. Go to [**Model Catalog → Add Provider**](https://app.portkey.ai/model-catalog/providers)
2. Enable **"Local/Privately hosted provider"** toggle
3. Select **Ollama** as the provider type
4. Enter your Ollama URL in **Custom Host**: `https://your-ollama.ngrok-free.app`
5. Name your provider (e.g., `my-ollama`)
See all setup options
```python Python theme={"system"}
from portkey_ai import Portkey
portkey = Portkey(
api_key="PORTKEY_API_KEY",
provider="@my-ollama"
)
response = portkey.chat.completions.create(
model="llama3",
messages=[{"role": "user", "content": "Hello!"}]
)
print(response.choices[0].message.content)
```
```javascript Node.js theme={"system"}
import Portkey from 'portkey-ai';
const portkey = new Portkey({
apiKey: 'PORTKEY_API_KEY',
provider: '@my-ollama'
});
const response = await portkey.chat.completions.create({
model: 'llama3',
messages: [{ role: 'user', content: 'Hello!' }]
});
console.log(response.choices[0].message.content);
```
**Or use custom host directly:**
```python Python theme={"system"}
from portkey_ai import Portkey
portkey = Portkey(
api_key="PORTKEY_API_KEY",
provider="ollama",
custom_host="https://your-ollama.ngrok-free.app"
)
```
```javascript Node.js theme={"system"}
import Portkey from 'portkey-ai';
const portkey = new Portkey({
apiKey: 'PORTKEY_API_KEY',
provider: 'ollama',
customHost: 'https://your-ollama.ngrok-free.app'
});
```
**Important:** For Ollama integration, you only need to pass the base URL to `customHost` **without** the version identifier (such as `/v1`) - Portkey handles the rest!
***
## Supported Models
Ollama supports a wide range of models including:
* Llama 3, Llama 3.1, Llama 3.2
* Mistral, Mixtral
* Gemma, Gemma 2
* Phi-3
* Qwen 2
* And many more!
Check [Ollama's model library](https://ollama.com/library) for the complete list.
***
## Next Steps
Add retries, timeouts, and fallbacks
Monitor your Ollama requests
Learn more about custom host setup
Complete guide for private LLMs
For complete SDK documentation:
Complete Portkey SDK documentation