> ## Documentation Index
> Fetch the complete documentation index at: https://portkey-docs-feat-rerank-documentation.mintlify.site/llms.txt
> Use this file to discover all available pages before exploring further.
# LocalAI
> Integrate LocalAI-hosted models with Portkey for local LLM deployment with observability.
Portkey provides a robust gateway to facilitate the integration of your **locally hosted models through** [**LocalAI**](https://localai.io/).
## Integration Steps
Ensure your LocalAI API is externally accessible. If running on `http://localhost`, use a tool like `ngrok` to create a public URL.
```sh theme={"system"}
ngrok http 8080
```
1. Go to [**Model Catalog → Add Provider**](https://app.portkey.ai/model-catalog/providers)
2. Enable **"Local/Privately hosted provider"** toggle
3. Select **OpenAI** as the provider type (LocalAI follows OpenAI API schema)
4. Enter your LocalAI URL with `/v1` in **Custom Host**: `https://your-localai.ngrok-free.app/v1`
5. Name your provider (e.g., `my-localai`)
See all setup options
```python Python theme={"system"}
from portkey_ai import Portkey
portkey = Portkey(
api_key="PORTKEY_API_KEY",
provider="@my-localai"
)
response = portkey.chat.completions.create(
model="ggml-koala-7b-model-q4_0-r2.bin",
messages=[{"role": "user", "content": "Hello!"}]
)
print(response.choices[0].message.content)
```
```javascript Node.js theme={"system"}
import Portkey from 'portkey-ai';
const portkey = new Portkey({
apiKey: 'PORTKEY_API_KEY',
provider: '@my-localai'
});
const response = await portkey.chat.completions.create({
model: 'ggml-koala-7b-model-q4_0-r2.bin',
messages: [{ role: 'user', content: 'Hello!' }]
});
console.log(response.choices[0].message.content);
```
**Or use custom host directly:**
```python Python theme={"system"}
from portkey_ai import Portkey
portkey = Portkey(
api_key="PORTKEY_API_KEY",
provider="openai",
custom_host="https://your-localai.ngrok-free.app/v1"
)
```
```javascript Node.js theme={"system"}
import Portkey from 'portkey-ai';
const portkey = new Portkey({
apiKey: 'PORTKEY_API_KEY',
provider: 'openai',
customHost: 'https://your-localai.ngrok-free.app/v1'
});
```
**Important:**
* Don't forget to include the version identifier (`/v1`) in the Custom Host URL
* Portkey supports all endpoints that adhere to the OpenAI specification
***
## LocalAI Endpoints Supported
| Endpoint | Resource |
| :---------------------------------------------- | :---------------------------------------------------------------- |
| /chat/completions (Chat, Vision, Tools support) | [Doc](/api-reference/inference-api/chat) |
| /images/generations | [Doc](/api-reference/inference-api/images/create-image) |
| /embeddings | [Doc](/api-reference/inference-api/embeddings) |
| /audio/transcriptions | [Doc](/product/ai-gateway/multimodal-capabilities/speech-to-text) |
***
## Next Steps
Add retries, timeouts, and fallbacks
Monitor your LocalAI requests
Learn more about custom host setup
Complete guide for private LLMs
For complete SDK documentation:
Complete Portkey SDK documentation