> ## Documentation Index
> Fetch the complete documentation index at: https://portkey-docs-feat-rerank-documentation.mintlify.site/llms.txt
> Use this file to discover all available pages before exploring further.
# Deepinfra
> Integrate Deepinfra models with Portkey's AI Gateway
Portkey provides a robust and secure gateway to integrate various Large Language Models (LLMs) into applications, including [Deepinfra's hosted models](https://deepinfra.com/models/text-generation).
With Portkey, take advantage of features like fast AI gateway access, observability, prompt management, and more, while securely managing API keys through [Model Catalog](/product/model-catalog).
## Quick Start
Get Deepinfra working in 3 steps:
```python Python icon="python" theme={"system"}
from portkey_ai import Portkey
# 1. Install: pip install portkey-ai
# 2. Add @deepinfra provider in model catalog
# 3. Use it:
portkey = Portkey(api_key="PORTKEY_API_KEY")
response = portkey.chat.completions.create(
model="@deepinfra/nvidia/Nemotron-4-340B-Instruct",
messages=[{"role": "user", "content": "Say this is a test"}]
)
print(response.choices[0].message.content)
```
```js Javascript icon="square-js" theme={"system"}
import Portkey from 'portkey-ai'
// 1. Install: npm install portkey-ai
// 2. Add @deepinfra provider in model catalog
// 3. Use it:
const portkey = new Portkey({
apiKey: "PORTKEY_API_KEY"
})
const response = await portkey.chat.completions.create({
model: "@deepinfra/nvidia/Nemotron-4-340B-Instruct",
messages: [{ role: "user", content: "Say this is a test" }]
})
console.log(response.choices[0].message.content)
```
```python OpenAI Py icon="python" theme={"system"}
from openai import OpenAI
from portkey_ai import PORTKEY_GATEWAY_URL
# 1. Install: pip install openai portkey-ai
# 2. Add @deepinfra provider in model catalog
# 3. Use it:
client = OpenAI(
api_key="PORTKEY_API_KEY", # Portkey API key
base_url=PORTKEY_GATEWAY_URL
)
response = client.chat.completions.create(
model="@deepinfra/nvidia/Nemotron-4-340B-Instruct",
messages=[{"role": "user", "content": "Say this is a test"}]
)
print(response.choices[0].message.content)
```
```js OpenAI JS icon="square-js" theme={"system"}
import OpenAI from "openai"
import { PORTKEY_GATEWAY_URL } from "portkey-ai"
// 1. Install: npm install openai portkey-ai
// 2. Add @deepinfra provider in model catalog
// 3. Use it:
const client = new OpenAI({
apiKey: "PORTKEY_API_KEY", // Portkey API key
baseURL: PORTKEY_GATEWAY_URL
})
const response = await client.chat.completions.create({
model: "@deepinfra/nvidia/Nemotron-4-340B-Instruct",
messages: [{ role: "user", content: "Say this is a test" }]
})
console.log(response.choices[0].message.content)
```
```sh cURL icon="square-terminal" theme={"system"}
# 1. Add @deepinfra provider in model catalog
# 2. Use it:
curl https://api.portkey.ai/v1/chat/completions \
-H "Content-Type: application/json" \
-H "x-portkey-api-key: $PORTKEY_API_KEY" \
-d '{
"model": "@deepinfra/nvidia/Nemotron-4-340B-Instruct",
"messages": [
{ "role": "user", "content": "Say this is a test" }
]
}'
```
**Tip:** You can also set `provider="@deepinfra"` in `Portkey()` and use just `model="nvidia/Nemotron-4-340B-Instruct"` in the request.
## Add Provider in Model Catalog
1. Go to [**Model Catalog → Add Provider**](https://app.portkey.ai/model-catalog/providers)
2. Select **Deepinfra**
3. Choose existing credentials or create new by entering your [Deepinfra API key](https://deepinfra.com/dash/api_keys)
4. Name your provider (e.g., `deepinfra-prod`)
See all setup options, code examples, and detailed instructions
## Supported Endpoints
| Endpoint | Supported |
| ------------------- | --------- |
| `/chat/completions` | ✅ |
| `/completions` | ✅ |
| `/embeddings` | ✅ |
## Tool Calling
DeepInfra supports tool calling (function calling) for compatible models. Use the standard OpenAI `tools` format:
```python Python theme={"system"}
from portkey_ai import Portkey
portkey = Portkey(api_key="PORTKEY_API_KEY")
tools = [
{
"type": "function",
"function": {
"name": "get_weather",
"description": "Get the current weather for a location",
"parameters": {
"type": "object",
"properties": {
"location": {"type": "string", "description": "City name"}
},
"required": ["location"]
}
}
}
]
response = portkey.chat.completions.create(
model="@deepinfra/meta-llama/Meta-Llama-3.1-70B-Instruct",
messages=[{"role": "user", "content": "What's the weather in London?"}],
tools=tools,
tool_choice="auto"
)
print(response.choices[0].message.tool_calls)
```
```js Javascript theme={"system"}
import Portkey from 'portkey-ai'
const portkey = new Portkey({ apiKey: "PORTKEY_API_KEY" })
const tools = [
{
type: "function",
function: {
name: "get_weather",
description: "Get the current weather for a location",
parameters: {
type: "object",
properties: {
location: { type: "string", description: "City name" }
},
required: ["location"]
}
}
}
]
const response = await portkey.chat.completions.create({
model: "@deepinfra/meta-llama/Meta-Llama-3.1-70B-Instruct",
messages: [{ role: "user", content: "What's the weather in London?" }],
tools,
tool_choice: "auto"
})
console.log(response.choices[0].message.tool_calls)
```
## Supported Models
Deepinfra hosts a wide range of open-source models for text generation. View the complete list:
Browse all available models on Deepinfra
Popular models include:
* `nvidia/Nemotron-4-340B-Instruct`
* `meta-llama/Meta-Llama-3.1-405B-Instruct`
* `Qwen/Qwen2.5-72B-Instruct`
## Next Steps
Add metadata to your Deepinfra requests
Add gateway configs to your Deepinfra requests
Trace your Deepinfra requests
Setup fallback from OpenAI to Deepinfra
For complete SDK documentation:
Complete Portkey SDK documentation