Skip to content
Osarai
Get started

DocsSDKs and frameworks

LiteLLM

Call your Osarai GPU through LiteLLM's Python SDK, or put it behind a LiteLLM proxy next to other providers.

LiteLLM gives many providers one interface. Osarai is an OpenAI-compatible provider: prefix the model with openai/ and set the base URL.

Python SDK

Terminal
pip install litellm
Python
import litellm, os

reply = litellm.completion(
    model="openai/qwen3.8-27b",
    api_base="https://osarai.xyz/v1",
    api_key=os.environ["OSARAI_API_KEY"],
    messages=[{"role": "user", "content": "Explain recursion in one paragraph."}],
)
print(reply.choices[0].message.content)

stream=True, tools and reasoning_effort work as in the OpenAI SDK. Embeddings use litellm.embedding(model="openai/qwen3-embedding-8b", …) with the same base URL.

Proxy

Run a LiteLLM proxy to route several apps and providers through one place, with Osarai as one backend:

litellm_config.yaml
model_list:
  - model_name: qwen3.8-27b
    litellm_params:
      model: openai/qwen3.8-27b
      api_base: https://osarai.xyz/v1
      api_key: os.environ/OSARAI_API_KEY
    model_info:
      max_input_tokens: 131072
      max_output_tokens: 16384
      input_cost_per_token: 0
      output_cost_per_token: 0

  - model_name: qwen3-embedding-8b
    litellm_params:
      model: openai/qwen3-embedding-8b
      api_base: https://osarai.xyz/v1
      api_key: os.environ/OSARAI_API_KEY
Terminal
pip install 'litellm[proxy]'
export OSARAI_API_KEY="qck_your_key_here"
litellm --config litellm_config.yaml --port 4000

Clients then call http://localhost:4000/v1 with the proxy's own key and model: "qwen3.8-27b". Token costs are zero because Osarai bills the GPU by the hour.

Troubleshooting

SymptomFix
LLM Provider NOT providedKeep the openai/ prefix on the model
NotFoundErrorapi_base must end with /v1
AuthenticationErrorOSARAI_API_KEY is not set where LiteLLM runs