Skip to main content

✨ Enterprise Features

Features here are behind a commercial license in our /enterprise folder. See Code

info

Get Started with Enterprise here

Features:

  • Content Moderation with LlamaGuard
  • Content Moderation with Google Text Moderations
  • Tracking Spend for Custom Tags

Content Moderation with LlamaGuard​

Currently works with Sagemaker's LlamaGuard endpoint.

How to enable this in your config.yaml: 

litellm_settings:
   callbacks: ["llamaguard_moderations"]
   llamaguard_model_name: "sagemaker/jumpstart-dft-meta-textgeneration-llama-guard-7b"

Make sure you have the relevant keys in your environment, eg.: 

os.environ["AWS_ACCESS_KEY_ID"] = ""
os.environ["AWS_SECRET_ACCESS_KEY"] = ""
os.environ["AWS_REGION_NAME"] = ""

Content Moderation with Google Text Moderation​

Requires your GOOGLE_APPLICATION_CREDENTIALS to be set in your .env (same as VertexAI).

How to enable this in your config.yaml: 

litellm_settings:
   callbacks: ["google_text_moderation"]

Set custom confidence thresholds​

Google Moderations checks the test against several categories. Source

Set global default confidence threshold​

By default this is set to 0.8. But you can override this in your config.yaml.

litellm_settings: 
    google_moderation_confidence_threshold: 0.4

Set category-specific confidence threshold​

Set a category specific confidence threshold in your config.yaml. If none set, the global default will be used. 

litellm_settings: 
    toxic_confidence_threshold: 0.1

Here are the category specific values:

CategorySetting
"toxic"toxic_confidence_threshold: 0.1
"insult"insult_confidence_threshold: 0.1
"profanity"profanity_confidence_threshold: 0.1
"derogatory"derogatory_confidence_threshold: 0.1
"sexual"sexual_confidence_threshold: 0.1
"death_harm_and_tragedy"death_harm_and_tragedy_threshold: 0.1
"violent"violent_threshold: 0.1
"firearms_and_weapons"firearms_and_weapons_threshold: 0.1
"public_safety"public_safety_threshold: 0.1
"health"health_threshold: 0.1
"religion_and_belief"religion_and_belief_threshold: 0.1
"illicit_drugs"illicit_drugs_threshold: 0.1
"war_and_conflict"war_and_conflict_threshold: 0.1
"politics"politics_threshold: 0.1
"finance"finance_threshold: 0.1
"legal"legal_threshold: 0.1

Tracking Spend for Custom Tags​

Requirements:

  • Virtual Keys & a database should be set up, see virtual keys

Usage - /chat/completions requests with request tags​

Set extra_body={"metadata": { }} to metadata you want to pass

import openai
client = openai.OpenAI(
    api_key="anything",
    base_url="http://0.0.0.0:8000"
)

# request sent to model set on litellm proxy, `litellm --model`
response = client.chat.completions.create(
    model="gpt-3.5-turbo",
    messages = [
        {
            "role": "user",
            "content": "this is a test request, write a short poem"
        }
    ],
    extra_body={
        "metadata": {
            "tags": ["model-anthropic-claude-v2.1", "app-ishaan-prod"]
        }
    }
)

print(response)

Viewing Spend per tag​

/spend/tags Request Format​

curl -X GET "http://0.0.0.0:4000/spend/tags" \
-H "Authorization: Bearer sk-1234"

/spend/tagsResponse Format​

[
  {
    "individual_request_tag": "model-anthropic-claude-v2.1",
    "log_count": 6,
    "total_spend": 0.000672
  },
  {
    "individual_request_tag": "app-ishaan-local",
    "log_count": 4,
    "total_spend": 0.000448
  },
  {
    "individual_request_tag": "app-ishaan-prod",
    "log_count": 2,
    "total_spend": 0.000224
  }
]