Oaib
Use the OpenAI Batch tool to make async batch requests to the OpenAI API.
Install / Use
/learn @SpellcraftAI/OaibREADME
OpenAI Batcher (oaib)
A Python library for making rate-limited, async batch requests to the OpenAI API.
- Use
Batchto make batch requests as quickly as possible given TPM/RPM limits. - Use
Autoto automatically read your rate limits from OpenAI's response headers, and run the job as fast as possible.
This notebook is available at README.ipynb.
Usage
Set OpenAI API Key
import os
os.environ['OPENAI_API_KEY'] = input()
Using the Batch class
You can mix and match endpoints as needed for regular Batch, but it's not
recommended as the rate limits for different endpoints/models will differ. For
maximum efficiency, use Auto.
from oaib import Batch
# Use low rate limits for this example.
batch = Batch(rpm=100, tpm=1_000, workers=5)
# Creating a batch with 20 chat completions.
for i in range(20):
await batch.add(
"chat.completions.create",
model="gpt-3.5-turbo",
messages=[{"role": "user", "content": "say hello"}]
)
await batch.run()
Using the Auto class
Automatically use the given TPM/RPM rate limits provided by OpenAI API responses.
from oaib import Auto
# Automatically set rate limits.
batch = Auto(workers=8)
# Fetch 1,000 chat completions as quickly as possible, setting rate limits
# automatically from OpenAI's response headers.
for i in range(1000):
await batch.add(
"chat.completions.create",
model="gpt-3.5-turbo",
messages=[{"role": "user", "content": "say hello"}]
)
await batch.run()
Metadata and Index
You can add custom metadata to your observations with add(metadata={...}, and
set the index for the DataFrame with Batch(index=[...]) and
Auto(index=[...]).
from oaib import Batch
n = 5
batch = Batch(rpm=1000, tpm=10000, workers=5, index=["difficulty", "i"])
difficulties = ["easy", "medium", "hard"]
for difficulty in difficulties:
for i in range(n):
await batch.add(
"chat.completions.create",
