Build a Document Q&A Bot with OpenAI Assistants API

Creating a question‑answering system that can read your own documents and give accurate, sourced answers used to require stitching together several pieces: a file parser, a chunking strategy, an embedding model, a vector database, and a retrieval pipeline. OpenAI’s Assistants API removes much of that plumbing by handling document ingestion, embedding, and vector search for you. In this guide we’ll walk through building a functional doc‑qa bot with the Assistants API, discuss pricing and limits, and share best practices to keep your bot reliable and cost‑effective.

Why the Assistants API simplifies document Q&A

The Assistants API bundles three capabilities that traditionally required separate services:

1. File handling – You upload PDFs, DOCX, TXT, or other supported files once; the API extracts text, creates embeddings, and stores them in a hosted vector store.
2. Retrieval tool – When a user asks a question, the assistant automatically runs a similarity search over the stored vectors and returns the most relevant snippets as context.
3. Thread management – Conversations are stored as threads, so the assistant retains chat history without you needing to manage a separate message store.

Because OpenAI takes care of the heavy lifting, you can focus on designing the assistant’s instructions, shaping the user experience, and integrating the bot into your application. The trade‑off is that you cede some control over where the data lives and how it’s indexed, and you incur storage costs for the vector store.

Prerequisites

Before you start coding, make sure you have:

• An OpenAI account with API access (you’ll need a secret key).
• Python 3.8+ (or Node.js if you prefer a JavaScript stack).
• The official OpenAI SDK (pip install openai).
• A folder of documents you want the bot to answer questions about (PDF, DOCX, TXT, MD, CSV, etc.).

Step 1: Create the assistant

The assistant is the core object that holds the model, instructions, and tools. You can create it once via the API or through the OpenAI Playground and reuse its ID across sessions.

from openai import OpenAI

client = OpenAI(api_key="sk‑your‑key‑here")

assistant = client.beta.assistants.create(
    name="Doc QA Assistant",
    instructions=(
        "You are a helpful assistant that answers questions using only the "
        "provided documents. If the answer is not in the files, say you "
        "cannot find that information. Keep responses concise and cite the "
        "source when possible."
    ),
    model="gpt-4o-mini",          # or gpt-4o, gpt-4-turbo, etc.
    tools=[{"type": "file_search"}],  # enables retrieval over uploaded files
)
assistant_id = assistant.id
print(f"Assistant ID: {assistant_id}")

Key points

• Instructions guide the model’s behavior. Tell it to rely solely on the uploaded files and to admit when it doesn’t know something.
• Model choice affects both quality and cost. gpt-4o-mini offers a good balance for most Q&A tasks; upgrade to gpt-4o if you need deeper reasoning.
• Tools – the file_search tool triggers the internal retrieval pipeline. You don’t need to manage embeddings yourself.

Step 2: Upload your documents

Each file you want the assistant to reference must be uploaded with the purpose "assistants". The API will automatically chunk the content, compute embeddings, and store them in a vector store tied to your assistant.

def upload_file(path: str) -> str:
    with open(path, "rb") as f:
        file_obj = client.files.create(file=f, purpose="assistants")
    return file_obj.id

# Example: upload all PDFs in a folder
file_ids = []
for fname in os.listdir("docs"):
    if fname.lower().endswith(".pdf"):
        fid = upload_file(os.path.join("docs", fname))
        file_ids.append(fid)
        print(f"Uploaded {fname} → {fid}")

After uploading, attach the files to the assistant:

for fid in file_ids:
    client.beta.assistants.files.create(assistant_id=assistant_id, file_id=fid)

What happens behind the scenes

• The file is processed: text is extracted (OCR is not applied unless you use a model that supports images; for plain PDFs the API extracts the text layer).
• The content is split into chunks, embedded with OpenAI’s embedding model, and stored in a vector database that OpenAI hosts.
• You are charged $0.10 per GB per day for the stored vectors (first GB free). The actual storage needed is often far smaller than the raw file size because embeddings are compact.

Step 3: Manage conversation threads

A thread represents a single conversation session. When a user sends a message, you add it to a thread, then ask the assistant to run on that thread. The assistant will use the file_search tool to retrieve relevant chunks and generate an answer.

def create_thread() -> str:
    thread = client.beta.threads.create()
    return thread.id

def add_message(thread_id: str, content: str):
    client.beta.threads.messages.create(
        thread_id=thread_id,
        role="user",
        content=content,
    )

Step 4: Run the assistant and poll for results

The Assistants API runs are asynchronous. After you create a run, you must poll its status until it reaches "completed" (or a terminal error state). A simple polling loop with a short sleep works fine for most applications.

def run_assistant(thread_id: str) -> str:
    run = client.beta.threads.runs.create(
        thread_id=thread_id,
        assistant_id=assistant_id,
    )

    # Poll until finished
    while True:
        run_status = client.beta.threads.runs.retrieve(
            thread_id=thread_id,
            run_id=run.id,
        )
        if run_status.status in ("completed", "failed", "cancelled", "expired"):
            break
        time.sleep(1)  # adjust based on latency tolerance

    if run_status.status != "completed":
        raise RuntimeError(f"Run ended with status {run_status.status}")

    # Retrieve the newest assistant messages
    messages = client.beta.threads.messages.list(
        thread_id=thread_id,
        order="asc",
        after=None,  # get all messages; you can filter by lastMessageId if you want only new ones
    )
    # The last message is typically the assistant’s reply
    for msg in reversed(messages.data):
        if msg.role == "assistant":
            for block in msg.content:
                if block.type == "text":
                    return block.text.value
    return "No answer returned."

Putting it together – a minimal chat loop

def chat():
    thread_id = create_thread()
    print("Doc QA Bot ready. Type 'exit' to quit.")
    while True:
        user_input = input("\nYou: ")
        if user_input.lower() in ("exit", "quit"):
            break
        add_message(thread_id, user_input)
        answer = run_assistant(thread_id)
        print(f"Assistant: {answer}")

if __name__ == "__main__":
    chat()

This loop creates a persistent thread so the assistant remembers earlier turns within the same session. If you need to support multiple users simultaneously, generate a new thread per user (or per session) and store the thread ID in your application’s session store.

Cost considerations

Tips to keep costs low

1. Select the smallest capable model – gpt-4o-mini often answers factual questions adequately.
2. Limit conversation length – Trim old messages or start a new thread after a few exchanges to reduce input token count.
3. Pre‑filter files – If you know only a subset of documents is relevant for a given topic, upload only those files for that assistant.
4. Cache frequent queries – If your users ask repetitive questions, store the answer in your own cache to avoid unnecessary runs.

Limitations and when to consider a manual RAG pipeline

While the Assistants API is convenient, it isn’t a universal solution.

If any of the above constraints are deal‑breakers for your project, a traditional RAG architecture (document loader → chunker → embedding model → vector store → retriever → LLM) offers full control at the cost of more engineering effort.

Best practices for a reliable doc‑qa bot

1. Craft precise instructions

• Explicitly forbid hallucination: “Answer only using the provided files. If the answer is not present, say you don’t know.”
• Ask for citations when possible: “Include the file name or a short quote to support your answer.”

1. Validate file ingestion

• After uploading, run a quick test query that you know appears in the document and verify the assistant returns a correct answer.
• Check the file_search tool’s usage in the run steps (visible in the run object) to confirm retrieval happened.

1. Handle edge cases gracefully

• If the assistant returns “I couldn’t find that information,” present a friendly fallback or suggest rephrasing the question.
• Catch API errors (rate limits, invalid file types) and present user‑friendly messages.

1. Monitor usage

• Enable OpenAI usage tracking or log token counts and file search calls to stay within budget.
• Set up alerts for sudden spikes, which could indicate a runaway loop or abusive traffic.

1. Secure your API key

• Never expose the secret key in client‑side code. Use a backend service (e.g., Azure Functions, AWS Lambda, or a simple FastAPI endpoint) that holds the key and exposes only a thin chat endpoint to the front end.

1. Test with realistic data

• Use a mixture of document types (PDFs with tables, scanned pages requiring OCR, markdown, code) to ensure the assistant’s built‑in extraction works for your content.
• If OCR is needed, consider preprocessing files with Azure Form Recognizer or Tesseract and uploading the resulting text instead of relying on the API’s native extraction.

Example: Adding citations to the answer

The assistant’s message objects contain annotations when the file_search tool is used. You can extract those to show which file(s) contributed to the answer.

def get_answer_with_citations(thread_id: str) -> str:
    run = client.beta.threads.runs.create(
        thread_id=thread_id,
        assistant_id=assistant_id,
    )
    while True:
        run_status = client.beta.threads.runs.retrieve(thread_id=thread_id, run_id=run.id)
        if run_status.status == "completed":
            break
        time.sleep(1)

    messages = client.beta.threads.messages.list(thread_id=thread_id, order="asc")
    for msg in reversed(messages.data):
        if msg.role == "assistant":
            text_parts = []
            citations = []
            for block in msg.content:
                if block.type == "text":
                    text_parts.append(block.text.value)
                    # annotations may contain file_citation objects
                    if hasattr(block.text, "annotations"):
                        for ann in block.text.annotations:
                            if ann.type == "file_citation":
                                file_id = ann.file_citation.file_id
                                # retrieve file metadata to show name
                                file_obj = client.files.retrieve(file_id)
                                citations.append(file_obj.filename)
            answer = " ".join(text_parts)
            if citations:
                answer += f"\n\nSources: {', '.join(set(citations))}"
            return answer
    return "No answer."

This approach gives users transparency about where the information came from, boosting trust in the bot.

Alternative: Manual RAG with LangChain (quick comparison)

If you decide the Assistants API doesn’t fit your needs, here’s a condensed version of a manual RAG pipeline using LangChain and ChromaDB:

from langchain_community.document_loaders import PyPDFLoader
from langchain.text_splitter import RecursiveCharacterTextSplitter
from langchain_community.vectorstores import Chroma
from langchain_openai import OpenAIEmbeddings, ChatOpenAI
from langchain.chains import RetrievalQA

# 1. Load & split
loader = PyPDFLoader("docs/manual.pdf")
pages = loader.load()
splitter = RecursiveCharacterTextSplitter(chunk_size=1000, chunk_overlap=200)
chunks = splitter.split_documents(pages)

# 2. Embed & store
vectordb = Chroma.from_documents(chunks, OpenAIEmbeddings(model="text-embedding-3-small"))

# 3. QA chain
llm = ChatOpenAI(model="gpt-4o-mini", temperature=0)
qa = RetrievalQA.from_chain_type(llm=llm, retriever=vectordb.as_retriever(), return_source_documents=True)

# 4. Ask
result = qa.invoke({"query": "What is the warranty period?"})
print(result["result"])
print("Sources:", [doc.metadata for doc in result["source_documents"]])

When to choose this route

• You need custom chunking (e.g., splitting by sections rather than fixed tokens).
• You want to control the embedding model (perhaps a domain‑specific fine‑tuned embedder).
• You require air‑gapped or on‑premises storage for compliance.
• You wish to inspect the retrieved chunks directly for debugging or UI purposes.

If none of those apply, the Assistants API offers a faster path to a working bot with less code.

Wrap‑up

Building a document‑centric Q&A bot has become markedly simpler with OpenAI’s Assistants API. By uploading files once, letting the platform handle embeddings and vector search, and managing conversation threads, you can focus on the user experience and the specific knowledge you want to surface. Keep an eye on costs, respect the API’s limits, and follow the best practices outlined here to deliver a reliable, helpful assistant that truly leverages your own documentation.

Whether you’re powering an internal help desk, enabling customers to self‑serve product docs, or creating a research companion for your team, the pattern described above will get you up and running quickly—while still giving you room to grow into a more customized solution if your needs evolve. Happy building!