How to Integrate ChatGPT Into an Existing SaaS Product

Adding an LLM capability to an existing SaaS product is usually a 2–8 week engineering effort depending on which pattern you're implementing and how much of your data needs to be involved. The hard parts are not the OpenAI API calls — those are straightforward. The hard parts are prompt management, cost controls, error handling, and making sure you're not leaking user data in ways that cause security or compliance problems.

This guide covers the three integration patterns you'll actually use, what each costs to operate, and the implementation decisions that matter.

Three integration patterns

Pattern 1: Inline response generation

What it is: User submits a form or types a query. Your application sends that input to the OpenAI API and streams the response back in real time. The response appears character-by-character as it generates.

Example use cases:

• Customer-facing chat or Q&A widget
• In-app writing assistant ("rewrite this email in a professional tone")
• Live content generation (product descriptions, summaries)
• Conversational feature inside an existing workflow

How it works architecturally:

1. User input reaches your backend API endpoint
2. Backend sanitizes input, constructs the prompt with appropriate system context
3. Backend makes a streaming API call to OpenAI using stream: true
4. Backend forwards the stream to the frontend using Server-Sent Events (SSE) or WebSockets
5. Frontend renders the stream token-by-token

Implementation time: 1–2 weeks for a solid production implementation. A prototype takes a day; the remaining time is SSE infrastructure, error handling, rate limiting, and testing edge cases.

Operational cost: For GPT-5 at $2.50/1M input tokens and $10/1M output tokens:

• A typical conversational exchange (500 tokens in, 300 tokens out) costs ~$0.004
• 1,000 conversations/day = ~$4/day = ~$120/month
• At 10,000 conversations/day with heavier context: $400–$1,200/month

GPT-5 mini reduces this by ~95% — appropriate for simpler tasks.

When to use it: The task requires real-time, interactive generation. Latency matters — users are waiting for the response.

Pattern 2: Async background tasks

What it is: User triggers an action that queues an LLM task. The task runs asynchronously. When complete, results are available (via notification, email, or UI update) without the user waiting.

Example use cases:

• "Analyze these 50 support tickets and summarize the themes"
• "Generate a weekly report from my data"
• "Process this document and extract structured fields"
• Post-processing after a user upload (generate tags, summaries, or metadata)

How it works architecturally:

1. User triggers action (upload, button click, scheduled job)
2. Your application enqueues a job (Redis/BullMQ, AWS SQS, Celery, etc.)
3. Worker process picks up the job, calls OpenAI API (non-streaming)
4. Results stored in your database
5. User notified (email, in-app notification, or they check back)

Implementation time: 2–4 weeks, primarily because you're implementing a job queue if you don't have one, handling failures and retries, and surfacing results appropriately in the UI.

Operational cost: Same token costs as inline generation, but you can optimize more aggressively since latency isn't as constrained. Batch similar operations. Use longer context windows to reduce round-trips. Cheaper models are more viable here.

Consider OpenAI's Batch API for truly async workloads — it offers a 50% cost reduction with a 24-hour completion window. At scale, this is a significant saving.

When to use it: The task is long-running, expensive, or doesn't require immediate output. The user experience works as "request → notification" rather than "request → wait → see result."

Pattern 3: RAG-powered feature

What it is: User queries are answered using a combination of retrieval from your application's data and LLM-generated synthesis. The model's response is grounded in your specific content — not just general knowledge.

Example use cases:

• "Search our knowledge base and answer this customer's question"
• "Find relevant policies and summarize what applies to my situation"
• "Answer this question based on this user's account history"
• Internal search that surfaces contextually relevant documents

How it works architecturally:

1. User query received
2. Query embedded using an embedding model (text-embedding-3-small, ~$0.02/1M tokens)
3. Vector similarity search against your document store (pgvector, Pinecone, Qdrant)
4. Top-k retrieved chunks assembled into context
5. Query + context sent to LLM for synthesis
6. Response returned (streaming or not)

Implementation time: 4–8 weeks for a production-quality implementation including document ingestion pipeline, embedding management, vector database setup, retrieval tuning, and eval infrastructure.

Operational cost: Adds embedding costs on top of generation costs, plus vector database hosting ($70–$300/month for managed Pinecone; $20–$60/month for a self-hosted Qdrant or pgvector instance on a small VPS).

When to use it: Your users need answers grounded in your specific content. General LLM knowledge isn't sufficient. See our RAG development services for a more detailed breakdown of this pattern.

API key management: what not to do

Never expose your OpenAI API key in client-side code. This gets re-stated repeatedly because it keeps happening. A key in your JavaScript bundle, in a mobile app binary, or in a localStorage value is a key anyone can extract and use. The financial exposure is real — a leaked key used by someone scraping at scale can generate thousands of dollars in charges within hours.

What to do instead:

• API calls to OpenAI always originate from your backend server
• The client calls your own API endpoint (e.g., POST /api/generate)
• Your backend authenticates the user, checks rate limits, constructs the prompt, and makes the OpenAI call
• The client never sees the API key

Use environment variables: OPENAI_API_KEY in a .env file that is never committed to version control. In production, inject it via your deployment platform's secrets management (Vercel environment variables, AWS Secrets Manager, Railway environment variables, etc.).

Rotate keys periodically and immediately if you suspect exposure. OpenAI allows multiple keys per project — use separate keys for development and production so you can rotate one without affecting the other.

Streaming responses with Server-Sent Events

For inline response generation, streaming dramatically improves perceived performance. A response that takes 8 seconds to generate feels fast when tokens start appearing in 200ms. The same response delivered as a single JSON payload at 8 seconds feels slow.

Backend implementation (Node.js/TypeScript):

import OpenAI from 'openai';

const client = new OpenAI({ apiKey: process.env.OPENAI_API_KEY });

export async function POST(req: Request) {
  const { message } = await req.json();
  
  const stream = await client.chat.completions.create({
    model: 'gpt-5',
    messages: [{ role: 'user', content: message }],
    stream: true,
  });

  const encoder = new TextEncoder();
  const readable = new ReadableStream({
    async start(controller) {
      for await (const chunk of stream) {
        const text = chunk.choices[0]?.delta?.content ?? '';
        if (text) {
          controller.enqueue(encoder.encode(`data: ${JSON.stringify({ text })}\n\n`));
        }
      }
      controller.enqueue(encoder.encode('data: [DONE]\n\n'));
      controller.close();
    },
  });

  return new Response(readable, {
    headers: {
      'Content-Type': 'text/event-stream',
      'Cache-Control': 'no-cache',
      'Connection': 'keep-alive',
    },
  });
}

Frontend consumption (React):

async function streamResponse(message: string, onChunk: (text: string) => void) {
  const response = await fetch('/api/generate', {
    method: 'POST',
    headers: { 'Content-Type': 'application/json' },
    body: JSON.stringify({ message }),
  });

  const reader = response.body!.getReader();
  const decoder = new TextDecoder();

  while (true) {
    const { done, value } = await reader.read();
    if (done) break;
    
    const lines = decoder.decode(value).split('\n');
    for (const line of lines) {
      if (line.startsWith('data: ') && line !== 'data: [DONE]') {
        const { text } = JSON.parse(line.slice(6));
        onChunk(text);
      }
    }
  }
}

Error handling and fallbacks

OpenAI's API has real failure modes: rate limit errors (429), server errors (500, 503), and timeout errors. Production implementations need to handle all three.

Rate limits: The default tier allows a fixed requests-per-minute quota for GPT-5. If your feature might hit this, implement exponential backoff with jitter on 429 responses. For async tasks, retry with a delay. For streaming, show the user a "try again" prompt rather than failing silently.

Fallback strategy for user-facing features: Have a non-LLM fallback that still provides value. A search feature that falls back to keyword search when the LLM is unavailable is better than a feature that shows an error. A summarization feature that returns "summary unavailable, showing full text" is better than a blank state.

Timeouts: Set explicit timeouts on your API calls (30 seconds is reasonable for most generation tasks; 60 seconds for long outputs). Don't leave requests open indefinitely — it creates resource leaks in your backend.

Circuit breaker pattern: If OpenAI returns errors for 10 consecutive requests, stop sending requests for 30 seconds. This prevents your application from hammering a degraded service and piling up queued requests.

Cost controls

Without per-user rate limiting, a single aggressive user (or a bot) can generate significant API costs before you notice.

Per-user rate limiting: Track API calls per user per hour or per day at your application layer. Redis is the standard tool for this. For a free tier, 20 generations/day is a reasonable limit. Paid tiers can get more.

Token limits: Set max_tokens on every API call. Don't allow open-ended generation. If you're generating summaries, 300 tokens is usually enough. If you're generating long-form content, set a ceiling and communicate it to users.

Monitor costs by feature: Tag your OpenAI API calls with metadata so you can see which features are generating costs. OpenAI's usage dashboard doesn't do this natively — track it in your own database by logging token counts per call per feature.

Set a spending alert: OpenAI allows you to configure usage alerts (email notification when monthly spend hits a threshold) and hard limits (API calls stop above a monthly ceiling). Configure both. The default is no limit, which means a runaway feature can accumulate significant charges without warning.

What not to do with user data

Don't put raw user data into prompts without sanitization. If a user submits text that becomes part of your prompt, that text can include prompt injection attacks — instructions designed to override your system prompt and cause the model to behave unexpectedly. Sanitize inputs that become part of system-level context. For user message content that goes in the user role, the risk is lower but still worth understanding.

Don't log full prompt content if it contains sensitive user data. If your prompts include account information, user preferences, or private content, logging full prompt text to a centralized logging service creates a secondary data exposure risk. Log prompt structure (template ID, parameter names) and token counts, not full content, unless you've specifically designed the logging system for sensitive content.

Understand OpenAI's data usage policy. OpenAI does not use API inputs for training by default (opt-out is the default for API customers as of March 2023). If your organization has stricter requirements, look at Azure OpenAI (which has additional contractual protections) or self-hosted models.

Don't persist conversation history without a data retention policy. Storing every user conversation creates growing data liabilities. Define a retention period (90 days, 1 year) and implement deletion. If any of those conversations contain PII, you need to be able to honor deletion requests.

The bottom line

Integrating ChatGPT or any OpenAI model into an existing SaaS product is genuinely achievable in a few weeks for the inline and async patterns. The RAG pattern takes longer but delivers more differentiated value.

The technical decisions that matter most are: keep API keys on the server, stream inline responses for user experience, implement per-user rate limits immediately (before launch, not after a cost spike), and build error handling that degrades gracefully.

The integration itself is not the hard part. The hard part is making the feature compelling enough that users actually rely on it — which requires good prompts, an appropriate use case, and iteration based on real usage patterns.

If you're evaluating which integration pattern fits your product or want to scope the effort more precisely, our AI agent development services covers how we approach LLM feature implementation.