Answers that cite their source

Turning a business's own PDFs into a support chatbot it can embed on its website, where every reply points to the page it relied on.

RAGcat · CAB432 Assessment 3·Engineering notes·4 min read

Many businesses answer the same questions again and again. The refund policy is already on the website, yet customers still ask support because they can't find it. RAGcat lets the business upload those documents, review what was extracted, and add a short script to its homepage to get a chatbot that answers from them. Every answer cites the document and page it came from. I built it for CAB432 Cloud Computing at QUT, and these notes are condensed from my submission report. The live demo is at ragcat.jayliu.dev.

An answer only counts when it cites at least one passage. Otherwise the visitor gets the owner's contact details, not a guess.

Two services, scaled on requests

The application runs as two ECS Fargate services behind an Application Load Balancer. The management API handles accounts, assistants, documents and uploads, and the QA service answers questions. Question traffic is the part expected to spike, so giving it its own service keeps a burst of visitors from slowing down the owners' dashboard. CloudFront serves the React app and the widget from a private S3 bucket through Origin Access Control, and the ECS tasks sit in private subnets, so the ALB is the only way in.

Both services are I/O bound. A question mostly waits on Bedrock, so a QA task can be saturated with in-flight model calls while its CPU stays low. Scaling therefore tracks requests per target instead of CPU. The API runs one to two tasks at 300 requests per task per minute, and QA runs one to four at 60, because a question holds a task far longer than a CRUD call. Each task is 0.25 vCPU and 512 MB on ARM64, and the maximums cap both cost and Bedrock call volume.

Owners sign in through Cognito, using the app's own sign-in page rather than the hosted UI. Each service verifies tokens locally, ownership checks decide what an owner may change, and every retrieval query is scoped to a single assistant. Widget visitors are anonymous, so their access depends on the assistant being published, a website allowlist, a per-visitor rate limit and the owner's answer quota.

Document processing, off the request path

Parsing a PDF is slow and CPU heavy, so it never runs inside a request. The browser uploads straight to S3 with a presigned POST. An EventBridge rule matches only new objects under the uploads/ prefix, so the page previews the worker writes back don't trigger parsing again, and sends the event to an SQS queue that invokes the ingest Lambda. SQS delivers at least once, so the function first claims the document with a conditional write from awaiting_upload to parsing and ignores duplicates. It then extracts the text and tables, renders a preview of each page, and has a vision model describe the figures that carry information the text layer doesn't.

Those descriptions can hallucinate, so every document stops at a review step. The owner reads and edits the passages next to the original page, and nothing becomes searchable until they publish.

Publishing queues a reconcile job for the embed Lambda, which re-embeds only the passages whose hash has changed. Each passage gets a short context and entity tags from an LLM, then Titan Text Embeddings V2 turns it into a 1024-dimension vector in S3 Vectors. The new hash is recorded only after the vector is written, so a failed run resumes where it stopped. Concurrency is capped at five ingest and two embed functions, so bursts wait in the queue instead of piling onto Bedrock, and a message that fails three times moves to a dead-letter queue.

Answering from cache first

Widget requests are rate limited in Valkey to ten per IP per minute. Before anything is streamed, the QA service validates the question, checks that the assistant is published and checks the owner's quota, so those failures come back as ordinary HTTP errors instead of a stream that breaks halfway.

The answer cache key combines the model configuration, the assistant id, the assistant's content version and a hash of the normalised question. Cached answers are never deleted. Each publish bumps the content version, so new questions look up a fresh key and old entries expire after 24 hours. A hit returns in about 1 ms on the server, without touching Bedrock or S3 Vectors. On a miss, the question is embedded, with its vector cached for seven days, the five closest passages are retrieved, and Gemma on Bedrock streams the answer back as server-sent events.

The model has to cite each sentence as [n] or reply with a fixed NO_ANSWER marker. The service holds back the first characters of the stream until it knows the reply isn't that marker, so visitors never see it, and a reply without a citation falls back to the owner's contact details.

What I'd change

Exact-match caching looked like the obvious way to cut model calls, but it keys on the normalised question text, so the same question worded differently still misses, and entries that are rarely reused just take up memory. A synthetic test can't show how often real visitors actually repeat a question. Next I'd compare it with semantic caching, which reuses answers by embedding similarity, kept within the same assistant, published version and language. I'd measure incorrect reuse, latency, memory and total cost before deciding the extra complexity is worth it.

Two smaller gaps are on the list too. The embed workers run outside the VPC and can't reach Valkey to clear the cached configuration, so answers can stay up to 300 seconds stale after a publish. And each service runs a single task to save cost, so a failure interrupts service until ECS replaces it; production would run at least two across zones.