AI for creating and maintaining technical documentation

Technical documentation in a SaaS company ages faster than code. A runbook written for v2 deployment describes procedures that are already different in v4. The May onboarding wiki mentions endpoints whose parameters changed in July. A new engineer spends their first days verifying whether what they’re reading is still true.

At Cashcrown, we’re researching how a RAG-based assistant can shorten the distance between code and documentation without eliminating the human role as the final verifier. This article describes the pattern of drafting from code, the documentation drift problem, and the boundary no automation should cross.

The classic approach to automatic documentation generation relies on instructing the language model with prompts like “describe this endpoint.” The model responds based on general knowledge encoded in its weights. The result is grammatically correct but factually uncertain—because the model may have never seen your API.

The pattern we’re researching is inverted: the assistant receives a question or drafting task, retrieval extracts relevant fragments from indexed code and existing documentation, and the LLM formulates a document draft exclusively based on those. Every sentence is linked to a specific source fragment.

Practical implications for indexing:

A guardrail in the generation layer enforces one rule: if retrieval doesn’t return a fragment with similarity above the threshold (approximately 0.65 for semantic search), the assistant drafts nothing. Instead, it responds directly: “no coverage found in the indexed source version.” A parameter description not present in OpenAPI won’t make it into the draft.

Read more about code indexing strategies in the article on RAG for code and technical documentation.

Generating the first version of documentation is a solvable problem. Keeping it up to date after six months of intensive development is a problem most teams solve by... not updating documents.

Drift has a concrete mechanism: an endpoint’s parameter changes in code review, and no one updates the wiki. A month later, a new developer wastes two hours debugging an error that would look like a configuration issue in up-to-date documentation. The scale of drift grows linearly with the number of microservices and inversely with the discipline of updating.

Three layers we apply to address this problem:

Incremental reindexing after commit. A GitLab or GitHub webhook (post-receive hook) triggers reindexing of changed files. git diff --name-only HEAD~1 HEAD provides the list. The assistant knows which file was modified after the last draft and can flag the related documentation section as requiring review.

Freshness metadata in every response. Every chunk in the index stores last_commit_sha and indexing date. The assistant reveals them in the citation: “source: src/payments/refund.py, line 43, commit b2f1a09, indexed 3 hours ago.” The developer sees whether they can trust the response before applying it.

TTL for documents without webhooks. Runbooks in Confluence or Notion don’t send edit events. We set a reindexing TTL: 24 hours for actively changing documentation, 7–14 days for stable architectural specifications. After TTL expires, the assistant adds a warning about potential staleness.

The human decides the TTL threshold for each document category. This isn’t a technical configuration—it’s a risk decision made by the engineer or tech lead responsible for the area.

The assistant doesn’t independently decide a documentation section is outdated and overwrites it. That would violate the human-oversight principle. Drift detection works differently: the assistant flags, the human evaluates.

The concrete drift detection pattern:

With every query about a documentation fragment, the assistant compares the wiki page’s last modification date with the last commit date in the associated code file. If the commit is newer by more than a defined threshold (e.g., 48 hours), the assistant appends a note to the response: “code source changed 3 days after the last update to this documentation section. Verification recommended before use.”

For a more advanced approach, an agent can generate a periodic report: a list of documentation sections whose associated code files changed since the last review, sorted by number of changes. The tech lead reviews the list every sprint and delegates updates to specific individuals.

The hard boundary: the assistant never publishes a documentation change without human approval. It proposes edits, drafts changes for review, and shows the diff between what’s in the wiki and what the current code implies. The decision to accept the draft belongs to the documentation owner.

Read more about the assistant workflow pattern for long documents in the article on summarizing long documents.

Hallucination in technical documentation carries higher costs than in most other applications. A description of a nonexistent API parameter will lead a developer to an integration error. A runbook with a fictional command will cause an incident during an overnight operation.

That’s why every documentation fragment generated by the assistant must include a citation trace: where it came from, which commit, when it was indexed. The internal format we use:

[draft from source: src/payments/refund.ts, lines 67-89, commit a3f9c12, indexed 2026-06-18 08:41]

This trace appears in the documentation review tool, not in the published article. The reviewer sees the origin of every sentence and can jump to the source file in the repository with one click. If the code has changed since then, they’ll see it before approval.

Guardrails enforcing structured output should require citation in every draft. If the model attempts to generate a parameter description without a linked retrieval fragment, generation is rejected. An empty draft is better than one with an invented parameter.

The pattern of a company-wide RAG-based assistant is described in the article company GPT based on knowledge.

With every assistant deployment for documentation creation, we repeat the same conversation with the team: what the assistant does and what belongs to the engineer.

The assistant drafts based on code. The engineer evaluates whether the draft reflects architectural intent, not just code mechanics. These two things are different.

The assistant flags sections whose associated code has changed. The engineer evaluates whether the code change actually affects the documented behavior’s semantics. Often, it doesn’t—a variable name refactor doesn’t change the API.

The assistant generates a runbook skeleton from code procedures. The engineer adds operational knowledge not present in any file: what to check first at 3 AM, when to call the DBA, which errors NOT to fix without consultation. This knowledge lives in people’s heads, not in source files.

The boundary no automation crosses: the publication decision. No draft reaches official documentation without approval from the person responsible for the area. This is an operational requirement, not just ethical. Best practices for preparing data for AI are described in the article how to prepare company data for AI.

It shouldn’t, at least not without a review step. The assistant can automatically generate a change draft and create a pull request for documentation linked to the code commit. But accepting that draft requires approval from the engineer responsible for the area. Automatically overwriting the wiki without review carries real risk: internal refactoring that doesn’t change public behavior can generate misleading documentation changes.

This is one of the most common problems with RAG over technical documentation. Retrieval returns several fragments, and the model may try to combine them, leading to conflicting descriptions. The pattern we recommend: before deploying the assistant, audit the knowledge base and designate a single source of truth for each endpoint. Mark duplicates as deprecated with a reference to the canonical source. The assistant indexes only the canonical source.

Two metrics make operational sense: the percentage of drafts accepted without changes by the reviewer (target: above 50 percent after the first month of calibration) and the time an engineer spends reviewing the draft (target: under 15 minutes for a typical section). A low acceptance rate signals a problem with index quality or the system prompt. Long review times signal drafts that are too large or insufficiently specific source citations.

This signals that the citation guardrail isn’t working correctly. A properly configured assistant shouldn’t generate a description without coverage in the index. If this happens, check two things: whether the guardrail verifies citation presence in every draft, and whether the retrieval similarity threshold is too low (allowing weak matches). A parameter description that doesn’t exist in the source is a symptom of missing guardrails, not model weakness.

It can draft onboarding document skeletons from code and existing procedures. In practice, onboarding requires knowledge not present in any file: team conventions, historical architectural decisions, fragile areas known only to “veteran” team members. The assistant reduces document preparation time by tens of percent, but the final version always requires input from an engineer experienced in the area.