Interdisciplinary discoveries have a long history. The discovery of DNA’s structure combined physics (X-ray crystallography), chemistry, and biology. Penicillin emerged from microbiological observation, which pharmacology later developed. Each time, someone had to read enough from two different fields to notice that something fit together.
The problem is simple: the number of published works grows faster than any researcher’s ability to track them. PubMed alone adds over a million new records annually. In this context, AI as a preliminary search tool makes sense—provided we understand what it actually does and what it cannot do.
What is "hidden public knowledge"#
The term originates from Donald Swanson’s concept in the 1980s. Swanson demonstrated that literature on fish oil and literature on Raynaud’s disease were isolated from each other, despite sharing a common pathophysiology. No author had read research from the other side. The connection existed in the literature, but no one discovered it.
Swanson performed the synthesis manually, searching through indexes. Today, the same operation can be scaled using semantic search: instead of searching by keywords, the model compares the meaning of concepts encoded as embeddings. Two articles from different fields may describe the same mechanism using different words, and the similarity of their vector representations will reveal this.
It’s worth stating upfront what this means in practice: the model indicates statistical similarity in the representation space. It does not understand biology, chemistry, or physics. Assessing whether the identified similarity has substantive significance is the researcher’s responsibility.
How semantic search works in practice#
A typical pipeline for interdisciplinary discovery looks as follows:
- Abstracts or full texts of articles are converted into vectors by an embedding model. In environments requiring data privacy, we use local models like BGE-M3.
- Vectors are indexed in a vector database (e.g., Qdrant or pgvector).
- The researcher inputs a query or text fragment. The model converts it into a vector and searches for the nearest neighbors in the representation space.
- Results are reranked by relevance and delivered with source citations.
The RAG (Retrieval-Augmented Generation) system goes a step further: the language model reads the retrieved fragments and attempts to formulate an answer or identify a pattern. This is the stage most prone to hallucinations: the model may "attach" a false interpretation to a real citation. Therefore, every research system based on RAG should always provide sources, not just syntheses.
| Approach | What it does well | Where it fails | Human verification |
|---|---|---|---|
| Keyword search | Terminological precision | Different vocabulary across fields | Low complexity |
| Semantic search | Finding thematically close works despite different terminology | False similarity without substantive connection | Expert relevance assessment |
| RAG with LLM | Synthesis of conclusions from multiple sources at once | Hallucinations, incorrect citation attributes | Verification of each claim against the source |
| Knowledge graph | Structural relationships between entities | Requires manual curation or good NER | Verification of relationship accuracy |
Real-world applications and their limitations#
At Cashcrown, we observe several patterns in projects where clients use AI to work with literature:
Literature search during scoping review. The researcher provides a few articles central to the problem. The model searches for semantically similar works in the corpus and identifies candidates for exclusion. The preliminary review time may be reduced, depending on the field and index quality. The decision on which works to include in the review remains with the researcher.
Terminology mapping. Different fields describe the same phenomena using different words. A "signaling cascade" in biology and a "feedback loop" in systems theory may have similar vector representations. Do they describe the same mechanism? That’s a question for the expert, not the model.
Preliminary identification of research gaps. If two fields are semantically close but the number of articles connecting them is small, it’s a potential niche. Models can detect such asymmetries, but this is a clue, not a ready-made hypothesis.
Where it consistently fails. Models lack access to data from closed repositories, raw data, or unpublished results. Interdisciplinary knowledge hidden in laboratories, patents without full texts, or clinical data is out of reach. Additionally, models inherit the gaps in their training corpora: there is significantly more English-language literature than Polish- or Japanese-language, which systematically favors certain research traditions.
Human oversight: where humans must step in#
Human oversight in interdisciplinary discovery is not an option or an obstacle—it’s a prerequisite for results to have scientific value.
Here are the key points where human intervention is necessary:
Input corpus selection. The quality of results depends on what is in the index. The decision on which databases to index, which years and document types to include, is substantive. Searching for connections in a corpus that deliberately excludes certain research traditions or languages makes no sense.
Assessment of identified links. The model returns a list of similar articles or identifies a pattern. An expert in the field must assess whether the semantic similarity corresponds to a real substantive connection. There’s no shortcut here: domain knowledge is required.
Hypothesis formulation. AI can indicate that something is similar. It cannot formulate a testable hypothesis in the scientific sense because it lacks a causal model of reality. A hypothesis that can be experimentally tested emerges in the researcher’s mind.
Pre-publication verification. Every claim based on AI-generated synthesis must be verified against the primary source. Hallucinations in the context of scientific citations are a serious problem: models can generate convincingly sounding, non-existent article titles with real author names.
In our analytical agent deployments, we use a human-gate pattern: every action that could be used outside the verification context (sending a report, saving to a knowledge base) requires explicit confirmation. In a research context, the equivalent is requiring every AI result to be labeled as "candidate for verification," not as a "fact."
More on why this matters in the article about the role of humans in the loop.
Pitfalls and limitations that cannot be ignored#
Algorithmic bias. Embedding models trained primarily on English-language literature represent concepts from non-English research traditions poorly. The effect: connections between Arabic-, Chinese-, or Polish-language works will be searched less effectively. This is not a bug fixable with a single parameter—it’s a consequence of the training data structure.
False similarity. Two articles may be close in the embedding space because they use similar academic styles, not because they describe related phenomena. The problem intensifies when comparing reviews from different fields.
The problem of negative publications. Public knowledge is heavily affected by publication bias: positive results are published much more frequently. The model replicates this error: a connection indicated by AI may appear strong because negative experiments never made it into the index.
Intellectual property and data issues. Processing articles through external APIs may violate licenses of databases like Web of Science or Scopus. Local processing, without transmitting full texts externally, typically complies with licenses. More on the architecture of such solutions in the article about data governance for AI.
2026 outlook: what’s real, what’s hype#
Below is a summary of the current practical state for research teams:
| Application | State in 2026 | Who decides |
|---|---|---|
| Semantic search in large corpora | Mature, scalable tool | Researcher selects corpus and evaluates results |
| Automatic interdisciplinary connection suggestions | Works as a selection assistant, not an oracle | Researcher verifies each connection |
| Literature synthesis with citations | Useful, but with citation hallucination risk | Verification of each citation against the source |
| Autonomous generation of publication-ready hypotheses | Does not exist as a credible function | Researcher formulates hypothesis based on AI clues |
| Replacing domain experts | Beyond current systems’ capabilities | Expert remains essential at every stage |
AI-driven interdisciplinarity is a real change in work pace, not in the structure of the scientific process. A researcher who understands the limitations of models gains a time advantage. A researcher treating model results as ready-made discoveries risks serious errors. More in the article about scientists working with AI and AI as an autonomous scientist.
FAQ#
Can AI independently discover interdisciplinary connections without researcher involvement?#
The model can indicate statistical similarity between articles from different fields. It cannot assess whether this similarity has substantive significance. Every potential connection requires verification by someone who understands the context of both fields. Systems operating without human oversight may generate false discoveries with high confidence.
How to avoid the hallucination problem in literature synthesis?#
The most important safeguard is requiring the system to provide exact citations with sources, not just syntheses. Every claim in an AI-generated report should have a link or identifier to the source document. The researcher checks the cited fragments in the original before using the claims in their work. Details of this approach in the article about limiting AI hallucinations.
Which embedding models work best for Polish-language scientific literature?#
Multilingual models like BGE-M3 (1024 dimensions) or multilingual-e5 handle Polish text significantly better than models trained exclusively on English corpora. For mixed literature (Polish abstracts, English full texts), BGE-M3 offers a good compromise between representation quality and the ability to run locally without sending data to external APIs.
Does using AI for literature analysis violate copyright or database licenses?#
It depends on the processing model. Sending full texts to external APIs may violate licenses of databases restricting bulk processing. Local processing with embeddings generated on your own infrastructure typically complies with licenses. Before implementation, it’s worth checking the terms of the specific database and consulting with the legal department.
How to measure whether an interdisciplinary knowledge discovery system actually works?#
The basic benchmark is retrospective validation: would the system have indicated known historical connections, having access only to literature before their discovery? A complement is precision@k: what proportion of the top k highest-rated candidates does the expert consider substantively accurate? This number is more informative than a general vector similarity measure.