Prepared by ASI AI Committee
IndexStudio, an LLM-based indexing service, claims on its website that it allows users to “create professional book indexes in a fraction of the time” and that its AI1 generates “comprehensive entries.” It is marketed to professional indexers, academic authors, and corporate editors for $29/book, $49/month, or custom enterprise pricing.
Our tests have shown LLM-generated book indexes to be seriously deficient.2 In particular, the systems we’ve tested severely underindex books, omitting 60%–80% of the main headings (access points) of a normal index. They fail to use cross-references effectively to guide the user to related topics and subtopics. They can also contain hallucinated entries and page references.
To test the accuracy of IndexStudio, we took a chapter from Daniel S. Halacy’s Computers—The Machines We Think With, published in 1962 and now in the public domain. The chapter (chapter 2, on computer history) was first indexed by a professional indexer (one of the AI Committee members), then uploaded to IndexStudio to generate an LLM-generated index for comparison.
The results are below. In line with other LLMs, IndexStudio’s LLM-generated index did not create comprehensive entries, nor did it create a professional book index. It severely underindexed the chapter we provided, including only 29 main headings in comparison to the professionally created index’s 122 main headings—24% as many as the professionally created index. It failed to index over half the names of people; it also failed to index the four companies discussed in the chapter and did not include any of the 23 titles of works present. It included no cross-references. There were various other less pervasive issues, such as failure to collapse subheadings and page spans appropriately, altering the capitalization of a name, indexing of a passing mention, and a typo.
Based on these results, IndexStudio does not appear to provide indexes that will serve the reader or author well.
Professionally created index
|
IndexStudio index
Other observations:
|
Click to view side-by-side comparison of professionally created entries vs IndexStudio entries.
1. The AI used by the website appears to be Gemini 2.5 Pro, created by Google DeepMind.↩
2. Bartmess, Elizabeth, “AI: Where You Can Use It and Where You Shouldn’t,” May 31, 2025, ISC/SCI Conference 2025 “Location! Location! Location.”↩