We’ve spoken to many developers who are feeling empowered by the ability of large-language models like OpenAI’s GPT-3 to parse input and deliver natural language responses. Many of them want to develop interfaces similar to ChatGPT for their users to be able to query documents of various types — including HTML, JSON, Plain Text. This is a powerful application of LLMs that lets teams of all types kinds gain insights into large volumes of text, such as recruiting call transcripts, request tickets from customers, and medical records.
Developers are finding, however, that this is not an easy problem to tackle in a robust manner. Often, because they are trained on extremely large, generalized datasets, LLMs will hallucinate, making up responses to queries if the true response is not present in the input document. Moreover, developers are passing large documents into GPT-3 in the form of a prompt, which quickly eats up the input token limit established by OpenAI and others. Additionally, developers see latency issues as prompts get larger, preventing them from delivering an optimal experience to their users.
At Meru, we specialize in building infrastructure around LLMs that makes it possible to deploy them in a production environment where accuracy, latency, and experience are of the utmost importance. Today, we are releasing a new API for dense data retrieval.
When a client submits a document to our API, we index the document and store the index on our servers. Clients can then query the index via an LLM, such as GPT-3 and obtain a response. This approach is more robust than passing the document as a part of the prompt and highly effective at reducing hallucinations. Moreover, because you are querying the index, you don’t have to waste tokens on passing the document in as a prompt, and can therefore build much larger and complex queries. Documents larger than 4096 tokens can also be queried effectively.
We’re excited to launch our Dense Data Retrieval API and hope it is useful. For more information on how to integrate the API into your application, please view our docs.
As always, feel free to get in touch at rohan@usemeru.com if you have any questions!
Back