HeatWave Release Notes
This section describes the Vector Store functionality available in HeatWave.
HeatWave vector store is a relational database that lets you load unstructured data to HeatWave Lakehouse. It automatically parses unstructured data formats, which include PDF (including scanned PDF files), PPT, TXT, HTML, and DOC file formats, from Object Storage. Then, it segments the parsed data, creates vector embeddings, and stores them for HeatWave GenAI to perform semantic searches.
HeatWave vector store uses the native
VECTOR data type to store
unstructured data in a multi-dimensional space. Each point in
a vector store represents the vector embedding of the
corresponding data. Semantically similar data is placed closer
in the vector space.
The large language models (LLMs) available in HeatWave GenAI are trained on publicly available data. Therefore, the responses generated by these LLMs are based on publicly available information. To generate content relevant to your proprietary data, you must store your proprietary enterprise data, which has been converted to vector embeddings, in a vector store. This enables the in-database retrieval-augmented generation (RAG) system to perform a semantic search in the proprietary data stored in the vector stores to find appropriate content, which is then fed to the LLM for generating more accurate and relevant responses.
To create vector embeddings, HeatWave GenAI uses in-database embedding models, which are encoders that converts sequence of words and sentences from documents into numerical representations. These numerical values are stored as vector embeddings in the vector store and capture the semantics of the data and relationships to other data.
A vector distance function measures the similarity between vectors by calculating the mathematical distance between two multi-dimensional vectors.
HeatWave GenAI encodes your queries using the same embedding model that is used to encode the ingested data to create the vector store. It then uses the right distance function to find relevant content with similar semantic meaning from the vector store to perform RAG.
Optical Character Recognition (OCR) in HeatWave GenAI lets you extract and encode text from images stored in unstructured documents. The text extracted from images is converted into vector embeddings and stored in a vector store the same way regular text in unstructured documents is encoded and stored in a vector store.
OCR is supported in MySQL 9.1.0 and later versions.
HeatWave GenAI supports OCR in the following unstructured data formats: PDF (including scanned PDF files), DOC, DOCX, PPT, and PPTX. However, HeatWave GenAI doesn't support OCR in TXT and HTML files. Images stored in TXT and HTML files are ignored while ingesting the files.
OCR in HeatWave GenAI also has the following limitations:
HeatWave GenAI might not be able to extract and process the text from images with 100% accuracy. However, if there are minor character recognition errors, the overall meaning of the text is still preserved.
In some cases, text-like figures in images might incorrectly be treated as regular text.
HeatWave GenAI doesn't support OCR for Scalable Vector Graphic (SVG) images in PDF files.
OCR is not supported in Asynchronous Load. To enable OCR, you must ingest files using Auto Parallel Load.