29----txtai 索引剖析

柔情只为你懂 2023-09-27 19:56 9阅读 0赞

本文检查了 txtai 嵌入索引的文件系统并给出了结构的概述。

安装依赖  
安装txtai和所有依赖项。  
pip install txtai  
创建索引  
让我们首先创建一个索引来检查。我们将使用经典的 txtai 示例。  
from txtai.embeddings import Embeddings

data = \[“US tops 5 million confirmed virus cases”,  
“Canada’s last fully intact ice shelf has suddenly collapsed, forming a Manhattan-sized iceberg”,  
“Beijing mobilises invasion craft along coast as Taiwan tensions escalate”,  
“The National Park Service warns against sacrificing slower friends in a bear attack”,  
“Maine man wins $1M from $25 lottery ticket”,  
“Make huge profits without work, earn up to $100,000 a day”\]

## Create embeddings index with content enabled. The default behavior is to only store indexed vectors. ##

embeddings = Embeddings(\{“path”: “sentence-transformers/nli-mpnet-base-v2”, “content”: True, “objects”: True\})

## Create an index for the list of text ##

embeddings.index(\[(uid, text, None) for uid, text in enumerate(data)\])

## Run a search ##

embeddings.search(“feel good story”, 1)  
\[\{‘id’: ‘4’,  
‘score’: 0.08329004049301147,  
‘text’: ‘Maine man wins $1M from $25 lottery ticket’\}\]  
打印索引信息  
嵌入索引有一种info方法可以打印有关索引的元数据。这可用于查看索引的构建时间、使用的设置以及上次更新的时间。

## Print metadata ##

embeddings.info()  
\{  
“backend”: “faiss”,  
“build”: \{  
“create”: “2022-03-02T15:18:41Z”,  
“python”: “3.7.12”,  
“settings”: \{  
“components”: “IDMap,Flat”  
\},  
“system”: “Linux (x86\_64)”,  
“txtai”: “4.3.0”  
\},  
“content”: “sqlite”,  
“dimensions”: 768,  
“objects”: true,  
“offset”: 6,  
“path”: “sentence-transformers/nli-mpnet-base-v2”,  
“update”: “2022-03-02T15:18:41Z”  
\}  
保存索引并查看文件结构  
接下来让我们保存索引并查看文件结构。此部分打印每个文件，并运行命令以显示

## Save the index ##

embeddings.save(“index”)

## Show basic details about index files ##

for f in \[“confi