[Question]: Why are Japanese characters being saved as escaped Unicode?

[tmforai60]

Question Validation

• I have searched both the documentation and discord for an answer.

Question

Hello,
I create saved a docstore in the way below.
It has texts written in unicode with escape, not in Japanese character(or UTF-8) which I want.

How can I solve the problem?
Thank you for your help.

Code:
from langchain import text_splitter
from llama_index.indices.loading import load_index_from_storage
from llama_index import StorageContext
from llama_index.storage.docstore import SimpleDocumentStore
from llama_index.storage.index_store import SimpleIndexStore
from llama_index.vector_stores import SimpleVectorStore

from llama_index import ServiceContext
from llama_index.node_parser import SimpleNodeParser
from llama_index.embeddings.openai import OpenAIEmbedding
from llama_index import LLMPredictor
from llama_index.indices.prompt_helper import PromptHelper
from llama_index.logger.base import LlamaLogger
from llama_index.callbacks.base import CallbackManager

from llama_index import VectorStoreIndex
from llama_index import ResponseSynthesizer
from llama_index.optimization.optimizer import SentenceEmbeddingOptimizer
from llama_index import SimpleDirectoryReader
from llama_index.indices.vector_store.retrievers import VectorIndexRetriever
from llama_index.vector_stores.types import VectorStoreQueryMode
from llama_index.query_engine.retriever_query_engine import RetrieverQueryEngine
from llama_index import LLMPredictor

from llama_index import GPTListIndex
from llama_index.llm_predictor.chatgpt import ChatOpenAI

documents = SimpleDirectoryReader('data').load_data()

llm_predictor = ChatGPTLLMPredictor(
    llm=ChatOpenAI(
        temperature=0,
        model_name='gpt-3.5-turbo',
        streaming=True
        )
)

'''
storage_context = StorageContext.from_defaults(
    docstore=SimpleDocumentStore(),
    vector_store=SimpleVectorStore(),
    index_store=SimpleIndexStore()
)

from llama_index.langchain_helpers.text_splitter import TokenTextSplitter
from llama_index.constants import DEFAULT_CHUNK_OVERLAP, DEFAULT_CHUNK_SIZE
import tiktoken

text_splitter = TokenTextSplitter(separator=" ", chunk_size=DEFAULT_CHUNK_SIZE
    , chunk_overlap=DEFAULT_CHUNK_OVERLAP
    , tokenizer=tiktoken.get_encoding("cl100k_base").encode)

service_context = ServiceContext.from_defaults(
    node_parser=SimpleNodeParser(text_splitter=text_splitter),
    embed_model=OpenAIEmbedding(),
    llm_predictor=llm_predictor,
    prompt_helper=PromptHelper.from_llm_metadata(llm_predictor.get_llm_metadata()),
    llama_logger=LlamaLogger(),
    callback_manager=CallbackManager([])
)

index.set_index_id("test")
index.storage_context.persist("./storage")

docstore
{"docstore/metadata":...},` "docstore/data": {"...": {"data": {"text": "\u6c11\u4e8b\u88c1\u5224\u306b\u304a\u3044\u3066...", "doc_id": ...

[Disiok]

Why do you need to inspect the docstore? We don't do any special encoding when saving to disk.

[quarhodron]

Why do you need to inspect the docstore? We don't do any special encoding when saving to disk.

I have the same question because in my docstore Norwegian diacritic characters are also Unicode escaped. I didn't specify any parameter that could cause this. Docstore is UTF-8 encoded so I don't know why they are escaped?
It's mainly the question if it has any negative impact on the data retrieval process - so if unicode escapes are converted back to the right characters before analysis?
But I think it would be nice if we could have docstore in UTF-8 characters.

[bl-taira-m]

There is a way to avoid Unicode escaping, by giving the argument ensure_ascii=False to json.dump(). However, I believe that there would be no negative impacts as it is.

[dosubot[bot]]

Hi, @tmforai60! I'm Dosu, and I'm helping the LlamaIndex team manage their backlog. I wanted to let you know that we are marking this issue as stale.

From what I understand, the issue you raised is about Japanese characters being saved as escaped Unicode instead of UTF-8. There have been some discussions in the comments about inspecting the docstore and the impact of Unicode escapes on data retrieval. One user suggested using the argument ensure_ascii=False to avoid Unicode escaping.

Before we close this issue, we wanted to check with you if it is still relevant to the latest version of the LlamaIndex repository. If it is, please let the LlamaIndex team know by commenting on this issue. Otherwise, feel free to close the issue yourself, or it will be automatically closed in 7 days.

Thank you for your contribution to LlamaIndex!

[dosubot[bot]]

🤖

Hello,

Thank you for reaching out with your question. It seems like you're experiencing an issue with Japanese characters being saved as escaped Unicode in your document store.

Based on the information you've provided and the context from similar issues, it appears that the issue might be related to how the text encoding is handled during the data loading process. In LlamaIndex, the SimpleDirectoryReader and HTMLTagReader classes handle text encoding by specifying the encoding type when opening the file. They assume that the text files they are reading are encoded in UTF-8.

Here is the relevant code from the HTMLTagReader class:

with open(file, "r", encoding="utf-8") as html_file:
    soup = BeautifulSoup(html_file, "html.parser")

And from the SimpleDirectoryReader class:

def load_data(
        self, file: Path, extra_info: Optional[Dict] = None
    ) -> List[Document]:
        """Parse file into string."""
        with open(file, encoding="utf-8") as f:
            content = f.read()
        metadata = {"filename": file.name, "extension": file.suffix}
        if extra_info:
            metadata = {**metadata, **extra_info}

        return [Document(text=content, metadata=metadata)]

In both cases, the encoding is set to "utf-8" when opening a file for reading. If the file's content is not compatible with the provided encoding, a UnicodeDecodeError will be raised.

However, it's important to note that these classes are used for reading data from files, not for saving data to the document store. The issue you're experiencing might be related to how the data is saved to the document store, not how it's read from the files.

Unfortunately, without more information about how you're saving the data to the document store, it's difficult to provide a more specific solution. Could you please provide more details about this part of your code? Specifically, it would be helpful to know how you're creating the documents that you're saving to the document store, and how you're saving them.

Thank you for your cooperation.

Sources

• [Bug]: JSONReader messes with text encoding
• Chinese Support
• llama_index/readers/file/html_reader.py
• llama_index/readers/file/base.py
• llama_index/readers/file/flat_reader.py

---

This response is meant to be useful and save you time. It is not meant to be a precise solution, but rather a starting point for your own research.

Help me be more useful! Please leave a 👍 if this is helpful and 👎 if it is irrelevant.

If you want to continue the conversation, start your reply with @dosu-bot.

[plumqz]

Hello, I am facing the same issue for the vector_store utils (llama_index/core/vector_stores/utils.py) method node_to_metadata_dict. when adding content to Pinecone. It uses unicode typing rather than giving ensure_ascii = False, and makes the text content show as all unicode on Pinecone page, which is not easy to debug from the Pinecone page.
Shown the function with problem as below, the modification is just metadata["_node_content"] = json.dumps(node_dict, ensure_ascii=False).

Besides, I would also suggest to add the metadata key depending on the vector store object created. Now the following 3 keys are created regardless the vector store used, although they are clearly not shared by all the vector stores as commented. This is causing confusion for all the similar but different naming for doc_id.
metadata["document_id"] = node.ref_doc_id or "None" # for Chroma metadata["doc_id"] = node.ref_doc_id or "None" # for Pinecone, Qdrant, Redis metadata["ref_doc_id"] = node.ref_doc_id or "None" # for Weaviate

Modified method:
`def node_to_metadata_dict(
node: BaseNode,
remove_text: bool = False,
text_field: str = DEFAULT_TEXT_KEY,
flat_metadata: bool = False,
) -> Dict[str, Any]:
"""Common logic for saving Node data into metadata dict."""
node_dict = node.dict()
metadata: Dict[str, Any] = node_dict.get("metadata", {})

if flat_metadata:
    _validate_is_flat_dict(metadata)

# store entire node as json string - some minor text duplication
if remove_text:
    node_dict[text_field] = ""

# remove embedding from node_dict
node_dict["embedding"] = None

# dump remainder of node_dict to json string
metadata["_node_content"] = json.dumps(node_dict, ensure_ascii=False)
metadata["_node_type"] = node.class_name()

# store ref doc id at top level to allow metadata filtering
# kept for backwards compatibility, will consolidate in future
metadata["document_id"] = node.ref_doc_id or "None"  # for Chroma
metadata["doc_id"] = node.ref_doc_id or "None"  # for Pinecone, Qdrant, Redis
metadata["ref_doc_id"] = node.ref_doc_id or "None"  # for Weaviate`