Inserting a multi-part document results in duplicate nodes and edges

[jin38324]

I want to convert a long document into a Lightrage knowledge base.
If I convert the entire document directly, one error will lead to a total failure. For the purpose of risk control, the document is divided into multiple parts and insert is performed separately.
From the result data, it is found that there are a lot of duplications in nodes and edges.

After investigation, the reason is in the relevant code below.
LightRAG will merge nodes and edges with the same name when executing insert; but this is only valid for the same execution of _process_single_content. If you insert multiple documents like I did, the results will not be merged.

LightRAG/lightrag/operate.py

Line 391 in 874f3b3

_merge_nodes_then_upsert(k, v, knowledge_graph_inst, global_config)

[xandery-geek]

@jin38324 Hi, have you solved this issue?
I want to process multiple documents individually and merge the duplicated nodes or edges into one knowledge graph.

[jin38324]

@jin38324 Hi, have you solved this issue? I want to process multiple documents individually and merge the duplicated nodes or edges into one knowledge graph.

I dug deeper into the code and found that the problem was not from _merge_nodes_then_upsert.
The problem was caused by the data storage part from kg, which set too strict constraints when trying to get existing objects, resulting in the inability to find existing objects, so an insert was performed.

So, if you use the default storage, there should be no problem.

[jin38324]

The problem was not from _merge_nodes_then_upsert.
The problem was caused by the data storage part from kg, which set too strict constraints when trying to get existing objects, resulting in the inability to find existing objects, so an insert was performed.