I have what appears to be a large memory leak coming from libxml2. While I find this to be difficult to believe, it is what seems to be happening, and I wonder if anyone here has any ideas.
The system is a freebsd 12.0 system that is used in an embedded application. The program is multithreaded and written in C and is compiled using clang.
The sequence of events is this: Upon startup, the program will read in an XML file from disk using this command:
Along the way, the program will download another config file across the internet. This new config file arrives encrypted.
The program reads this encrypted file into a buffer, then writes it to disk and frees the buffer. The copy on disk is then passed to gpg for decryption via a popen() call, and the stream out of gpg is captured in a memory buffer.
This memory buffer is also written to disk, then the buffer is freed.
The decrypted xml file on disk is then passed through "xmllint --format infile > outfile" using a system() call. There is no memory buffer in use here, at least, not that I define.
The program loads the newly decrypted and lint'ed xml file like this:
So, now there are two XML documents loaded into memory by this program. The program, depending on circumstances, may choose to switch these documents by swapping pointers:
It ordinarily does this only one time, on the first startup, but might be compelled to do so again at rare intervals (rare meaning months apart).
Usually, it won't swap XML files. In normal operation it will download this new XML file approximately every 4 minutes, and go through the decryption/lint/load process.
After loading the new XML file as new_document, the program will do some validation on it to ensure it is complete and correct. When the file passes validation, the program then will compare it against the existing XML that is loaded as document.
If any differences are found, the section that has the difference in it is moved from new_document to document (replacing the section that is in document) like this:
Finally, the new document is cleared:
When the new document is cleared, the memory that was allocated for it is not returned to the system. It continues to show with my program with the result that the program's footprint is growing and growing and growing.
I have considered the possibility that the way I am moving the node from document to document could be behind the problem. But I have the same problem whether or not there is a difference in the XML that causes a section to be moved. In other words, the common circumstance is that there is no change to the XML so nothing is moved.
But, also, nothing is released.
Does anyone see what I might be doing wrong, or what I might do to solve this problem?
The system is a freebsd 12.0 system that is used in an embedded application. The program is multithreaded and written in C and is compiled using clang.
The sequence of events is this: Upon startup, the program will read in an XML file from disk using this command:
C:
document = xmlReadFile(filename, NULL, 0 );
Along the way, the program will download another config file across the internet. This new config file arrives encrypted.
The program reads this encrypted file into a buffer, then writes it to disk and frees the buffer. The copy on disk is then passed to gpg for decryption via a popen() call, and the stream out of gpg is captured in a memory buffer.
This memory buffer is also written to disk, then the buffer is freed.
The decrypted xml file on disk is then passed through "xmllint --format infile > outfile" using a system() call. There is no memory buffer in use here, at least, not that I define.
The program loads the newly decrypted and lint'ed xml file like this:
C:
new_document = xmlReadFile(filename, NULL, 0 );
So, now there are two XML documents loaded into memory by this program. The program, depending on circumstances, may choose to switch these documents by swapping pointers:
Code:
keepdoc = document;
keeproot = root;
document = new_document;
root = new_root;
new_document = keepdoc;
new_root = keeproot;
It ordinarily does this only one time, on the first startup, but might be compelled to do so again at rare intervals (rare meaning months apart).
Usually, it won't swap XML files. In normal operation it will download this new XML file approximately every 4 minutes, and go through the decryption/lint/load process.
After loading the new XML file as new_document, the program will do some validation on it to ensure it is complete and correct. When the file passes validation, the program then will compare it against the existing XML that is loaded as document.
If any differences are found, the section that has the difference in it is moved from new_document to document (replacing the section that is in document) like this:
Code:
xmlNode *copy_new_node(int nargs, char *nodename, ...)
{
xmlNode *oldnode, *newnode, *copynode, *siblingnode, *parentnode;
va_list nodenames;
if(!new_root || !root)
return(0);
pthread_mutex_lock(&xml_access_mutex);
newnode = __find_xml_entry(new_root, nargs, nodename, nodenames);
oldnode = __find_xml_entry(root, nargs, nodename, nodenames);
va_end(nodenames);
if(!newnode) {
pthread_mutex_unlock(&xml_access_mutex);
return 0x0;
}
siblingnode = oldnode->next;
parentnode = oldnode->parent;
xmlUnlinkNode(oldnode);
xmlFreeNode(oldnode);
xmlUnlinkNode(newnode);
if(siblingnode) {
xmlAddPrevSibling(siblingnode, newnode);
} else {
xmlAddChild(parentnode, newnode);
}
copynode = xmlDocCopyNode(newnode, document, 1);
pthread_mutex_unlock(&xml_access_mutex);
return(copynode);
}
Finally, the new document is cleared:
Code:
void clear_new_xml()
{
if(new_document)
{
pthread_mutex_lock(&xml_access_mutex);
xmlFreeDoc(new_document);
pthread_mutex_unlock(&xml_access_mutex);
}
new_root = 0;
new_document = 0;
}
When the new document is cleared, the memory that was allocated for it is not returned to the system. It continues to show with my program with the result that the program's footprint is growing and growing and growing.
I have considered the possibility that the way I am moving the node from document to document could be behind the problem. But I have the same problem whether or not there is a difference in the XML that causes a section to be moved. In other words, the common circumstance is that there is no change to the XML so nothing is moved.
But, also, nothing is released.
Does anyone see what I might be doing wrong, or what I might do to solve this problem?