Btfj.dat – huge size and memory consumption

Notice: This thread is very old.
josef.sabl
Member | 153
+
0
-

Hi, how is btfj.dat supposed to behave? It is bloated (500 MB) after few weeks of running in production. It also appears to cause troubles with out of memory fatal errors as scripts consume more and more and when fatal error occurs, it is usually in FileJournal.php. What can cause it to be like that. What are we doing wrong? Thanks for any advice.

Milo
Nette Core | 1283
+
0
-

The btjf.dat file is a B+ Tree index for storing tags and priority cache parameters. It makes cache very fast. Its size depends on how many items with tags (or priority) you store into cache. The file is created by FileJournal.

The 500MB is a pretty big index, but it should not behave oddly as you describe. There is a long-persist bug in the FileJournal (last report) and it is hard to find.

So, you don't do wrong anything, I guess.

josef.sabl
Member | 153
+
0
-

Thanks a lot for your response.

Let me ask one more question, please: Does filejournal has some fallback or self-cleaning mechanism that would throw away really old entries? I mean when I create an app that will push items into cache with tags like “entry” . $timestamp without any expiry, will the file grow indefinetly? Will it have some impact on the memory consumption or io operations? What if I set the expiry to those entries, will they be purged.

And on related note, I noticed that cleaning cache doesn't do anything to filejournal's file size. I have to clean cache, delete file and clean cache again, or else I get bugs :-)

Thanks again, appreciated!

And good luck hunting that bug :-)

Last edited by josef.sabl (2014-11-26 15:27)

Milo
Nette Core | 1283
+
0
-

Hard to answer… Cache has garbage collector implemented, but I'm not sure if it cleans journal too. Maybe not. I cannot answer other questions, I don't know code at required level.

Jan Tvrdík
Nette guru | 2595
+
+1
-

The FileJournal uses 4kB for each node, therefore size 500 MB means that it contains about 128 000 nodes. Can you guess how many tags / items to you store in cache?

Filip Procházka
Moderator | 4668
+
0
-

If you have so much data, it would be probably wise to start using something that might be a bit faster, have a look at Kdyby/Redis journal.