Its always been my belief that large folders cause multi level directory
block chaining in traditional UNIX fs. This itself incurs costs and
consequences on how the cross-system file buffer cache works. Basically,
any operation which requires all the directory blocks to be walked in
sequence flood kernel file buffers. It has impacts on other uses of the OS.
It is likely more modern FS like ZFS handle this differently but I don't
know, I've never seen an analysis.
Your system has cronjobs doing things like find . -type f -mtime <x> which
may run slower, you may be causing general systems slowdowns.
I think it would make sense to filter out the things you want.
I Share your problem, mails from now dead relatives it is exquisitely
painful for me to read but I am unwilling to delete, and the thought of
having to write filters to find and file them doesn't fill me with joy. On
the other hand, I have replicated the data because you have other risks:
disk media is fragile.
Don't have only one copy of these mails. A cloud mail provider like Google
might be a good backup, and has filter, search and tag options.
Cheers
G
On Sun, 6 Jun 2021, 7:10 am , <norm(_at_)dad(_dot_)org> wrote:
Starting in late 2014 I have stopped deleting messages, putting them in a
directory, +gone, which now contains 465,147 messages and uses about 17
gigabytes. The bulk of these messages were of transitory or of less
interest
to me. But they include 1,702 messages from my daughter. They were almost
all
of no interest or use to me within a day or two of when she sent them. But
she
recently died (the worst thing by far that's ever happened to me). Now
every
byte she ever wrote is precious to me. So I am glad that I stopped deleting
messages that I no longer care about.
In practice this large folder has little impact on performance. For
example,
whenever I do a pick which is, or in a script which might be, +gone, I
give it
an argument like last:100000. I could, if necessary split +gone into
several
smaller folders, but I would rather not. But I'm concerned that a bug in
nmh
might cause a problem. For example, some kind of a buffer overflow.
So, what is the likelihood of such a bug? Does anybody have any experience
dealing with such large folders?
Norman Shapiro
----------
Starting in late 2014 I have stopped deleting messages, putting them in
a directory, +gone, which now contains 465,147 messages and uses
about 17 gigabytes. The bulk of these messages were of transitory or of
less
interest to me. But they include 1,702 messages from my daughter. They
were almost
all of no interest or use to me within a day or two of when she sent them.
But she recently died (the worst thing by far that's ever happened to me).
Now every byte she ever wrote is precious to me. So I am glad that I
stopped
deleting messages that I no longer care about.
In practice this large folder has little impact on performance. For
example,
whenever I do a pick which is, or in a script which might be, +gone I give
it an argument like last:100000. I could, if necessary split +gone into
several smaller folders, but I would rather not. But I'm concerned that a
bug
in nmh might cause a problem. For example, some kind of a buffer overflow.
So, what is the likelihood of such a bug? Does anybody have any experience
dealing with such large folders?
such a large folder might
Norman Shapiro