Re: Finegrained a/c/mtime was Re: Directory notification problem

[Jamie Lokier]

This note explains how to implement file timestamps in such a way that
modifications to file can always be detect reliably.  Currently,
programs such as `make' and other interesting applications cannot give
absolute guarantees of detecting changed files.

Andi Kleen says we can ignore the risk; I disagree, as there are some
applications that cannot be trusted if the risk is plausible, and it can
be fixed easily.

Bernd Eckenfels wrote:
> If you simply check the mtime and the file size you have the two most
> relevant parts. If neighter of those changes this means that programs using
> the dnotify api most likely do not need to act.
                  ^^^^^^^^^^^

In other words, the API is broken for certain applications.

One that springs to mind is transparent caching of JIT-compiled code
between interpreter invocations.  If dnotify misses the notification
sometimes, the caching ceases to be transparent, and you have to switch
it off for reliable behaviour, a major efficiency loss.

Microsecond resolution, of course, does not fix this problem.

Alex Larsson wrote:
> Is a nanoseconds field the right choice though? In reality you might not 
> have a nanosecond resolution timer, so you would miss changes that appear
> on shorter timescale than the timer resolution. Wouldn't a generation 
> counter, increased when ctime was updated, be a better solution?

As has been pointed out, it would not be compatible with other unix
systems and existing software, and timestamps have nice audit trail
possibilities.

I didn't realise there was enough precision left in ext2 inodes for
nanosecond timestamps.

Timestamps have _many_ problems: the main problem is that you can't
guarantee to reliably detect a changed file.  For some interesting
applications this is fatal.

However, you can fix timestamps and keep the best benefits of timestamps
and counters:

  - high resolution timestamps.

  - whenever there is a change event, check whether the timestamp
    would be advanced.  If not, delay the change (i.e. inside the
    write() call) until the clock time has advanced to the next
    high-resolution unit.

  - if you use nanoseconds, this will never occur on current machines
    and only rarely on faster machines.

  - spinning is an acceptable way to delay for such a short time.

  - it's not necessary to delay if nobody read the mtime since the last
    timestamp update, which will nearly always be the case.  So even on
    extremely fast future machines, you would hardly ever pause.

cheers,
-- Jamie