What Linux Logs Do and Do Not Record

Introduction

LX0.4 covered the kernel answering questions about what is true at this moment. This section is about the other large category of evidence on a Linux host: the records written by programs about the past.

They behave nothing like kernel files. Each one was written by a different program, entirely for that program's own purposes, in whatever format its authors preferred, and an investigation trying to assemble a single timeline has to reconcile every bit of it.

The difference is who wrote them and when. A kernel file is the system answering a question you just asked; a log is a program recording what it thought was worth keeping, at the time, for reasons that had nothing to do with an investigation.

You will finish able to name what the format differences actually cost, date a line from a source that does not record the year, and say which gaps are structural rather than somebody's oversight.

The differences look cosmetic written out and they are not. Each one is a field a timeline needs and a program did not write, which means somebody or something has to supply it before the entries can be placed in a single sequence.

Comparing what each log writes for the same moment shows the problem directly.

$ ./timestamp-census.py
  source            year?  zone?  sub-sec?  format
  syslog            NO     NO     no        Mon D HH:MM:SS
  web access        yes    yes    no        [D/Mon/YYYY:HH:MM:SS +ZZZZ]
  journald          NO     NO     no        Mon DD HH:MM:SS
  application json  yes    yes    yes       ISO 8601
  audit             yes    implied yes      epoch.millis
  dpkg              yes    NO     no        YYYY-MM-DD HH:MM:SS

Six sources and no two agree. Three fields vary and every combination of them appears somewhere in the six. They differ on year, on time zone and on precision, and every one of them is correct for the program that writes it.

The variation is not carelessness. Each format was chosen for a program's own logging needs long before anybody thought about assembling them into a single sequence, which is why no amount of configuration makes them agree.

It is also not fixable by an estate. You can standardize what you deploy and you cannot change what a program writes about itself, so reconciling formats is permanent work rather than a configuration debt somebody will eventually pay off.

Two of those six sources omit the year entirely, and one of them is the most commonly read log on the whole system.

$ head -1 /var/log/messages
Mar  2 09:14:22 web01 sshd[2841]: Accepted publickey for deploy
$ ./parse-line.py
  month=Mar  day=2  time=09:14:22  year=UNKNOWN

A complete line with no year in it. The format is decades old and was designed for reading on the machine that wrote it. Nothing in the record identifies which March it describes, and the line is not defective: the format predates the problem it causes.

Something external has to supply the year, and the usual candidates are weak.

Weak is not the same as useless. Each of the three is fine as corroboration and none is sound as the sole basis for a date, which is a distinction worth holding because tools routinely use one of them silently.

$ ./year-sources.py
  the file's mtime         a property of the FILE, not of the line
  "it must be this year"   breaks for anything near 01 January
  the rotation filename    only if the scheme encodes a date

Three candidates and none is reliable alone. All three are used silently by tools that do not report which they chose. A rotated file holds months of lines and only one modification time, so the file cannot date any individual line inside it.

Setting the cases out separately matters because the mid-year one is so nearly always right. A rule that works for eleven months of the year gets trusted, and the month where it fails is the month nobody re-examines.

That note is the scenario's whole correction, and the worked example makes the size of the error obvious.

The size is the useful part. An error of a few minutes invites a search for drift or clock skew, and an error of exactly 365 days has one cause, which makes it easy to recognize once somebody knows to look.

$ ./year-boundary.py --today '04 Jan 2026'
  'Dec 30'  assume current year -> 30 Dec 2026   correct is 30 Dec 2025   WRONG by 365 days
  'Jan  2'  assume current year -> 02 Jan 2026   correct is 02 Jan 2026   ok

One right and one wrong by a year. Only the failing one crosses a boundary, which is what makes it rare and undetectable. Both were produced by the same rule applied to the same file, which is why the failure is not visible from the output.

There is a second field worth the same scrutiny as the year, which is the time zone. Two of the six sources record one, one implies it by definition, and three write a local time with nothing saying which local means.

That matters less than the year and it matters in a specific case. An estate whose hosts are not all in one zone produces timelines where entries from two machines appear to interleave and do not, and the error is hours rather than a year so it looks plausible.

The fix is the same in shape. Either the zone came from the record or somebody supplied it, and writing down which is what lets a reader check rather than trust.

The format problems are solvable and the absences are not.

That distinction is worth holding onto because effort spent on the wrong side of it is wasted. Reconciling six timestamp conventions is an afternoon of parsing; recovering a record nobody wrote is not a harder version of the same task.

Process execution is not logged by default. Nothing on a standard host records that a program ran, which LX0.2 showed costs an investigation the entire middle of an intrusion, and enabling it is a decision an estate makes in advance or not at all.

That single absence explains most of the difficulty in LX0.2. Four of six stages in that intrusion were processes starting and exiting, and on a standard host not one of them wrote anything anywhere.

Two yes and three no, and the three absent rows are the ones an intrusion lives in. The two that are recorded are both about administration rather than activity.

File access is not logged either. Reading a file leaves an access time if the mount options allow it and no log entry anywhere, so who read what is usually unanswerable after the fact.

Network connections are logged by what handles them rather than by the host. An outbound connection appears in a firewall or proxy record elsewhere, and the machine that made it may have written nothing.

That is why LX0.3 put those requests in the first ten minutes. The record exists, it is good, it belongs to somebody else, and the constraint on getting it is a queue rather than anything happening on the machine.

Those three cover most of what an investigation wants to know. None of the three can be recovered later by looking harder. They are absent by design rather than by accident, because logging all of it on every host would cost more than most estates will spend.

Drawing the six sources converging on one timeline is the shape worth keeping, with the absences hanging below it. Everything above the box is work; everything below it is a boundary, and no amount of effort moves an item from the second group to the first.

The dashed boxes are the harder half. Format differences are work and the absences are a boundary, and no amount of skill reads a record that was never written.

They are also the half worth stating plainly in a report. An investigation that says process execution was not recorded on this host has explained a gap; one that simply has nothing to say about what ran leaves a reader to assume nobody looked.

Seeing what it produces is worth the space before the exercise, because the output is not obviously broken. It is well formed, correctly sorted for the values it holds, and wrong only if somebody notices that a December entry has landed after a January one.

One framing before the exercise. Supplying the current year is the right default and it is correct for the overwhelming majority of lines anybody will ever parse. What makes it dangerous is that it is applied silently, so the small minority where it fails arrives in a timeline looking exactly like every entry that worked.

$ ./timeline.sh --host web01 --window '28 Dec - 04 Jan'
  2026-12-30 03:14:22  sshd    Accepted publickey for deploy      <- from syslog
  2026-01-02 09:14:22  sshd    Accepted publickey for deploy      <- from syslog
  2026-01-03 11:02:08  dpkg    install curl:amd64
  2026-01-04 10:00:00  audit   collection begins
  4 entries, sorted ascending

Four entries and the sort put December after January. The tool applied one rule consistently and produced one wrong row. The first row is the scenario: a yearless line given the current year, landing eleven months in the future and sorting to the wrong end of its own timeline.

The exercise above gives you a timeline assembled from all six sources.

Listing the absences first makes it worth being clear about what actually remains.

The four below are enough to reconstruct a great deal. Most of what LX0.2 recovered eleven days after the fact came from exactly these sources, which is why the absences are a constraint rather than a defeat.

$ ./what-is-here.sh --host web01
  authentication   present   year yes, account yes, source address yes
  package changes  present   year yes, dated, long retention
  service starts   present   brackets activity without describing it
  web requests     present   application routes only; static paths not logged
  process exec     ABSENT    not enabled on this host
  file access      ABSENT    no such log exists
  outbound conns   ELSEWHERE firewall and proxy, requested 16:01

Seven rows: four present, two absent and one belonging to somebody else. The fourth row carries the qualification that matters, since half a web log reads as a complete one.

Authentication is recorded well. Who connected, from where, by what method and whether it succeeded is written by default on every standard host, and LX9.8 used exactly that as independent corroboration.

It is worth knowing that this is the strongest record on most Linux hosts. It has a year, it survives rotation longer than almost anything else, it names an account and a source address, and it is written by default on machines nobody configured for logging.

Package changes are recorded reliably. Installations and removals are dated in a database and a log, which is one of the few records with a year, a date and a durable retention.

They are also useful for something other than packages. A dated entry from a source that records the year is exactly the anchor the yearless logs need, which makes this log valuable well beyond what it is about.

Service starts and failures are there. A program crashing, restarting or being reconfigured writes something, which frequently brackets an intrusion even when the intrusion itself wrote nothing.

Those entries bracket rather than describe. Something restarting at an odd hour says a great deal about when to look without saying anything about what happened, which is often the most an unaudited host will give.

And web requests are usually complete. A web server records what it served, which for the class of host in LX0.2 is the only record of how anybody arrived.

The qualification is which routes. A server logging its application routes and not its static paths has a complete record of one half, and an investigation that does not check which half it has will read an absence as evidence.

Normalizing six conventions into one is routine, and the record of what was assumed is what makes it defensible.

Every tool that assembles timelines does the normalizing. Very few of them report what they supplied while doing it, which means the work is done correctly and the provenance is discarded on the way through.

Every normalization is a small claim. Converting a yearless line to a full timestamp claims a year; converting a zoneless one claims a zone; and both are usually right and neither came from the record.

Nobody thinks of them as claims, which is the problem. Parsing feels mechanical, and a parser filling a missing field is making the same kind of decision an analyst would be asked to justify if they had made it in prose.

$ cat timeline-provenance.txt
  syslog lines        year SUPPLIED from rotation filename messages-20260302
  journald lines      year SUPPLIED from the journal's own index
  web access          year, zone from the record
  application json    year, zone, milliseconds from the record
  audit               epoch converted; zone is UTC by definition
  dpkg                year from the record; zone ASSUMED host local
  boundary check      0 lines fall within 7 days of 01 January

Seven lines and four of them mark something supplied. The three unmarked ones came entirely from their own records. The last is the check that would have caught the scenario, and it costs one comparison.

That level of detail is not bureaucracy. A timeline where every entry is either from the record or marked as supplied can be argued with precisely, and one where the distinction was lost cannot be defended at all.

It is also cheap at the point of parsing and impossible afterwards. A parser knows exactly which fields it invented while it is inventing them, and a finished timeline has no way to distinguish them from fields that were read.

One more thing the anchoring technique gives that is worth stating, because it generalizes past the year problem. Any field a source omits can sometimes be recovered from a source that records it, provided the two can be matched on something they share.

That is the same move as the socket join in LX0.4 and the authentication corroboration in LX9.8. Two records describing one event, matched on a field they agree about, let the richer one supply what the poorer one left out.

It is also the only form of supplying that is not an assumption. A year taken from a matched entry in a dated log came from the record; a year taken from the calendar came from a rule, and only the first survives somebody asking where it came from.

The same four questions answer most of what you need before reading anything.

$ ./log-survey.sh --host web01 --incident '02 Mar'
  source         exists  window covers 02 Mar  records year
  auth           yes     yes, oldest 06 Feb    NO
  web access     yes     yes                   yes
  application    yes     NO, oldest 06 Mar     yes
  package        yes     yes                   yes
  process exec   NO      n/a                   n/a
  anchor         package log dates the yearless auth lines

Six rows against four questions, and the last line is the output. One source that records a year dates the two that do not.

Which logs exist here. Sources are optional, so the first pass is an inventory rather than a search, and an absent source is a fact about the estate.

The inventory is worth writing down rather than holding in your head. Six sources is enough that a reader of the eventual report cannot reconstruct which ones were consulted, and an unexamined log and an absent one look identical in a finding.

What window does each cover. LX0.3 established that rotation moves a boundary rather than decaying content, and a log whose window has passed looks identical to one with nothing in it.

Doing this before reading rather than after is the whole value. A search returning nothing means something quite different depending on whether the window covered the period, and the check costs one command per log.

Does each record the year. Two of six did not on the measured host, and those two need a supplied year that somebody has to justify.

The answer is a property of the format rather than of the host, so it transfers. Once you know which sources on your platform carry a year, the question becomes which of those sources are present rather than a fresh investigation each time.

Ask whether anything forwards off the host. A copy on a central collector answers the retention question differently and often carries a normalized timestamp, which removes several of the problems above at once.

And what is missing entirely. Process execution, file access and outbound connections are the usual three, and naming them in the record is what stops a reader assuming nobody looked.

The yearless problem has a proper solution, and it does not involve assuming anything at all.

Lines within a file are in order. A log is written by appending, so entry positions carry a sequence even when the entries themselves carry no year, and that ordering is a fact from the record rather than a supposition.

That is worth relying on because it is a property of how logs are written rather than a convention a program chose. Appending is what makes a log a log, so the ordering holds across every source in the census regardless of format.

$ ./anchor.sh --file messages-20260302 --anchor-from dpkg.log
  line     1  Dec 28 03:11:02  ...
  line   840  Dec 31 23:58:41  ...
  line   841  Jan  1 00:00:12  ...   <- year increments here, from ordering alone
  line  4102  Mar  2 09:14:22  ...
  anchored   line 4102 matches a dpkg entry dated 2026-03-02 09:14:22
  resolved   lines 1-840 are 2025; lines 841 onward are 2026
  supplied   nothing. Both years follow from one dated anchor plus order.

One anchor and the whole file resolves. The anchor can be any entry at all, provided its source records a year. A single entry from a source that does record the year fixes the position, and the ordering carries that year outward in both directions.

The increment is visible in the data. A file whose dates run from December into January contains the boundary itself, so the year change is something the record shows rather than something a rule imposes.

A file that does not span a boundary is easier still. All its lines share one year, so a single anchor anywhere in it dates every entry, and the whole problem reduces to finding one matching record elsewhere.

Practice

There is an exercise worth running once on your own estate that costs an afternoon and answers a question most teams cannot. Take the three absences, check each against a representative host per class, and write down which of them your machines would actually record.

The answers vary more than people expect within one organization. Hosts built at different times under different templates disagree about auditing, and the machine somebody most wants command logging on is frequently the one that does not have it.

Doing it in advance also changes what gets asked for. A team that knows process execution is unrecorded on its web servers can argue for enabling it with a specific case, rather than discovering the gap during an incident when it is far too late to be useful.

Take a rotated log on a host you use and check whether its lines carry a year. On most systems the answer for the main system log is no, which makes every date in it something a tool decided rather than something the host recorded.

One thing to carry forward. The absences here are why so much of an investigation leans on things that persist rather than things that were recorded, and the most reliable of those is whatever an intruder built to survive a reboot.

The gaps are now established. Where persistence lives on a Linux host, and why finding one instance is not the end of the search, is LX0.6.