Log Rotation and Cleanup Job¶

A recurring cleanup process — instantiates Layer Decay and Expiration Management

A recurring job that closes the current append-only log, shifts each older generation one slot down the numbered chain, and compresses, archives, or purges whatever falls off the end.

Append-only streams — logs, journals, ledgers — grow forever if left alone, and the newest entries are the ones you almost always want. Log Rotation and Cleanup Job is the scheduled process that keeps such a stream bounded by working on its sequence rather than on any single item's value. Its defining move is generational shifting: on each run it seals the active file, renames the chain of prior generations one step older (log → log.1 → log.2 → …), starts a fresh empty file for new writes, and disposes of whatever slid past the last slot. It is not deciding whether a given layer is worth keeping — it is enforcing a fixed depth of history on an ordered, ever-growing deposit, and executing the compress/archive/delete at the tail.

Example¶

A web server writes access.log continuously. A nightly rotation job runs: it renames access.log to access.log.1, bumps access.log.1 to access.log.2, and so on, keeps 14 generations, gzips everything older than one day, ships .log.7 and beyond to an archive bucket, and deletes anything past generation 14. A signal tells the server to reopen and write to a fresh access.log. The result is a rolling two weeks of readily-searchable logs on the box, older weeks compressed and archived off it, and disk that never fills from log growth — all without anyone ever judging an individual log line. The position in the chain, driven purely by age of deposit, decides each generation's fate.

How it works¶

Its substance is the ordered shift-and-dispose cycle, not any per-item judgment:

Trigger on a schedule or a size threshold (rotate nightly, or whenever the file crosses N megabytes).
Shift the generation chain — the deposition-order index made concrete: each file's number is its age rank, and rotation renames the whole chain one step older.
Apply staged disposition down the chain — compress recent generations, archive middle ones, purge the tail past the retention depth.
Signal writers to reopen onto the fresh file so no entries are lost across the swap.

Tuning parameters¶

Rotation trigger — time-based, size-based, or both. Size-based bounds disk tightly under bursty load; time-based gives predictable, aligned generations.
Retention depth — how many generations to keep before the tail is purged. Deeper keeps more searchable history on hand at direct storage cost.
Compression point — how early in the chain to gzip. Compress sooner to save space, at the cost of on-the-fly decompression when reading recent logs.
Archive vs. delete at the tail — whether generations that fall off are shipped to cold storage or simply removed, trading long-term recoverability against total cleanup.
Post-rotation action — how writers are told to reopen (signal, copytruncate); the wrong choice drops or duplicates entries across the swap.

When it helps, and when it misleads¶

Its strength is dead-simple, reliable boundedness for high-volume ordered streams: it turns "the log will eventually fill the disk" into "the log holds a fixed, known window," with no per-entry decision and negligible overhead. For anything append-only and mostly-recent, it is the right tool.

Its weakness is that it is blind to content — it disposes by position in the chain, so a burst of once-in-a-year critical entries is purged on exactly the same schedule as routine noise. Under a log flood, size-based rotation can spin so fast that the window collapses to minutes and the entries you need are gone before you look. And the tail action is genuinely destructive when set to delete, so a job pointed at logs that also serve as an audit or forensic record can quietly erase evidence a retention rule required. The discipline is to route audit-bearing streams through an archive tail (not delete) and to let a real retention schedule, not the rotation depth alone, govern anything with a compliance obligation.

How it implements the components¶

Log Rotation and Cleanup Job realises the ordered-stream execution side of the archetype — enforcing bounded depth on an append-only deposit:

deposition_order_and_age_index — the numbered generation chain is the age index: each file's slot encodes its deposit order, and rotation is that index advancing.
pruning_or_archival_pathway — it executes the concrete tail disposition: compress, ship to archive, or delete generations past the retention depth.

It does not weigh a layer's individual value (that's Age-Weighted Value Score) or check dependencies before purging (Dependency-Safe Delete Check); this job disposes by chain position, trusting those mechanisms and the retention schedule to have set the depth safely.

Instantiates: Layer Decay and Expiration Management — the scheduled boundedness path for append-only streams.
Consumes: Retention Schedule — the retention depth it enforces should come from there, not be invented at the job.
Sibling mechanisms: Retention Schedule · Lifecycle Storage Tiering Policy · Age-Weighted Value Score · Cache Eviction Rule · Time-to-Live (TTL) Policy · Dependency-Safe Delete Check · Soft-Delete Quarantine Window · Stale-Layer Detection Dashboard · Archive-Restore Test · Tombstone or Deletion Marker