Keeping and Organizing Your Source Data

Preserve individual subframes and use an Archive, Active, Projects, and Results structure so future processing remains possible.

The finished JPEG is not the durable source of an astrophotography project. The individual subframes, calibration data, linear masters, and processing record preserve choices you may want to revisit later.

New software can improve registration, rejection, gradient removal, drizzle, or channel extraction. A stronger processing idea may require the original linear data. If only the finished image remains, most of those options are gone.

What to keep

When practical, retain:

accepted and rejected light frames;
darks, flats, bias, or dark flats used by the workflow;
the automatic or live stack;
clean linear masters for each night, filter, and channel;
registration or stacking logs when useful;
and final display exports.

Rejected frames can still explain what went wrong, but they need not occupy premium storage forever. Review them after the project and distinguish diagnostic value from hoarding.

A four-role folder model

I use Archive → Active → Projects → Results, with a fast scratch workspace inside the active/project side when the data volume justifies it.

Active

Active is a disposable working copy for inspection, calibration, and frame curation. Software can create temporary files here without changing the archive.

For large stacks, I often add Scratch on the fastest local storage. The first calibration pass may read from Active, but intermediate calibrated and registered files are written to Scratch so the repeated processing work happens on the fastest disk. Scratch is a workspace, not a backup.

Projects

Projects contains calibrated frames, registrations, stacks, channel masters, masks, and editable processing files. Use descriptive names such as M42_Ha_2026-01-12_stack_linear.fit instead of final2_reallyfinal.fit.

Results

Results contains finished versions for the web, print, comparison, or publication. Keep the master export and document major display choices, especially narrowband channel mappings.

Think in hot, warm, and cold data

The folder names are less important than the roles:

Cold data is the immutable history: original source on large, economical storage, with another verified copy elsewhere.
Warm data is the organized library you expect to revisit: selected subjects, sessions, and calibration sets on reasonably fast storage.
Hot data is the current working set: the project and temporary files on the fastest storage available.

Each layer above the cold archive can be more transient because it can be rebuilt from the layer below. That is what makes a scratch quota or an aggressive cleanup policy safe: the deletion target is a reproducible working copy, not the only copy of the measurements.

An automated ingest can make the model easier to maintain. A verified copy job can mirror new FITS files when a telescope, removable drive, or remote host becomes available, then file them by subject, filter, date, and frame type. Tags can provide a second route through the same library—for example, collecting every subject in Cygnus without changing the underlying folders.

Automation is not verification. Log failures, compare counts and sizes, and periodically restore a sample from backup. A synchronized mistake or deletion is still a mistake unless an independent copy or version history can recover it.

Metadata that makes the data reusable

Record enough context to answer:

What telescope, camera, filter, and mount mode produced this?
What gain, sub length, and accepted integration were used?
Were calibration frames applied, and which set?
Which nights and orientations were combined?
What software and major workflow steps produced the master?
Is the displayed color broadband, HOO, SHO, or another mapping?

File names do not need to contain everything. A small text or JSON manifest beside the project is often clearer.

Backups and storage tiers

Use at least two copies of irreplaceable source data on different devices, with one copy disconnected or off-site when feasible. Fast local storage is valuable for Active and Projects; Archive can move to slower, larger storage after verification.

Checksums or a verified copy tool help confirm that a transfer completed intact. Before deleting a card, phone, or smart-telescope copy, open several destination files and verify the expected count and size.

What the archive enables

Keeping subs makes it possible to:

reject weak frames with better criteria;
recalibrate with improved masters;
combine multiple nights;
drizzle data with sufficient dither diversity;
extract new channels from original OSC data;
and compare processing methods without repeatedly degrading an export.

The aim is not to preserve every temporary file forever. Preserve the measurements and the decisions needed to reconstruct the result.