40 lines
1.6 KiB
Markdown
40 lines
1.6 KiB
Markdown
shoebox
|
|
=======
|
|
|
|
binary data archiving library - supports uploading to object storage
|
|
|
|
There are ArchiveReaders and ArchiveWriters which are managed
|
|
by RollManager. "Roll" comes from "roll over". When does a file roll-over
|
|
from one to the next? There is only one Archiver active at a time.
|
|
|
|
The RollManager opens and closes Archivers as
|
|
needed. "As needed" is determined by which RollChecker that was
|
|
passed into the RollManager. Archive files can roll over based
|
|
on file size or elapsed time (for writing). For reading, archive
|
|
files are only rolled over when the EOF is reached.
|
|
|
|
Roll Managers also take care of filename creation, compression
|
|
of completed archives and transfer of archive files to remote
|
|
storage locations.
|
|
|
|
The RollChecker's have a reference to the current Archive so
|
|
they can ask file-related questions (like "how big are you?")
|
|
|
|
Usage:
|
|
|
|
# Make a roll checker of whatever strategy you choose.
|
|
checker = roll_checker.NeverRollChecker() # one big file.
|
|
# Make a roll manager for reading or writing.
|
|
# Give the filename template and the checker.
|
|
# (and an optional working directory for new files)
|
|
x = roll_manager.WritingRollManager("template_%s", checker)
|
|
# Write metadata and payload ...
|
|
for index in range(10):
|
|
x.write({"index": str(index)}, "payload_%d" % index)
|
|
# WritingRollManager.write(metadata, payload) where
|
|
# metadata = string:string dict
|
|
# payload = string of data. Most likely a json structure.
|
|
|
|
TODO: How will the ReadingRollManager know which files to read
|
|
from, and in which order, if the filename is templated?
|