==Change 1==
Fix drift in when bklog starts reading data via `while read` by
enforcing buffer termination when SECONDS (script variable that
increments every second) rises above a number determined by the
current bufferRound number and bufferTTL.
==Change 2==
Also initially advance SECONDS so it aligns with the remainder of
dividing the current seconds remaining in the day by bufferTTL. The
result should be that, if the user specifies a bufferTTL value that is
a clean fraction of a time element (ex: a bufferTTL of 5 minutes = 300
seconds), then each buffer round should start at a clean fraction of
absolute time with respect to the day (ex: 02:05, 02:10, 02:15,
etc.). This may assist searching through saved files since instead of
files named:
Fix how the ISO-8601 duration string is calculated. Instead of using a
fixed value of `bufferTTL` seconds, record the epoch start and end
times (in seconds since 1970-01-01) of the main data gathering loop
and then calculate the difference. Now, actual deviations in buffer
gathering durations (ex: the first buffer round will likely be shorter
than a normal one) will be accurately represented.
test(bklog):Uncomment cleanup line since script functional
Use of appendFileTar() to append a file to a tar using the file's same
location as a temporary working file caused problems. More direct
'eval' command string formation and 'tar --append' usage solved empty
file problem.
fix(bklog):Fix how procStrings and procFileExts arrays assembled
Default raw data was not being included in output file processing
since procStrings and procFileExts arrays were being completely
overwritten instead of being appended by argProcStrings array
assembled in magicParseProcessStrings() function.
feat(bkgpslog):Add PID control loop for buffer start sched
Add PID loop to adjust time delay between when buffer read-write jobs
are spun off in main while loop. The adjustments are made in order to
keep startup times of write jobs BUFFER_TTL seconds apart. This PID
loop calculation dynamically accounts for any future synchronous
calculations that may be added in the main while loop (including
itself).
feat(bkgpslog):Add PID correction of buffer round timing
Add function to apply PID (proportional, integral, derivative)
adjustments to the sleep period between each buffer round in order to
compensate for lag caused by synchronous calculations between each
buffer round (including the PID adjustment calculation itself).
feat(bkgpslog):Add functions, group code into functions
Add several fucntions to add future new feature (recipient dir watch)
as well as to group existing code blocks into their own functions for
readability. Changes include:
* Add checkAgePubkey() function: checks that a provided string doesn't
throw an error when fed as a recipient string to `age`.
* Pushed several code blocks into their own "magic" functions
** magicParseRecipientDir() : (future feature): To be run at each
buffer session to update the recipient array in response to changes in
a specified recipient directory.
** magicParseRecipientArgs(): Parses strings provided by '-r'
option and format output accordingly. Existing code.
** magicParseCompressionArgs(): Responds to '-c' option by specifying
compression function and output file suffix.
* Update showUsage() to document future '-R' recipient dir option.
chore(bkgpslog):Confirm age can encrypt to ssh pubkey string
Ssh pubkey strings have a space in them which was breaking
appendFileTar() function. 'eval' was used to fix how appendFileTar()
and appendArgTar() execute command strings.
Use eval to build command strings instead of simply putting bare
unquoted variable. This is necessary because the `recipients` variable
used to build a command to encrypt files may contain quotes if one of
the recipients is an ssh key which may have a space:
ssh-rsa AAAAB...
This should fix the inability of `bkgpslog` to accept quoted ssh
pubkey strings as a specified `-r` option argument.
fix(bkgpslog):Reduce logging downtime between buffer rounds
Reduce logging downtime between buffer rounds by moving commands into
the asynchronous `magicWriteBuffer()` function that is spun off
immediately after `gpspipe` has collected data.
Add functionality to `dateTimeShort()` and `dateShort` so the start
time of each saved chunk appended to the `tar` archive can be
calculated indirectly from when `magicWriteBuffer()` starts rather
than performing slow `date` and associated bash variable changes in
the main loop. This start time is needed to name each inserted file as
well as to identify which `tar` file to insert into.
Save gpspipe buffer data `/dev/shm` instead of a Bash variable.
Add `appendFileTar()` function to facilitate reading and processing of
buffer data from `/dev/shm`.
- File writes asynchronous from main script forced to be sequential to
one another to avoid simultaneous writes to same file. Tar write code
consolidated.
- VERSION file format changed to include more fields. VERSION file
generation code cosolidated.
projects / EVA-2020-02.git / log