Steven Baltakatei Sandoval [Sat, 11 Jul 2020 01:19:19 +0000 (01:19 +0000)]
fix(bklog):Add missing function timeDuration() from bkgpslog
Rearrange function declarations
Steven Baltakatei Sandoval [Sat, 11 Jul 2020 01:11:02 +0000 (01:11 +0000)]
feat(bklog):Copy bkgpslog functions, rewrite main
Steven Baltakatei Sandoval [Fri, 10 Jul 2020 05:42:25 +0000 (05:42 +0000)]
Merge branch 'develop' into feature/bash-logging-filter/BK-2020-02
Steven Baltakatei Sandoval [Fri, 10 Jul 2020 05:29:19 +0000 (05:29 +0000)]
style(bkgpslog):Change VAR_NAMES to varNames
Change all-caps variable names to mix of lowercase and uppercase names
in order to avoid naming conflicts with environment variables.
Ref: https://mywiki.wooledge.org/BashGuide/InputAndOutput
Steven Baltakatei Sandoval [Fri, 10 Jul 2020 02:44:45 +0000 (02:44 +0000)]
chore(bkgpslog):Change allcaps varnames to lower/uppercase
Steven Baltakatei Sandoval [Fri, 10 Jul 2020 02:32:50 +0000 (02:32 +0000)]
chore(bklog):Add more metadata
Steven Baltakatei Sandoval [Fri, 10 Jul 2020 02:26:56 +0000 (02:26 +0000)]
feat(bklog):Add metadata, version
Steven Baltakatei Sandoval [Fri, 10 Jul 2020 02:10:29 +0000 (02:10 +0000)]
feat(bklog):Create barebones logging script for general logger
Steven Baltakatei Sandoval [Tue, 7 Jul 2020 23:59:56 +0000 (23:59 +0000)]
fix(bkgpslog):Recreate working dir if missing
Steven Baltakatei Sandoval [Tue, 7 Jul 2020 23:51:32 +0000 (23:51 +0000)]
test(bkgpslog):Add sticky bit to DIR_TMP
Add debug message when DIR_TMP removal is completed.
Steven Baltakatei Sandoval [Tue, 7 Jul 2020 20:32:00 +0000 (20:32 +0000)]
style(bkgpslog):Minor boilerplate modification
Steven Baltakatei Sandoval [Tue, 7 Jul 2020 20:10:47 +0000 (20:10 +0000)]
fix(bkgpslog):Fix PATHOUT_TAR filename dateShort() call
Steven Baltakatei Sandoval [Tue, 7 Jul 2020 19:47:15 +0000 (19:47 +0000)]
test(bkgpslog):Test date format bug
Steven Baltakatei Sandoval [Tue, 7 Jul 2020 18:55:21 +0000 (18:55 +0000)]
test(bkgpslog):Add debug messages
Bug still exists if `bkgpslog` run by cron instead of directly by
user.
Steven Baltakatei Sandoval [Tue, 7 Jul 2020 13:38:43 +0000 (13:38 +0000)]
test(bkgpslog):Minor debug msg
Steven Baltakatei Sandoval [Tue, 7 Jul 2020 12:32:25 +0000 (12:32 +0000)]
test(bkgpslog):Add debug tests
Steven Baltakatei Sandoval [Tue, 7 Jul 2020 12:15:58 +0000 (12:15 +0000)]
chore(bkgpslog):Minor verbose message fix
Steven Baltakatei Sandoval [Tue, 7 Jul 2020 12:02:47 +0000 (12:02 +0000)]
fix(bkgpslog):Make static initial BUFFER_TTL dynamic
Static value was used during debugging but should have been made
dynamic.
Steven Baltakatei Sandoval [Tue, 7 Jul 2020 11:45:52 +0000 (11:45 +0000)]
chore(bkgpslog):Add minor comment
Steven Baltakatei Sandoval [Tue, 7 Jul 2020 11:37:53 +0000 (11:37 +0000)]
feat(bkgpslog):Add PID control loop for buffer start sched
Add PID loop to adjust time delay between when buffer read-write jobs
are spun off in main while loop. The adjustments are made in order to
keep startup times of write jobs BUFFER_TTL seconds apart. This PID
loop calculation dynamically accounts for any future synchronous
calculations that may be added in the main while loop (including
itself).
Steven Baltakatei Sandoval [Tue, 7 Jul 2020 11:35:36 +0000 (11:35 +0000)]
Merge branch 'feature/location-logging/EVA-2020-02' into develop
Steven Baltakatei Sandoval [Tue, 7 Jul 2020 10:39:43 +0000 (10:39 +0000)]
fix(bkgpslog):Remove debug sleep
Steven Baltakatei Sandoval [Tue, 7 Jul 2020 10:36:03 +0000 (10:36 +0000)]
test(bkgpslog):More debug msgs
Steven Baltakatei Sandoval [Tue, 7 Jul 2020 10:16:38 +0000 (10:16 +0000)]
test(bkgpslog):Add debug messages
Steven Baltakatei Sandoval [Tue, 7 Jul 2020 10:04:57 +0000 (10:04 +0000)]
fix(bkgpslog):fix pid float calc
Steven Baltakatei Sandoval [Tue, 7 Jul 2020 10:02:51 +0000 (10:02 +0000)]
test(bkgpslog):more pid float testing
Steven Baltakatei Sandoval [Tue, 7 Jul 2020 09:59:26 +0000 (09:59 +0000)]
fix(bkgpslog):Fix float calc in buffer PID adjust
Steven Baltakatei Sandoval [Tue, 7 Jul 2020 09:37:07 +0000 (09:37 +0000)]
test(bkgpslog):More PID testing
Steven Baltakatei Sandoval [Tue, 7 Jul 2020 09:30:07 +0000 (09:30 +0000)]
test(bkgpslog):Fix debug msgs
Steven Baltakatei Sandoval [Tue, 7 Jul 2020 09:27:46 +0000 (09:27 +0000)]
test(bkgpslog):Test different PID calc
Steven Baltakatei Sandoval [Tue, 7 Jul 2020 09:13:17 +0000 (09:13 +0000)]
test(bkgpslog):debug:Shift magicBufferSleepPID
Steven Baltakatei Sandoval [Tue, 7 Jul 2020 09:04:33 +0000 (09:04 +0000)]
test(bkgpslog):Add debug code and negative int catch
Steven Baltakatei Sandoval [Tue, 7 Jul 2020 08:45:42 +0000 (08:45 +0000)]
test(bkgpslog):Debug testing PID loop
Steven Baltakatei Sandoval [Tue, 7 Jul 2020 08:31:59 +0000 (08:31 +0000)]
feat(bkgpslog):Add PID correction of buffer round timing
Add function to apply PID (proportional, integral, derivative)
adjustments to the sleep period between each buffer round in order to
compensate for lag caused by synchronous calculations between each
buffer round (including the PID adjustment calculation itself).
Steven Baltakatei Sandoval [Tue, 7 Jul 2020 01:48:52 +0000 (01:48 +0000)]
fix(bkgpslog):Remove nuisance error if -R not set
Error was appearing if -e were set but -R wasn't. -R is not a required
option.
Steven Baltakatei Sandoval [Mon, 6 Jul 2020 20:01:18 +0000 (20:01 +0000)]
fix(bkgpslog):Fix recPubKeysValid array bloat
Implemented recPubKeysValidStatic array for magicParseRecipientDir to
process instead of recPubKeysValid array.
Steven Baltakatei Sandoval [Mon, 6 Jul 2020 19:47:03 +0000 (19:47 +0000)]
fix(bkgpslog):Actually add recipient watch dir check
Steven Baltakatei Sandoval [Mon, 6 Jul 2020 19:10:32 +0000 (19:10 +0000)]
test(bkgpslog):Add verbose debug msg to recipient dir code
Steven Baltakatei Sandoval [Mon, 6 Jul 2020 19:02:09 +0000 (19:02 +0000)]
fix(bkgpslog):Fix error in encryption option handling, bump ver
Note: 0.4.0 version includes new '-b', '-B' options for specifying buffer and
script time-to-live values.
Note: 0.4.0 version includes new '-R' option for specifying directory
containing files specifying recipient pubkeys.
Steven Baltakatei Sandoval [Mon, 6 Jul 2020 18:42:14 +0000 (18:42 +0000)]
feat(bkgpslog):Enable custom TTL for buffer and script
New options: '-b' (buffer), '-B' (script).
Updated showUsage().
Steven Baltakatei Sandoval [Mon, 6 Jul 2020 16:55:37 +0000 (16:55 +0000)]
fix(bkgpslog):Fix broken tempdir var name
Steven Baltakatei Sandoval [Sun, 5 Jul 2020 17:31:07 +0000 (17:31 +0000)]
fix(bkgpslog):Add '-e'/'-R' checks in -R parse
Also add comments to magic functions in main()
Steven Baltakatei Sandoval [Sun, 5 Jul 2020 17:17:36 +0000 (17:17 +0000)]
fix(bkgpslog):Check -e, -r, -R option combinations; update usage
Steven Baltakatei Sandoval [Sun, 5 Jul 2020 16:56:56 +0000 (16:56 +0000)]
feat(bkgpslog):Add functions, group code into functions
Add several fucntions to add future new feature (recipient dir watch)
as well as to group existing code blocks into their own functions for
readability. Changes include:
* Add checkAgePubkey() function: checks that a provided string doesn't
throw an error when fed as a recipient string to `age`.
* Pushed several code blocks into their own "magic" functions
** magicParseRecipientDir() : (future feature): To be run at each
buffer session to update the recipient array in response to changes in
a specified recipient directory.
** magicParseRecipientArgs(): Parses strings provided by '-r'
option and format output accordingly. Existing code.
** magicParseCompressionArgs(): Responds to '-c' option by specifying
compression function and output file suffix.
* Update showUsage() to document future '-R' recipient dir option.
Steven Baltakatei Sandoval [Sat, 4 Jul 2020 22:13:09 +0000 (22:13 +0000)]
Merge branch 'feature/location-logging/EVA-2020-02' into develop
Steven Baltakatei Sandoval [Sat, 4 Jul 2020 22:12:43 +0000 (22:12 +0000)]
chore(bkgpslog):Bump version number to include recent change
Steven Baltakatei Sandoval [Sat, 4 Jul 2020 22:11:45 +0000 (22:11 +0000)]
Merge branch 'feature/location-logging/EVA-2020-02' into develop
Steven Baltakatei Sandoval [Sat, 4 Jul 2020 21:46:37 +0000 (21:46 +0000)]
feat(bkgpslog):Add additional validation for age pubkeys
Steven Baltakatei Sandoval [Sat, 4 Jul 2020 00:32:15 +0000 (00:32 +0000)]
chore(bkgpslog):Confirm age can encrypt to ssh pubkey string
Ssh pubkey strings have a space in them which was breaking
appendFileTar() function. 'eval' was used to fix how appendFileTar()
and appendArgTar() execute command strings.
Steven Baltakatei Sandoval [Sat, 4 Jul 2020 00:22:45 +0000 (00:22 +0000)]
fix(bkgpslog):Use eval to build command strings
Use eval to build command strings instead of simply putting bare
unquoted variable. This is necessary because the `recipients` variable
used to build a command to encrypt files may contain quotes if one of
the recipients is an ssh key which may have a space:
ssh-rsa AAAAB...
This should fix the inability of `bkgpslog` to accept quoted ssh
pubkey strings as a specified `-r` option argument.
Steven Baltakatei Sandoval [Fri, 3 Jul 2020 22:54:50 +0000 (22:54 +0000)]
test(bkgpslog):Test recipient string quoting
recipient strings with spaces in them don't fare well with `\"` in
them when such strings are used to create new strings.
Steven Baltakatei Sandoval [Fri, 3 Jul 2020 22:43:05 +0000 (22:43 +0000)]
fix(bkgpslog):Fix pubkey strings with spaces
ssh pubkeys have a space in them and caused age encryption operation
to fail.
Steven Baltakatei Sandoval [Fri, 3 Jul 2020 22:02:17 +0000 (22:02 +0000)]
doc(bkgpslog):Add TODO for future features
- recipient watch folder
- simplify gps data
Steven Baltakatei Sandoval [Fri, 3 Jul 2020 20:58:01 +0000 (20:58 +0000)]
doc(bkgpslog-plan):Closed TODO item
Steven Baltakatei Sandoval [Fri, 3 Jul 2020 20:46:46 +0000 (20:46 +0000)]
style(bkgpslog):Minor semicolons
Steven Baltakatei Sandoval [Fri, 3 Jul 2020 20:38:01 +0000 (20:38 +0000)]
chore(bkgpslog):Minor comment cleanup
Steven Baltakatei Sandoval [Fri, 3 Jul 2020 20:31:53 +0000 (20:31 +0000)]
fix(bkgpslog):Simplify main loop further
Move `gpspipe` command itself into write function.
Steven Baltakatei Sandoval [Fri, 3 Jul 2020 20:12:59 +0000 (20:12 +0000)]
fix(bkgpslog):Add missing gpspipe option
Need `-o` to write to file.
Steven Baltakatei Sandoval [Fri, 3 Jul 2020 19:49:50 +0000 (19:49 +0000)]
fix(bkgpslog):Reduce logging downtime between buffer rounds
Reduce logging downtime between buffer rounds by moving commands into
the asynchronous `magicWriteBuffer()` function that is spun off
immediately after `gpspipe` has collected data.
Add functionality to `dateTimeShort()` and `dateShort` so the start
time of each saved chunk appended to the `tar` archive can be
calculated indirectly from when `magicWriteBuffer()` starts rather
than performing slow `date` and associated bash variable changes in
the main loop. This start time is needed to name each inserted file as
well as to identify which `tar` file to insert into.
Save gpspipe buffer data `/dev/shm` instead of a Bash variable.
Add `appendFileTar()` function to facilitate reading and processing of
buffer data from `/dev/shm`.
Steven Baltakatei Sandoval [Fri, 3 Jul 2020 17:16:28 +0000 (17:16 +0000)]
doc(bkgpslog):Add note to simplify main loop
Steven Baltakatei Sandoval [Thu, 2 Jul 2020 23:46:10 +0000 (23:46 +0000)]
fix(bkgpslog):Increase buffer period 60s to 5m to reduce write freq
Steven Baltakatei Sandoval [Thu, 2 Jul 2020 22:15:45 +0000 (22:15 +0000)]
fix(bkgpslog):Remove debug yells
Steven Baltakatei Sandoval [Thu, 2 Jul 2020 22:12:58 +0000 (22:12 +0000)]
style(bkgpslog):checkMakeTar():Add/remove debug code
Steven Baltakatei Sandoval [Thu, 2 Jul 2020 22:00:42 +0000 (22:00 +0000)]
test(bkgpslog):Bump version number to 0.3.0
Incompatible changes from 0.2:
- File writes asynchronous from main script forced to be sequential to
one another to avoid simultaneous writes to same file. Tar write code
consolidated.
- VERSION file format changed to include more fields. VERSION file
generation code cosolidated.
Steven Baltakatei Sandoval [Thu, 2 Jul 2020 21:55:50 +0000 (21:55 +0000)]
fix(bkgpslog):Clarify echoed newlines per Shellcheck
Changed echo "blah""\n""blah" into echo "blah""\\n""blah"
Steven Baltakatei Sandoval [Thu, 2 Jul 2020 21:54:26 +0000 (21:54 +0000)]
style(bkgpslog):Update timeDuration
Steven Baltakatei Sandoval [Thu, 2 Jul 2020 21:23:34 +0000 (21:23 +0000)]
feat(bkgpslog):Include more fields in VERSION
These fields are included to automatically-generated VERSION file
included when a new tar is generated or the script is restarted:
- SCRIPT_VERSION : Version of bkgpslog being run.
- SCRIPT_NAME : Name of the script being run ("bkgpslog").
- SCRIPT_URL : URL where script can be downloaded.
- AGE_VERSION : Version of age (encryption program) used.
- AGE_URL : URL where age can be downloaded.
- DATE : Creation timestamp (ISO-8601).
- HOSTNAME : Hostname of device running script.
- PUBKEY_[n] : Pubkey string used by age to encrypt.
Steven Baltakatei Sandoval [Thu, 2 Jul 2020 20:52:09 +0000 (20:52 +0000)]
chore(bkgpslog):bump to version 0.2.1 from 0.2.0
Steven Baltakatei Sandoval [Thu, 2 Jul 2020 20:51:06 +0000 (20:51 +0000)]
feat(bkgpslog):Change format in VERSION file
From one line to multiline
Steven Baltakatei Sandoval [Thu, 2 Jul 2020 20:22:37 +0000 (20:22 +0000)]
doc(bkgpslog):Update plan doc
Steven Baltakatei Sandoval [Thu, 2 Jul 2020 20:13:03 +0000 (20:13 +0000)]
feat(bkgpslog): Consolidate VERSION writing, update functions
Bump version to 0.2.0 from 0.1.0.
style(timeUntilNextDay()). Set as version 1.0.0. Lowercased
returnState variable messages in function return code handling.
fix(timeDuration()). Fix missing $ in bash variable. Set to Version
1.0.0. Removed unused boolean variables.
feat(magicWriteVersion()): Add function to consolidate how VERSION
file is generated; VERSION file can be generated at start or if output
tar goes missing.
feat(magicWriteBuffer()): Take checkMakeTar() exit status into account
in order to determine whether or not to rerun
magicWriteVersion(). Relevant if output tar archive goes missing.
Steven Baltakatei Sandoval [Thu, 2 Jul 2020 19:12:31 +0000 (19:12 +0000)]
feat(bkgpslog):Include exit codes in checkMakeTar
The change is necessary to decide if VERSION file must be added in
case output tar is deleted between writes.
Steven Baltakatei Sandoval [Thu, 2 Jul 2020 18:34:46 +0000 (18:34 +0000)]
feat(bkgpslog):Consolidate tar validation/creation
Consolidate how output tar file is validated (if it already exists)
and/or created. checkMakeTar() function added for this purpose.
Add checkMakeTar() in `magicWriteBuffer()` in case output tar file is
deleted between writes.
Steven Baltakatei Sandoval [Thu, 2 Jul 2020 16:05:04 +0000 (16:05 +0000)]
feat(bkgpslog):Force ordered write operations with wait
Use `wait` built-in to force each detached process that appends
location data to the output tar to wait until all other detached
processes complete. This will prevent the magicWriteBuffer() and
appendArgTar() functions from attempting to append to the same tar at
the same time, causing script failure.
Steven Baltakatei Sandoval [Thu, 2 Jul 2020 14:54:12 +0000 (14:54 +0000)]
debug(bkgpslog):Minor syntax formatting: add semicolons
Steven Baltakatei Sandoval [Thu, 2 Jul 2020 01:30:41 +0000 (01:30 +0000)]
fix(bkgpslog):Fix command strings in magicWriteBuffer
Steven Baltakatei Sandoval [Thu, 2 Jul 2020 01:24:30 +0000 (01:24 +0000)]
fix(bkgpslog):Move chunk cleanup into magicWriteBuffer func
Steven Baltakatei Sandoval [Thu, 2 Jul 2020 01:22:12 +0000 (01:22 +0000)]
fix(bkgpslog):Disable old tar append code
Steven Baltakatei Sandoval [Thu, 2 Jul 2020 01:19:24 +0000 (01:19 +0000)]
debug(bkgpslog):Add debug messages for functions
Steven Baltakatei Sandoval [Thu, 2 Jul 2020 01:10:02 +0000 (01:10 +0000)]
fix(bkgpslog):Arg index error in appendArgTar()
Steven Baltakatei Sandoval [Thu, 2 Jul 2020 01:08:10 +0000 (01:08 +0000)]
debug(bkgpslog):Add more debug data
Steven Baltakatei Sandoval [Thu, 2 Jul 2020 00:55:23 +0000 (00:55 +0000)]
feat(bkgpslog):Consolidate write ops into one function
Create metafunction 'magicWriteBuffer' to be triggered for
asynchronous appending of location files into tar file via
appendArgTar function (new) once buffer filled in main logging loop.
Create appendArgTar function to append a bash variable as a file to a
tar file.
Steven Baltakatei Sandoval [Wed, 1 Jul 2020 22:06:58 +0000 (22:06 +0000)]
debug(bkgpslog):testing debug of gpx kml output
Steven Baltakatei Sandoval [Wed, 1 Jul 2020 21:42:21 +0000 (21:42 +0000)]
fix(bkgpslog):Fix output files having the same names
Steven Baltakatei Sandoval [Wed, 1 Jul 2020 21:25:00 +0000 (21:25 +0000)]
feat(bkgpslog):Added debug message when main function completes
Steven Baltakatei Sandoval [Wed, 1 Jul 2020 21:23:28 +0000 (21:23 +0000)]
fix(bkgpslog):Clean up how tar appended, version file included
Steven Baltakatei Sandoval [Wed, 1 Jul 2020 21:18:02 +0000 (21:18 +0000)]
chore(bkgpslog):Add timestamp to version content.
Steven Baltakatei Sandoval [Wed, 1 Jul 2020 21:15:32 +0000 (21:15 +0000)]
chore(bkgpslog):Include version in appendtest
Steven Baltakatei Sandoval [Wed, 1 Jul 2020 21:07:54 +0000 (21:07 +0000)]
debug(bkgpslog):Clarified debug messages
Steven Baltakatei Sandoval [Wed, 1 Jul 2020 21:03:59 +0000 (21:03 +0000)]
fix(bkgpslog):Fix secure chunk cleanup code
Steven Baltakatei Sandoval [Wed, 1 Jul 2020 20:59:08 +0000 (20:59 +0000)]
debug(bkgpslog):Add more debug messages; clarified tar workdir
Steven Baltakatei Sandoval [Wed, 1 Jul 2020 20:51:42 +0000 (20:51 +0000)]
debug(bkgpslog):Add more verbose debug messages
Steven Baltakatei Sandoval [Wed, 1 Jul 2020 20:47:02 +0000 (20:47 +0000)]
debug(bkgpslog):Add more verbose debug code
Steven Baltakatei Sandoval [Wed, 1 Jul 2020 20:42:23 +0000 (20:42 +0000)]
debug(bkgpslog):Add additional debug for output file path vars
Steven Baltakatei Sandoval [Wed, 1 Jul 2020 20:29:22 +0000 (20:29 +0000)]
fix(bkgpslog):Minor formatting
Steven Baltakatei Sandoval [Wed, 1 Jul 2020 20:27:41 +0000 (20:27 +0000)]
feat(bkgpslog):Append to preexisting tar if possible
Steven Baltakatei Sandoval [Wed, 1 Jul 2020 20:04:26 +0000 (20:04 +0000)]
fix(bkgpslog):Add missing shifts in processArguments()
Steven Baltakatei Sandoval [Wed, 1 Jul 2020 20:02:09 +0000 (20:02 +0000)]
feat(bkgpslog):Add more verbose error message in processArguments()
Steven Baltakatei Sandoval [Wed, 1 Jul 2020 19:57:29 +0000 (19:57 +0000)]
fix(bkgpslog):Fix incorrectly selected argument
Steven Baltakatei Sandoval [Wed, 1 Jul 2020 19:51:23 +0000 (19:51 +0000)]
doc(bkgpslog): Update usage