From e879cdc33c40fb5b871ad7376094193925963e8c Mon Sep 17 00:00:00 2001 From: Steven Baltakatei Sandoval Date: Sun, 12 Jul 2020 21:35:16 +0000 Subject: [PATCH] doc(exec):Create coding plan for bklog, retire bkgpslog plan Create coding plan for `bklog` at `exec/bklog-plan.org`. Retire coding plan for `bkgpslog` at `exec/bkgpslog-plan.org`. --- exec/bkgpslog-plan.org | 34 ++++++++++++++++++++++--- exec/bklog-plan.org | 58 ++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 88 insertions(+), 4 deletions(-) create mode 100644 exec/bklog-plan.org diff --git a/exec/bkgpslog-plan.org b/exec/bkgpslog-plan.org index e100962..cc5c839 100644 --- a/exec/bkgpslog-plan.org +++ b/exec/bkgpslog-plan.org @@ -1,3 +1,6 @@ +2020-07-12T21:16Z; bktei> Note: This file is now retired since ~bklog~ +has replaced ~bkgpslog~. + * bkgpslog task list ** DONE Add job control for short buffer length CLOSED: [2020-07-02 Thu 16:04] @@ -60,7 +63,8 @@ done Raspberry Pi Zero W shows approximately 71ms of drift per buffer round with 10s buffer. -** TODO Feature: Recipient watch folder +** DONE Feature: Recipient watch folder + CLOSED: [2020-07-12 Sun 21:08] 2020-07-03T21:28Z; bktei> This feature would be to scan the contents of a specified directory at the start of every buffer round in order to determine encryption (age) recipients. This would allow a device to @@ -95,7 +99,11 @@ to watch. When examining the directory, check for a file with the appropriate file extension (ex: .pubkey) and then read the first line into the script's pubKey array. -** TODO Feature: Simplify option to reduce output size +2020-07-12T21:08Z; bktei> ~-R~ watch directory option added in ~bkgpslog~ ver +~0.4.0~. + +** DONE Feature: Simplify option to reduce output size + CLOSED: [2020-07-12 Sun 21:15] ~gpsbabel~ [[https://www.gpsbabel.org/htmldoc-development/filter_simplify.html][features]] a ~simplify~ option to trim data points from GPS data. There are several methods for prioritizing which points to keep @@ -126,7 +134,16 @@ about 450. 152K relerror-1000.kml #+END_EXAMPLE -** TODO Feature: Generalize bkgpslog to bklog function +2020-07-12T21:13Z; bktei> Instead of programming data simplification +in ~bkgpslog~, the data simplification step should be performed via +~bklog~'s ~-p~ option which specifies a processing command string to +be ~eval~'d before data is compressed, encrypted, and written to +disk. In other words, handling the simplification of data beyond +allowing for a general command string specified by ~-p~ is outside the +scope of ~bkgpslog~ or its successor ~bklog~. + +** DONE Feature: Generalize bkgpslog to bklog function + CLOSED: [2020-07-12 Sun 21:11] 2020-07-05T02:42Z; bktei> Transform ~bkgpslog~ into a modular component called ~bklog~ such that it processes a stdout stream of any external command, not just ~gpspipe -r~. This would permit reuse of @@ -160,7 +177,12 @@ unsure in my ability to program script that could run for long periods of time without causing a runaway usage of memory. I still think it's a good idea to offer a script TTL option to the user but I think the default should be to simply run forver. -** TODO: Evaluate ~rsyslog~ as stand-in for this work + +2020-07-12T21:11Z; bktei> ~bklog~ script created and tested as of +commit ~aedd19f~. + +** DONE TODO: Evaluate ~rsyslog~ as stand-in for this work + CLOSED: [2020-07-12 Sun 21:09] 2020-07-05T02:57Z; bktei> I searched for "debian iot logging" ("iot" as in "Internet of Things", the current buzzword for small low-power computers being used to provide microservices for owners in their own @@ -222,6 +244,10 @@ think converting ~bkgpslog~ into a ~bklog~ script that appends encrypted and compressed data to a tar file for later extraction (possibly the same script with future features) would be best. +2020-07-12T21:10Z; bktei> rsyslog is outside the scope of what +~bkgpslog~ does (record location observations). A different tool +should be used to retrieve and synchronize data. The dumb storage +method of "tar files in a syncthing folder" works for now. ** TODO: Place persistent recip. updates in asynchronous coproc 2020-07-06T19:37Z; bktei> In order to update the recipient list, the magicParseRecipientDir() function needs to be run each buffer period diff --git a/exec/bklog-plan.org b/exec/bklog-plan.org new file mode 100644 index 0000000..9908efb --- /dev/null +++ b/exec/bklog-plan.org @@ -0,0 +1,58 @@ +* bklog task list +** TODO Adjust filename duration dynamically +2020-07-12T21:17Z; bktei> Currently, the "duration" component of the +output filename for a given chunk is calculated from the ~bufferTTL~ +variable which does not necessarily reflect the amount of buffer lines +being read into a given chunk, especially since lines of stdin may be +received while synchronous processing occurs before a new +~magicProcessWriteBuffer &~ job is spun off. + +A better way to specify the duration is to maintain a pair of +timestamp varriables before each ~magicProcessWriteBuffer~ command is +run. Within the asynchronous write job the time difference between the +two time stamps may be evaluated in order to determine how much time +has passed since the last write operation started. The last line read +into the buffer when the ~while read~ loop finishes should be the most +current value and so one of the two timestamps should be recorded +then. The other time stamp should simply be the previous loop's +timestamp value. + +For example: + +#+BEGIN_EXAMPLE + # MAIN LOOP: Run until script TTL seconds pass + bufferRound=0; + while [[ $SECONDS -lt "scriptTTL" ]]; do + vbm "STATUS:$fn:Starting buffer round:$bufferRound"; + bufferTOD="$((SECONDS + bufferTTL))"; # Set buffer round time-of-death + # Consume stdin to fill buffer until buffer time-of-death (TOD) arrives + while read -r -t "$bufferTTL" line && [[ $SECONDS -lt "$bufferTOD" ]]; do + # Append line to buffer array + buffer+=("$line"); + done; + #====BEGIN NEW CODE HERE==== + bufferTimestampOld="$bufferTimestampNew"; + bufferTimeStampNew="$(date --iso-8601=seconds)"; + #====END NEW CODE HERE==== + # Create dir_tmp if missing + if ! [[ -d "$dir_tmp" ]]; then + yell "ERROR:$fn:dir_tmp existence failure:$dir_tmp"; + try mkdir "$dir_tmp" && vbm "DEBUG :$fn:Working dir recreated dir_tmp:$dir_tmp"; fi + # Update cmd_encrypt, cmd_encrypt_suffix + magicParseRecipients; + # Export buffer to asynchronous processing. + magicProcessWriteBuffer & + unset buffer; # Clear buffer array for next bufferRound + # Increment buffer round + ((bufferRound++)); + done; +#+END_EXAMPLE + +Then, within the ~magicProcessWriteBuffer()~ function, the difference +in seconds between ~bufferTimestampOld~ and ~bufferTimestampNew~ may +be calculated and an appropriate duration string generated from the +~timeDuration()~ function. + + +* bklog narrative + -- 2.30.2