From d5f5aa4e0f362dd76471b5440339ef249a5552aa Mon Sep 17 00:00:00 2001 From: Steven Baltakatei Sandoval Date: Mon, 13 Jul 2020 22:31:14 +0000 Subject: [PATCH 01/16] fix(bklog):Add 2nd try to checkMakeTar() if tar --list fails --- exec/bklog | 11 +++++++---- 1 file changed, 7 insertions(+), 4 deletions(-) diff --git a/exec/bklog b/exec/bklog index 0d24898..fbe28e0 100644 --- a/exec/bklog +++ b/exec/bklog @@ -11,7 +11,7 @@ dirTmpDefault="/dev/shm"; # Default parent of working directory # Script Metadata scriptName="bklog"; # Define basename of script file. -scriptVersion="0.1.23"; # Define version of script. +scriptVersion="0.1.24"; # Define version of script. scriptURL="https://gitlab.com/baltakatei/ninfacyzga-01"; # Define wesite hosting this script. scriptTimeStart="$(date +%Y%m%dT%H%M%S.%N)"; # YYYYmmddTHHMMSS.NNNNNNNNN scriptHostname=$(hostname); # Save hostname of system running this script. @@ -357,17 +357,20 @@ checkMakeTar() { if tar --list --file="$pathTar" 1>/dev/null 2>&1; then ## T1: return success returnFlag0="tar valid"; + elif { sleep 2; tar --list --file="$pathTar" 1>/dev/null 2>&1; }; then + ## F1: Check tar archive again after 2-second sleep + returnFlag0="tar valid"; else - ## F1: Check if file exists + ## F2-1: Check if file exists if [[ -f "$pathTar" ]]; then ### T: Rename file - try mv "$pathTar" "$pathTar""--broken--""$(date +%Y%m%dT%H%M%S)" && \ + try mv "$pathTar" "$pathTar""--broken--""$(date +%Y%m%dT%H%M%S%z)" && \ returnFlag1="tar moved"; else ### F: - : fi; - ## F2: Create tar archive, return 0 + ## F2-1: Create tar archive, return 0 try tar --create --file="$pathTar" --files-from=/dev/null && \ returnFlag2="tar created"; fi; -- 2.30.2 From 89c5dead8a4eaba8408cdc6ad8c16a5ee9e02585 Mon Sep 17 00:00:00 2001 From: Steven Baltakatei Sandoval Date: Tue, 14 Jul 2020 01:03:27 +0000 Subject: [PATCH 02/16] doc(bklog-plan):Add TODO task: correct drift of SECONDS --- exec/bklog-plan.org | 13 +++++++++++++ 1 file changed, 13 insertions(+) diff --git a/exec/bklog-plan.org b/exec/bklog-plan.org index 9908efb..9e5e54d 100644 --- a/exec/bklog-plan.org +++ b/exec/bklog-plan.org @@ -53,6 +53,19 @@ in seconds between ~bufferTimestampOld~ and ~bufferTimestampNew~ may be calculated and an appropriate duration string generated from the ~timeDuration()~ function. +** TODO Update ~SECONDS~ variable during while read loop +2020-07-14T00:58Z; bktei> The starting timestamps of each output file +still drifts against system time. Although the ~while read~ loop does +not lose data, the drift causes the output files to be named weirdly. +A possible solution is to correct the ~SECONDS~ variable against the +current system time. Because ~date~ calls are slow, this correction +should not be made after every line. At a minimum, the correction +should occur once per buffer round, possibly after the buffer round +has completed. If more frequent corrections are required, then the +number of lines being read in each buffer round should be tracked and +a modulus comparison may be implemented within the ~while read~ loop +so that a correction is made after some fraction of the expected lines +to be read are read. * bklog narrative -- 2.30.2 From f5024030c11cd53a1cfdb067f128e3047dff34a2 Mon Sep 17 00:00:00 2001 From: Steven Baltakatei Sandoval Date: Tue, 14 Jul 2020 02:08:49 +0000 Subject: [PATCH 03/16] fix(bklog):Fix buffer drift via SECONDS ==Change 1== Fix drift in when bklog starts reading data via `while read` by enforcing buffer termination when SECONDS (script variable that increments every second) rises above a number determined by the current bufferRound number and bufferTTL. ==Change 2== Also initially advance SECONDS so it aligns with the remainder of dividing the current seconds remaining in the day by bufferTTL. The result should be that, if the user specifies a bufferTTL value that is a clean fraction of a time element (ex: a bufferTTL of 5 minutes = 300 seconds), then each buffer round should start at a clean fraction of absolute time with respect to the day (ex: 02:05, 02:10, 02:15, etc.). This may assist searching through saved files since instead of files named: 20200714T000019+0000--PT5M0S..hostname_location.nmea.gz.age 20200714T000521+0000--PT5M0S..hostname_location.nmea.gz.age 20200714T001023+0000--PT5M0S..hostname_location.nmea.gz.age 20200714T001525+0000--PT5M0S..hostname_location.nmea.gz.age 20200714T002027+0000--PT5M0S..hostname_location.nmea.gz.age 20200714T002529+0000--PT5M0S..hostname_location.nmea.gz.age 20200714T003031+0000--PT5M0S..hostname_location.nmea.gz.age 20200714T003533+0000--PT5M0S..hostname_location.nmea.gz.age they will be named something more like: 20200714T000000+0000--PT5M0S..hostname_location.nmea.gz.age 20200714T000500+0000--PT5M0S..hostname_location.nmea.gz.age 20200714T001000+0000--PT5M0S..hostname_location.nmea.gz.age 20200714T001500+0000--PT5M0S..hostname_location.nmea.gz.age 20200714T002000+0000--PT5M0S..hostname_location.nmea.gz.age 20200714T002500+0000--PT5M0S..hostname_location.nmea.gz.age 20200714T003000+0000--PT5M0S..hostname_location.nmea.gz.age 20200714T003500+0000--PT5M0S..hostname_location.nmea.gz.age ==Change 3== Fix how the ISO-8601 duration string is calculated. Instead of using a fixed value of `bufferTTL` seconds, record the epoch start and end times (in seconds since 1970-01-01) of the main data gathering loop and then calculate the difference. Now, actual deviations in buffer gathering durations (ex: the first buffer round will likely be shorter than a normal one) will be accurately represented. --- exec/bklog | 33 ++++++++++++++++++++++++--------- 1 file changed, 24 insertions(+), 9 deletions(-) diff --git a/exec/bklog b/exec/bklog index fbe28e0..9b7fd67 100644 --- a/exec/bklog +++ b/exec/bklog @@ -11,7 +11,7 @@ dirTmpDefault="/dev/shm"; # Default parent of working directory # Script Metadata scriptName="bklog"; # Define basename of script file. -scriptVersion="0.1.24"; # Define version of script. +scriptVersion="0.1.25"; # Define version of script. scriptURL="https://gitlab.com/baltakatei/ninfacyzga-01"; # Define wesite hosting this script. scriptTimeStart="$(date +%Y%m%dT%H%M%S.%N)"; # YYYYmmddTHHMMSS.NNNNNNNNN scriptHostname=$(hostname); # Save hostname of system running this script. @@ -1218,14 +1218,15 @@ magicWriteVersion() { } # write version data to pathout_tar via appendArgTar() magicProcessWriteBuffer() { # Desc: process and write buffer - # In : vars: bufferTTL bufferTTL_STR scriptHostname label dir_tmp SECONDS + # In : vars: bufferTTL scriptHostname label dir_tmp SECONDS + # : vars: timeBufferStartEpoch timeBufferEndEpoch # : arry: buffer # Out: file:(pathout_tar) # Depends: Bash 5.0.3, date 8.30, yell(), vbm(), dateTimeShort(), ### Note: These arrays should all have the same number of elements: ### pathouts, fileouts, procFileExts, procStrings - local fn timeBufferStartLong timeBufferStart fileoutBasename + local fn timeBufferStartLong timeBufferStart bufferDuration bufferDurationStr fileoutBasename local -a fileouts pathouts local writeCmd1 writeCmd2 writeCmd3 writeCmd4 @@ -1239,12 +1240,17 @@ magicProcessWriteBuffer() { # Determine file paths (time is start of buffer period) ## Calculate start time - timeBufferStartLong="$(date --date="$bufferTTL seconds ago" --iso-8601=seconds)" && \ - vbm "DEBUG :$fn:timeBufferStartLong:$timeBufferStartLong"; + timeBufferStartLong="$(date --date="@$timeBufferStartEpoch" --iso-8601=seconds)" && \ + vbm "DEBUG :$fn:timeBufferStartLong:$timeBufferStartLong"; # Note start time in 'date' parsable ISO-8601 timeBufferStart="$(dateTimeShort "$timeBufferStartLong" )" && \ vbm "DEBUG :$fn:timeBufferStart:$timeBufferStart"; # Note start time YYYYmmddTHHMMSS+zzzz (no separators) + ## Calculate buffer duration string (ISO-8601 duration) + bufferDuration="$((timeBufferEndEpoch - timeBufferStartEpoch))" && \ + vbm "DEBUG :$fn:bufferDuration:$bufferDuration"; # length of time (seconds) stdin was read + bufferDurationStr="$(timeDuration "$bufferDuration")" && \ + vbm "DEBUG :$fn:bufferDurationStr:$bufferDurationStr"; # buffer duration (ISO-8601) ## Set common basename - fileoutBasename="$timeBufferStart""--""$bufferTTL_STR""..""$scriptHostname""$label" && \ + fileoutBasename="$timeBufferStart""--""$bufferDurationStr""..""$scriptHostname""$label" && \ vbm "STATUS:$fn:Set fileoutBasename to:$fileoutBasename"; ## Determine output file name array ### in: fileOutBasename cmd_compress_suffix cmd_encrypt_suffix procFileExts @@ -1400,8 +1406,13 @@ main() { # Perform secondary setup operations ## Set script lifespan (scriptTTL from scriptTTL_TE) magicSetScriptTTL "$scriptTTL_TE"; - ## File name substring (ISO-8601 duration from bufferTTL) - bufferTTL_STR="$(timeDuration "$bufferTTL")" && vbm "DEBUG :$fn:bufferTTL_STR:$bufferTTL_STR"; + ## Adjust SECONDS so buffer rounds align with time elements + ### Advance SECONDS the remainder seconds for dividend timeUntilNextDay, divisor bufferTTL + if [[ "$(timeUntilNextDay)" -gt "$bufferTTL" ]]; then + vbm "DEBUG :$fn:SECONDS currently :$SECONDS"; + SECONDS="$(( $(timeUntilNextDay) % bufferTTL ))" && \ + vbm "DEBUG :$fn:SECONDS advanced to:$SECONDS"; + fi; ## Init temp working dir try mkdir "$dir_tmp" && vbm "DEBUG :$fn:Working dir created at dir_tmp:$dir_tmp"; ## Initialize output tar (set pathout_tar) @@ -1418,12 +1429,16 @@ main() { bufferRound=0; while [[ $SECONDS -lt "scriptTTL" ]]; do vbm "STATUS:$fn:Starting buffer round:$bufferRound"; - bufferTOD="$((SECONDS + bufferTTL))"; # Set buffer round time-of-death + bufferTOD="$(( (1+bufferRound)*bufferTTL ))"; # Set buffer round time-of-death + # Note start time of data collection + timeBufferStartEpoch="$(date +%s)"; # Consume stdin to fill buffer until buffer time-of-death (TOD) arrives while read -r -t "$bufferTTL" line && [[ $SECONDS -lt "$bufferTOD" ]]; do # Append line to buffer array buffer+=("$line"); done; + # Note end time of data collection + timeBufferEndEpoch="$(date +%s)"; # Create dir_tmp if missing if ! [[ -d "$dir_tmp" ]]; then yell "ERROR:$fn:dir_tmp existence failure:$dir_tmp"; -- 2.30.2 From be31cb91af743e9586026185f6746f12f25d3045 Mon Sep 17 00:00:00 2001 From: Steven Baltakatei Sandoval Date: Tue, 14 Jul 2020 02:31:40 +0000 Subject: [PATCH 04/16] debug(bklog):Add debug messages bklog is malfunctioning. Adding debug messages. --- exec/bklog | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/exec/bklog b/exec/bklog index 9b7fd67..584c22c 100644 --- a/exec/bklog +++ b/exec/bklog @@ -1429,16 +1429,16 @@ main() { bufferRound=0; while [[ $SECONDS -lt "scriptTTL" ]]; do vbm "STATUS:$fn:Starting buffer round:$bufferRound"; - bufferTOD="$(( (1+bufferRound)*bufferTTL ))"; # Set buffer round time-of-death + bufferTOD="$(( (1+bufferRound)*bufferTTL ))" && vbm "DEBUG :$fn:bufferTOD:$bufferTOD"; # Set buffer round time-of-death # Note start time of data collection - timeBufferStartEpoch="$(date +%s)"; + timeBufferStartEpoch="$(date +%s)" && vbm "DEBUG :$fn:timeBufferStartEpoch:$timeBufferStartEpoch"; # Consume stdin to fill buffer until buffer time-of-death (TOD) arrives while read -r -t "$bufferTTL" line && [[ $SECONDS -lt "$bufferTOD" ]]; do # Append line to buffer array buffer+=("$line"); done; # Note end time of data collection - timeBufferEndEpoch="$(date +%s)"; + timeBufferEndEpoch="$(date +%s)" && vbm "DEBUG :$fn:timeBufferEndEpoch:$timeBufferEndEpoch"; # Create dir_tmp if missing if ! [[ -d "$dir_tmp" ]]; then yell "ERROR:$fn:dir_tmp existence failure:$dir_tmp"; -- 2.30.2 From 99e199b255630ca54defe2abbc245bd9e259759c Mon Sep 17 00:00:00 2001 From: Steven Baltakatei Sandoval Date: Tue, 14 Jul 2020 02:39:33 +0000 Subject: [PATCH 05/16] debug(bklog):Add minor debug msg --- exec/bklog | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/exec/bklog b/exec/bklog index 584c22c..188910d 100644 --- a/exec/bklog +++ b/exec/bklog @@ -11,7 +11,7 @@ dirTmpDefault="/dev/shm"; # Default parent of working directory # Script Metadata scriptName="bklog"; # Define basename of script file. -scriptVersion="0.1.25"; # Define version of script. +scriptVersion="0.1.26"; # Define version of script. scriptURL="https://gitlab.com/baltakatei/ninfacyzga-01"; # Define wesite hosting this script. scriptTimeStart="$(date +%Y%m%dT%H%M%S.%N)"; # YYYYmmddTHHMMSS.NNNNNNNNN scriptHostname=$(hostname); # Save hostname of system running this script. @@ -1412,6 +1412,7 @@ main() { vbm "DEBUG :$fn:SECONDS currently :$SECONDS"; SECONDS="$(( $(timeUntilNextDay) % bufferTTL ))" && \ vbm "DEBUG :$fn:SECONDS advanced to:$SECONDS"; + vbm "DEBUG :$fn:current time:$(date --iso-8601=seconds)"; fi; ## Init temp working dir try mkdir "$dir_tmp" && vbm "DEBUG :$fn:Working dir created at dir_tmp:$dir_tmp"; -- 2.30.2 From a4823900b419f0fd59ee64c791243470f88d5cb8 Mon Sep 17 00:00:00 2001 From: Steven Baltakatei Sandoval Date: Tue, 14 Jul 2020 02:44:05 +0000 Subject: [PATCH 06/16] debug(bklog):Fix SECONDS advancing --- exec/bklog | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/exec/bklog b/exec/bklog index 188910d..45e6996 100644 --- a/exec/bklog +++ b/exec/bklog @@ -11,7 +11,7 @@ dirTmpDefault="/dev/shm"; # Default parent of working directory # Script Metadata scriptName="bklog"; # Define basename of script file. -scriptVersion="0.1.26"; # Define version of script. +scriptVersion="0.1.27"; # Define version of script. scriptURL="https://gitlab.com/baltakatei/ninfacyzga-01"; # Define wesite hosting this script. scriptTimeStart="$(date +%Y%m%dT%H%M%S.%N)"; # YYYYmmddTHHMMSS.NNNNNNNNN scriptHostname=$(hostname); # Save hostname of system running this script. @@ -1410,7 +1410,7 @@ main() { ### Advance SECONDS the remainder seconds for dividend timeUntilNextDay, divisor bufferTTL if [[ "$(timeUntilNextDay)" -gt "$bufferTTL" ]]; then vbm "DEBUG :$fn:SECONDS currently :$SECONDS"; - SECONDS="$(( $(timeUntilNextDay) % bufferTTL ))" && \ + SECONDS="$(( bufferTTL - ($(timeUntilNextDay) % bufferTTL) ))" && \ vbm "DEBUG :$fn:SECONDS advanced to:$SECONDS"; vbm "DEBUG :$fn:current time:$(date --iso-8601=seconds)"; fi; -- 2.30.2 From ceb6d15e2f20de620fcbd6c2ddbca5eaeb914296 Mon Sep 17 00:00:00 2001 From: Steven Baltakatei Sandoval Date: Tue, 14 Jul 2020 03:08:18 +0000 Subject: [PATCH 07/16] doc(bklog-plan):Add TODO item:account for input script failure --- exec/bklog-plan.org | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/exec/bklog-plan.org b/exec/bklog-plan.org index 9e5e54d..597f653 100644 --- a/exec/bklog-plan.org +++ b/exec/bklog-plan.org @@ -67,5 +67,11 @@ number of lines being read in each buffer round should be tracked and a modulus comparison may be implemented within the ~while read~ loop so that a correction is made after some fraction of the expected lines to be read are read. +** TODO Account for "early-exit" bug in input script +2020-07-14T03:00Z; bktei> What happens if the script piping its stdout +into ~bklog~ immediately exits without providing any stdout (ex: a +python script with a missing module)? ~bklog~ should be able to detect +the latest exit code and exit early. It should also be able to detect +if the incoming pipe is closed. * bklog narrative -- 2.30.2 From cf0947a89e3b69264c0dd3c2727657c94cadac08 Mon Sep 17 00:00:00 2001 From: Steven Baltakatei Sandoval Date: Tue, 14 Jul 2020 03:23:37 +0000 Subject: [PATCH 08/16] fix(bklog):timeDuration():Fix duration 0 error. Duration of "0" was erroneously being converted to "P" instead of "PT0S". --- exec/bklog | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/exec/bklog b/exec/bklog index 45e6996..62c0901 100644 --- a/exec/bklog +++ b/exec/bklog @@ -495,7 +495,7 @@ timeDuration(){ # Ref/Attrib: ISO-8601:2004(E), §4.4.4.2 Representations of time intervals by duration and context information # Note: "1 month" ("P1M") is assumed to be "30 days" (see ISO-8601:2004(E), §2.2.1.2) # Usage: timeDuration [1:seconds] ([2:precision]) - # Version: 1.0.4 + # Version: 1.0.5 # Input: arg1: seconds as base 10 integer >= 0 (ex: 3601) # arg2: precision level (optional; default=2) # Output: stdout: ISO-8601 duration string (ex: "P1H1S", "P2Y10M15DT10H30M20S") @@ -571,7 +571,7 @@ timeDuration(){ if [[ $fullDays -gt 0 ]]; then hasDays="true"; else hasDays="false"; fi if [[ $fullHours -gt 0 ]]; then hasHours="true"; else hasHours="false"; fi if [[ $fullMinutes -gt 0 ]]; then hasMinutes="true"; else hasMinutes="false"; fi - if [[ $fullSeconds -gt 0 ]]; then hasSeconds="true"; else hasSeconds="false"; fi + if [[ $fullSeconds -ge 0 ]]; then hasSeconds="true"; else hasSeconds="false"; fi ## Determine which fields to display (see ISO-8601:2004 §4.4.3.2) witherPrecision="false" -- 2.30.2 From b75f454fc443c1c1f51cc27ee8a4db136e6a1e72 Mon Sep 17 00:00:00 2001 From: Steven Baltakatei Sandoval Date: Tue, 14 Jul 2020 16:58:48 +0000 Subject: [PATCH 09/16] fix(bklog):Fix gap in duration times Gap caused by not accounting for synchronous calcs between timestamps marking duration. Using method planned in `bklog-plan.org`. --- exec/bklog | 9 ++++----- 1 file changed, 4 insertions(+), 5 deletions(-) diff --git a/exec/bklog b/exec/bklog index 62c0901..2cadbc5 100644 --- a/exec/bklog +++ b/exec/bklog @@ -11,7 +11,7 @@ dirTmpDefault="/dev/shm"; # Default parent of working directory # Script Metadata scriptName="bklog"; # Define basename of script file. -scriptVersion="0.1.27"; # Define version of script. +scriptVersion="0.1.28"; # Define version of script. scriptURL="https://gitlab.com/baltakatei/ninfacyzga-01"; # Define wesite hosting this script. scriptTimeStart="$(date +%Y%m%dT%H%M%S.%N)"; # YYYYmmddTHHMMSS.NNNNNNNNN scriptHostname=$(hostname); # Save hostname of system running this script. @@ -1431,15 +1431,14 @@ main() { while [[ $SECONDS -lt "scriptTTL" ]]; do vbm "STATUS:$fn:Starting buffer round:$bufferRound"; bufferTOD="$(( (1+bufferRound)*bufferTTL ))" && vbm "DEBUG :$fn:bufferTOD:$bufferTOD"; # Set buffer round time-of-death - # Note start time of data collection - timeBufferStartEpoch="$(date +%s)" && vbm "DEBUG :$fn:timeBufferStartEpoch:$timeBufferStartEpoch"; # Consume stdin to fill buffer until buffer time-of-death (TOD) arrives while read -r -t "$bufferTTL" line && [[ $SECONDS -lt "$bufferTOD" ]]; do # Append line to buffer array buffer+=("$line"); done; - # Note end time of data collection - timeBufferEndEpoch="$(date +%s)" && vbm "DEBUG :$fn:timeBufferEndEpoch:$timeBufferEndEpoch"; + # Mark time for buffer + timeBufferEndEpoch="$timeBufferStartEpoch"; + timeBufferStartEpoch="$(date +%s)" && vbm "DEBUG :$fn:timeBufferStartEpoch"; # Create dir_tmp if missing if ! [[ -d "$dir_tmp" ]]; then yell "ERROR:$fn:dir_tmp existence failure:$dir_tmp"; -- 2.30.2 From 7ce2010f5c738f11945efb2ec6c7573b5ca9a78f Mon Sep 17 00:00:00 2001 From: Steven Baltakatei Sandoval Date: Tue, 14 Jul 2020 17:20:42 +0000 Subject: [PATCH 10/16] debug(bklog):Debugging duration code --- exec/bklog | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/exec/bklog b/exec/bklog index 2cadbc5..819af4b 100644 --- a/exec/bklog +++ b/exec/bklog @@ -11,7 +11,7 @@ dirTmpDefault="/dev/shm"; # Default parent of working directory # Script Metadata scriptName="bklog"; # Define basename of script file. -scriptVersion="0.1.28"; # Define version of script. +scriptVersion="0.1.28-test1"; # Define version of script. scriptURL="https://gitlab.com/baltakatei/ninfacyzga-01"; # Define wesite hosting this script. scriptTimeStart="$(date +%Y%m%dT%H%M%S.%N)"; # YYYYmmddTHHMMSS.NNNNNNNNN scriptHostname=$(hostname); # Save hostname of system running this script. @@ -1437,7 +1437,7 @@ main() { buffer+=("$line"); done; # Mark time for buffer - timeBufferEndEpoch="$timeBufferStartEpoch"; + timeBufferEndEpoch="$timeBufferStartEpoch" && vbm "DEBUG :$fn:timeBufferEndEpoch"; timeBufferStartEpoch="$(date +%s)" && vbm "DEBUG :$fn:timeBufferStartEpoch"; # Create dir_tmp if missing if ! [[ -d "$dir_tmp" ]]; then -- 2.30.2 From 3b1b51216d530ed47ed84aabb2437ffbd18ea82e Mon Sep 17 00:00:00 2001 From: Steven Baltakatei Sandoval Date: Tue, 14 Jul 2020 17:23:40 +0000 Subject: [PATCH 11/16] debug(bklog):Testing duration code --- exec/bklog | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/exec/bklog b/exec/bklog index 819af4b..68d81ea 100644 --- a/exec/bklog +++ b/exec/bklog @@ -11,7 +11,7 @@ dirTmpDefault="/dev/shm"; # Default parent of working directory # Script Metadata scriptName="bklog"; # Define basename of script file. -scriptVersion="0.1.28-test1"; # Define version of script. +scriptVersion="0.1.28-test2"; # Define version of script. scriptURL="https://gitlab.com/baltakatei/ninfacyzga-01"; # Define wesite hosting this script. scriptTimeStart="$(date +%Y%m%dT%H%M%S.%N)"; # YYYYmmddTHHMMSS.NNNNNNNNN scriptHostname=$(hostname); # Save hostname of system running this script. @@ -1437,8 +1437,8 @@ main() { buffer+=("$line"); done; # Mark time for buffer - timeBufferEndEpoch="$timeBufferStartEpoch" && vbm "DEBUG :$fn:timeBufferEndEpoch"; - timeBufferStartEpoch="$(date +%s)" && vbm "DEBUG :$fn:timeBufferStartEpoch"; + timeBufferEndEpoch="$timeBufferStartEpoch" && vbm "DEBUG :$fn:timeBufferEndEpoch:$timeBufferEndEpoch"; + timeBufferStartEpoch="$(date +%s)" && vbm "DEBUG :$fn:timeBufferStartEpoch:$timeBufferStartEpoch"; # Create dir_tmp if missing if ! [[ -d "$dir_tmp" ]]; then yell "ERROR:$fn:dir_tmp existence failure:$dir_tmp"; -- 2.30.2 From 7814d1643b916c04bafd9c728027b421fe7b6020 Mon Sep 17 00:00:00 2001 From: Steven Baltakatei Sandoval Date: Tue, 14 Jul 2020 19:40:03 +0000 Subject: [PATCH 12/16] fix(bklog):Fix flip-flopped buffer start/end timestamps --- exec/bklog | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/exec/bklog b/exec/bklog index 68d81ea..b4c8257 100644 --- a/exec/bklog +++ b/exec/bklog @@ -11,7 +11,7 @@ dirTmpDefault="/dev/shm"; # Default parent of working directory # Script Metadata scriptName="bklog"; # Define basename of script file. -scriptVersion="0.1.28-test2"; # Define version of script. +scriptVersion="0.1.28-test3"; # Define version of script. scriptURL="https://gitlab.com/baltakatei/ninfacyzga-01"; # Define wesite hosting this script. scriptTimeStart="$(date +%Y%m%dT%H%M%S.%N)"; # YYYYmmddTHHMMSS.NNNNNNNNN scriptHostname=$(hostname); # Save hostname of system running this script. @@ -1437,8 +1437,8 @@ main() { buffer+=("$line"); done; # Mark time for buffer - timeBufferEndEpoch="$timeBufferStartEpoch" && vbm "DEBUG :$fn:timeBufferEndEpoch:$timeBufferEndEpoch"; - timeBufferStartEpoch="$(date +%s)" && vbm "DEBUG :$fn:timeBufferStartEpoch:$timeBufferStartEpoch"; + timeBufferStartEpoch="$timeBufferEndEpoch" && vbm "DEBUG :$fn:timeBufferStartEpoch:$timeBufferStartEpoch"; + timeBufferEndEpoch="$(date +%s)" && vbm "DEBUG :$fn:timeBufferEndEpoch:$timeBufferEndEpoch"; # Create dir_tmp if missing if ! [[ -d "$dir_tmp" ]]; then yell "ERROR:$fn:dir_tmp existence failure:$dir_tmp"; -- 2.30.2 From 52fd74fc6af6a5c39927d7121b2845915dff9ad4 Mon Sep 17 00:00:00 2001 From: Steven Baltakatei Sandoval Date: Tue, 14 Jul 2020 19:47:38 +0000 Subject: [PATCH 13/16] fix(bklog):Fix initial edge case bug in duration time --- exec/bklog | 13 +++++++++++-- 1 file changed, 11 insertions(+), 2 deletions(-) diff --git a/exec/bklog b/exec/bklog index b4c8257..a99468a 100644 --- a/exec/bklog +++ b/exec/bklog @@ -11,8 +11,9 @@ dirTmpDefault="/dev/shm"; # Default parent of working directory # Script Metadata scriptName="bklog"; # Define basename of script file. -scriptVersion="0.1.28-test3"; # Define version of script. +scriptVersion="0.1.28-test4"; # Define version of script. scriptURL="https://gitlab.com/baltakatei/ninfacyzga-01"; # Define wesite hosting this script. +scriptTimeStartEpoch="$(date +%s)"; # Save start time of script in epoch seconds scriptTimeStart="$(date +%Y%m%dT%H%M%S.%N)"; # YYYYmmddTHHMMSS.NNNNNNNNN scriptHostname=$(hostname); # Save hostname of system running this script. PATH="$HOME/.local/bin:$PATH"; # Add "$(systemd-path user-binaries)" path in case user apps saved there @@ -1437,7 +1438,15 @@ main() { buffer+=("$line"); done; # Mark time for buffer - timeBufferStartEpoch="$timeBufferEndEpoch" && vbm "DEBUG :$fn:timeBufferStartEpoch:$timeBufferStartEpoch"; + ## Initial time + if [[ ! -z "$timeBufferEndEpoch" ]]; then + ### Usual case + timeBufferStartEpoch="$timeBufferEndEpoch" && vbm "DEBUG :$fn:timeBufferStartEpoch:$timeBufferStartEpoch"; + else + ### Edge case: initial startup + timeBufferStartEpoch="$scriptTimeStartEpoch" && vbm "DEBUG :$fn:timeBufferStartEpoch:$timeBufferStartEpoch"; + fi; + ## End Time timeBufferEndEpoch="$(date +%s)" && vbm "DEBUG :$fn:timeBufferEndEpoch:$timeBufferEndEpoch"; # Create dir_tmp if missing if ! [[ -d "$dir_tmp" ]]; then -- 2.30.2 From b773158ab3376261c6b132f04d2d3bff9644be8a Mon Sep 17 00:00:00 2001 From: Steven Baltakatei Sandoval Date: Tue, 14 Jul 2020 22:27:50 +0000 Subject: [PATCH 14/16] fix(bklog):Update plan, bump version (bklog working) Complete fix for `bklog` duration string bug. --- exec/bklog | 2 +- exec/bklog-plan.org | 28 +++++++++++++++++++++------- 2 files changed, 22 insertions(+), 8 deletions(-) diff --git a/exec/bklog b/exec/bklog index a99468a..504ecbc 100644 --- a/exec/bklog +++ b/exec/bklog @@ -11,7 +11,7 @@ dirTmpDefault="/dev/shm"; # Default parent of working directory # Script Metadata scriptName="bklog"; # Define basename of script file. -scriptVersion="0.1.28-test4"; # Define version of script. +scriptVersion="0.1.29"; # Define version of script. scriptURL="https://gitlab.com/baltakatei/ninfacyzga-01"; # Define wesite hosting this script. scriptTimeStartEpoch="$(date +%s)"; # Save start time of script in epoch seconds scriptTimeStart="$(date +%Y%m%dT%H%M%S.%N)"; # YYYYmmddTHHMMSS.NNNNNNNNN diff --git a/exec/bklog-plan.org b/exec/bklog-plan.org index 597f653..36dd08b 100644 --- a/exec/bklog-plan.org +++ b/exec/bklog-plan.org @@ -1,5 +1,6 @@ * bklog task list -** TODO Adjust filename duration dynamically +** DONE Adjust filename duration dynamically + CLOSED: [2020-07-14 Tue 22:17] 2020-07-12T21:17Z; bktei> Currently, the "duration" component of the output filename for a given chunk is calculated from the ~bufferTTL~ variable which does not necessarily reflect the amount of buffer lines @@ -53,7 +54,11 @@ in seconds between ~bufferTimestampOld~ and ~bufferTimestampNew~ may be calculated and an appropriate duration string generated from the ~timeDuration()~ function. -** TODO Update ~SECONDS~ variable during while read loop +2020-07-14T22:17:16Z; bktei> Initial adjustment of SECONDS +implemented. Ongoing monitoring of end time of each buffer round +~while read~ loop checked. +** DONE Update ~SECONDS~ variable during while read loop + CLOSED: [2020-07-14 Tue 16:22] 2020-07-14T00:58Z; bktei> The starting timestamps of each output file still drifts against system time. Although the ~while read~ loop does not lose data, the drift causes the output files to be named weirdly. @@ -67,11 +72,20 @@ number of lines being read in each buffer round should be tracked and a modulus comparison may be implemented within the ~while read~ loop so that a correction is made after some fraction of the expected lines to be read are read. + +2020-07-14T16:21Z; bktei> I ran a test to see if SECONDS drifts and it +does not. The lag is caused by other synchronous commands. The +solution will be to adjust the variables against which SECONDS is +compared. ** TODO Account for "early-exit" bug in input script -2020-07-14T03:00Z; bktei> What happens if the script piping its stdout -into ~bklog~ immediately exits without providing any stdout (ex: a -python script with a missing module)? ~bklog~ should be able to detect -the latest exit code and exit early. It should also be able to detect -if the incoming pipe is closed. +*** 2020-07-14T03:00Z; bktei> +What happens if the script piping its stdout into ~bklog~ immediately +exits without providing any stdout (ex: a python script with a missing +module)? ~bklog~ should be able to detect the latest exit code and +exit early. It should also be able to detect if the incoming pipe is +closed. + +*** 2020-07-14T22:25Z; bktei> +Possible solution using ~dd~, ~od~, and ~if [ -z string ]~ [[https://unix.stackexchange.com/a/33055][here]]. * bklog narrative -- 2.30.2 From 0301313aef3a20aca68f23e146fc514a3ae9710f Mon Sep 17 00:00:00 2001 From: Steven Baltakatei Sandoval Date: Wed, 15 Jul 2020 23:51:21 +0000 Subject: [PATCH 15/16] doc(bklog-plan):Note possible future feature: user config file --- exec/bklog-plan.org | 8 ++++++++ 1 file changed, 8 insertions(+) diff --git a/exec/bklog-plan.org b/exec/bklog-plan.org index 36dd08b..9afaea9 100644 --- a/exec/bklog-plan.org +++ b/exec/bklog-plan.org @@ -87,5 +87,13 @@ closed. *** 2020-07-14T22:25Z; bktei> Possible solution using ~dd~, ~od~, and ~if [ -z string ]~ [[https://unix.stackexchange.com/a/33055][here]]. +** TODO Support configuration file for storing options +*** 2020-07-15T23:48Z;bktei> +Support storing options in a ~$HOME/.config/bklog/options.conf~ file +so individual calls to ~bklog~ don't have to be so long. Aliases can +be used instead to make ~bklog~ calls shorter, but cron jobs don't +necessarily source alias files. Explicitly reading options from a +configuration file would be more reliable, although it complicates +usage of ~bklog~. * bklog narrative -- 2.30.2 From 1ce28642473fcd0372a23e1b6863c984440a1bb7 Mon Sep 17 00:00:00 2001 From: Steven Baltakatei Sandoval Date: Thu, 16 Jul 2020 01:01:44 +0000 Subject: [PATCH 16/16] feat(README):Convert main README to org file from markdown --- README.md | 103 ----------------------------------------------------- README.org | 92 +++++++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 92 insertions(+), 103 deletions(-) delete mode 100644 README.md create mode 100644 README.org diff --git a/README.md b/README.md deleted file mode 100644 index b7724e9..0000000 --- a/README.md +++ /dev/null @@ -1,103 +0,0 @@ - - - -# Environment Sensor ninfacyzga-01 - -Created by [Steven Baltakatei Sandoval][bktei_2020_homepage] on -2020-06-07T21:56Z under a [CC BY-SA 4.0][cc_20131125_bysa] license and -last updated on 2020-06-28T16:41Z. - - -**Table of Contents** - -- [Environment Sensor ninfacyzga-01](#environment-sensor-ninfacyzga-01) - - [Abstract](#abstract) - - [Setup](#setup) - - [Location](#location) - - [Name Etymology](#name-etymology) - - [Origin](#origin) - - [Meaning](#meaning) - - [Pronunciation](#pronunciation) - - [Directory structure](#directory-structure) - - [Subsection](#subsection) - - [References](#references) - - - - -## Abstract - -This git repository contains software and documentation necessary to -set up a sensor package for logging new fact observe. - -Sensor data collected include: - -- location (GPS WGS84) - -## Setup - -See the following documents for instructions on setting up hardware -for each type of data collected. - -### Location - -See files within [`doc/location`](doc/location). - -## Name Etymology - -### Origin - -The name "ninfacyzga" is a -lojban[[1]](#lojwiki_20200212_main) -lujvo[[2]](#lojwiki_20140930_lujvo) consisting of the tanru -"cnino fatci zgana". The three gismu in the tanru contain the -following meanings: - -* **cnino**: new -* **fatci**: fact -* **zgana**: observe - -### Meaning - -Hence, the primary mission of a ninfacyzga device is to observe facts -of the new. The primary mission is NOT to process the facts into -stories since that is the mission of a temlisru'e device. Neither is -the primary mission to network with other devices over distances of -space for the purpose of sharing information. - -In practice, this means a ninfacyzga device records data about the -environment but does not attempt to generate summaries or averages of -the data. It simply transmits what it sees (ex: camera), hears (ex: -microphone), feels (ex: accelerometer), smells (ex: gas detector), -mags (ex: magnetometer). - -### Pronunciation - -"ninfacyzga" is pronounced /ninfaʃəzga/. - -## Directory structure - -Directories in this repository are structured as follows: - -* `doc`: Contains documentation explaining in human-readable format - what each file does. - -* `archive`: Contains legacy files from previous iterations of the - project. - -* `exec`: Contains executable components of the project. - -### Subsection - -## References -- 1. ["Lojban"][1]. 2020-02-12. [Logical Language Group"](https://mw.lojban.org/papri/Logical_Language_Group). Date Accessed: 2020-06-07. [Archive link](https://web.archive.org/web/20200304102316/https://mw.lojban.org/papri/Logical_Language_Group). Archive date: 2020-03-04. -- 2. ["lujvo"][2]. 2014-09-30. [Logical Language Group"](https://mw.lojban.org/papri/Logical_Language_Group). Date Accessed: 2020-06-07. [Archive link](https://web.archive.org/web/20160401100124/https://mw.lojban.org/papri/lujvo). Archive date: 2016-04-01. - -[1]: https://mw.lojban.org/papri/Lojban -[2]: https://mw.lojban.org/papri/lujvo -[bktei_2020_homepage]: http://baltakatei.com -[cc_20131125_bysa]: http://creativecommons.org/licenses/by-sa/4.0/ - - -
-

This work by Steven Baltakatei Sandoval is licensed under CC BY-SA 4.0

diff --git a/README.org b/README.org new file mode 100644 index 0000000..d2f47a6 --- /dev/null +++ b/README.org @@ -0,0 +1,92 @@ +* Environment Sensor ninfacyzga-01 +** Introduction +This document created by Steven Baltakatei Sandoval on +2020-06-07T21:56Z under a [[http://creativecommons.org/licenses/by-sa/4.0/][CC BY-SA 4.0]] license and last updated on +2020-07-16T00:37Z. +** Abstract +This git repository contains software and documentation necessary to +set up a sensor package for logging observables in the local +environment. + +Collected sensor data include: + +- Location (GPS WGS84) +- Air Pressure (pascals) +- Alignment (degrees X, degrees Y, degrees from magnetic north) + +** Setup + +See the following documents for instructions on setting up hardware +for each type of data collected. + +*** Location + +See the location [[file:doc/location/README.org][readme]]. + +*** Pressure + +See the pressure [[file:doc/pressure/README.org][readme]]. + +*** Alignment + +See the alignment [[file:doc/alignment/README.org][readme]]. + +** Name Etymology + +*** Origin + +The name ~ninfacyzga~ is a [[https://mw.lojban.org/papri/Logical_Language_Group][lojban]] [[https://mw.lojban.org/papri/lujvo][lujvo]] consisting of the tanru "cnino +fatci zgana". The three gismu in the tanru contain the following +meanings: + +**** ~cnino~: new +**** ~fatci~: fact +**** ~zgana~: observe + +*** Meaning + +Hence, the primary mission of a ~ninfacyzga~ device is to observe facts +of the new. The primary mission is NOT to process facts into +stories. Neither is the primary mission to network with other devices +for the purpose of sharing information. This means a ~ninfacyzga~ +device records data about the environment but does not attempt to +generate summaries or reports of archived data as a historian +would. Instead, the device simply transmits what it sees (ex: camera), +hears (ex: microphone), feels (ex: accelerometer, gyroscope, or +magnetometer), smells (ex: gas detector), etc. + +The observations of a ~nincafyzga~ device are the input of another +class of devices called ~temlisru'e~ whose primary mission is to +process observations into stories over time. + +The transmission and verification of observations and analyses +produced by ~ninfacyzga~ or ~temlisru'e~ devices is managed by another +class of devices called ~calkuptcana~. + +*** Pronunciation + +~ninfacyzga~ is pronounced ~/ninfaʃəzga/~. + +~temlisru'e~ is pronounced ~/tɛmlisruhe/~. + +~calkuptcana~ is pronounced ~/ʃalkuptʃana/~. + +** Directory structure + +Directories in this repository are structured as follows: + +- ~doc~: Contains documentation explaining in human-readable format + what each file does. + +- ~archive~: Contains legacy files from previous iterations of the + project. + +- ~exec~: Contains executable components of the project. + +** References + +1. "Lojban". 2020-02-12. [[https://mw.lojban.org/papri/Logical_Language_Group]["Logical Language Group"]]. Date Accessed: + 2020-06-07. [[https://web.archive.org/web/20200304102316/https://mw.lojban.org/papri/Logical_Language_Group][Archive link]]. Archive date: 2020-03-04. + +2. "lujvo". 2014-09-30. [[https://mw.lojban.org/papri/Logical_Language_Group]["Logical Language Group"]]. Date Accessed: + 2020-06-07. [[https://web.archive.org/web/20160401100124/https://mw.lojban.org/papri/lujvo][Archive link]]. Archive date: 2016-04-01. -- 2.30.2