exec/bklog-plan.org

   1 * bklog task list
   2 ** DONE Adjust filename duration dynamically
   3    CLOSED: [2020-07-14 Tue 22:17]
   4 2020-07-12T21:17Z; bktei> Currently, the "duration" component of the
   5 output filename for a given chunk is calculated from the ~bufferTTL~
   6 variable which does not necessarily reflect the amount of buffer lines
   7 being read into a given chunk, especially since lines of stdin may be
   8 received while synchronous processing occurs before a new
   9 ~magicProcessWriteBuffer &~ job is spun off.
  10
  11 A better way to specify the duration is to maintain a pair of
  12 timestamp varriables before each ~magicProcessWriteBuffer~ command is
  13 run. Within the asynchronous write job the time difference between the
  14 two time stamps may be evaluated in order to determine how much time
  15 has passed since the last write operation started. The last line read
  16 into the buffer when the ~while read~ loop finishes should be the most
  17 current value and so one of the two timestamps should be recorded
  18 then. The other time stamp should simply be the previous loop's
  19 timestamp value.
  20
  21 For example:
  22
  23 #+BEGIN_EXAMPLE
  24     # MAIN LOOP: Run until script TTL seconds pass
  25     bufferRound=0;
  26     while [[ $SECONDS -lt "scriptTTL" ]]; do
  27         vbm "STATUS:$fn:Starting buffer round:$bufferRound";
  28         bufferTOD="$((SECONDS + bufferTTL))"; # Set buffer round time-of-death
  29         # Consume stdin to fill buffer until buffer time-of-death (TOD) arrives
  30         while read -r -t "$bufferTTL" line && [[ $SECONDS -lt "$bufferTOD" ]]; do
  31             # Append line to buffer array
  32             buffer+=("$line");
  33         done;
  34         #====BEGIN NEW CODE HERE====
  35         bufferTimestampOld="$bufferTimestampNew";
  36         bufferTimeStampNew="$(date --iso-8601=seconds)";
  37         #====END NEW CODE HERE====
  38         # Create dir_tmp if missing
  39         if ! [[ -d "$dir_tmp" ]]; then
  40             yell "ERROR:$fn:dir_tmp existence failure:$dir_tmp";
  41             try mkdir "$dir_tmp" && vbm "DEBUG :$fn:Working dir recreated dir_tmp:$dir_tmp"; fi
  42         # Update cmd_encrypt, cmd_encrypt_suffix
  43         magicParseRecipients;
  44         # Export buffer to asynchronous processing.
  45         magicProcessWriteBuffer &
  46         unset buffer; # Clear buffer array for next bufferRound
  47         # Increment buffer round
  48         ((bufferRound++));
  49     done;
  50 #+END_EXAMPLE
  51
  52 Then, within the ~magicProcessWriteBuffer()~ function, the difference
  53 in seconds between ~bufferTimestampOld~ and ~bufferTimestampNew~ may
  54 be calculated and an appropriate duration string generated from the
  55 ~timeDuration()~ function.
  56
  57 2020-07-14T22:17:16Z; bktei> Initial adjustment of SECONDS
  58 implemented. Ongoing monitoring of end time of each buffer round
  59 ~while read~ loop checked.
  60 ** DONE Update ~SECONDS~ variable during while read loop
  61    CLOSED: [2020-07-14 Tue 16:22]
  62 2020-07-14T00:58Z; bktei> The starting timestamps of each output file
  63 still drifts against system time. Although the ~while read~ loop does
  64 not lose data, the drift causes the output files to be named weirdly.
  65
  66 A possible solution is to correct the ~SECONDS~ variable against the
  67 current system time. Because ~date~ calls are slow, this correction
  68 should not be made after every line. At a minimum, the correction
  69 should occur once per buffer round, possibly after the buffer round
  70 has completed. If more frequent corrections are required, then the
  71 number of lines being read in each buffer round should be tracked and
  72 a modulus comparison may be implemented within the ~while read~ loop
  73 so that a correction is made after some fraction of the expected lines
  74 to be read are read.
  75
  76 2020-07-14T16:21Z; bktei> I ran a test to see if SECONDS drifts and it
  77 does not. The lag is caused by other synchronous commands. The
  78 solution will be to adjust the variables against which SECONDS is
  79 compared.
  80 ** TODO Account for "early-exit" bug in input script
  81 *** 2020-07-14T03:00Z; bktei>
  82 What happens if the script piping its stdout into ~bklog~ immediately
  83 exits without providing any stdout (ex: a python script with a missing
  84 module)? ~bklog~ should be able to detect the latest exit code and
  85 exit early. It should also be able to detect if the incoming pipe is
  86 closed.
  87
  88 *** 2020-07-14T22:25Z; bktei>
  89 Possible solution using ~dd~, ~od~, and ~if [ -z string ]~ [[https://unix.stackexchange.com/a/33055][here]].
  90 * bklog narrative
  91