exec/bklog-plan.org

   1 * bklog task list
   2 ** TODO Adjust filename duration dynamically
   3 2020-07-12T21:17Z; bktei> Currently, the "duration" component of the
   4 output filename for a given chunk is calculated from the ~bufferTTL~
   5 variable which does not necessarily reflect the amount of buffer lines
   6 being read into a given chunk, especially since lines of stdin may be
   7 received while synchronous processing occurs before a new
   8 ~magicProcessWriteBuffer &~ job is spun off.
   9
  10 A better way to specify the duration is to maintain a pair of
  11 timestamp varriables before each ~magicProcessWriteBuffer~ command is
  12 run. Within the asynchronous write job the time difference between the
  13 two time stamps may be evaluated in order to determine how much time
  14 has passed since the last write operation started. The last line read
  15 into the buffer when the ~while read~ loop finishes should be the most
  16 current value and so one of the two timestamps should be recorded
  17 then. The other time stamp should simply be the previous loop's
  18 timestamp value.
  19
  20 For example:
  21
  22 #+BEGIN_EXAMPLE
  23     # MAIN LOOP: Run until script TTL seconds pass
  24     bufferRound=0;
  25     while [[ $SECONDS -lt "scriptTTL" ]]; do
  26         vbm "STATUS:$fn:Starting buffer round:$bufferRound";
  27         bufferTOD="$((SECONDS + bufferTTL))"; # Set buffer round time-of-death
  28         # Consume stdin to fill buffer until buffer time-of-death (TOD) arrives
  29         while read -r -t "$bufferTTL" line && [[ $SECONDS -lt "$bufferTOD" ]]; do
  30             # Append line to buffer array
  31             buffer+=("$line");
  32         done;
  33         #====BEGIN NEW CODE HERE====
  34         bufferTimestampOld="$bufferTimestampNew";
  35         bufferTimeStampNew="$(date --iso-8601=seconds)";
  36         #====END NEW CODE HERE====
  37         # Create dir_tmp if missing
  38         if ! [[ -d "$dir_tmp" ]]; then
  39             yell "ERROR:$fn:dir_tmp existence failure:$dir_tmp";
  40             try mkdir "$dir_tmp" && vbm "DEBUG :$fn:Working dir recreated dir_tmp:$dir_tmp"; fi
  41         # Update cmd_encrypt, cmd_encrypt_suffix
  42         magicParseRecipients;
  43         # Export buffer to asynchronous processing.
  44         magicProcessWriteBuffer &
  45         unset buffer; # Clear buffer array for next bufferRound
  46         # Increment buffer round
  47         ((bufferRound++));
  48     done;
  49 #+END_EXAMPLE
  50
  51 Then, within the ~magicProcessWriteBuffer()~ function, the difference
  52 in seconds between ~bufferTimestampOld~ and ~bufferTimestampNew~ may
  53 be calculated and an appropriate duration string generated from the
  54 ~timeDuration()~ function.
  55
  56 ** TODO Update ~SECONDS~ variable during while read loop
  57 2020-07-14T00:58Z; bktei> The starting timestamps of each output file
  58 still drifts against system time. Although the ~while read~ loop does
  59 not lose data, the drift causes the output files to be named weirdly.
  60
  61 A possible solution is to correct the ~SECONDS~ variable against the
  62 current system time. Because ~date~ calls are slow, this correction
  63 should not be made after every line. At a minimum, the correction
  64 should occur once per buffer round, possibly after the buffer round
  65 has completed. If more frequent corrections are required, then the
  66 number of lines being read in each buffer round should be tracked and
  67 a modulus comparison may be implemented within the ~while read~ loop
  68 so that a correction is made after some fraction of the expected lines
  69 to be read are read.
  70 ** TODO Account for "early-exit" bug in input script
  71 2020-07-14T03:00Z; bktei> What happens if the script piping its stdout
  72 into ~bklog~ immediately exits without providing any stdout (ex: a
  73 python script with a missing module)? ~bklog~ should be able to detect
  74 the latest exit code and exit early. It should also be able to detect
  75 if the incoming pipe is closed.
  76 * bklog narrative
  77