exec/bkgpslog-plan.org

   1 * bkgpslog task list
   2 ** DONE Add job control for short buffer length
   3    CLOSED: [2020-07-02 Thu 16:04]
   4 2020-07-02T14:56Z; bktei> File write operations were bundled into a
   5 magicWriteBuffer function that is called then detached from the script
   6 shell (job control), but the detached job is not tracked by the main
   7 script. A problem may arise if two instances of magicWriteBuffer
   8 attempt to write to the same tar simultaneously. Two instances of
   9 magicWriteBuffer may exist if the buffer length is low (ex: 1 second);
  10 the default buffer length of 60 seconds should reduce the probability
  11 of a collision but it should be possible for the main script to track
  12 the process ID of a magicWriteBuffer() as soon as it detaches and then
  13 checking (via ~$!~ as described [[https://bashitout.com/2013/05/18/Ampersands-on-the-command-line.html][here]]) that the process is still alive.
  14 2020-07-02T15:23Z; bktei> I found that the Bash ~wait~ built-in can be
  15 used to delay processing until a specified job completes. The ~wait~
  16 command will pause script execution until all backgrounded processes
  17 complete.
  18 2020-07-02T16:03Z; bktei> Added ~wait~.
  19 ** DONE Rewrite tar initialization function
  20    CLOSED: [2020-07-02 Thu 17:23]
  21 2020-07-02T17:23Z; bktei> Simplify tar initialization function so
  22 VERSION file is used to test appendability of tar as well as to mark
  23 when a new session is started.
  24 ** DONE Consolidate tar checking/creation into function
  25    CLOSED: [2020-07-02 Thu 18:33]
  26 2020-07-02T18:33Z; bktei> Simplify how the output tar file's existence
  27 is checked and its status as a valid tar file is validated. This was
  28 done using a new function ~checkMakeTar~.
  29 ** DONE Add VERSION if output tar deleted between writes
  30
  31    CLOSED: [2020-07-02 Thu 20:22]
  32 2020-07-02T20:21Z; bktei> Added bkgpslog-specified function
  33 magicWriteVersion() to be called whenever a new time-stamped ~VERSION~
  34 file needs to be generated and appended to the output tar file
  35 ~PATHOUT_TAR~.
  36 ** DONE Rewrite buffer loop to reduce lag between gpspipe runs
  37    CLOSED: [2020-07-03 Fri 20:57]
  38 2020-07-03T17:10Z; bktei> As is, there is still a 5-6 second lag
  39 between when ~gpspipe~ times out at the end of a buffer round and when
  40 ~gpspipe~ is called by the subsequent buffer round. I believe this can
  41 be reduced by moving variable manipulations inside the
  42 asynchronously-executed magicWriteBuffer() function. Ideally, the
  43 while loop should look like:
  44
  45 #+BEGIN_EXAMPLE
  46 while( $SECONDS < $SCRIPT_TTL); do
  47     gpspipe-r > "$DIR_TMP"/buffer.nmea
  48     writeBuffer &
  49 done
  50 #+END_EXAMPLE
  51 2020-07-03T20:56Z; bktei> I simplified it futher to something like
  52 this:
  53 #+BEGIN_EXAMPLE
  54 while( $SECONDS < $SCRIPT_TTL); do
  55     writeBuffer &
  56     sleep $SCRIPT_TTL
  57 done
  58 #+END_EXAMPLE
  59 Raspberry Pi Zero W shows approximately 71ms of drift per buffer round
  60 with 10s buffer.
  61 * bkgpslog narrative
  62 ** Initialize environment
  63 *** Init variables
  64 **** Save timeStart (YYYYmmddTHHMMSS±zz)
  65 *** Define Functions
  66 **** Define Debugging functions
  67 **** Define Argument Processing function
  68 **** Define Main function
  69 ** Run Main Function
  70 *** Process Arguments
  71 *** Set output encryption and compression option strings
  72 *** Check that critical apps and dirs are available, displag missing ones.
  73 *** Set lifespans of script and buffer
  74 *** Init temp working dir ~DIR_TMP~
  75 Make temporary dir in tmpfs dir: ~/dev/shm/$(nonce)..bkgpslog/~ (~DIR_TMP~)
  76 *** Initialize ~tar~ archive
  77 **** Write ~bkgpslog~ version to ~$DIR_TMP/VERSION~
  78 **** Create empty ~tar~ archive in ~DIR_OUT~ at ~PATHOUT_TAR~
  79
  80 Set output file name to:
  81 : PATHOUT_TAR="$DIR_OUT/YYYYmmdd..hostname_location.gz.age.tar"
  82 Usage: ~iso8601Period $timeStart $timeEnd~
  83
  84 **** Append ~VERSION~ file to ~PATHOUT_TAR~
  85
  86 Append ~$DIR_TMP/VERSION~ to ~PATHOUT_TAR~ via ~tar --append~
  87
  88 *** Read/Write Loop (Record gps data until script lifespan ends)
  89 **** Determine output file paths
  90 **** Define GPS conversion commands
  91 **** Fill Bash variable buffer from ~gpspipe~
  92 **** Process bufferBash, save secured chunk set to ~DIR_TMP~
  93 **** Append each secured chunk to ~PATHOUT_TAR~
  94 : tar --append --directory=DIR_TMP --file=PATHOUT_TAR $(basename PATHOUT_{NMEA,GPX,KML} )
  95 **** Remove secured chunk from ~DIR_TMP~