exec/bkgpslog-plan.org

   1 * bkgpslog task list
   2 ** DONE Add job control for short buffer length
   3    CLOSED: [2020-07-02 Thu 16:04]
   4 2020-07-02T14:56Z; bktei> File write operations were bundled into a
   5 magicWriteBuffer function that is called then detached from the script
   6 shell (job control), but the detached job is not tracked by the main
   7 script. A problem may arise if two instances of magicWriteBuffer
   8 attempt to write to the same tar simultaneously. Two instances of
   9 magicWriteBuffer may exist if the buffer length is low (ex: 1 second);
  10 the default buffer length of 60 seconds should reduce the probability
  11 of a collision but it should be possible for the main script to track
  12 the process ID of a magicWriteBuffer() as soon as it detaches and then
  13 checking (via ~$!~ as described [[https://bashitout.com/2013/05/18/Ampersands-on-the-command-line.html][here]]) that the process is still alive.
  14 2020-07-02T15:23Z; bktei> I found that the Bash ~wait~ built-in can be
  15 used to delay processing until a specified job completes. The ~wait~
  16 command will pause script execution until all backgrounded processes
  17 complete.
  18 2020-07-02T16:03Z; bktei> Added ~wait~.
  19 ** DONE Rewrite tar initialization function
  20    CLOSED: [2020-07-02 Thu 17:23]
  21 2020-07-02T17:23Z; bktei> Simplify tar initialization function so
  22 VERSION file is used to test appendability of tar as well as to mark
  23 when a new session is started.
  24 ** DONE Consolidate tar checking/creation into function
  25    CLOSED: [2020-07-02 Thu 18:33]
  26 2020-07-02T18:33Z; bktei> Simplify how the output tar file's existence
  27 is checked and its status as a valid tar file is validated. This was
  28 done using a new function ~checkMakeTar~.
  29 ** DONE Add VERSION if output tar deleted between writes
  30
  31    CLOSED: [2020-07-02 Thu 20:22]
  32 2020-07-02T20:21Z; bktei> Added bkgpslog-specified function
  33 magicWriteVersion() to be called whenever a new time-stamped ~VERSION~
  34 file needs to be generated and appended to the output tar file
  35 ~PATHOUT_TAR~.
  36 ** DONE Rewrite buffer loop to reduce lag between gpspipe runs
  37
  38    CLOSED: [2020-07-03 Fri 20:57]
  39 2020-07-03T17:10Z; bktei> As is, there is still a 5-6 second lag
  40 between when ~gpspipe~ times out at the end of a buffer round and when
  41 ~gpspipe~ is called by the subsequent buffer round. I believe this can
  42 be reduced by moving variable manipulations inside the
  43 asynchronously-executed magicWriteBuffer() function. Ideally, the
  44 while loop should look like:
  45
  46 #+BEGIN_EXAMPLE
  47 while( $SECONDS < $SCRIPT_TTL); do
  48     gpspipe-r > "$DIR_TMP"/buffer.nmea
  49     writeBuffer &
  50 done
  51 #+END_EXAMPLE
  52 2020-07-03T20:56Z; bktei> I simplified it futher to something like
  53 this:
  54 #+BEGIN_EXAMPLE
  55 while( $SECONDS < $SCRIPT_TTL); do
  56     writeBuffer &
  57     sleep $SCRIPT_TTL
  58 done
  59 #+END_EXAMPLE
  60
  61 Raspberry Pi Zero W shows approximately 71ms of drift per buffer round
  62 with 10s buffer.
  63 ** TODO Feature: Recipient watch folder
  64 2020-07-03T21:28Z; bktei> This feature would be to scan the contents
  65 of a specified directory at the start of every buffer round in order
  66 to determine encryption (age) recipients. This would allow a device to
  67 dynamically encrypt location data in response to automated changes
  68 made by other tools. For example, if such a directory were
  69 synchronized via Syncthing and changes to such a directory were
  70 managed by a trusted remote server, then that server could respond to
  71 human requests to secure location data.
  72
  73 Two specific privacy subfeatures come to mind:
  74
  75 1. Parallel encryption: Given a set of ~n~ public keys, encrypt data
  76    with a single ~age~ command with options causing all ~n~ pubkeys to
  77    be recipients. In order to decrypt the data, any individual private
  78    key could be used. No coordination between key owners would be
  79    required to decrypt.
  80
  81 2. Sequential encryption: Given a set of ~n~ public keys, encrypt data
  82    with ~n~ sequential ~age~ commands all piped in series with each
  83    ~age~ command utilizing only one of the ~n~ public keys. In order
  84    to decrypt the data, all ~n~ private keys would be required to
  85    decrypt the data. Since coordination is required, it is less
  86    convenient than parallel encryption.
  87
  88 In either case, a directory would be useful for holding configuration
  89 files specifying how to execute which or combination of which features
  90 at the start of every buffer round.
  91
  92 I don't yet know how to program the rules, although I think it'd be
  93 easier to simply add an option providing ~bkgpslog~ with a directory
  94 to watch. When examining the directory, check for a file with the
  95 appropriate file extension (ex: .pubkey) and then read the first line
  96 into the script's pubKey array.
  97
  98 ** TODO Feature: Simplify option to reduce output size
  99
 100 ~gpsbabel~ [[https://www.gpsbabel.org/htmldoc-development/filter_simplify.html][features]] a ~simplify~ option to trim data points from GPS
 101 data. There are several methods for prioritizing which points to keep
 102 and which to trim, although the following seems useful given some
 103 sample data I've recorded in a test run of ninfacyzga-01:
 104
 105 #+BEGIN_EXAMPLE
 106 gpsbabel -i nmea -f all.nmea -x simplify,error=10,relative -o gpx \
 107 -F all-simp-rel-10.gpx
 108 #+END_EXAMPLE
 109
 110 An error level of "10" with the "relative" option seems to retain all
 111 desireable features for GPS data while reducing the number of points
 112 along straightaways. File size is reduced by a factor of
 113 about 11. Noise from local stay-in-place drift isn't removed; a
 114 relative error of about 1000 is required to remove stay-in-place drift
 115 noise but this also trims all but 100m-size features of the recorded
 116 path. A relative error of 1000 reduces file size by a factor of
 117 about 450.
 118
 119 #+BEGIN_EXAMPLE
 120  67M relerror-0.001.kml
 121  66M relerror-0.01.kml
 122  58M relerror-0.1.kml
 123  21M relerror-1.kml
 124 5.8M relerror-10.kml
 125 797K relerror-100.kml
 126 152K relerror-1000.kml
 127 #+END_EXAMPLE
 128
 129 * bkgpslog narrative
 130 ** Initialize environment
 131 *** Init variables
 132 **** Save timeStart (YYYYmmddTHHMMSS±zz)
 133 *** Define Functions
 134 **** Define Debugging functions
 135 **** Define Argument Processing function
 136 **** Define Main function
 137 ** Run Main Function
 138 *** Process Arguments
 139 *** Set output encryption and compression option strings
 140 *** Check that critical apps and dirs are available, displag missing ones.
 141 *** Set lifespans of script and buffer
 142 *** Init temp working dir ~DIR_TMP~
 143 Make temporary dir in tmpfs dir: ~/dev/shm/$(nonce)..bkgpslog/~ (~DIR_TMP~)
 144 *** Initialize ~tar~ archive
 145 **** Write ~bkgpslog~ version to ~$DIR_TMP/VERSION~
 146 **** Create empty ~tar~ archive in ~DIR_OUT~ at ~PATHOUT_TAR~
 147
 148 Set output file name to:
 149 : PATHOUT_TAR="$DIR_OUT/YYYYmmdd..hostname_location.gz.age.tar"
 150 Usage: ~iso8601Period $timeStart $timeEnd~
 151
 152 **** Append ~VERSION~ file to ~PATHOUT_TAR~
 153
 154 Append ~$DIR_TMP/VERSION~ to ~PATHOUT_TAR~ via ~tar --append~
 155
 156 *** Read/Write Loop (Record gps data until script lifespan ends)
 157 **** Determine output file paths
 158 **** Define GPS conversion commands
 159 **** Fill Bash variable buffer from ~gpspipe~
 160 **** Process bufferBash, save secured chunk set to ~DIR_TMP~
 161 **** Append each secured chunk to ~PATHOUT_TAR~
 162 : tar --append --directory=DIR_TMP --file=PATHOUT_TAR $(basename PATHOUT_{NMEA,GPX,KML} )
 163 **** Remove secured chunk from ~DIR_TMP~