doc(bkgpslog):Add TODO for future features

[EVA-2020-02.git] / exec / bkgpslog-plan.org

* bkgpslog task list
** DONE Add job control for short buffer length
   CLOSED: [2020-07-02 Thu 16:04]
2020-07-02T14:56Z; bktei> File write operations were bundled into a
magicWriteBuffer function that is called then detached from the script
shell (job control), but the detached job is not tracked by the main
script. A problem may arise if two instances of magicWriteBuffer
attempt to write to the same tar simultaneously. Two instances of
magicWriteBuffer may exist if the buffer length is low (ex: 1 second);
the default buffer length of 60 seconds should reduce the probability
of a collision but it should be possible for the main script to track
the process ID of a magicWriteBuffer() as soon as it detaches and then
checking (via ~$!~ as described [[https://bashitout.com/2013/05/18/Ampersands-on-the-command-line.html][here]]) that the process is still alive.
2020-07-02T15:23Z; bktei> I found that the Bash ~wait~ built-in can be
used to delay processing until a specified job completes. The ~wait~
command will pause script execution until all backgrounded processes
complete.
2020-07-02T16:03Z; bktei> Added ~wait~.
** DONE Rewrite tar initialization function
   CLOSED: [2020-07-02 Thu 17:23]
2020-07-02T17:23Z; bktei> Simplify tar initialization function so
VERSION file is used to test appendability of tar as well as to mark
when a new session is started.
** DONE Consolidate tar checking/creation into function
   CLOSED: [2020-07-02 Thu 18:33]
2020-07-02T18:33Z; bktei> Simplify how the output tar file's existence
is checked and its status as a valid tar file is validated. This was
done using a new function ~checkMakeTar~.
** DONE Add VERSION if output tar deleted between writes

   CLOSED: [2020-07-02 Thu 20:22]
2020-07-02T20:21Z; bktei> Added bkgpslog-specified function
magicWriteVersion() to be called whenever a new time-stamped ~VERSION~
file needs to be generated and appended to the output tar file
~PATHOUT_TAR~.
** DONE Rewrite buffer loop to reduce lag between gpspipe runs

   CLOSED: [2020-07-03 Fri 20:57]
2020-07-03T17:10Z; bktei> As is, there is still a 5-6 second lag
between when ~gpspipe~ times out at the end of a buffer round and when
~gpspipe~ is called by the subsequent buffer round. I believe this can
be reduced by moving variable manipulations inside the
asynchronously-executed magicWriteBuffer() function. Ideally, the
while loop should look like:

#+BEGIN_EXAMPLE
while( $SECONDS < $SCRIPT_TTL); do
    gpspipe-r > "$DIR_TMP"/buffer.nmea
    writeBuffer &
done
#+END_EXAMPLE
2020-07-03T20:56Z; bktei> I simplified it futher to something like
this:
#+BEGIN_EXAMPLE
while( $SECONDS < $SCRIPT_TTL); do
    writeBuffer &
    sleep $SCRIPT_TTL
done
#+END_EXAMPLE

Raspberry Pi Zero W shows approximately 71ms of drift per buffer round
with 10s buffer.
** TODO Feature: Recipient watch folder
2020-07-03T21:28Z; bktei> This feature would be to scan the contents
of a specified directory at the start of every buffer round in order
to determine encryption (age) recipients. This would allow a device to
dynamically encrypt location data in response to automated changes
made by other tools. For example, if such a directory were
synchronized via Syncthing and changes to such a directory were
managed by a trusted remote server, then that server could respond to
human requests to secure location data.

Two specific privacy subfeatures come to mind:

1. Parallel encryption: Given a set of ~n~ public keys, encrypt data
   with a single ~age~ command with options causing all ~n~ pubkeys to
   be recipients. In order to decrypt the data, any individual private
   key could be used. No coordination between key owners would be
   required to decrypt.

2. Sequential encryption: Given a set of ~n~ public keys, encrypt data
   with ~n~ sequential ~age~ commands all piped in series with each
   ~age~ command utilizing only one of the ~n~ public keys. In order
   to decrypt the data, all ~n~ private keys would be required to
   decrypt the data. Since coordination is required, it is less
   convenient than parallel encryption.

In either case, a directory would be useful for holding configuration
files specifying how to execute which or combination of which features
at the start of every buffer round.

I don't yet know how to program the rules, although I think it'd be
easier to simply add an option providing ~bkgpslog~ with a directory
to watch. When examining the directory, check for a file with the
appropriate file extension (ex: .pubkey) and then read the first line
into the script's pubKey array.

** TODO Feature: Simplify option to reduce output size

~gpsbabel~ [[https://www.gpsbabel.org/htmldoc-development/filter_simplify.html][features]] a ~simplify~ option to trim data points from GPS
data. There are several methods for prioritizing which points to keep
and which to trim, although the following seems useful given some
sample data I've recorded in a test run of ninfacyzga-01:

#+BEGIN_EXAMPLE
gpsbabel -i nmea -f all.nmea -x simplify,error=10,relative -o gpx \
-F all-simp-rel-10.gpx
#+END_EXAMPLE

An error level of "10" with the "relative" option seems to retain all
desireable features for GPS data while reducing the number of points
along straightaways. File size is reduced by a factor of
about 11. Noise from local stay-in-place drift isn't removed; a
relative error of about 1000 is required to remove stay-in-place drift
noise but this also trims all but 100m-size features of the recorded
path. A relative error of 1000 reduces file size by a factor of
about 450.

#+BEGIN_EXAMPLE
 67M relerror-0.001.kml
 66M relerror-0.01.kml
 58M relerror-0.1.kml
 21M relerror-1.kml
5.8M relerror-10.kml
797K relerror-100.kml
152K relerror-1000.kml
#+END_EXAMPLE

* bkgpslog narrative
** Initialize environment
*** Init variables
**** Save timeStart (YYYYmmddTHHMMSS±zz)
*** Define Functions
**** Define Debugging functions
**** Define Argument Processing function
**** Define Main function
** Run Main Function
*** Process Arguments
*** Set output encryption and compression option strings
*** Check that critical apps and dirs are available, displag missing ones.
*** Set lifespans of script and buffer
*** Init temp working dir ~DIR_TMP~
Make temporary dir in tmpfs dir: ~/dev/shm/$(nonce)..bkgpslog/~ (~DIR_TMP~)
*** Initialize ~tar~ archive
**** Write ~bkgpslog~ version to ~$DIR_TMP/VERSION~
**** Create empty ~tar~ archive in ~DIR_OUT~ at ~PATHOUT_TAR~

Set output file name to:
: PATHOUT_TAR="$DIR_OUT/YYYYmmdd..hostname_location.gz.age.tar"
Usage: ~iso8601Period $timeStart $timeEnd~ 

**** Append ~VERSION~ file to ~PATHOUT_TAR~

Append ~$DIR_TMP/VERSION~ to ~PATHOUT_TAR~ via ~tar --append~

*** Read/Write Loop (Record gps data until script lifespan ends)
**** Determine output file paths
**** Define GPS conversion commands
**** Fill Bash variable buffer from ~gpspipe~
**** Process bufferBash, save secured chunk set to ~DIR_TMP~
**** Append each secured chunk to ~PATHOUT_TAR~
: tar --append --directory=DIR_TMP --file=PATHOUT_TAR $(basename PATHOUT_{NMEA,GPX,KML} )
**** Remove secured chunk from ~DIR_TMP~