+++ /dev/null
-2020-07-12T21:16Z; bktei> Note: This file is now retired since ~bklog~
-has replaced ~bkgpslog~.
-
-* bkgpslog task list
-** DONE Add job control for short buffer length
- CLOSED: [2020-07-02 Thu 16:04]
-2020-07-02T14:56Z; bktei> File write operations were bundled into a
-magicWriteBuffer function that is called then detached from the script
-shell (job control), but the detached job is not tracked by the main
-script. A problem may arise if two instances of magicWriteBuffer
-attempt to write to the same tar simultaneously. Two instances of
-magicWriteBuffer may exist if the buffer length is low (ex: 1 second);
-the default buffer length of 60 seconds should reduce the probability
-of a collision but it should be possible for the main script to track
-the process ID of a magicWriteBuffer() as soon as it detaches and then
-checking (via ~$!~ as described [[https://bashitout.com/2013/05/18/Ampersands-on-the-command-line.html][here]]) that the process is still alive.
-2020-07-02T15:23Z; bktei> I found that the Bash ~wait~ built-in can be
-used to delay processing until a specified job completes. The ~wait~
-command will pause script execution until all backgrounded processes
-complete.
-2020-07-02T16:03Z; bktei> Added ~wait~.
-** DONE Rewrite tar initialization function
- CLOSED: [2020-07-02 Thu 17:23]
-2020-07-02T17:23Z; bktei> Simplify tar initialization function so
-VERSION file is used to test appendability of tar as well as to mark
-when a new session is started.
-** DONE Consolidate tar checking/creation into function
- CLOSED: [2020-07-02 Thu 18:33]
-2020-07-02T18:33Z; bktei> Simplify how the output tar file's existence
-is checked and its status as a valid tar file is validated. This was
-done using a new function ~checkMakeTar~.
-** DONE Add VERSION if output tar deleted between writes
-
- CLOSED: [2020-07-02 Thu 20:22]
-2020-07-02T20:21Z; bktei> Added bkgpslog-specified function
-magicWriteVersion() to be called whenever a new time-stamped ~VERSION~
-file needs to be generated and appended to the output tar file
-~PATHOUT_TAR~.
-** DONE Rewrite buffer loop to reduce lag between gpspipe runs
-
- CLOSED: [2020-07-03 Fri 20:57]
-2020-07-03T17:10Z; bktei> As is, there is still a 5-6 second lag
-between when ~gpspipe~ times out at the end of a buffer round and when
-~gpspipe~ is called by the subsequent buffer round. I believe this can
-be reduced by moving variable manipulations inside the
-asynchronously-executed magicWriteBuffer() function. Ideally, the
-while loop should look like:
-
-#+BEGIN_EXAMPLE
-while( $SECONDS < $SCRIPT_TTL); do
- gpspipe-r > "$DIR_TMP"/buffer.nmea
- writeBuffer &
-done
-#+END_EXAMPLE
-2020-07-03T20:56Z; bktei> I simplified it futher to something like
-this:
-#+BEGIN_EXAMPLE
-while( $SECONDS < $SCRIPT_TTL); do
- writeBuffer &
- sleep $SCRIPT_TTL
-done
-#+END_EXAMPLE
-
-Raspberry Pi Zero W shows approximately 71ms of drift per buffer round
-with 10s buffer.
-** DONE Feature: Recipient watch folder
- CLOSED: [2020-07-12 Sun 21:08]
-2020-07-03T21:28Z; bktei> This feature would be to scan the contents
-of a specified directory at the start of every buffer round in order
-to determine encryption (age) recipients. This would allow a device to
-dynamically encrypt location data in response to automated changes
-made by other tools. For example, if such a directory were
-synchronized via Syncthing and changes to such a directory were
-managed by a trusted remote server, then that server could respond to
-human requests to secure location data.
-
-Two specific privacy subfeatures come to mind:
-
-1. Parallel encryption: Given a set of ~n~ public keys, encrypt data
- with a single ~age~ command with options causing all ~n~ pubkeys to
- be recipients. In order to decrypt the data, any individual private
- key could be used. No coordination between key owners would be
- required to decrypt.
-
-2. Sequential encryption: Given a set of ~n~ public keys, encrypt data
- with ~n~ sequential ~age~ commands all piped in series with each
- ~age~ command utilizing only one of the ~n~ public keys. In order
- to decrypt the data, all ~n~ private keys would be required to
- decrypt the data. Since coordination is required, it is less
- convenient than parallel encryption.
-
-In either case, a directory would be useful for holding configuration
-files specifying how to execute which or combination of which features
-at the start of every buffer round.
-
-I don't yet know how to program the rules, although I think it'd be
-easier to simply add an option providing ~bkgpslog~ with a directory
-to watch. When examining the directory, check for a file with the
-appropriate file extension (ex: .pubkey) and then read the first line
-into the script's pubKey array.
-
-2020-07-12T21:08Z; bktei> ~-R~ watch directory option added in ~bkgpslog~ ver
-~0.4.0~.
-
-** DONE Feature: Simplify option to reduce output size
- CLOSED: [2020-07-12 Sun 21:15]
-
-~gpsbabel~ [[https://www.gpsbabel.org/htmldoc-development/filter_simplify.html][features]] a ~simplify~ option to trim data points from GPS
-data. There are several methods for prioritizing which points to keep
-and which to trim, although the following seems useful given some
-sample data I've recorded in a test run of ninfacyzga-01:
-
-#+BEGIN_EXAMPLE
-gpsbabel -i nmea -f all.nmea -x simplify,error=10,relative -o gpx \
--F all-simp-rel-10.gpx
-#+END_EXAMPLE
-
-An error level of "10" with the "relative" option seems to retain all
-desireable features for GPS data while reducing the number of points
-along straightaways. File size is reduced by a factor of
-about 11. Noise from local stay-in-place drift isn't removed; a
-relative error of about 1000 is required to remove stay-in-place drift
-noise but this also trims all but 100m-size features of the recorded
-path. A relative error of 1000 reduces file size by a factor of
-about 450.
-
-#+BEGIN_EXAMPLE
- 67M relerror-0.001.kml
- 66M relerror-0.01.kml
- 58M relerror-0.1.kml
- 21M relerror-1.kml
-5.8M relerror-10.kml
-797K relerror-100.kml
-152K relerror-1000.kml
-#+END_EXAMPLE
-
-2020-07-12T21:13Z; bktei> Instead of programming data simplification
-in ~bkgpslog~, the data simplification step should be performed via
-~bklog~'s ~-p~ option which specifies a processing command string to
-be ~eval~'d before data is compressed, encrypted, and written to
-disk. In other words, handling the simplification of data beyond
-allowing for a general command string specified by ~-p~ is outside the
-scope of ~bkgpslog~ or its successor ~bklog~.
-
-** DONE Feature: Generalize bkgpslog to bklog function
- CLOSED: [2020-07-12 Sun 21:11]
-2020-07-05T02:42Z; bktei> Transform ~bkgpslog~ into a modular
-component called ~bklog~ such that it processes a stdout stream of any
-external command, not just ~gpspipe -r~. This would permit reuse of
-the ~bkgpslog~ code for logging not just GPS data but things like
-pressure, temperature, system statistics, etc.
-2020-07-05T16:35Z; bktei>
-: bklog -r age1asdf -o log.tar # encrypt/compress stdin to log.tar
-: bklog -x -f log.tar -i age.key -O /tmp # extract and decrypt
-
-Making ~bklog~ follow the [[https://en.wikipedia.org/wiki/Unix_philosophy][Unix philosophy]] means that it shouldn't care
-what kind of text is fed to it.
-
-*** ~bklog~ Design vs. Unix Philosophy
-**** Pubkey dir watching
-The feature of periodically checking a directory for changes in the
-pubkeys it contains should be justified by its usefulness; if the
-complexity cannot be justified then the feature should be removed.
-**** Defaults vs options
-Many options can cause the tool to become complex in unjustifiable
-ways. Currently I am adding options because I want the ability to
-modify the script's behavior without having to modify the source code
-on the machine in which the code is running. I should consider
-removing features at some point and having the program force defaults
-on the user. For example, allowing the specification of a temporary
-directory, while useful for me, is probably not useful for most people
-who don't know or care about the difference between ~/tmp~ and
-~/dev/shm~.
-**** Script time to live (TTL)
-I initially implemented a script time-to-live feature because I was
-unsure in my ability to program script that could run for long periods
-of time without causing a runaway usage of memory. I still think it's
-a good idea to offer a script TTL option to the user but I think the
-default should be to simply run forver.
-
-2020-07-12T21:11Z; bktei> ~bklog~ script created and tested as of
-commit ~aedd19f~.
-
-** DONE TODO: Evaluate ~rsyslog~ as stand-in for this work
- CLOSED: [2020-07-12 Sun 21:09]
-2020-07-05T02:57Z; bktei> I searched for "debian iot logging" ("iot"
-as in "Internet of Things", the current buzzword for small low-power
-computers being used to provide microservices for owners in their own
-home) and came across several search results mentioning ~syslog~ and
-~rsyslog~.
-
-https://www.thissmarthouse.net/consolidating-iot-logs-into-mysql-using-rsyslog/
-https://rsyslog.readthedocs.io/en/latest/tutorials/tls.html
-https://serverfault.com/questions/20840/how-would-you-send-syslog-securely-over-the-public-internet
-https://www.rsyslog.com/
-
-My impression is that ~rsyslog~ is a complex software package designed
-to offer many features, some of which possibly might satisfy my
-needs.
-
-However, as stated in the repository README, the objective of the
-~ninfacyzga-01~ project is "Observing facts of the new". This means
-that the goal is not only to record location data but any data that
-can be captured by a sensor. This means the capture of the following
-environmental phenomena are within the scope of this device:
-
-*** Sounds (microphone)
-*** Light (camera)
-*** Temperature (thermocouple)
-*** Air Pressure (barometer)
-*** Acceleration Vector (acceleromter / gyroscope)
-*** Magnetic Field Vector (magnetometer)
-
-This brings up the issue of respecting privacy of others in shared
-spaces through which ~ninfacyzga-01~ may pass through. ~ninfacyzga-01~
-should encrypt data it records according to rules set by its
-owner.
-
-One permissive rule could be that if ~ninfacyzga-01~ detects that a
-person (let's call her Alice) enters a room, it should add Alice's
-encryption public key to the list of recipients against which it
-encrypts data without Alice having to know how ~ninfacyzga-01~ is
-programmed (she might have a ~calkuptcana~ agent on her person that
-broadcasts her privacy preferences). Meanwhile, ~ninfacyzga-01~ may
-publish its observations to a repository that Alice and other members
-of the shared communal space have access to (ex: a read-only shared
-directory on a local network WiFi). Alice could download all the files
-in the shared repository but she would only be able to decrypt files
-generated when she was physically near enough to ~ninfacyzga-01~ for
-it to detect that her presence was within some spatial boundary.
-
-A more restrictive rule could resemble the permissive rule in that
-~ninfacyzga-01~ uses Alice's encryption public key only when she is
-physically near by, except that it encrypts logged files against
-public keys in a sequential manner. This would mean that all people
-who were near ~ninfacyzga-01~ would have to pass around each log file
-to eachother so that they could decrypt the content.
-
-That said, according to [[https://www.rsyslog.com/doc/master/tutorials/database.html][this ~rsyslog~ page]], ~rsyslog~ is more a data
-wrangling system for collecting data from disparate sources of
-different types and outputting data to text files on disk than a
-system committed to the server-client model of database storage. So, I
-think converting ~bkgpslog~ into a ~bklog~ script that appends
-encrypted and compressed data to a tar file for later extraction
-(possibly the same script with future features) would be best.
-
-2020-07-12T21:10Z; bktei> rsyslog is outside the scope of what
-~bkgpslog~ does (record location observations). A different tool
-should be used to retrieve and synchronize data. The dumb storage
-method of "tar files in a syncthing folder" works for now.
-** TODO: Place persistent recip. updates in asynchronous coproc
-2020-07-06T19:37Z; bktei> In order to update the recipient list, the
-magicParseRecipientDir() function needs to be run each buffer period
-in order to scan for changes in the recipient list. However, such a
-scan takes time; if the magicGatherWriteBuffer() function must pause
-until magicParseRecipientDir() completes, then a significant pause
-between buffer sessions may occur, causing detectable gaps in location
-data between buffer rounds.
-
-I looked for ways in which I might start magicParseRecipientDir()
-asynchronously immediately before running the data collection command
-and then collect its output at the start of the next buffer round. One
-way using the ~coproc~ Bash built-in is described [[https://stackoverflow.com/a/20018504/10850071][here]]. I'd have to
-make the asynchronous function output the recipient list to stdout
-which would then be ~read~ into the ~recPubKeysValid~ array in the
-main loop. However, for now, I'm putting the magicParseRecipientDir()
-as-is in the main loop and accepting the delay for now.
-* bkgpslog narrative
-** Initialize environment
-*** Init variables
-**** Save timeStart (YYYYmmddTHHMMSS±zz)
-*** Define Functions
-**** Define Debugging functions
-**** Define Argument Processing function
-**** Define Main function
-** Run Main Function
-*** Process Arguments
-*** Set output encryption and compression option strings
-*** Check that critical apps and dirs are available, displag missing ones.
-*** Set lifespans of script and buffer
-*** Init temp working dir ~DIR_TMP~
-Make temporary dir in tmpfs dir: ~/dev/shm/$(nonce)..bkgpslog/~ (~DIR_TMP~)
-*** Initialize ~tar~ archive
-**** Write ~bkgpslog~ version to ~$DIR_TMP/VERSION~
-**** Create empty ~tar~ archive in ~DIR_OUT~ at ~PATHOUT_TAR~
-
-Set output file name to:
-: PATHOUT_TAR="$DIR_OUT/YYYYmmdd..hostname_location.gz.age.tar"
-Usage: ~iso8601Period $timeStart $timeEnd~
-
-**** Append ~VERSION~ file to ~PATHOUT_TAR~
-
-Append ~$DIR_TMP/VERSION~ to ~PATHOUT_TAR~ via ~tar --append~
-
-*** Read/Write Loop (Record gps data until script lifespan ends)
-**** Determine output file paths
-**** Define GPS conversion commands
-**** Fill Bash variable buffer from ~gpspipe~
-**** Process bufferBash, save secured chunk set to ~DIR_TMP~
-**** Append each secured chunk to ~PATHOUT_TAR~
-: tar --append --directory=DIR_TMP --file=PATHOUT_TAR $(basename PATHOUT_{NMEA,GPX,KML} )
-**** Remove secured chunk from ~DIR_TMP~
projects / EVA-2020-02.git / blobdiff