fix(bklog):Fix flip-flopped buffer start/end timestamps

[EVA-2020-02.git] / doc / location / README.org

* Location Logging
This document was created by Steven Baltakatei Sandoval on
~2020-06-29T12:14Z~ under a [[https://creativecommons.org/licenses/by-sa/4.0/][Creative Commons BY-SA 4.0 license]]. It
was updated by Steven Baltakatei Sandoval on ~2020-07-12T21:04Z~
#+TITLE: Ninfacyzga-01 Manual
#+AUTHOR: Steven Baltakatei Sandoval
#+EMAIL: baltakatei@gmail.com
** Narrative
Ninfacyzga-01 records (logs) its position in time and space using a
[[https://en.wikipedia.org/wiki/Satellite_navigation_device][GPS receiver]]. The NMEA location data produced by the receiver is
converted into the more commonly used GPS data storage formats of GPX
and KML. All three types of data are then compressed and encrypted
against a set of public keys. The encrypted data is then written to
disk. Data produced by the receiver is segmented into 60-second chunks
before being processed and written to disk.
** Description
*** Hardware
**** Raspberry Pi Zero W
See the [[https://www.raspberrypi.org/pi-zero-w/][OEM]] webpage for this product.
**** PiZ UpTime 2.0
See the [[https://alchemy-power.com/piz-uptime-2-0/][OEM]] webpage for this product.
*** Software
~bklog~ : A bash script that saves its stdin stream to a tar file. The
file may be compressed by ~gzip~ and encrypted by ~age~. It is an
executable file contained within this repository at ~exec/bklog~. It
should be copied to ~$HOME/.local/bin~.

~bkgpslog~ : A legacy bash script similar to ~bklog~ but narrower in
scope in that it only records output from ~gpspipe~.

~gpsd~ : A background daemon app capable of interfacing with the
Ozzmaker BerryGPS-IMU's GPS submodule. Installed and initialized by
~apt~.

~gpspipe~ : A command line app that polls ~gpsd~ and produces a stream
stdout consisting of GPS data lines in NMEA format. Installed via
~apt~.

~gpsbabel~ : A command line app that converts GPS data from one format
into another. ~bklog~ may be used to convert NMEA data into GPX and
KML. Installed via ~apt~.

~gzip~ : A simple command line app that compresses stdin into a
smaller stdout stream.

~age~ : A simple command line app that encrypts stdin against public
keys specified in its options. Produces encrypted stdout. Is an
executable file contained within this repository at ~exec/age~. It
should be copied to ~$HOME/.local/bin~.

**** Narrative
~bklog~ may be used to log location data by receiving stdout produced
by ~gpspipe~. ~bklog~ contains options that allow file writes to be
performed at adjustable time intervals (default: 300 seconds),
compressed (with ~gzip~), and encrypted (with ~age~). Files are
written in the form of appendages to a ~tar~ archive saved to a
specified location. The NMEA data produced by ~gpspipe~ may be
processed via a ~gpsbabel~ command string specified as an option to
~bklog~, assuming ~gpsbabel~ is installed.

*** Output

Several output file formats have been tested with ~bklog~.

**** File Formats
***** NMEA
See the [[https://en.wikipedia.org/wiki/NMEA_0183][Wikipedia page]] for this. This file format is the default
output of the ~gpspipe -r~ command.
***** GPX
See the [[https://en.wikipedia.org/wiki/GPS_Exchange_Format][Wikipedia page]] for this. [[http://wiki.gis.com/wiki/index.php/WGS84][WGS84]] is the datum used. An NMEA file
may be converted to this format using ~gpsbabel~.
***** KML
See the [[https://en.wikipedia.org/wiki/Keyhole_Markup_Language][Wikipedia page]] for this. [[http://wiki.gis.com/wiki/index.php/WGS84][WGS84]] is the datum used. An NMEA file
may be converted to this format using ~gpsbabel~.

**** Encryption Method
Files produced by the bkgpslog script are encrypted against a set of
public keys using [[https://github.com/FiloSottile/age][~age~]], a simple command line encryption tool
selected over ~gpg~ because of ~age~'s deliberate lack of
configurability.

The public keys are bech32 strings supplied as options to bkgpslog
when called. The secret key should *NOT* be stored in Ninfacyzga-01.

If a key pair was generated using ~age-keygen~, then it is an [[https://en.wikipedia.org/wiki/Curve25519][~X25519~]]
key pair. See the [[https://age-encryption.org/v1][~age~ Version 1 specification]].

An ~ssh-rsa~ or ~ssh-ed25519~ SSH public key string may be used instead of
the bech32 public key string produced by ~age-keygen~ for convenience.

Help information for ~age~ is available by running ~$ age --help~.
***** Encryption Commands
****** Encryption through ~age~
In order to illustrate how ~bklog~ encrypts files, below is an example
command illustrating how ~age~ may be used to encrypt a file.

#+BEGIN_EXAMPLE
$ echo "asdf" | age -r \
age1kza7pfshy7xwygf9349zgmk7x53mquvedgw9r98qwyyqhssh830qqjzlsw \
> "$HOME/secret_file"
#+END_EXAMPLE

The resulting ~secret-file~ is a binary blob with a plaintext header
indicating how the blob was encrypted (which version of age was used,
which public key was used).

****** Encryption through ~bklog~
~bklog~ may instructed to encrypt files via the ~-e~ and ~-r [pubkey
string]~ options. An example is shown below:

#+BEGIN_EXAMPLE
$ gpspipe -r | bklog -e \
-r age1kza7pfshy7xwygf9349zgmk7x53mquvedgw9r98qwyyqhssh830qqjzlsw \
-r age1ce3pvzrqfcn2pc6zqzglc8ac8yjk3fzukpy08cesqjjwns53xywqmaq7xw \
-r age1pu5usxm743sx7rf22985xv2f4s0luzv6r6yx4fa7p8c2zyvp9fvqus2xr5 \
-o "$HOME/Location"
#+END_EXAMPLE

~bklog~ may be instructed via the ~-e~ and ~-R~ options to watch a
directory in order to locate public key strings in its files. ~bklog~
reads the first line of each file and interprets it as a public key
string.

In this example, the strings beginning with ~age1...~ are
bech32-formatted public key strings. Please see the [[*Key Generation][Key Generation]]
section for an explanation.

Since ~age~ also accepts ~ssh~ public key strings, these may also be
used if they are of the following form (no comment).

: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABA…AACAQDLnJbPs7CjwPT+OxXd

***** Decryption Commands
Files may be decrypted using a command similar to:

#+BEGIN_EXAMPLE
cat location.gpx.age | age -d -i key.txt > location.gpx
#+END_EXAMPLE

The version of ~age~ used to perform the encryption 

** Operating Procedures
*** Initial Startup
See OEM (Ozzmaker) [[https://ozzmaker.com/berrygps-berrygps-imu-quick-start-guide/][quickstart guide for the BerryGPS-IMU]].

**** Physical Setup

BerryGPS-IMU must be electrically connected to the correct pins on the
GPIO header of a Raspberry Pi Zero W.

*Optional*: stack together with PiZ Uptime 2.0 module. No GPIO pins
conflict so a simple stacking and soldering with long header pins is
possible.

**** Software Setup
***** Install Executables

Install Raspbian 10 Buster onto an SD card image. See the Raspberry Pi
Foundation [[https://www.raspberrypi.org/documentation/installation/installing-images/README.md][installation instructions]]. Configure WiFi to permit log
file transfer. Configure SSH to permit remote administration via the
command line interface.

Make sure to install the ~unattended-upgrades~ package to make sure
the latest security patches for packages are installed. See [[https://linux-audit.com/using-unattended-upgrades-on-debian-and-ubuntu/][this page]]
for a description of how ~unattended-upgrades~ works.

Install ~gpsd~, ~gpspipe~, ~git~, and this repository for location
logging capability.

Install ~syncthing~ for log file transfer capability.

Place ~age~ binary (the one compiled for ARM CPU architecture for
Linux) in ~$HOME/.local/bin~.

***** Disable Swap File
Since standard Raspbian 10 (Buster) install involves copying
unencrypted file system image to SD card which is mounted by the
Raspberry Pi, system memory may be written to disk in the form of a
Swap file as described [[https://ideaheap.com/2013/07/stopping-sd-card-corruption-on-a-raspberry-pi/][here]]. In order to reduce the chance that
location log data is ever written to disk, swap file functionality
must be disabled[fn:ideaheap_20130731_disableswap].

Raspbian 10 uses dphys-swapfile to manage a swap file. It may be
disabled persistently[fn:rpf_20190702_disableswappersist] by running
the following command:

: sudo systemctl disable dphys-swapfile.service

To view the status of the swap file in Raspbian 10, run ~free -m~:

#+BEGIN_EXAMPLE
pi@ninfacyzga-01:~$ free -m
          total    used    free  shared  buff/cache   available
Mem:        432      86      36      21         309         268
Swap:        99       0      99
#+END_EXAMPLE

After disabling the swap file and rebooting:

#+BEGIN_EXAMPLE
pi@ninfacyzga-01:~$ free -m
          total    used    free  shared  buff/cache   available
Mem:        432      89     214       3         128         289
Swap:         0       0       0
#+END_EXAMPLE

[fn:ideaheap_20130731_disableswap] Explanation:
https://ideaheap.com/2013/07/stopping-sd-card-corruption-on-a-raspberry-pi/

[fn:rpf_20190702_disableswappersist] Persistant disabling of swap in
Raspbian 10 Buster:
https://www.raspberrypi.org/forums/viewtopic.php?p=1490692&sid=5c596a124b7805d6b10dab8d3d7caf16#p1490692

***** Automatic Start Configuration

It is recommended to create a daily ~cron~ job that executes a bash
script that starts ~bklog~ jobs to record location along with other
types of data recording. An example of such a script is below:

#+BEGIN_EXAMPLE
#!/bin/bash

# Log location
gpspipe -r | /bin/bash "$HOME/.local/bin/bklog" -c -e -z "UTC" -t "/dev/shm" \
          -r age1kza7pfshy7xwygf9349zgmk7x53mquvedgw9r98qwyyqhssh830qqjzlsw \
          -r age1ce3pvzrqfcn2pc6zqzglc8ac8yjk3fzukpy08cesqjjwns53xywqmaq7xw \
          -R "$HOME/.config/bklog/recipients" -w ".nmea" -b "300" -B "day" \
	  -o "$HOME/Sync/Evanescent_Location" -l "location" \
          -p "gpsbabel -i nmea -f - -o gpx -F - " ".gpx" \
          -p "gpsbabel -i nmea -f - -o kml -F - " ".kml" &
# Log pressure
python ~/.local/bin/bmp388.py | /bin/bash "$HOME/git-OC/ninfacyzga-01/exec/bklog" \
          -c -e -z "UTC" -t "/dev/shm" \
          -r age1kza7pfshy7xwygf9349zgmk7x53mquvedgw9r98qwyyqhssh830qqjzlsw \
          -r age1ce3pvzrqfcn2pc6zqzglc8ac8yjk3fzukpy08cesqjjwns53xywqmaq7xw \
          -R "$HOME/.config/bklog/recipients" -w ".txt" -b "300" -B "day" \
	  -o "$HOME/Sync/Evanescent_Location" -l "pressure" &
#+END_EXAMPLE

This script, if it were saved at
~"$HOME/.local/bin/cron/dailylog.sh"~ would then be added as a
line in the ~crontab~ file as shown below:

#+BEGIN_EXAMPLE
$ crontab -e
0 0 * * * /bin/bash ~/.local/bin/cron/dailylog.sh
@reboot /bin/bash ~/.local/bin/cron/dailylog.sh
#+END_EXAMPLE

In the example script, the options are:

: -c : tells bklog to compress output
: -e : tells bklog log to encrypt output
: -r : tells bklog to interpret the next argument as a pubkey string
: -R : tells bklog to interpret the next argument as a directory
:        where public keys may be found (first line of each file is
:        read).
: -o : tells bklog to write output files to the directory represented
: -t : tells bklog to interpret the next argument as a directory
:        for storing temporary files
:        by the next argument
: -p : tells bklog a command string through which output is piped
:        before being compressed and encrypted. Also expected is a
:        file extension to be appended before the compression and
:        encryption file extensions.
: -w : tells bklog to save the unprocessed stdin with a specified
:        file extension (instead of the default '.stdin').
: -b : tells bklog how long each buffer round (time between file
:        writes) lasts in seconds.
: -B : specifies the time-to-live for the bklog script. A valid value may
:        one of the time elements such as "day" or "hour".
: -l : specfies a custom string to be used in output file names to
:        help differentiate tar files produced via bklog from different
:        sources of data.
: -z : specifies a time zone to be used to determine the script time-to-live.
:        By default, bklog uses whatever time is specified by the TZ
:        environment variable.

***** Log Transfer Configuration
Log files may be shared to other machines via ~syncthing~. See [[https://docs.syncthing.net/][this]]
manual for how to set up a shared folder and add Ninfacyzga-01 as a
device. Syncthing's directory synchronization capability allows a
remote machine to delete files from Ninfacyzga-01 by deleting from the
shared folder that they both share.

When log files are removed from Ninfacyzga-01 is not within the scope
of this document.

***** Key Generation
An ~age~ encryption key may be generated like so:
#+BEGIN_EXAMPLE
$ umask          # Gets current umask
0022             # Note: This is the default umask for Raspbian 10
$ umask 066      # So key.txt will have no perms except for owner (you)
$ umask          # Confirm umask set to 066
0066
$ age-keygen > key.txt
Public key: age1pu5usxm743sx7rf22985xv2f4s0luzv6r6yx4fa7p8c2zyvp9fvqus2xr5
$ ls -al key.txt
-rw------- 1 baltakatei baltakatei 184 Jun 29 18:28 key.txt
$ umask 0022     # Return umask to default value
$ umask          
0022
#+END_EXAMPLE

The resulting public/private keypair data looks like:
#+BEGIN_EXAMPLE
$ cat key.txt
# created: 2020-06-29T18:01:56Z
# public key: age1pu5usxm743sx7rf22985xv2f4s0luzv6r6yx4fa7p8c2zyvp9fvqus2xr5
AGE-SECRET-KEY-1NEUU5U2XGZGL9UYWNPU5DL99TGJJHFSN4F2E2WCCSDJJ6L5ZMLESNTVTU0
#+END_EXAMPLE

The file ~key.txt~ is not password-protected by default and should be
secured like an SSH public key should. The ~$ umask 066~ command run
before the ~$ age-keygen > key.txt~ command ensures ~key.txt~ will not
be readable, writeable, or executable to anyone except the owner
(you).

*** Normal Startup
Turn on Ninfacyzga-01 by supplying 5VDC power to the Raspberry Pi. No
further interaction should be required.
*** Normal Operation
No interaction beyond continually supplying approximately 100mA of
5VDC power and occasionally removing log files to conserve disk space
is required.
**** Log Transfer
Log files may be transferred by use of ~syncthing~ shared folders.
**** Automatic Updates
The ~automatic-upgrades~ package, if installed, should automatically
install security patches to packages installed via ~apt~.
*** Normal Shutdown
The system may be shutdown via SSH by running:

: $ sudo shutdown -r 0

*** Unscheduled Shutdown
Ninfacyzga-01 as described and setup should tolerate unscheduled power
loss. Log files being written every 60 seconds means, at most, 60
seconds worth of location data may be lost.
*** End of Life Disposal
LiPo batteries used by the PiZ Uptime 2.0 module should be disposed of
properly with their potential ignitability in mind, especially if they
are not fully discharged.

Consult your local municipality for its "E-Waste Disposal" (or
equivalent) policy. Metals used in the Raspberry Pi and related
components may be recycled.

Take extra precuation if lead solder was used in assembling the
electronics. Consumer electronics in early 21st century should use
lead-free solder.