doc(doc/time):Document time tracking setup

[EVA-2020-02.git] / doc / location / README.org

diff --git a/doc/location/README.org b/doc/location/README.org
index 50a0b5ed5d0e7f449a8d3746abed31f4a148edbd..3676e3225b0afb712144610b98804a02690da9ad 100644 (file)
--- a/doc/location/README.org
+++ b/doc/location/README.org
@@ -1,7 +1,12 @@
 * Location Logging
 * Location Logging

+#+TITLE: Ninfacyzga-01 Manual
+#+AUTHOR: Steven Baltakatei Sandoval
+#+EMAIL: baltakatei@gmail.com
+** About

 This document was created by Steven Baltakatei Sandoval on
 This document was created by Steven Baltakatei Sandoval on

-<2020-06-29 Mon 12:14> under a [[https://creativecommons.org/licenses/by-sa/4.0/][Creative Commons BY-SA 4.0 license]]. It
-was updated by Steven Baltakatei Sandoval on <2020-06-29 Mon 22:06>.
+~2020-06-29T12:14Z~ under a [[https://creativecommons.org/licenses/by-sa/4.0/][Creative Commons BY-SA 4.0 license]]. It
+was updated by Steven Baltakatei Sandoval on ~2020-10-16T22:33Z~
+

 ** Narrative
 Ninfacyzga-01 records (logs) its position in time and space using a
 [[https://en.wikipedia.org/wiki/Satellite_navigation_device][GPS receiver]]. The NMEA location data produced by the receiver is
 ** Narrative
 Ninfacyzga-01 records (logs) its position in time and space using a
 [[https://en.wikipedia.org/wiki/Satellite_navigation_device][GPS receiver]]. The NMEA location data produced by the receiver is


@@ -17,21 +22,21 @@ See the [[https://www.raspberrypi.org/pi-zero-w/][OEM]] webpage for this product
 **** PiZ UpTime 2.0
 See the [[https://alchemy-power.com/piz-uptime-2-0/][OEM]] webpage for this product.
 *** Software
 **** PiZ UpTime 2.0
 See the [[https://alchemy-power.com/piz-uptime-2-0/][OEM]] webpage for this product.
 *** Software

-~bkgpslog~ : The bash script that performs the location data
-collection and processing. Is an executable file contained within this
-repository at ~exec/bkgpslog~. It should be copied to
-~$HOME/.local/bin~.
+~bklog~ : A bash script that saves its stdin stream to a tar file. The
+file may be compressed by ~gzip~ and encrypted by ~age~. It is an
+executable file contained within this repository at ~exec/bklog~. It
+should be copied to ~$HOME/.local/bin~.

 
 

-~gpsd~ : A background daemon app capable of interfacing with the
-Ozzmaker BerryGPS-IMU's GPS submodule. Installed and initialized by
-~apt~.
+[[https://tracker.debian.org/pkg/gpsd][~gpsd~]] : A background daemon app capable of interfacing with the
+[[https://ozzmaker.com/berrygps-berrygps-imu-quick-start-guide/][Ozzmaker BerryGPS-IMU]]'s GPS submodule. Installed and initialized by
+~apt~. Should be installed along with the ~gpsd-clients~ package.

 
 ~gpspipe~ : A command line app that polls ~gpsd~ and produces a stream
 stdout consisting of GPS data lines in NMEA format. Installed via
 ~apt~.
 
 ~gpsbabel~ : A command line app that converts GPS data from one format
 
 ~gpspipe~ : A command line app that polls ~gpsd~ and produces a stream
 stdout consisting of GPS data lines in NMEA format. Installed via
 ~apt~.
 
 ~gpsbabel~ : A command line app that converts GPS data from one format

-into another. ~bkgpslog~ uses it to convert NMEA data into GPX and
+into another. ~bklog~ may be used to convert NMEA data into GPX and

 KML. Installed via ~apt~.
 
 ~gzip~ : A simple command line app that compresses stdin into a
 KML. Installed via ~apt~.
 
 ~gzip~ : A simple command line app that compresses stdin into a


@@ -43,61 +48,44 @@ executable file contained within this repository at ~exec/age~. It
 should be copied to ~$HOME/.local/bin~.
 
 **** Narrative
 should be copied to ~$HOME/.local/bin~.
 
 **** Narrative

-~bkgpslog~ populates a 60-second buffer with NMEA data from ~gpsd~ via
-~gpspipe~. This buffer is used by ~gpsbabel~ to produce GPX and KML
-versions of the buffer. All 3 buffers are then comprssed with ~gzip~,
-encrypted with ~age~, and then written to disk.
+~bklog~ may be used to log location data by receiving stdout produced
+by ~gpspipe~. ~bklog~ contains options that allow file writes to be
+performed at adjustable time intervals (default: 300 seconds),
+compressed (with ~gzip~), and encrypted (with ~age~). Files are
+written in the form of appendages to a ~tar~ archive saved to a
+specified location. The NMEA data produced by ~gpspipe~ may be
+processed via a ~gpsbabel~ command string specified as an option to
+~bklog~, assuming ~gpsbabel~ is installed.

 
 *** Output
 
 *** Output

+
+Several output file formats have been tested with ~bklog~.
+

 **** File Formats
 ***** NMEA
 **** File Formats
 ***** NMEA

-See the [[https://en.wikipedia.org/wiki/NMEA_0183][Wikipedia page]] for this.
+~NMEA~ is an acronym for National marine Electronics Association. The
+NMEA format described in this document follows the NMEA 0183
+standard. It is a newline-delimited streaming text format that encodes
+global positioning system (GPS) data such as WGS84 location, time and
+date information, satellite count, accuracy, and other
+information. Each line is an "NMEA sentence". Descriptions of various
+NMEA sentences can be found on [[http://aprs.gids.nl/nmea/][this]] webpage.
+
+See the [[https://en.wikipedia.org/wiki/NMEA_0183][Wikipedia page for NMEA 0183]] for this. This file format is the
+default output of the ~gpspipe -r~ command.

 ***** GPX
 ***** GPX

-See the [[https://en.wikipedia.org/wiki/GPS_Exchange_Format][Wikipedia page]] for this. [[http://wiki.gis.com/wiki/index.php/WGS84][WGS84]] is the datum used.
+See the [[https://en.wikipedia.org/wiki/GPS_Exchange_Format][Wikipedia page]] for this. [[http://wiki.gis.com/wiki/index.php/WGS84][WGS84]] is the datum used. An NMEA file
+may be converted to this format using ~gpsbabel~.

 ***** KML
 ***** KML

-See the [[https://en.wikipedia.org/wiki/Keyhole_Markup_Language][Wikipedia page]] for this. [[http://wiki.gis.com/wiki/index.php/WGS84][WGS84]] is the datum used.
-**** Encryption Method
-Files produced by the bkgpslog script are encrypted against a set of
-public keys using [[https://github.com/FiloSottile/age][~age~]], a simple command line encryption tool
-selected over ~gpg~ because of ~age~'s deliberate lack of
-configurability.
+See the [[https://en.wikipedia.org/wiki/Keyhole_Markup_Language][Wikipedia page]] for this. [[http://wiki.gis.com/wiki/index.php/WGS84][WGS84]] is the datum used. An NMEA file
+may be converted to this format using ~gpsbabel~.

 
 

-The public keys are bech32 strings supplied as options to bkgpslog
-when called. The secret key should *NOT* be stored in Ninfacyzga-01.
-
-If a key pair was generated using ~age-keygen~, then it is an [[https://en.wikipedia.org/wiki/Curve25519][~X25519~]]
-key pair. See the [[https://age-encryption.org/v1][~age~ Version 1 specification]].
-
-An ~ssh-rsa~ or ~ssh-ed25519~ SSH public key string may be used instead of
-the bech32 public key string produced by ~age-keygen~ for convenience.
-
-Help information for ~age~ is available by running ~$ age --help~.
-***** Encryption
-Files may be encrypted to several recipients using a command similar to:
-#+BEGIN_EXAMPLE
-timeout "60s" gpspipe -r | gpsbabel -i nmea -f - -o gpx -F | age \
--r age1kza7pfshy7xwygf9349zgmk7x53mquvedgw9r98qwyyqhssh830qqjzlsw \
--r age1ce3pvzrqfcn2pc6zqzglc8ac8yjk3fzukpy08cesqjjwns53xywqmaq7xw \
--r age1pu5usxm743sx7rf22985xv2f4s0luzv6r6yx4fa7p8c2zyvp9fvqus2xr5 \
-> location.gpx.age
-#+END_EXAMPLE
-
-In this example, the strings beginning with ~age1...~ are
-bech32-formatted public key strings. 
-
-
-***** Decryption
-Files may be decrypted using a command similar to:
-
-#+BEGIN_EXAMPLE
-cat location.gpx.age | age -d -i key.txt > location.gpx
-#+END_EXAMPLE
+**** Encryption Method
+See [[file:../setup/README.org][Main Setup]] procedures.

 
 

-The version of ~age~ used to perform the encryption 

 ** Operating Procedures
 *** Initial Startup
 See OEM (Ozzmaker) [[https://ozzmaker.com/berrygps-berrygps-imu-quick-start-guide/][quickstart guide for the BerryGPS-IMU]].
 ** Operating Procedures
 *** Initial Startup
 See OEM (Ozzmaker) [[https://ozzmaker.com/berrygps-berrygps-imu-quick-start-guide/][quickstart guide for the BerryGPS-IMU]].

-

 **** Physical Setup
 
 BerryGPS-IMU must be electrically connected to the correct pins on the
 **** Physical Setup
 
 BerryGPS-IMU must be electrically connected to the correct pins on the


@@ -108,66 +96,127 @@ conflict so a simple stacking and soldering with long header pins is
 possible.
 
 **** Software Setup
 possible.
 
 **** Software Setup

-

 ***** Install Executables
 ***** Install Executables

+Follow the [[file:../setup/README.org][Main Setup]] procedures to obtain required files from this
+repository.

 
 

-Install Raspbian 10 Buster onto an SD card image. See the Raspberry Pi
-Foundation [[https://www.raspberrypi.org/documentation/installation/installing-images/README.md][installation instructions]]. Configure WiFi to permit log
-file transfer. Configure SSH to permit remote administration via the
-command line interface.
+Install ~gpsd~, ~gpsd-clients~, and ~gpsbabel~.

 
 

-Make sure to install the ~unattended-upgrades~ package to make sure
-the latest security patches for packages are installed. See [[https://linux-audit.com/using-unattended-upgrades-on-debian-and-ubuntu/][this page]]
-for a description of how ~unattended-upgrades~ works.
+: $ sudo apt install gpsd gpsd-clients gpsbabel

 
 

-Install ~gpsd~, ~gpspipe~, ~git~, and this repository for location
-logging capability.
+***** Setup Serial for BerryGPS
+The Ozzmaker BerryGPS-IMU unit requires that the serial console be
+disabled and the serial port enabled. (see [[https://ozzmaker.com/berrygps-setup-guide-raspberry-pi/][ref]]).

 
 

-Install ~syncthing~ for log file transfer capability.
+: $ sudo raspi-config

 
 

-Place ~age~ binary (the one compiled for ARM CPU architecture for
-Linux) in ~$HOME/.local/bin~.
+Navigate to ~5 Interfacing Options~, then ~P6 Serial~.

 
 

-***** Automatic Start Configuration
+When prompted "Would you like a login shell to be accessible over
+serial?", answer ~No~.

 
 

-Edit the user cron job list with ~$ crontab -e~ to add the following
-lines:
+When prompted "Would you like the serial port hardware to be
+enabled?", answer ~Yes~.

 
 

-#+BEGIN_EXAMPLE
-0 * * * * /bin/bash ~/bkgpslog --output ~/dir
+***** Setup ~gpsd~
+~gpsd~ needs to know which serial port to look at for NMEA location
+data generated by the GPS unit. This can be done by modifying the
+~gpsd~ configuration file at ~/etc/default/gpsd~.
+
+: sudo nano /etc/default/gpsd
+
+Make sure the following lines are present:

 
 

-@reboot /bin/bash ~/bkgpslog --output ~/dir
+#+BEGIN_EXAMPLE
+START_DAEMON="true"
+USBAUTO="false"
+DEVICES="/dev/ttyAMA0"
+GPSD_OPTIONS="-n"

 #+END_EXAMPLE
 
 #+END_EXAMPLE
 

-The first line will run ~bkgpslog~ at the start of every hour and save
-output files to the ~dir~ directory in your home folder.
+NOTE: The ~DEVICES=~ line must be adjusted if the the ~ninfacyzga~
+device is configured to also track time.

 
 

-The second line will run ~bkgpslog~ when the system starts up.
+NOTE: The ~-n~ option causes ~gpsd~ to begin polling GPS devices
+without waiting for a client to connect. This is important if ~gpsd~
+provides data to other applications (ex: ~gpspipe~, ~ntpsec~,
+~chrony~).

 
 

-~/bin/bash~ tells ~cron~ to run ~bkgpslog~ with Bash.
+Restart ~gpsd~ via:

 
 

-If encryption and compression are required, then the appropriate
-options must be added. The lines that must be added via ~$ crontab -e~
-may resemble:
+: $ sudo systemctl restart gpsd.service

 
 

-#+BEGIN_EXAMPLE
-0 * * * * /bin/bash ~/bkgpslog -c -e -r age1z2...qkv6p -o ~/dir
+Run ~gpsmon~ to verify that location data is being recorded.
+
+: $ gpsmon

 
 

-@reboot /bin/bash ~/bkgpslog -c -e -r age1z2...qkv6p -o ~/dir
+***** Automatic Start Configuration
+It is recommended to create a daily ~cron~ job that executes a bash
+script that starts ~bklog~ jobs to record location along with other
+types of data recording. An example of such a script is below:
+
+#+BEGIN_EXAMPLE
+#!/bin/bash
+
+# Log location
+gpspipe -r | /bin/bash "$HOME/.local/bin/bklog" -c -e -z "UTC" -t "/dev/shm" \
+          -r age1kza7pfshy7xwygf9349zgmk7x53mquvedgw9r98qwyyqhssh830qqjzlsw \
+          -r age1ce3pvzrqfcn2pc6zqzglc8ac8yjk3fzukpy08cesqjjwns53xywqmaq7xw \
+          -R "$HOME/.config/bklog/recipients" -w ".nmea" -b "300" -B "day" \
+         -o "$HOME/Sync/Evanescent_Location" -l "location" \
+          -p "gpsbabel -i nmea -f - -o gpx -F - " ".gpx" \
+          -p "gpsbabel -i nmea -f - -o kml -F - " ".kml" &
+# Log pressure
+python ~/.local/bin/bmp388.py | /bin/bash "$HOME/git-OC/ninfacyzga-01/exec/bklog" \
+          -c -e -z "UTC" -t "/dev/shm" \
+          -r age1kza7pfshy7xwygf9349zgmk7x53mquvedgw9r98qwyyqhssh830qqjzlsw \
+          -r age1ce3pvzrqfcn2pc6zqzglc8ac8yjk3fzukpy08cesqjjwns53xywqmaq7xw \
+          -R "$HOME/.config/bklog/recipients" -w ".txt" -b "300" -B "day" \
+         -o "$HOME/Sync/Evanescent_Location" -l "pressure" &

 #+END_EXAMPLE
 
 #+END_EXAMPLE
 

-The ~age1z2...qkv6p~ is an ~age~ public key string. See the Encryption
-Methods section for an explanation.
+This script, if it were saved at
+~"$HOME/.local/bin/cron/dailylog.sh"~ would then be added as a
+line in the ~crontab~ file as shown below:

 
 

-The options are:
+#+BEGIN_EXAMPLE
+$ crontab -e
+0 0 * * * /bin/bash ~/.local/bin/cron/dailylog.sh
+@reboot /bin/bash ~/.local/bin/cron/dailylog.sh
+#+END_EXAMPLE

 
 

-: -c : tells bkgpslog to compress output
-: -e : tells bkgpslog log to encrypt output
-: -r : tells bkgpslog to interpret the next argument as a pubkey string
-: -o : tells bkgpslog to write output files to the directory represented
+In the example script, the options are:
+
+: -c : tells bklog to compress output
+: -e : tells bklog log to encrypt output
+: -r : tells bklog to interpret the next argument as a pubkey string
+: -R : tells bklog to interpret the next argument as a directory
+:        where public keys may be found (first line of each file is
+:        read).
+: -o : tells bklog to write output files to the directory represented
+: -t : tells bklog to interpret the next argument as a directory
+:        for storing temporary files

 :        by the next argument
 :        by the next argument

+: -p : tells bklog a command string through which output is piped
+:        before being compressed and encrypted. Also expected is a
+:        file extension to be appended before the compression and
+:        encryption file extensions.
+: -w : tells bklog to save the unprocessed stdin with a specified
+:        file extension (instead of the default '.stdin').
+: -b : tells bklog how long each buffer round (time between file
+:        writes) lasts in seconds.
+: -B : specifies the time-to-live for the bklog script. A valid value may
+:        one of the time elements such as "day" or "hour".
+: -l : specfies a custom string to be used in output file names to
+:        help differentiate tar files produced via bklog from different
+:        sources of data.
+: -z : specifies a time zone to be used to determine the script time-to-live.
+:        By default, bklog uses whatever time is specified by the TZ
+:        environment variable.

 
 ***** Log Transfer Configuration
 
 ***** Log Transfer Configuration

+See [[file:../setup/README.org][Main Setup]] procedures.
+

 Log files may be shared to other machines via ~syncthing~. See [[https://docs.syncthing.net/][this]]
 manual for how to set up a shared folder and add Ninfacyzga-01 as a
 device. Syncthing's directory synchronization capability allows a
 Log files may be shared to other machines via ~syncthing~. See [[https://docs.syncthing.net/][this]]
 manual for how to set up a shared folder and add Ninfacyzga-01 as a
 device. Syncthing's directory synchronization capability allows a


@@ -178,35 +227,7 @@ When log files are removed from Ninfacyzga-01 is not within the scope
 of this document.
 
 ***** Key Generation
 of this document.
 
 ***** Key Generation

-An ~age~ encryption key may be generated like so:
-#+BEGIN_EXAMPLE
-$ umask          # Gets current umask
-0022             # Note: This is the default umask for Raspbian 10
-$ umask 066      # Sets umask so key.txt will have no permissions except for owner (you)
-$ umask          # Confirm umask set to 066
-0066
-$ age-keygen > key.txt
-Public key: age1pu5usxm743sx7rf22985xv2f4s0luzv6r6yx4fa7p8c2zyvp9fvqus2xr5
-$ ls -al key.txt
--rw------- 1 baltakatei baltakatei 184 Jun 29 18:28 key.txt
-$ umask 0022     # Return umask to default value
-$ umask          
-0022
-#+END_EXAMPLE
-
-The resulting public/private keypair data looks like:
-#+BEGIN_EXAMPLE
-$ cat key.txt
-# created: 2020-06-29T18:01:56Z
-# public key: age1pu5usxm743sx7rf22985xv2f4s0luzv6r6yx4fa7p8c2zyvp9fvqus2xr5
-AGE-SECRET-KEY-1NEUU5U2XGZGL9UYWNPU5DL99TGJJHFSN4F2E2WCCSDJJ6L5ZMLESNTVTU0
-#+END_EXAMPLE
-
-The file ~key.txt~ is not password-protected by default and should be
-secured like an SSH public key should. The ~$ umask 066~ command run
-before the ~$ age-keygen > key.txt~ command ensures ~key.txt~ will not
-be readable, writeable, or executable to anyone except the owner
-(you).
+See [[file:../setup/README.org][Main Setup]] procedures.

 
 *** Normal Startup
 Turn on Ninfacyzga-01 by supplying 5VDC power to the Raspberry Pi. No
 
 *** Normal Startup
 Turn on Ninfacyzga-01 by supplying 5VDC power to the Raspberry Pi. No


@@ -230,15 +251,4 @@ Ninfacyzga-01 as described and setup should tolerate unscheduled power
 loss. Log files being written every 60 seconds means, at most, 60
 seconds worth of location data may be lost.
 *** End of Life Disposal
 loss. Log files being written every 60 seconds means, at most, 60
 seconds worth of location data may be lost.
 *** End of Life Disposal

-LiPo batteries used by the PiZ Uptime 2.0 module should be disposed of
-properly with their potential ignitability in mind, especially if they
-are not fully discharged.
-
-Consult your local municipality for its "E-Waste Disposal" (or
-equivalent) policy. Metals used in the Raspberry Pi and related
-components may be recycled.
-
-Take extra precuation if lead solder was used in assembling the
-electronics. Consumer electronics in early 21st century should use
-lead-free solder.
-
+See [[file:../setup/README.org][Main Setup]] procedures.