From: Steven Baltakatei Sandoval Date: Sun, 5 Jul 2020 16:56:56 +0000 (+0000) Subject: feat(bkgpslog):Add functions, group code into functions X-Git-Url: https://zdv2.bktei.com/gitweb/EVA-2020-02.git/commitdiff_plain/320ac29c93864f2d5f3229bbc8b628ddac5371f9?ds=inline feat(bkgpslog):Add functions, group code into functions Add several fucntions to add future new feature (recipient dir watch) as well as to group existing code blocks into their own functions for readability. Changes include: * Add checkAgePubkey() function: checks that a provided string doesn't throw an error when fed as a recipient string to `age`. * Pushed several code blocks into their own "magic" functions ** magicParseRecipientDir() : (future feature): To be run at each buffer session to update the recipient array in response to changes in a specified recipient directory. ** magicParseRecipientArgs(): Parses strings provided by '-r' option and format output accordingly. Existing code. ** magicParseCompressionArgs(): Responds to '-c' option by specifying compression function and output file suffix. * Update showUsage() to document future '-R' recipient dir option. --- diff --git a/exec/bkgpslog b/exec/bkgpslog index 69401ee..6daa541 100755 --- a/exec/bkgpslog +++ b/exec/bkgpslog @@ -23,8 +23,7 @@ AGE_URL="https://github.com/FiloSottile/age/releases/tag/v1.0.0-beta2"; # Define declare -Ag appRollCall # Associative array for storing app status declare -Ag fileRollCall # Associative array for storing file status declare -Ag dirRollCall # Associative array for storing dir status -declare -a recPubKeys # for processArguments function -declare recipients # for main function +declare -a argRecPubKeys # for processArguments function ## Initialize variables OPTION_VERBOSE=""; OPTION_ENCRYPT=""; OPTION_COMPRESS=""; OPTION_TMPDIR=""; @@ -147,35 +146,31 @@ showUsage() { echoerr "OPTIONS:" echoerr " -h, --help" echoerr " Display help information." - echoerr echoerr " --version" echoerr " Display script version." - echoerr echoerr " -v, --verbose" echoerr " Display debugging info." - echoerr echoerr " -e, --encrypt" echoerr " Encrypt output." - echoerr echoerr " -r, --recipient [ pubkey string ]" echoerr " Specify recipient. May be age or ssh pubkey." echoerr " See https://github.com/FiloSottile/age" - echoerr echoerr " -o, --output [ directory ]" echoerr " Specify output directory to save logs." - echoerr echoerr " -c, --compress" echoerr " Compress output with gzip (before encryption if enabled)." - echoerr echoerr " -z, --time-zone" echoerr " Specify time zone. (ex: \"America/New_York\")" - echoerr echoerr " -t, --temp-dir" echoerr " Specify parent directory for temporary working directory." echoerr " Default: \"/dev/shm\"" + echoerr " -R, --recipient-dir" + echoerr " Specify directory containing files whose first lines are" + echoerr " to be interpreted as pubkey strings (see \'-r\' option)." echoerr echoerr "EXAMPLE: (bash script lines)" - echoerr "/bin/bash bkgpslog -e -c \\" + echoerr "/bin/bash bkgpslog -v -e -c \\" + echoerr "-z \"UTC\" -t \"/dev/shm\" \\" echoerr "-r age1mrmfnwhtlprn4jquex0ukmwcm7y2nxlphuzgsgv8ew2k9mewy3rs8u7su5 \\" echoerr "-r age1ala848kqrvxc88rzaauc6vc5v0fqrvef9dxyk79m0vjea3hagclswu0lgq \\" echoerr "-o ~/Sync/Location" @@ -216,10 +211,11 @@ processArguments() { -o | --output) if [ -d "$2" ]; then DIR_OUT="$2"; vbm "DEBUG:DIR_OUT:$DIR_OUT"; shift; fi ;; # Define output directory. -e | --encrypt) OPTION_ENCRYPT="true"; vbm "DEBUG:Encrypted output mode enabled.";; -r | --recipient) # Add 'age' recipient via public key string - recPubKeys+=("$2"); vbm "STATUS:pubkey added:""$2"; shift;; + argRecPubKeys+=("$2"); vbm "STATUS:pubkey added:""$2"; shift;; -c | --compress) OPTION_COMPRESS="true"; vbm "DEBUG:Compressed output mode enabled.";; -z | --time-zone) try setTimeZoneEV "$2"; shift;; - -t | --temp-dir) OPTION_TMPDIR="true" && TMP_DIR_PRIORITY="$2"; shift;; + -t | --temp-dir) OPTION_TMPDIR="true" && argTmpDirPriority="$2"; shift;; + -R | --recipient-dir) OPTION_RECDIR="true" && argRecDir="$2"; shift;; *) echoerr "ERROR: Unrecognized argument: $1"; echoerr "STATUS:All arguments:$*"; exit 1;; # Handle unrecognized options. esac shift @@ -819,6 +815,24 @@ appendFileTar(){ try tar --append --directory="$TMP_DIR" --file="$TAR_PATH" "$FILENAME"; #yell "DEBUG:STATUS:$FN:Finished appendFileTar()." } # Append file to Tar archive +checkAgePubkey() { + # Desc: Checks if string is an age-compatible pubkey + # Usage: checkAgePubkey [str pubkey] + # Version: 0.1.2 + # Input: arg1: string + # Output: return code 0: string is age-compatible pubkey + # return code 1: string is NOT an age-compatible pubkey + # age stderr (ex: there is stderr if invalid string provided) + # Depends: age (v0.1.0-beta2; https://github.com/FiloSottile/age/releases/tag/v1.0.0-beta2 ) + + argPubkey="$1"; + + if echo "test" | age -a -r "$argPubkey" 1>/dev/null; then + return 0; + else + return 1; + fi; +} # Check age pubkey validateInput() { # Desc: Validates Input # Usage: validateInput [str input] [str input type] @@ -937,41 +951,61 @@ magicGatherWriteBuffer() { rm "$PATHOUT_BUFFER" "$PATHOUT_NMEA" "$PATHOUT_GPX" "$PATHOUT_KML"; vbm "DEBUG:STATUS:$FN:Finished magicWriteBuffer()."; } # write buffer to disk - -main() { - processArguments "$@" # Process arguments. +magicParseRecipientDir() { + # Desc: Updates recPubKeysValid with pubkeys in dir specified by '-R' option ("recipient directory") + # Inputs: vars: OPTION_RECDIR, argRecDir, + # arry: recPubKeysValid + # Outputs: arry: recPubKeysValid (modified with pubkeys in argRecDir if pubkeys valid) + # Depends: processArguments, + local recFileLine updateRecipients recipientDir + declare -a candRecPubKeysValid - # Determine working directory - ## Set DIR_TMP_PARENT to user-specified value if specified - if [[ "$OPTION_TMPDIR" = "true" ]]; then - if [[ -d "$TMP_DIR_PRIORITY" ]]; then - DIR_TMP_PARENT="$OPTION_TMPDIR"; + if [[ "$OPTION_RECDIR" = "true" ]]; then + ### Check that argRecDir is a directory. + if [[ -d "$argRecDir" ]]; then + recipientDir="$argRecDir"; + #### Initialize variable indicating outcome of pubkey review + unset updateRecipients + #### Add existing recipients + candRecPubKeysValid=(${recPubKeysValid[@]}); + #### Parse files in recipientDir + for file in "$recipientDir"/*; do + ##### Read first line of each file + recFileLine="$(cat "$file" | head -n1)"; + ##### check if first line is a valid pubkey + if checkAgePubkey "$recFileLine" && \ + ( validateInput "$recFileLine" "ssh_pubkey" || validateInput "$recFileLine" "age_pubkey"); then + ###### T: add candidate pubkey to candRecPubKeysValid + candRecPubKeysValid+=("$recFileLine"); + else + ###### F: throw warning; + yell "ERROR:Invalid recipient file detected. Not modifying recipient list." + updateRecipients="false"; + fi; + done + #### Write updated recPubKeysValid array to recPubKeysValid if no failure detected + if ! updateRecipients="false"; then + recPubKeysValid=(${candRecPubKeysValid[@]}); + fi; else - yell "WARNING:Specified temporary working directory not valid:$OPTION_TMPDIR"; - exit 1; - fi - fi - - ## Set DIR_TMP_PARENT to default or fallback otherwise - if [[ -d "$DIR_TMP_DEFAULT" ]]; then - DIR_TMP_PARENT="$DIR_TMP_DEFAULT"; - elif [[ -d /tmp ]]; then - yell "WARNING:/dev/shm not available. Falling back to /tmp ."; - DIR_TMP_PARENT="/tmp"; - else - yell "ERROR:No valid working directory available. Exiting."; - exit 1; - fi + yell "ERROR:$0:Recipient directory $argRecDir does not exist. Exiting."; exit 1; + fi; + fi; +} # Update recPubKeysValid with argRecDir +magicParseRecipientArgs() { + # Desc: Parses recipient arguments specified by '-r' option + # Input: vars: OPTION_ENCRYPT from processArguments() + # arry: argRecPubKeys from processArguments() + # Output: vars: CMD_ENCRYPT, CMD_ENCRYPT_SUFFIX + # arry: recPubKeysValid + # Depends: checkapp(), checkAgePubkey(), validateInput(), processArguments() + local recipients - ## Set DIR_TMP using DIR_TMP_PARENT and nonce (SCRIPT_TIME_START) - DIR_TMP="$DIR_TMP_PARENT"/"$SCRIPT_TIME_START""..bkgpslog" && vbm "DEBUG:Set DIR_TMP to:$DIR_TMP"; # Note: removed at end of main(). - - # Set output encryption and compression option strings if [[ "$OPTION_ENCRYPT" = "true" ]]; then # Check if encryption option active. if checkapp age; then # Check that age is available. - for pubkey in "${recPubKeys[@]}"; do # Validate recipient pubkey strings by forming test message + for pubkey in "${argRecPubKeys[@]}"; do # Validate recipient pubkey strings by forming test message vbm "DEBUG:Testing pubkey string:$pubkey"; - if echo "butts" | age -a -r "$pubkey" 1>/dev/null && + if checkAgePubkey "$pubkey" && \ ( validateInput "$pubkey" "ssh_pubkey" || validateInput "$pubkey" "age_pubkey"); then #### Form age recipient string recipients="$recipients""-r '$pubkey' "; @@ -983,7 +1017,7 @@ main() { yell "ERROR:Exit code ""$?"". Invalid recipient pubkey string. Exiting."; exit 1; fi done - vbm "DEBUG:Finished processing recPubKeys array"; + vbm "DEBUG:Finished processing argRecPubKeys array"; ## Form age command string CMD_ENCRYPT="age ""$recipients " && vbm "CMD_ENCRYPT:$CMD_ENCRYPT"; @@ -995,7 +1029,13 @@ main() { CMD_ENCRYPT="tee /dev/null " && vbm "CMD_ENCRYPT:$CMD_ENCRYPT"; CMD_ENCRYPT_SUFFIX="" && vbm "CMD_ENCRYPT_SUFFIX:$CMD_ENCRYPT_SUFFIX"; vbm "DEBUG:Encryption not enabled." - fi + fi +} # Populate recPubKeysValid with argRecPubKeys; form encryption cmd string and filename suffix +magicParseCompressionArg() { + # Desc: Parses compression arguments specified by '-c' option + # Input: vars: OPTION_COMPRESS + # Output: CMD_COMPRESS, CMD_COMPRESS_SUFFIX + # Depends: checkapp(), vbm(), gzip, if [[ "$OPTION_COMPRESS" = "true" ]]; then # Check if compression option active if checkapp gzip; then # Check if gzip available CMD_COMPRESS="gzip " && vbm "CMD_COMPRESS:$CMD_COMPRESS"; @@ -1007,10 +1047,57 @@ main() { CMD_COMPRESS="tee /dev/null " && vbm "CMD_COMPRESS:$CMD_COMPRESS"; CMD_COMPRESS_SUFFIX="" && vbm "CMD_COMPRESS_SUFFIX:$CMD_COMPRESS_SUFFIX"; vbm "DEBUG:Compression not enabled."; - fi + fi +} # Form compression cmd string and filename suffix +magicInitWorkingDir() { + # Desc: Determine temporary working directory from defaults or user input + # Input: vars: OPTION_TEMPDIR, argTmpDirPriority, DIR_TMP_DEFAULT + # Input: vars: SCRIPT_TIME_START + # Output: vars: DIR_TMP + # Depends: processArguments(), vbm(), yell() + # Parse '-t' option (user-specified temporary working dir) + ## Set DIR_TMP_PARENT to user-specified value if specified + local DIR_TMP_PARENT + + if [[ "$OPTION_TMPDIR" = "true" ]]; then + if [[ -d "$argTempDirPriority" ]]; then + DIR_TMP_PARENT="$argTempDirPriority"; + else + yell "WARNING:Specified temporary working directory not valid:$OPTION_TMPDIR"; + exit 1; # Exit since user requires a specific temp dir and it is not available. + fi; + else + ## Set DIR_TMP_PARENT to default or fallback otherwise + if [[ -d "$DIR_TMP_DEFAULT" ]]; then + DIR_TMP_PARENT="$DIR_TMP_DEFAULT"; + elif [[ -d /tmp ]]; then + yell "WARNING:$DIR_TMP_DEFAULT not available. Falling back to /tmp ."; + DIR_TMP_PARENT="/tmp"; + else + yell "ERROR:No valid working directory available. Exiting."; + exit 1; + fi + fi; + ## Set DIR_TMP using DIR_TMP_PARENT and nonce (SCRIPT_TIME_START) + DIR_TMP="$DIR_TMP_PARENT"/"$SCRIPT_TIME_START""..bkgpslog" && vbm "DEBUG:Set DIR_TMP to:$DIR_TMP"; # Note: removed at end of main(). +} # Sets working dir - # Check that critical apps and dirs are available, displag missing ones. - if ! checkapp gpspipe tar && ! checkdir "$DIR_OUT" "/dev/shm"; then +main() { + # Process arguments + processArguments "$@"; + ## Act upon arguments + ### Determine working directory + magicInitWorkingDir; + ### Set output encryption and compression option strings + #### React to "-r" ("encryption recipients") option + magicParseRecipientArgs; + #### React to "-c" ("compression") option + magicParseCompressionArg; + #### React to "-R" ("recipient directory") option + magicParseRecipientDir; + + # Check that critical apps and dirs are available, display missing ones. + if ! checkapp gpspipe tar && ! checkdir "$DIR_OUT" "DIR_TMP"; then yell "ERROR:Critical components missing."; displayMissing; yell "Exiting."; exit 1; fi diff --git a/exec/bkgpslog-plan.org b/exec/bkgpslog-plan.org index 4ba6806..3e43633 100644 --- a/exec/bkgpslog-plan.org +++ b/exec/bkgpslog-plan.org @@ -126,6 +126,102 @@ about 450. 152K relerror-1000.kml #+END_EXAMPLE +** TODO Feature: Generalize bkgpslog to bklog function +2020-07-05T02:42Z; bktei> Transform ~bkgpslog~ into a modular +component called ~bklog~ such that it processes a stdout stream of any +external command, not just ~gpspipe -r~. This would permit reuse of +the ~bkgpslog~ code for logging not just GPS data but things like +pressure, temperature, system statistics, etc. +2020-07-05T16:35Z; bktei> +: bklog -r age1asdf -o log.tar # encrypt/compress stdin to log.tar +: bklog -x -f log.tar -i age.key -O /tmp # extract and decrypt + +Making ~bklog~ follow the [[https://en.wikipedia.org/wiki/Unix_philosophy][Unix philosophy]] means that it shouldn't care +what kind of text is fed to it. + +*** ~bklog~ Design vs. Unix Philosophy +**** Pubkey dir watching +The feature of periodically checking a directory for changes in the +pubkeys it contains should be justified by its usefulness; if the +complexity cannot be justified then the feature should be removed. +**** Defaults vs options +Many options can cause the tool to become complex in unjustifiable +ways. Currently I am adding options because I want the ability to +modify the script's behavior without having to modify the source code +on the machine in which the code is running. I should consider +removing features at some point and having the program force defaults +on the user. For example, allowing the specification of a temporary +directory, while useful for me, is probably not useful for most people +who don't know or care about the difference between ~/tmp~ and +~/dev/shm~. +**** Script time to live (TTL) +I initially implemented a script time-to-live feature because I was +unsure in my ability to program script that could run for long periods +of time without causing a runaway usage of memory. I still think it's +a good idea to offer a script TTL option to the user but I think the +default should be to simply run forver. +** TODO: Evaluate ~rsyslog~ as stand-in for this work +2020-07-05T02:57Z; bktei> I searched for "debian iot logging" ("iot" +as in "Internet of Things", the current buzzword for small low-power +computers being used to provide microservices for owners in their own +home) and came across several search results mentioning ~syslog~ and +~rsyslog~. + +https://www.thissmarthouse.net/consolidating-iot-logs-into-mysql-using-rsyslog/ +https://rsyslog.readthedocs.io/en/latest/tutorials/tls.html +https://serverfault.com/questions/20840/how-would-you-send-syslog-securely-over-the-public-internet +https://www.rsyslog.com/ + +My impression is that ~rsyslog~ is a complex software package designed +to offer many features, some of which possibly might satisfy my +needs. + +However, as stated in the repository README, the objective of the +~ninfacyzga-01~ project is "Observing facts of the new". This means +that the goal is not only to record location data but any data that +can be captured by a sensor. This means the capture of the following +environmental phenomena are within the scope of this device: + +*** Sounds (microphone) +*** Light (camera) +*** Temperature (thermocouple) +*** Air Pressure (barometer) +*** Acceleration Vector (acceleromter / gyroscope) +*** Magnetic Field Vector (magnetometer) + +This brings up the issue of respecting privacy of others in shared +spaces through which ~ninfacyzga-01~ may pass through. ~ninfacyzga-01~ +should encrypt data it records according to rules set by its +owner. + +One permissive rule could be that if ~ninfacyzga-01~ detects that a +person (let's call her Alice) enters a room, it should add Alice's +encryption public key to the list of recipients against which it +encrypts data without Alice having to know how ~ninfacyzga-01~ is +programmed (she might have a ~calkuptcana~ agent on her person that +broadcasts her privacy preferences). Meanwhile, ~ninfacyzga-01~ may +publish its observations to a repository that Alice and other members +of the shared communal space have access to (ex: a read-only shared +directory on a local network WiFi). Alice could download all the files +in the shared repository but she would only be able to decrypt files +generated when she was physically near enough to ~ninfacyzga-01~ for +it to detect that her presence was within some spatial boundary. + +A more restrictive rule could resemble the permissive rule in that +~ninfacyzga-01~ uses Alice's encryption public key only when she is +physically near by, except that it encrypts logged files against +public keys in a sequential manner. This would mean that all people +who were near ~ninfacyzga-01~ would have to pass around each log file +to eachother so that they could decrypt the content. + +That said, according to [[https://www.rsyslog.com/doc/master/tutorials/database.html][this ~rsyslog~ page]], ~rsyslog~ is more a data +wrangling system for collecting data from disparate sources of +different types and outputting data to text files on disk than a +system committed to the server-client model of database storage. So, I +think converting ~bkgpslog~ into a ~bklog~ script that appends +encrypted and compressed data to a tar file for later extraction +(possibly the same script with future features) would be best. + * bkgpslog narrative ** Initialize environment *** Init variables