From 080ea4c0ff0d4e6b5ce86f664fa6645c1cb02bf0 Mon Sep 17 00:00:00 2001 From: Steven Baltakatei Sandoval Date: Tue, 14 Feb 2023 13:27:57 +0000 Subject: [PATCH 01/16] feat(doc/up/bkshuf/):Add TeXmacs documentation - Note: Compiled PDF uploaded to: https://reboil.com/res/2023/txt/20230214T1327+00..bkshuf_article.pdf --- doc/unitproc/bkshuf/article.tm | 374 +++++++++++++++++++++++++++++++++ 1 file changed, 374 insertions(+) create mode 100644 doc/unitproc/bkshuf/article.tm diff --git a/doc/unitproc/bkshuf/article.tm b/doc/unitproc/bkshuf/article.tm new file mode 100644 index 0000000..0030c11 --- /dev/null +++ b/doc/unitproc/bkshuf/article.tm @@ -0,0 +1,374 @@ + + + + +<\body> + |>>|> + + + + is a -like utility designed to output randomly + sized groups of lines with a group size distribution centered around some + characteristic value. + + + + The author desires to create a -like utility named + to mix line lists in order to produce output line lists with + the following somewhat conflicting properties: + + <\description> + An output line's position should not + contain information about its input line position. + + Two neighboring lines in the + input stream should have a high probability of remaining neighbors in the + output stream. + + + The objective is to improve the value of a short random scan of a small + fraction of a potentially large input list; output that demonstrates ROP as + well as some degree of PIR may achieve this objective. In contrast, the + utility provides PIR but no ROP: a line's neighbor in the + output of is equally likely to be any other random line from + the input.\ + + In other words, output produced by should group together + sequential segments of the input lines in order to partially preserve + relationships that may exist between sequential files. For example, this + could be done by jumping to a random position in the input lines, consuming + (i.e. reading, outputting, and marking a line not to be read again) some + amount of sequential lines, then repeating the process until every line is + consumed. The amount of sequential lines to read between jumps affects how + well the above desired properties are satisfied. + + The objective of is not to completely prevent the + possibility of reassembling the input given the output. Additionally, a + valuable property desired of is output which demonstrates + sufficiently high PIR compared to ROP such that only a short (compared to + the logarithm of the input list size) sequential scan of the output list + from a random starting position is required before a jump to a new group is + is encountered; this would permit the overal contents of very large input + line lists to be sampled. + + + + + + <\eqnarray*> + ||>>|>>||>>|>>||>>|||>>|||>>|>>>||>>|,0>|)>>||>>>|,0>>||>>>> + + + + + <\enumerate-numeric> + Acquire and count input lines (via or positional + arguments). + + Calculate line count >> . + + Calculate target group size . + + Select random unconsumed input line and consume it to + output. + + Consume the next sequential line with probability + >>. Otherwise if some input lines remain + unconsumed, go to step . Otherwise, exit. + + + + + + + The simultaneous presence of ROP and PIR properties in the output depends + upon the amount of sequential lines that are read before + jumps to a new random position in the input list. This amount is the + , ; it is the \Ptarget\Q since + represents the average of a distribution of group sizes that may be + selected, not a single group size. In this analysis, the total number of + lines in the input list is >>. For small input line + counts, (e.g. >\10>) the target group size + should be nearly one (e.g. 1>) since group sizes any larger + than this would have almost no PIR (e.g. a group size of for + >=10> would be 80% identical to the input). For + modest line input counts (e.g. >\100>), the + target group size may be allowed to be larger, such as a tenth of the input + line count (e.g. 10>); this would provide some PIR + (approximately permutations between the approximately + >|s>\\10> groups) + while each line in groups around size would have a low + probability of not being next to its neighbor ( of the 10 lines + would retain the same two neighbors while the two ends would retain one + each). For very large input line counts (e.g. + >\1000000>), + however, breaking up and randomizing the input into ten groups of + 000> offers very little PIR; the benefit + of the very high ROP is also lost since sequential scanning of tens of + thousands of lines is required before a random jump to a new group may be + encountered; therefore, the target group size should be a much smaller + fraction of >>, (e.g. 20>) while still + increasing. The relationship between a desireable target group size + and the input line count >> is non-linear. + The author believes a reasonable approach is to scale the group size to the + logarithm of input line count. + + > shows an example plot of + >|)>> that is tuned to achieve a target + group size of >=10|)>=25> for an + input list length of >=10> lines.\ + + <\big-figure||gr-frame|>|gr-geometry||gr-grid||1>|gr-grid-old||1>|gr-edit-grid-aspect|||>|gr-edit-grid||1>|gr-edit-grid-old||1>|gr-grid-aspect-props|||>|gr-grid-aspect|||>|magnify|1.18920711463847|gr-auto-crop|false|>|>|>|>|>|>|>||||>||>||>||>||>||>|>|>|>|>|>|>|>|>|>|||||>>> + A plot of a possible function that relates target + group size and input lines >> that provide + some ROP and PIR. The function is tuned to achieve + >=10|)>=25>.\ + + + The following is a set of equations that are used to derive a definition + for >|)>> that satisfies the plot in + >.\ + + <\eqnarray*> + || + >|)>=>|)>| + >>>|>||>>>|>|| + ,0>|)>=,0>|)>| + >>>|>>||,0>>>|||>|>||+1>>|=25>||\+1>>|||\+1>>||)>>|||)>+1>>||)>-1|x>>||>>||||)>-1|x>>>>||||)>-1>|x>>>|>|||)>-1|x>>>|>|)>>|||)>-1|x>|)>\>|)>| + >|)>+1>>||| + |ln ,0>|)>>|)>\|)>-1|)>\>|)>| + >|)>+1>>|>|)>>||,0>|)>-1|)>\>|)>|ln + ,0>|)>>|)>+1>>|>|)>>||,0>|)>-1|,0>|)>|]>>|)>\>|)>|]>+1>>>> + + + Equation defines a quadratic equation for the + linear range and the logarithmic domain . is + defined in terms of >> via a domain transformation + defined by >. The result is + > which defines + >|)>> as a function of + >> and parameters + ,0>|)>> and ,0>>. + The parameters define how quickly or slowly the quadratic equation grows. + In other words, if a user wishes for a 000000> + line input to be split into groups each containing, on average, + lines, then they should plug in ,0>=1000000> + and ,0>|)>=25> into > as is done in >. This equation can then be used to calculate target group + sizes as a function of other input line counts + >> besides >=1000000>. + For example, plugging >=500> into > yields > which specifies a target group size of + 6>. + + <\eqnarray*> + >|)>>||,0>|)>-1|,0>|)>|]>>|)>\>|)>|]>+1>>|>|)>>||000000|)>|]>>|)>\>|)>|]>+1>>|||>|>=500|)>>||000000|)>|]>>|)>\|]>+1>>|>=500|)>>||>>> + + + \; + + + + A method may employ to decide when read the next sequential + unconsumed input line is to simply do so with probability + >> such that the expected value of the average group + size trends towards . + + <\eqnarray*> + >>||>|)>>>|>>||>>>|||>>=>>>>|||>>>>|>>||>>|>-1>||>>|>>||>|)>>>>>|>>||>|)>>>>>|||>|>>|)>>||,0>|)>-1|,0>|)>|]>>|)>\>|)>|]>+1|]>>>>>> + + + + + Another method may employ to decide when to read the next + sequential unconsumed input line is to use an inverse gaussian + distribution. This may be done by generating from the distribution a float + sampled from the inverse gaussian with range 0 to infinity with mean + > whenever a new random position in the input list is selected; + the float is rounded to the nearest integer.<\footnote> + See "Generating Random Variates Using + Transformations with Multiple Roots". 1976. + . + Then, after consuming an input line, this integer is + decremented by one and another sequential line is consumed provided the + integer does not become less than or equal to zero. The inverse gaussian + distribution requires specifying the mean > and the shape + parameter >; a higher > results in a + greater spread. An upper bound may also be specified since the distribution + has none except for that imposed by its programming implementation. + + The result of using an inverse gaussian distribution is an output with + potentially much more regular group sizes than using the previously + mentioned expected value method. However, the implementation of the inverse + gaussian sampling operation described by (Michael, 1976) requires several + exponent calculations and a square root calculation in addition to various + multiplication and division operations. If sufficient processing power is + available, this may not necessarily be an issue. + + + + Regardless of whether group sizes are determined by the expected value + method or using variates of an inverse gaussian distribution, mimicking the + property of all input lines being present in the output, + albeit rearranged, results in a side effect: the first output lines are + more likely to contain groups with uninterrupted sequence runs (high ROP) + while groups in the last output lines are almost certain to contain + sequence jumps within a group (less ROP). The reason for this is that + , when it encounters an input line that has already been + consumed, will skip to the next available input line. The decision could be + made to skip to a new random line, but, it is simpler to simply read the + next available input line. The author's original intention of sampling only + a short initial portion of the output is compatible with the behavior that + ROP is preserved mostly at the beginning of the output. + + + + <\big-table||||>|>||>|.>>>>>> + A table listing versions of . + + + <\description> + Initial implementation in + with and + and tested on . Saved + to the author's repository<\footnote> + See commit at + . + . + + + + +> + +<\references> + <\collection> + > + > + > + > + > + > + > + > + > + > + > + > + > + > + > + > + > + > + > + > + > + > + > + > + > + > + > + > + + + +<\auxiliary> + <\collection> + <\associate|figure> + |1>|> + A plot of a possible function that relates target group size + |s> and input lines + |l|in>>> that + provide some ROP and PIR. The function is tuned to achieve + |s|in>>=10|)>=25>. + |> + + <\associate|table> + |1>|> + \; + |> + + <\associate|toc> + |math-font-series||1Summary> + |.>>>>|> + + + |math-font-series||2Objective> + |.>>>>|> + + + |math-font-series||3Design> + |.>>>>|> + + + |3.1Definitions + |.>>>>|> + > + + |3.2Process + |.>>>>|> + > + + |3.3Parameter analysis + |.>>>>|> + > + + |3.3.1Target group size + calculation |.>>>>|> + > + + |3.3.2Jump from expected value + |.>>>>|> + > + + |3.3.3Jump from random variate + of inverse gaussian distribution |.>>>>|> + > + + |3.3.4Output structure + |.>>>>|> + > + + |math-font-series||4Version + History> |.>>>>|> + + + + \ No newline at end of file -- 2.30.2 From 8d35dc96ccee8a30e5e54873e319395a4c6bfe53 Mon Sep 17 00:00:00 2001 From: Steven Baltakatei Sandoval Date: Tue, 14 Feb 2023 13:49:07 +0000 Subject: [PATCH 02/16] chore(doc/up/bkshuf/article.tm):Update GitLab URL in bkshuf doc --- doc/unitproc/bkshuf/article.tm | 61 +++++++++++++++++----------------- 1 file changed, 31 insertions(+), 30 deletions(-) diff --git a/doc/unitproc/bkshuf/article.tm b/doc/unitproc/bkshuf/article.tm index 0030c11..46f2c0e 100644 --- a/doc/unitproc/bkshuf/article.tm +++ b/doc/unitproc/bkshuf/article.tm @@ -6,7 +6,7 @@ |>>|> + Sandoval>>>||> @@ -267,7 +267,8 @@ with and and tested on . Saved to the author's repository<\footnote> - See commit at + See commit |https://gitlab.com/baltakatei/baltakatei-exdev/-/blob/080ea4c0ff0d4e6b5ce86f664fa6645c1cb02bf0/unitproc/bkshuf> + at . . @@ -278,34 +279,34 @@ <\references> <\collection> - > - > - > - > - > + > + > + > + > + > > - > - > - > - > - > - > - > - > - > - > - > - > - > - > - > - > - > - > - > - > - > - > + > + > + > + > + > + > + > + > + > + > + > + > + > + > + > + > + > + > + > + > + > + > @@ -322,7 +323,7 @@ <\associate|table> |1>|> - \; + A table listing versions of |prog-language||font-family||bkshuf>. |> <\associate|toc> -- 2.30.2 From b9e8b771e985fcdf26ba8b9ccb8e31b62da757d3 Mon Sep 17 00:00:00 2001 From: Steven Baltakatei Sandoval Date: Sat, 18 Feb 2023 15:31:56 +0000 Subject: [PATCH 03/16] feat(user/bk-copy-rand-music):Clump output with bkshuf - feat(unitproc/bkshuf):Utilize environment variables if available - doc(.../bkshuf/article.tm):update TeXmacs article --- doc/unitproc/bkshuf/article.tm | 21 ++- unitproc/bkshuf | 9 +- ...-copy-rand-music.sh => bk-copy-rand-music} | 158 +++++++++++------- 3 files changed, 115 insertions(+), 73 deletions(-) rename user/{bk-copy-rand-music.sh => bk-copy-rand-music} (74%) mode change 100755 => 100644 diff --git a/doc/unitproc/bkshuf/article.tm b/doc/unitproc/bkshuf/article.tm index 46f2c0e..3160815 100644 --- a/doc/unitproc/bkshuf/article.tm +++ b/doc/unitproc/bkshuf/article.tm @@ -6,7 +6,7 @@ |>>||> + Sandoval>>>||> @@ -204,11 +204,11 @@ size trends towards . <\eqnarray*> - >>||>|)>>>|>>||>>>|||>>=>>>>|||>>>>|>>||>>|>-1>||>>|>>||>|)>>>>>|>>||>|)>>>>>|||>|>>|)>>||,0>|)>-1|,0>|)>|]>>|)>\>|)>|]>+1|]>>>>>> + >>||>|)>>>|>>||>>>|||>>=>>>>|||>>>>|>>||>>|>-1>||>>|>>||>|)>>>>|>>||>|)>>>>|||>|>>|)>>||,0>|)>-1|,0>|)>|]>>|)>\>|)>|]>+1|]>>>>> @@ -280,14 +280,14 @@ <\references> <\collection> > - > + > > > > > > - > - > + > + > > > > @@ -299,6 +299,9 @@ > > > + > + > + > > > > diff --git a/unitproc/bkshuf b/unitproc/bkshuf index 4bf9f99..8aaac9d 100644 --- a/unitproc/bkshuf +++ b/unitproc/bkshuf @@ -1,12 +1,15 @@ #!/usr/bin/env bash # Desc: Mixes input lines while also preserving some neighbors # Usage: cat file | bkshuf arg1 -# Version 0.0.1 +# Version 0.1.0 # Depends: bc 1.07.1, GNU Coreutils 8.32 (shuf) # Input: var: arg1 initial lines to output -BKSHUF_PARAM_LINEC=1000000; -BKSHUF_PARAM_GSIZE=25 # lines per group for BKSHUF_PARAM_LINEC lines of input +# Load env vars +## For these numbers of lines of input... +if [[ ! -v BKSHUF_PARAM_LINEC ]]; then BKSHUF_PARAM_LINEC=1000000; fi; +## ... target this group size. +if [[ ! -v BKSHUF_PARAM_GSIZE ]]; then BKSHUF_PARAM_GSIZE=25; fi; yell() { echo "$0: $*" >&2; } # print script path and all args to stderr diff --git a/user/bk-copy-rand-music.sh b/user/bk-copy-rand-music old mode 100755 new mode 100644 similarity index 74% rename from user/bk-copy-rand-music.sh rename to user/bk-copy-rand-music index ea384fe..18f95f3 --- a/user/bk-copy-rand-music.sh +++ b/user/bk-copy-rand-music @@ -1,7 +1,8 @@ #!/usr/bin/env bash # Desc: Copies random audio files -# Usage: bk-copy-rand-music.sh [dir SOURCE] [dir DEST] [int DURATION] -# Version: 0.0.3 +# Usage: bk-copy-rand-music [dir SOURCE] [dir DEST] [int DURATION] ([int BYTES]) +# Version: 0.1.0 +# Depends: BK-2020-03: bkshuf v0.1.0 declare -Ag appRollCall # Associative array for storing app status declare -Ag fileRollCall # Associative array for storing file status @@ -10,13 +11,18 @@ declare -a music_codecs # Array for storing valid codec names (e.g. "aac" "mp3") # Adjustable parameters music_codecs=("vorbis" "aac" "mp3" "flac" "opus"); # whitelist of valid codec_names ffprobe might return -max_loops="1000000"; # max number of files to test whether are audio or not max_filename_length="255"; # max output filename length min_file_duration="10"; # minimum duration per music file +max_file_duration="3600"; # maximum duration per music file +min_file_size="100000"; # minimum size per music file (bytes) +max_file_size="100000000"; # maximum size per music file (bytes) +siz_dest="600000000"; # default destination size limit: 600 MB +max_find_depth="10"; # max find depth + yell() { echo "$0: $*" >&2; } # print script path and all args to stderr die() { yell "$*"; exit 111; } # same as yell() but non-zero exit status -try() { "$@" || die "cannot $*"; } # runs args as command, reports args if command fails +must() { "$@" || die "cannot $*"; } # runs args as command, reports args if command fails checkapp() { # Desc: If arg is a command, save result in assoc array 'appRollCall' # Usage: checkapp arg1 arg2 arg3 ... @@ -185,10 +191,11 @@ showUsage() { audio tracks from SOURCE to DEST. USAGE: - bk-copy-rand-music [dir SOURCE] [dir DEST] [int DURATION] + bk-copy-rand-music [dir SOURCE] [dir DEST] [int DURATION] (int BYTES) EXAMPLE: bk-copy-rand-music ~/Music /tmp/music-sample 3600 + bk-copy-rand-music ~/Music /tmp/music-sample 3600 680000000 DEPENDENCIES: ffprobe @@ -358,18 +365,26 @@ main() { # Input: arg1: path to source tree # arg2: path to destination tree # arg3: cumulative duration (seconds) of audio files in destination tree + # arg4: cumulative size (bytes) of audio files in destination tree (optional) # assoc arrays: appRollCall, fileRollCall, dirRollCall + # env.var: BKSHUF_PARAM_LINEC + # BKSHUF_PARAM_GSIZE + # arrays: music_codecs + # vars: max_filename_length, min_file_duration, max_file_duration, + # min_file_size, max_file_size, siz_dest, max_find_depth # Output: [none] - # Depends: yell(), checkdir() 0.1.2, displayMissing() 1.0.0, GNU Coreutils 8.30 (shuf) - local arg1 arg2 arg3 dur_dest dir_source dir_dest list_all + # Depends: yell(), checkdir() 0.1.2, displayMissing() 1.0.0, GNU Coreutils 8.30 + # BK-2020-03: bkshuf v0.1.0 + local arg1 arg2 arg3 dur_dest dir_source dir_dest declare -a list_files # array for files to be considered - declare -A list_copy # assoc array for files to be copied (key=path; value=duration) + declare -a list_copy_sa # simple array for files to be copied (string: "$dur,$path") # Parse args arg1="$1"; arg2="$2"; arg3="$3"; - if [[ $# -ne 3 ]]; then showUsage; die "ERROR:Invalid number of args."; fi; + arg4="$4"; + if ! ([[ $# -eq 3 ]] || [[ $# -eq 4 ]]); then showUsage; die "ERROR:Invalid number of args:$#"; fi; ## Check duration if checkInt "$arg3"; then @@ -377,6 +392,15 @@ main() { else yell "ERROR:Duration (seconds) not an int:$arg3" fi; + + ## Check size + if [[ -n "$arg4" ]]; then + if checkInt "$arg4"; then + siz_dest="$arg4"; + else + yell "ERROR:Size (bytes) not an int:$arg4"; + fi; + fi; ## Check directories if checkdir "$arg1" "$arg2"; then @@ -387,7 +411,7 @@ main() { fi; ## Check apps - checkapp ffprobe; + checkapp ffprobe bkshuf; if ! displayMissing; then showUsage; @@ -395,33 +419,26 @@ main() { fi; yell "STATUS:Working..."; - - # Generate file path list - list_all="$(find -L "$dir_source")"; - #yell "DEBUG:list_files_rel:$list_files_rel"; - # Prune list_all of non-files and save as array list_files + # Populate list_files array while read -r line; do - #yell "DEBUG:line:$line"; - if ! [[ -f $line ]]; then - #yell "DEBUG:Not a file:$line"; - #yell ""; # debug - continue; - fi; - list_files+=("$line"); - done < <(echo "$list_all"); + list_files+=("$line"); + done < <(find -L "$dir_source" -maxdepth "$max_find_depth" -type f | sort); - # Randomly test and add elements of list_files array to list_copy + # Test and add random elements of list_files to list_copy dur=0; # Initialize duration + siz=0; # Initialize size n=0; # Initialize loop counter + dur_cand_w=1; # Init duration digit width counter + siz_cand_w=1; # Init size digit width counter ## Get element count of list_files array - list_files_count="${#list_files[@]}"; - while [[ $dur -le $dur_dest ]]; do - #yell "DEBUG:list_copy building loop:$n"; - ### Select random element of list_files array - list_files_index="$(shuf -i 1-"$list_files_count" -n1)"; - list_files_index="$((list_files_index - 1))"; # bash arrays are zero-indexed - path_candfile="${list_files[$list_files_index]}"; # path of candidate file + file_count="${#list_files[@]}"; + while read -r line && \ + [[ $dur -le $dur_dest ]] && \ + [[ $siz -le $siz_dest ]] && \ + [[ $n -le $file_count ]]; do + #yell "DEBUG:list_copy building loop:$n"; + path_candfile="$line"; # path of candidate file ### Check if has valid codec if ! check_parsable_audio_ffprobe "$path_candfile"; then continue; fi; # reject @@ -433,60 +450,79 @@ main() { ### Check and save duration dur_cand="$(get_media_length "$path_candfile")"; dur_cand="${dur_cand%%.*}"; # convert float to int + if [[ "$((dur + dur_cand))" -gt "$dur_dest" ]]; then continue; fi; # reject + dur_cand_wnow="$(printf "%s" "$dur_cand" | wc -m)"; # duration width count + if [[ $dur_cand_wnow -gt $dur_cand_w ]]; then + dur_cand_w="$dur_cand_wnow"; fi; if ! checkInt "$dur_cand"; then continue; fi; # reject if [[ "$dur_cand" -lt "$min_file_duration" ]]; then continue; fi; # reject - - ### Add/update candfile to list_copy assoc. array (key=path; value=duration) + if [[ "$dur_cand" -gt "$max_file_duration" ]]; then continue; fi; # reject + + ### Check and save size + siz_cand="$(du -b "$path_candfile" | awk '{ print $1 }')"; # size in bytes + siz_cand_wnow="$(printf "%s" "$siz_cand" | wc -m)"; # size width count + if [[ $siz_cand_wnow -gt $siz_cand_w ]]; then + siz_cand_w="$siz_cand_wnow"; fi; + if ! checkInt "$siz_cand"; then continue; fi; # reject + if [[ "$siz_cand" -lt "$min_file_size" ]]; then continue; fi; # reject + if [[ "$siz_cand" -gt "$max_file_size" ]]; then continue; fi; # reject + + ### Add/update candfile to array: + ### list_copy_sa (simple array with only paths) #yell "DEBUG:Adding $path_candfile"; - list_copy["$path_candfile"]="$dur_cand"; + list_copy_sa+=("$dur_cand,$siz_cand,$path_candfile"); # for copying with order - ### Update total duration $dur by summing all list_copy assoc. array values - dur=0; - for value in "${list_copy[@]}"; do - dur="$((dur + value))"; - done; + ### Update total duration $dur and total size $siz + dur="$((dur + dur_cand))"; + siz="$((siz + siz_cand))"; #yell "DEBUG:dur:$dur"; + #yell "DEBUG:siz:$siz"; - ### Sanity check ((n++)); - if [[ $n -gt $max_loops ]]; then die "ERROR:Too many loops:$n"; fi; - done; + done < <(printf "%s\n" "${list_files[@]}" | bkshuf); n=0; # Initialize loop counter + num_w="$(printf "%s" "${#list_copy_sa[@]}" | wc -m)"; # init file number format + num_fmt="%0""$num_w""d"; + path_log_output="$dir_dest"/COPY.log; + printf "num,fingerprint,duration,size,original_path\n" >> "$path_log_output"; # Copy files in list_copy to dir_dest; - for key in "${!list_copy[@]}"; do - value="${list_copy[$key]}"; - ## Get basename of path - file_basename="$(basename "$key")"; - - ## Get 16-character b2sum fingerprint (for different files that share basename) - fingerprint="$(b2sum -l64 "$key" | cut -d' ' -f1)"; - - ## Form output filename - file_name="$fingerprint".."$file_basename"; + while read -r line; do + yell "DEBUG:line:$line"; # debug + fdur="$(printf "%s" "$line" | cut -d',' -f1)"; + fsize="$(printf "%s" "$line" | cut -d',' -f2)"; + fpath="$(printf "%s" "$line" | cut -d',' -f3-)"; + ## Get basename of path + file_basename="$(basename "$fpath")"; + + ## Get 16-character b2sum fingerprint (for different files that share basename) + fingerprint="$(b2sum -l32 "$fpath" | awk '{print $1}' )"; + + ## Form output filename + num="$(printf "$num_fmt" "$n")"; + file_name="$num"_"$fingerprint".."$file_basename"; file_name="${file_name:0:$max_filename_length}"; # Limit filename length (e.g. Windows has max of 255 characters) ## Form output path path_output="$dir_dest"/"$file_name"; ## Copy - try cp "$key" "$path_output" && yell "NOTICE:Copied ($value seconds): $key "; + must cp "$fpath" "$path_output" && yell "NOTICE:Copied ($fdur seconds): $fpath "; #yell "DEBUG:Copied $file_basename to $dur_dest."; ## Append log - path_log_output="$dir_dest"/COPY.log; - if [[ $n -le 0 ]]; then - echo "fingerprint","duration","original_path" >> "$path_log_output"; - else - echo "$fingerprint","$value","$key" >> "$path_log_output"; - fi; + fpath_can="$(readlink -f "$fpath")"; # resolve symlinks to canonical path + log_fmt="%s,%s,%""$dur_cand_w""d,%""$siz_cand_w""d,%s\n"; # e.g. "%s,%3d,%5d,%s" if dur_cand_w=3 and siz_cand_w=5 + #yell "DEBUG:log_fmt:$log_fmt"; sleep 10; # debug + printf "$log_fmt" "$num" "$fingerprint" "$fdur" "$fsize" "$fpath_can" >> "$path_log_output"; ((n++)); unset file_basename path_output - done; + done < <(printf "%s\n" "${list_copy_sa[@]}"); - # Report total duration + # Report total duration and size yell "NOTICE:Total duration (seconds):$dur"; + yell "NOTICE:Total size (bytes):$siz"; } # Main program -- 2.30.2 From 48dab430af0eb368f87d6e26eb7a7afa8b728a60 Mon Sep 17 00:00:00 2001 From: Steven Baltakatei Sandoval Date: Sat, 18 Feb 2023 17:59:52 +0000 Subject: [PATCH 04/16] feat(user/bk-copy-rand-music): Use env vars for bkshuf params - fix(bk-copy-rand-music):Fix upper size and duration cutoffs - feat(unitproc/bkshuf):Detect env vars --- unitproc/bkshuf | 9 +++++-- user/bk-copy-rand-music | 56 ++++++++++++++++++++++++++++++----------- 2 files changed, 48 insertions(+), 17 deletions(-) diff --git a/unitproc/bkshuf b/unitproc/bkshuf index 8aaac9d..e41d725 100644 --- a/unitproc/bkshuf +++ b/unitproc/bkshuf @@ -1,7 +1,7 @@ #!/usr/bin/env bash # Desc: Mixes input lines while also preserving some neighbors # Usage: cat file | bkshuf arg1 -# Version 0.1.0 +# Version 0.1.1 # Depends: bc 1.07.1, GNU Coreutils 8.32 (shuf) # Input: var: arg1 initial lines to output @@ -124,7 +124,12 @@ main() { else lc_out_max="$1"; # output line count fi; - + + # Check env vars + if ! checkInt "$BKSHUF_PARAM_LINEC"; then + die "FATAL:Not an int:BKSHUF_PARAM_LINEC:$BKSHUF_PARAM_LINEC"; fi; + if ! checkInt "$BKSHUF_PARAM_GSIZE"; then + die "FATAL:Not an int:BKSHUF_PARAM_LINEC:$BKSHUF_PARAM_GSIZE"; fi; # store input lines from stdin (like `shuf`) while read -r line; do diff --git a/user/bk-copy-rand-music b/user/bk-copy-rand-music index 18f95f3..bc21e5b 100644 --- a/user/bk-copy-rand-music +++ b/user/bk-copy-rand-music @@ -1,7 +1,7 @@ #!/usr/bin/env bash # Desc: Copies random audio files # Usage: bk-copy-rand-music [dir SOURCE] [dir DEST] [int DURATION] ([int BYTES]) -# Version: 0.1.0 +# Version: 0.1.1 # Depends: BK-2020-03: bkshuf v0.1.0 declare -Ag appRollCall # Associative array for storing app status @@ -19,6 +19,9 @@ max_file_size="100000000"; # maximum size per music file (bytes) siz_dest="600000000"; # default destination size limit: 600 MB max_find_depth="10"; # max find depth +# Load env vars (bkshuf defaults for typical music albums) +if [[ ! -v BKSHUF_PARAM_LINEC ]]; then export BKSHUF_PARAM_LINEC=1000000; fi; +if [[ ! -v BKSHUF_PARAM_GSIZE ]]; then export BKSHUF_PARAM_GSIZE=10; fi; yell() { echo "$0: $*" >&2; } # print script path and all args to stderr die() { yell "$*"; exit 111; } # same as yell() but non-zero exit status @@ -200,6 +203,10 @@ showUsage() { DEPENDENCIES: ffprobe GNU Coreutils 8.30 + + ENVIRONMENT VARIABLES + BKSHUF_PARAM_LINEC (see `bkshuf` in BK-2020-03) + BKSHUF_PARAM_GSIZE (see `bkshuf` in BK-2020-03) EOF } # Display information on how to use this script. check_parsable_audio_ffprobe() { @@ -367,8 +374,8 @@ main() { # arg3: cumulative duration (seconds) of audio files in destination tree # arg4: cumulative size (bytes) of audio files in destination tree (optional) # assoc arrays: appRollCall, fileRollCall, dirRollCall - # env.var: BKSHUF_PARAM_LINEC - # BKSHUF_PARAM_GSIZE + # env.var: BKSHUF_PARAM_LINEC (bkshuf) + # BKSHUF_PARAM_GSIZE (bkshuf) # arrays: music_codecs # vars: max_filename_length, min_file_duration, max_file_duration, # min_file_size, max_file_size, siz_dest, max_find_depth @@ -377,20 +384,28 @@ main() { # BK-2020-03: bkshuf v0.1.0 local arg1 arg2 arg3 dur_dest dir_source dir_dest declare -a list_files # array for files to be considered - declare -a list_copy_sa # simple array for files to be copied (string: "$dur,$path") + declare -a list_copy # array for files to be copied (string: "$dur,$fsize,$path") # Parse args arg1="$1"; arg2="$2"; arg3="$3"; arg4="$4"; - if ! ([[ $# -eq 3 ]] || [[ $# -eq 4 ]]); then showUsage; die "ERROR:Invalid number of args:$#"; fi; - + if ! { [[ $# -eq 3 ]] || [[ $# -eq 4 ]]; }; then + showUsage; + die "ERROR:Invalid number of args:$#"; fi; + + # Check env vars + if ! checkInt "$BKSHUF_PARAM_LINEC"; then + die "FATAL:Not an int:BKSHUF_PARAM_LINEC:$BKSHUF_PARAM_LINEC"; fi; + if ! checkInt "$BKSHUF_PARAM_GSIZE"; then + die "FATAL:Not an int:BKSHUF_PARAM_LINEC:$BKSHUF_PARAM_GSIZE"; fi; + ## Check duration if checkInt "$arg3"; then dur_dest="$arg3"; else - yell "ERROR:Duration (seconds) not an int:$arg3" + die "FATAL:Duration (seconds) not an int:$arg3" fi; ## Check size @@ -398,7 +413,7 @@ main() { if checkInt "$arg4"; then siz_dest="$arg4"; else - yell "ERROR:Size (bytes) not an int:$arg4"; + die "FATAL:Size (bytes) not an int:$arg4"; fi; fi; @@ -450,7 +465,7 @@ main() { ### Check and save duration dur_cand="$(get_media_length "$path_candfile")"; dur_cand="${dur_cand%%.*}"; # convert float to int - if [[ "$((dur + dur_cand))" -gt "$dur_dest" ]]; then continue; fi; # reject + if [[ "$((dur + dur_cand))" -gt "$dur_dest" ]]; then break; fi; # no more dur_cand_wnow="$(printf "%s" "$dur_cand" | wc -m)"; # duration width count if [[ $dur_cand_wnow -gt $dur_cand_w ]]; then dur_cand_w="$dur_cand_wnow"; fi; @@ -460,6 +475,7 @@ main() { ### Check and save size siz_cand="$(du -b "$path_candfile" | awk '{ print $1 }')"; # size in bytes + if [[ "$((siz + siz_cand))" -gt "$siz_dest" ]]; then break; fi; # no more siz_cand_wnow="$(printf "%s" "$siz_cand" | wc -m)"; # size width count if [[ $siz_cand_wnow -gt $siz_cand_w ]]; then siz_cand_w="$siz_cand_wnow"; fi; @@ -468,9 +484,12 @@ main() { if [[ "$siz_cand" -gt "$max_file_size" ]]; then continue; fi; # reject ### Add/update candfile to array: - ### list_copy_sa (simple array with only paths) + ### list_copy (array with "duration, size, path") #yell "DEBUG:Adding $path_candfile"; - list_copy_sa+=("$dur_cand,$siz_cand,$path_candfile"); # for copying with order + #printf "DEBUG:%8d,%8d,%s\n" "$dur_cand" "$siz_cand" "$path_candfile" 1>&2; + #printf "DEBUG:dur:%s\n" "$dur" 1>&2; + #printf "DEBUG:siz:%s\n" "$siz" 1>&2; + list_copy+=("$dur_cand,$siz_cand,$path_candfile"); # for copying with order ### Update total duration $dur and total size $siz dur="$((dur + dur_cand))"; @@ -481,14 +500,17 @@ main() { ((n++)); done < <(printf "%s\n" "${list_files[@]}" | bkshuf); + #yell "DEBUG:BKSHUF_PARAM_LINEC:$BKSHUF_PARAM_LINEC"; + #yell "DEBUG:BKSHUF_PARAM_GSIZE:$BKSHUF_PARAM_GSIZE"; + n=0; # Initialize loop counter - num_w="$(printf "%s" "${#list_copy_sa[@]}" | wc -m)"; # init file number format + num_w="$(printf "%s" "${#list_copy[@]}" | wc -m)"; # init file number format num_fmt="%0""$num_w""d"; path_log_output="$dir_dest"/COPY.log; printf "num,fingerprint,duration,size,original_path\n" >> "$path_log_output"; # Copy files in list_copy to dir_dest; while read -r line; do - yell "DEBUG:line:$line"; # debug + #yell "DEBUG:line:$line"; # debug fdur="$(printf "%s" "$line" | cut -d',' -f1)"; fsize="$(printf "%s" "$line" | cut -d',' -f2)"; fpath="$(printf "%s" "$line" | cut -d',' -f3-)"; @@ -513,12 +535,11 @@ main() { ## Append log fpath_can="$(readlink -f "$fpath")"; # resolve symlinks to canonical path log_fmt="%s,%s,%""$dur_cand_w""d,%""$siz_cand_w""d,%s\n"; # e.g. "%s,%3d,%5d,%s" if dur_cand_w=3 and siz_cand_w=5 - #yell "DEBUG:log_fmt:$log_fmt"; sleep 10; # debug printf "$log_fmt" "$num" "$fingerprint" "$fdur" "$fsize" "$fpath_can" >> "$path_log_output"; ((n++)); unset file_basename path_output - done < <(printf "%s\n" "${list_copy_sa[@]}"); + done < <(printf "%s\n" "${list_copy[@]}"); # Report total duration and size yell "NOTICE:Total duration (seconds):$dur"; @@ -530,3 +551,8 @@ main "$@"; # Author: Steven Baltakatei Sandoval # License: GPLv3+ + +# bkshuf v0.1.0 +# Author: Steven Baltakatei Sandoval +# License: GPLv3+ +# URL: https://gitlab.com/baltakatei/baltakatei-exdev/-/blob/b9e8b771e985fcdf26ba8b9ccb8e31b62da757d3/unitproc/bkshuf -- 2.30.2 From 8942e5363f55f2795a1de89162d99546262a0f07 Mon Sep 17 00:00:00 2001 From: Steven Baltakatei Sandoval Date: Mon, 20 Feb 2023 10:31:51 +0000 Subject: [PATCH 05/16] feat(user/space4):Add script to space out stdin every 4 lines --- user/space4 | 52 ++++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 52 insertions(+) create mode 100755 user/space4 diff --git a/user/space4 b/user/space4 new file mode 100755 index 0000000..823f412 --- /dev/null +++ b/user/space4 @@ -0,0 +1,52 @@ +#!/usr/bin/env bash +# Desc: Adds blank line every 4 lines +# Usage: cat file | space4 +# Example: seq 1 10 | space 4 +# Version: 0.0.1 +# Depends: Bash 5.1.16 + +yell() { echo "$0: $*" >&2; } # print script path and all args to stderr +die() { yell "$*"; exit 111; } # same as yell() but non-zero exit status +must() { "$@" || die "cannot $*"; } # runs args as command, reports args if command fails +read_stdin() { + # Desc: Consumes stdin; outputs as stdout lines + # Input: stdin (consumes) + # Output: stdout (newline delimited) + # Example: printf "foo\nbar\n" | read_stdin + # Depends: GNU bash (version 5.1.16) + # Version: 0.0.1 + local input_stdin output; + + # Store stdin + if [[ -p /dev/stdin ]]; then + input_stdin="$(cat -)"; + fi; + + # Store as output array elements + ## Read in stdin + if [[ -n $input_stdin ]]; then + while read -r line; do + output+=("$line"); + done < <(printf "%s\n" "$input_stdin"); + fi; + + # Print to stdout + printf "%s\n" "${output[@]}"; +}; # read stdin to stdout lines +main() { + # Depends: BK-2020-03: yell(), die(), must(), read_stdin() + if [[ $# -gt 0 ]]; then die "FATAL:This script uses no positional arguments.:$#"; fi; + n=0; + while read -r line; do + printf "%s\n" "$line"; + if ! (( (n + 1) % 4 )); then + printf "\n"; + fi; + ((n++)); + done < <(read_stdin); +}; # main program + +main "$@"; + +# Author: Steven Baltakatei Sandoval +# License: GPLv3+ -- 2.30.2 From f9779d87f3ff8a10737c3032c8594fce56761379 Mon Sep 17 00:00:00 2001 From: Steven Baltakatei Sandoval Date: Mon, 20 Feb 2023 12:39:14 +0000 Subject: [PATCH 06/16] feat(user/get_ytpljson.sh):Add script to get YT playlist metadata - Note: Requests YouTube playlist metadata in JSON format from `https://www.googleapis.com`. --- doc/user/get_ytpljson.sh.org | 36 +++++++++ user/get_ytpljson.sh | 140 +++++++++++++++++++++++++++++++++++ 2 files changed, 176 insertions(+) create mode 100644 doc/user/get_ytpljson.sh.org create mode 100755 user/get_ytpljson.sh diff --git a/doc/user/get_ytpljson.sh.org b/doc/user/get_ytpljson.sh.org new file mode 100644 index 0000000..5136424 --- /dev/null +++ b/doc/user/get_ytpljson.sh.org @@ -0,0 +1,36 @@ +* get_ytpljson.sh Information + +#+TITLE: get_ytpljson.sh Information +#+AUTHOR: Steven Baltakatei Sandoval +#+DATE:2023-02-20 +#+EMAIL:baltakatei@gmail.com +#+LANGUAGE: en +#+OPTIONS: toc:nil + +Created by [[https://baltakatei.com][Steven Baltakatei Sandoval]] on +2023-02-20T12:38+00 +under a [[https://creativecommons.org/licenses/by-sa/4.0/][CC BY-SA 4.0]] (🅭🅯🄎4.0) license and last updated on +2023-02-20T12:38+00. + +** Summary +This script downloads and saves YouTube playlist metadata as a JSON +file in the current working directory. + +** Versions +| Version | Description | +|---------+-------------------------------------------------| +| 0.0.1 | Initial version compatible with YouTUbe API v3. | +| | | + +** Background +Google provides an API for downloading playlist metadata in resposne +to receiving a correctly formatted URL containing the playlist ID and +an API key. + +** References +- "Obtaining authorization credentials" + https://developers.google.com/youtube/registering_an_application +- "Implementation: Playlists" + https://developers.google.com/youtube/v3/guides/implementation/playlists +- "Implementation: Pagination", + https://developers.google.com/youtube/v3/guides/implementation/pagination diff --git a/user/get_ytpljson.sh b/user/get_ytpljson.sh new file mode 100755 index 0000000..4f72265 --- /dev/null +++ b/user/get_ytpljson.sh @@ -0,0 +1,140 @@ +#!/bin/bash +# Usage: get_ytpljson.sh arg1 arg2 +# Input: posargs: arg1: YouTube playlist ID +# arg2: Google API key +# Output: file: JSON file +# Version: 0.0.1 + +max_api_calls="100"; + +yell() { echo "$0: $*" >&2; } # print script path and all args to stderr +die() { yell "$*"; exit 111; } # same as yell() but non-zero exit status +must() { "$@" || die "cannot $*"; } # runs args as command, reports args if command fails +get_response() { + # Input: arg1: YouTube playlist ID + # arg2: Google API key + # arg3: pageToken (optional) + # Output: stdout: JSON response from googleapis.com + # Depends: curl 7.81.0 + # BK-2020-03: die() + + local PLAYLIST_ID API_KEY PAGE_TOKEN URL; + + # Set the playlist ID and API key + PLAYLIST_ID="$1"; + API_KEY="$2"; + PAGE_TOKEN="$3"; + + # Check inputs + if [[ $# -lt 2 ]]; then die "FATAL:Incorrect arg count:$#"; fi; + + # Base URL + URL="https://www.googleapis.com/youtube/v3/playlistItems?part=snippet"; + + # Append playlist ID + URL="$URL""&playlistId=""$PLAYLIST_ID"; + + # Append API key + URL="$URL""&key=""$API_KEY"; + + # Append page token if it exists + if [[ -n "$PAGE_TOKEN" ]]; then + URL="$URL""&pageToken=""$PAGE_TOKEN"; + fi; + + curl -s "$URL"; + #curl -s "https://www.googleapis.com/youtube/v3/playlistItems?part=snippet&playlistId=$PLAYLIST_ID&key=$API_KEY"; # example + +}; # Stdout: JSON from YouTube v3 API +check_next_page() { + # Input: arg1: json string + # Depends: jq 1.6 + + # Checks if key ".nextPageToken" present + if jq -e '.nextPageToken' < <(printf "%s" "$1") 1>/dev/random 2>&1; then + return 0; + else + return 1; + fi; + +}; # returns true if '.nextPageToken' present +get_next_page() { + # Input: arg1: json string containing the key 'nextPageToken' + # Output: stdout: the value of the first 'nextPageToken' key + # exit code: 0: key '.nextPageToken' detected + # 1: key '.nextPageToken' not detected + # Depends: jq 1.6 + local output; + if [[ $# -ne 1 ]]; then die "Incorrect arg count:$#"; fi; + + if jq -e '.nextPageToken' < <(printf "%s" "$1") 1>/dev/random 2>&1; then + output="$(jq -r '.nextPageToken' < <(printf "%s" "$1") | head -n1)"; + printf "%s" "$output"; + return 0; + else + return 1; + fi; +}; # stdout: value from key "pageToken" + +main() { + # Depends: bash 5.1.16, GNU Coreutils 8.32 (date) + # BK-2020-03: yell() + # Ref/Attrib: [1]: "Obtaining authorization credentials" https://developers.google.com/youtube/registering_an_application + # [2]: "Implementation: Playlists" https://developers.google.com/youtube/v3/guides/implementation/playlists + # [3]: "Implementation: Pagination", https://developers.google.com/youtube/v3/guides/implementation/pagination + + local n out_path; + declare -a out_list; + + # Check input + if [[ $# -ne 2 ]]; then die "FATAL:Incorrect number of args:$#"; fi; + + # Set the playlist ID and API key + playlistId="$1"; # See ref [2] + apiKey="$2"; # See ref [1] + + # Set dynamic variables according to environment + out_dir="$(pwd)"; # output to present working directory + out_filename="$(date +%Y%m%dT%H%M%S%z)"_"$playlistId"..playlist_items.json; + out_path="$out_dir"/"$out_filename"; + + # Make initial curl request to the YouTube Data API + response="$(get_response "$playlistId" "$apiKey")"; + + # # debug + # if check_next_page "$response"; then + # yell "DEBUG:nextPageToken detected"; + # fi; + + # Make follow-up requests. See ref [3] + n=0; + while check_next_page "$response"; do + # Get page token from response + pageToken="$(get_next_page "$response")"; + # Update response + response="$(get_response "$playlistId" "$apiKey" "$pageToken")"; + # Record response + out_list+=("$response"); + + # Sanity check + if [[ $n -gt $max_api_calls ]]; then die "FATAL:Too many API calls:$n"; fi; + ((n++)); + done; + + # Write results + printf "%s\n" "${out_list[@]}" > "$out_path"; + + # Print stats + yell "STATUS:Performed $n API calls." + out_lc="$(printf "%s\n" "${out_list[@]}" | wc -l)"; + yell "STATUS:Wrote $out_lc lines to $out_path"; + + # Use jq to extract the publishedAt field for each playlist item + #PUBLISHED_AT=$(echo "$response" | jq -r '.items[].snippet.publishedAt') + +}; # main program + +main "$@"; + +# Author: Steven Baltakatei Sandoval +# License: GPLv3+ -- 2.30.2 From 196c91bb3fb9a469a90f6ed9c0fbf408d14fe040 Mon Sep 17 00:00:00 2001 From: Steven Baltakatei Sandoval Date: Mon, 20 Feb 2023 18:40:49 +0000 Subject: [PATCH 07/16] feat(user/rmsym):Add script to delete only symlinks --- user/rmsym | 24 ++++++++++++++++++++++++ 1 file changed, 24 insertions(+) create mode 100644 user/rmsym diff --git a/user/rmsym b/user/rmsym new file mode 100644 index 0000000..87f3549 --- /dev/null +++ b/user/rmsym @@ -0,0 +1,24 @@ +#!/usr/bin/env bash +# Desc: Delete symlinks; skips non-symlinks +# Usage: rmsym [paths] +# Version: 0.0.1 + +yell() { echo "$0: $*" >&2; } # print script path and all args to stderr +die() { yell "$*"; exit 111; } # same as yell() but non-zero exit status +must() { "$@" || die "cannot $*"; } # runs args as command, reports args if command fails +main () { + # Check args + for psarg in "$@"; do + if [[ -h "$psarg" ]]; then + rm "$psarg"; + else + yell "Not a symbolic link; not deleting:$psarg"; + continue; + fi; + done; +}; # main program + +main "$@"; + +# Author: Steven Baltakatei Sandoval +# License: GPLv3+ -- 2.30.2 From cffa431bc68533d72bea0fd3aaf15e57107ba60f Mon Sep 17 00:00:00 2001 From: Steven Baltakatei Sandoval Date: Mon, 20 Feb 2023 18:54:16 +0000 Subject: [PATCH 08/16] doc(user/rmsym.org):Describe rmsym --- doc/user/rmsym.org | 50 ++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 50 insertions(+) create mode 100644 doc/user/rmsym.org diff --git a/doc/user/rmsym.org b/doc/user/rmsym.org new file mode 100644 index 0000000..ce939e5 --- /dev/null +++ b/doc/user/rmsym.org @@ -0,0 +1,50 @@ +* rmsym - Symlink deleter +#+TITLE: rmsym - Symlink deleter +#+AUTHOR: Steven Baltakatei Sandoval +#+DATE:2023-02-20 +#+EMAIL:baltakatei@gmail.com +#+LANGUAGE: en +#+OPTIONS: toc:nil + +Created by [[https://baltakatei.com][Steven Baltakatei Sandoval]] on +2023-02-20T18:43+00 +under a [[https://creativecommons.org/licenses/by-sa/4.0/][CC BY-SA 4.0]] (🅭🅯🄎4.0) license and last updated on +2023-02-20T18:43+00 + +** Summary +~rmsym~ is a Bash script which deletes symbolic links specified by +positional arguments. + +** Versions +| Version | Description | +|---------+----------------------------------------------------------------| +| 0.0.1 | Initial version. Does not accept or pass on arguments to ~rm~. | +| | | + +** Background +I needed a script that would let me quickly delete only symlinks if I +were to provide it with a mix of paths of files, directories, or +symlinks; specifically, I want to be able to create music playlists +that any music player can parse; most music players can recursively +scan directories and a directory tree of symlinks would reduce the +need to maintain multiple copies of music files on the same file +system. However, in order to assure myself that when I prune symlinks +that I don't delete actual files targeted by the symlinks, I need a +program that will only delete symlinks. + +Bash supports testing whether a path is a symlink or not via the ~-h~ +test. For example, here is a simple Bash script that will report +whether a symlink is present at ~$HOME/foo.txt~ or not. + +#+begin_example +#!/bin/bash +path_myfile="$HOME/foo.txt"; +if [[ -h "$path_myfile" ]]; then + echo "STATUS:Is a symlink:$path_myfile"; +else + echo "STATUS:Is not a symlink:$path_myfile"; +fi; +#+end_example + +** Reference +- "Bash Reference Manual", Section: "[[https://www.gnu.org/software/bash/manual/bash.html#Bash-Conditional-Expressions][6.4 Bash Conditional Expressions]]". gnu.org. -- 2.30.2 From 13c46bd3783fb9ee553b541ea7267bbb0a4912a0 Mon Sep 17 00:00:00 2001 From: Steven Baltakatei Sandoval Date: Mon, 20 Feb 2023 22:36:13 +0000 Subject: [PATCH 09/16] feat(user/ping_offline.sh):Add script to ping until it succeeds once - Note: the script is meant to mark the time when internet connectivity is reestablished. - doc(user/ping_offline.sh.org):Add docs - doc(user/template.org):Add template for future executable documentation. --- doc/user/ping_offline.sh.org | 38 ++++++++++++++++++++++++++++++++++++ doc/user/template.org | 28 ++++++++++++++++++++++++++ user/ping_offline.sh | 16 +++++++++++++++ 3 files changed, 82 insertions(+) create mode 100644 doc/user/ping_offline.sh.org create mode 100644 doc/user/template.org create mode 100755 user/ping_offline.sh diff --git a/doc/user/ping_offline.sh.org b/doc/user/ping_offline.sh.org new file mode 100644 index 0000000..6b96e46 --- /dev/null +++ b/doc/user/ping_offline.sh.org @@ -0,0 +1,38 @@ +* ping_offline.sh documentation + +#+TITLE: ping_offline.sh documentation +#+AUTHOR: Steven Baltakatei Sandoval +#+DATE:2023-02-20 +#+EMAIL:baltakatei@gmail.com +#+LANGUAGE: en +#+OPTIONS: toc:nil + +Created by [[https://baltakatei.com][Steven Baltakatei Sandoval]] on +2023-02-20T22:27+00 +under a [[https://creativecommons.org/licenses/by-sa/4.0/][CC BY-SA 4.0]] (🅭🅯🄎4.0) license and last updated on +2023-02-20T22:34+00 + +** Summary + +~ping_offline.sh~ is a small Bash script meant to indicate when the +machine it runs on is able to ping a remote server. + +** Versions +| Version | Description | +|---------+-------------------------| +| 0.0.1 | Initial working script. | + +** Background +Sometimes, my machines are on a network that is temporarily +disconnected for various reasons. I'd like to know approximately when +they come back online by pinging a remote server; once a successful +ping is established, I can stop further pings. However, the ~ping~ +executable available (provided by the package ~iputils-ping~) on my +Pop!_OS 22.04 LTS machine throws an error if the machine is completely +offline; therefore, I wrote the script to regularly try the command +until it succeeds at least once. + +** Dependencies +- ~iputils-ping~ package (Version: 3:20211215-1) + +** References diff --git a/doc/user/template.org b/doc/user/template.org new file mode 100644 index 0000000..491153f --- /dev/null +++ b/doc/user/template.org @@ -0,0 +1,28 @@ +* template + +#+TITLE: template +#+AUTHOR: Steven Baltakatei Sandoval +#+DATE:YYYY-mm-dd +#+EMAIL:baltakatei@gmail.com +#+LANGUAGE: en +#+OPTIONS: toc:nil + +Created by [[https://baltakatei.com][Steven Baltakatei Sandoval]] on +YYYY-mm-ddTHH:MM+ZZ +under a [[https://creativecommons.org/licenses/by-sa/4.0/][CC BY-SA 4.0]] (🅭🅯🄎4.0) license and last updated on +YYYY-mm-ddTHH:MM+ZZ. + +** Summary + + +** Versions +| Version | Description | +|---------+-------------------------------------------------| +| | | + +** Background + + +** References +- foo +- bar diff --git a/user/ping_offline.sh b/user/ping_offline.sh new file mode 100755 index 0000000..e5b7804 --- /dev/null +++ b/user/ping_offline.sh @@ -0,0 +1,16 @@ +#!/bin/bash +# Desc:Marks the time when a remote server can be pinged. +# Depends: GNU Coreutils 8.32 (date, sleep), iputils-ping 3:20211215-1 (ping) +# Version: 0.0.1 + +delay=10; +domain="google.com" + +if [[ -n "$1" ]]; then domain="$1"; fi; + +printf "%s:STATUS:Starting ping...\n" "$(date -Is)"; +while ! ping -c1 "$domain"; do + sleep "$delay"; +done; +printf "%s:Ping resolved.\n" "$(date -Is)"; +exit 0; -- 2.30.2 From 10767133e479ce0fee75875c16fb544f9cbf0d48 Mon Sep 17 00:00:00 2001 From: Steven Baltakatei Sandoval Date: Fri, 24 Feb 2023 21:23:17 +0000 Subject: [PATCH 10/16] chore(README.org):Update with bktei project code --- README.org | 9 ++++++--- 1 file changed, 6 insertions(+), 3 deletions(-) diff --git a/README.org b/README.org index 943649e..373cf55 100644 --- a/README.org +++ b/README.org @@ -1,14 +1,14 @@ * Baltakatei Executables Development #+TITLE: Baltakatei Executables Development #+AUTHOR: Steven Baltakatei Sandoval -#+DATE:2021-03-04 +#+DATE:2023-02-24 #+EMAIL:baltakatei@gmail.com #+LANGUAGE: en #+OPTIONS: toc:nil ** Summary -This repository contains publicly-sharable executables used by Steven -Baltakatei Sandoval. produced by Steven Baltakatei Sandoval. +This repository contains publicly-sharable executables produced by +Steven Baltakatei Sandoval. ** Directory Structure *** ~archive~ @@ -58,3 +58,6 @@ Examples may include: alphabet and adjustable entropy. - A bash script that initializes a multiwindow terminal emulator (i.e. ~tilix~) with custom initial working directories. + +** Metadata +- Baltakatei Project No. ~BK-2020-03~ -- 2.30.2 From e65cf1ee23945b1115700d9d134f29a9f7ca7c95 Mon Sep 17 00:00:00 2001 From: Steven Baltakatei Sandoval Date: Thu, 2 Mar 2023 18:49:25 +0000 Subject: [PATCH 11/16] feat(user/git-bk-find-file):Show ISO 8601 time along with commit --- user/git-bk-find-file | 22 +++++++++++++++------- 1 file changed, 15 insertions(+), 7 deletions(-) mode change 100644 => 100755 user/git-bk-find-file diff --git a/user/git-bk-find-file b/user/git-bk-find-file old mode 100644 new mode 100755 index c31b31c..f4ef58e --- a/user/git-bk-find-file +++ b/user/git-bk-find-file @@ -1,9 +1,9 @@ #!/usr/bin/env bash # Desc: Grep search filenames of all git committed work trees -# Usage: git-bk-find-file [string regex] -# Example git-bk-find-file '*.txt' +# Usage: git-bk-find-file [string grep pattern] +# Example git-bk-find-file '.txt$' # Ref/Attrib: albfan "How can I search Git branches for a file or directory?" https://stackoverflow.com/a/16868704/10850071 -# Version: 0.0.1 +# Version: 0.1.0 # Depends: GNU bash v5.1.16, GNU parallel 20210822 yell() { echo "$0: $*" >&2; } # print script path and all args to stderr @@ -24,16 +24,24 @@ display_commit() { # Input: arg1: commit id # args2+: passed to grep # Output: stdout + # Depends: git 2.34.1 local commit results; commit="$1"; shift; # Decide if need to show commit at all if results="$(git ls-tree -r --name-only "$commit" | grep "$1")"; then - commit="${commit:(-8)}"; # get last 8 chars - # Prepend results with commit - results="$( echo "$results" | sed "s/^/$commit: /" )"; - printf "%s\n" "$results"; + # Get commit timestamp + time="$(git -c log.showSignature=false show -s --format=%cI "$commit")"; + + # Get last 8 chars of commit + short_commit="${commit:(-8)}"; + + # Output results + while read -r line; do + if [[ -z "$line" ]]; then continue; fi; # skip blank lines + printf "%s %s %s\n" "$time" "$short_commit" "$line"; + done < <(printf "%s\n" "$results"); fi; return 0; -- 2.30.2 From 58c4ea210f287d856ba98349115ec6e285a5c876 Mon Sep 17 00:00:00 2001 From: Steven Baltakatei Sandoval Date: Fri, 3 Mar 2023 23:31:54 +0000 Subject: [PATCH 12/16] update(unitproc/bkdstcountdown):Refactor --- unitproc/bkdstcountdown | 110 ++++++++++++++++++---------------------- 1 file changed, 50 insertions(+), 60 deletions(-) diff --git a/unitproc/bkdstcountdown b/unitproc/bkdstcountdown index 4e3efe8..17ff7fa 100755 --- a/unitproc/bkdstcountdown +++ b/unitproc/bkdstcountdown @@ -1,74 +1,64 @@ -#!/bin/bash +#!/usr/bin/env bash +# Description: Outputs days until next timezone discontinuity +# Depends: GNU Coreutils 8.32 (date), zdump 2.35, Bash 5.1.16 +# BK-2020-03: yell(), die(), must() +# Version: 0.1.0 -# Description: This script outputs days until the next time zone -# discontinuity using 'zdump' and 'date'. - -#=================ADJUST ME======================= -SCRIPT_TZ="America/Los_Angeles" # Timezone to check for discontinuities -#================================================= +# User config +script_tz="America/Los_Angeles"; # Timezone to check for discontinuities yell() { echo "$0: $*" >&2; } # Yell, Die, Try Three-Fingered Claw technique; # Ref/Attrib: https://stackoverflow.com/a/25515370 die() { yell "$*"; exit 111; } -try() { "$@" || die "cannot $*"; } - -# Declare variables -declare -a datesDiscontArray #array - -SCRIPT_YEAR=$(date +%Y) -SCRIPT_YEAR_NEXT=$(( $(date +%Y) + 1)) -SCRIPT_YEAR_NEXT2=$(( $(date +%Y) + 2)) -SCRIPT_RUN_DATE_SECONDS=$(date +%s) +must() { "$@" || die "cannot $*"; } +main() { + declare -a datesDiscontArray; #array -# Save zdump output for SCRIPT_YEAR (for how multiline data saved in variable, see https://stackoverflow.com/a/613580 ) -datesDiscont="$(zdump -c "$SCRIPT_YEAR","$SCRIPT_YEAR_NEXT2" -v $SCRIPT_TZ | grep "$SCRIPT_YEAR\|$SCRIPT_YEAR_NEXT" | awk '{ print $2 " " $3 " " $4 " " $5 " " $6 " " $7 }'; echo)" -if [ -z "$datesDiscont" ]; then - echo "No time discontinuity this year."; - exit 1; -fi; + # plumbing + script_year="$(date +%Y)"; + script_year_next="$(( script_year + 1 ))"; + script_year_next2="$(( script_year + 2 ))"; + script_run_date_seconds="$(date +%s)"; -###yell "DEBUG:datesDiscont----------"; -###echo "$datesDiscont" 1>&2; -###yell "DEBUG:-----------"; + # Get discontinuity dates from zdump (see https://stackoverflow.com/a/613580 ) + datesDiscont="$(zdump -c "$script_year","$script_year_next2" -v $script_tz | \ + grep "$script_year\|$script_year_next" | \ + awk '{ print $2 " " $3 " " $4 " " $5 " " $6 " " $7 }'; echo)"; # get dates + if [[ -z "$datesDiscont" ]]; then + yell "STATUS:No time discontinuity this year."; + exit 1; + fi; -# Count lines in datesDiscont (for counting lines in a variable, see https://stackoverflow.com/a/6314682 ) -#datesDiscontLineCount=$(echo "$datesDiscont" | wc -l) + # Convert datesDiscont into array + while read -r line; do + tunix="$(date --date="$line" +%s)"; # get unix epoch seconds + datesDiscontArray+=("$tunix"); + done < <(printf "%s\n" "$datesDiscont"); -# Convert datesDiscont into array (for how to process each line in a multiline variable, see https://superuser.com/a/284226 ) -while IFS= read -r line; do - #echo "$line" - #datesDiscontArray+="$line" - datesDiscontArray+=($(date --date="$line" +%s)) - #echo ${datesDiscontArray[-1]} -done <<< "$datesDiscont" -#yell ${datesDiscontArray[@]} + # Sort datesDiscontArray + while read -r line; do + datesDiscontArray+=("$line"); + done < <(printf "%s\n" "${datesDiscontArray[@]}" | sort); -# Sort datesDiscontArray (see https://stackoverflow.com/a/11789688 ) -IFS=$'\n' datesDiscontArray=($(sort <<<"${datesDiscontArray[*]}")) -unset IFS + # # Get earliest date in datesDiscontArray that isn't in the past. + for discontinuityDate in "${datesDiscontArray[@]}"; do + if [[ $discontinuityDate -gt $script_run_date_seconds ]]; then + nextDiscontDate="$discontinuityDate"; + break; + fi; + done; -###yell "DEBUG:"; -###yell "DEBUG:datesDiscontArray:"; -###yell "${datesDiscontArray[@]}"; -###yell "DEBUG:"; + # Check that nextDiscontDate is not empty + if [[ -z "$nextDiscontDate" ]]; then + die "FATAL:nextDiscontDate empty"; + fi; -# Get earliest date in datesDiscontArray that isn't in the past. -for discontinuityDate in "${datesDiscontArray[@]}"; do - ###yell "DEBUG:Discontinuity date (seconds):"$discontinuityDate - ###yell "DEBUG:SCRIPT_RUN_DATE_SECONDS :"$SCRIPT_RUN_DATE_SECONDS - if [ $discontinuityDate -gt $SCRIPT_RUN_DATE_SECONDS ]; then - ###yell "DEBUG:Date in future." - nextDiscontDate="$discontinuityDate" - break - fi -done + # Calculate days until next discontinuity date + output="$(( ( nextDiscontDate - script_run_date_seconds )/86400 )) days"; + printf "%s\n" "$output"; -# Check that nextDiscontDate is not empty -if [ -z "$nextDiscontDate" ]; then - echo "ERROR:nextDiscontDate empty" -fi; - +}; # main program -###yell "DEBUG:nextDiscontDate:$nextDiscontDate"; -###yell "DEBUG:SCRIPT_RUN_DATE_SECONDS:$SCRIPT_RUN_DATE_SECONDS"; +main "$@"; -echo $(( ( $nextDiscontDate - $SCRIPT_RUN_DATE_SECONDS )/86400 ))" days" # Days until next discontinuity date +# Author: Steven Baltakatei Sandoval +# License: GPLv3+ -- 2.30.2 From 8a4fc9ec681685229f902e2fe426fc7b3f851833 Mon Sep 17 00:00:00 2001 From: Steven Baltakatei Sandoval Date: Sat, 4 Mar 2023 00:12:49 +0000 Subject: [PATCH 13/16] feat(unitproc/isAudio):Add script to filter audio files from find results --- unitproc/isAudio | 195 +++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 195 insertions(+) create mode 100755 unitproc/isAudio diff --git a/unitproc/isAudio b/unitproc/isAudio new file mode 100755 index 0000000..943b0d1 --- /dev/null +++ b/unitproc/isAudio @@ -0,0 +1,195 @@ +#!/usr/bin/env bash +# Desc: Removes lines that aren't audio file paths +# Usage: find . -type f | isAudio +# Input: stdin +# Output: stdout +# Version: 0.0.1 +# Example: find ~/Music/ -type f | isAudio | mpv --playlist=- +# Depends: +# BK-2020-03: read_stdin() 0.0.1 + +declare -a music_codecs # Array for storing valid codec names (e.g. "aac" "mp3") + +# Adjustable parameters +music_codecs=("vorbis" "aac" "mp3" "flac" "opus"); # whitelist of valid codec_names ffprobe might return + +yell() { echo "$0: $*" >&2; } # print script path and all args to stderr +die() { yell "$*"; exit 111; } # same as yell() but non-zero exit status +must() { "$@" || die "cannot $*"; } # runs args as command, reports args if command fails +read_stdin() { + # Desc: Consumes stdin; outputs as stdout lines + # Input: stdin (consumes) + # Output: stdout (newline delimited) + # Example: printf "foo\nbar\n" | read_stdin + # Depends: GNU bash (version 5.1.16) + # Version: 0.0.1 + local input_stdin output; + + # Store stdin + if [[ -p /dev/stdin ]]; then + input_stdin="$(cat -)"; + fi; + + # Store as output array elements + ## Read in stdin + if [[ -n $input_stdin ]]; then + while read -r line; do + output+=("$line"); + done < <(printf "%s\n" "$input_stdin"); + fi; + + # Print to stdout + printf "%s\n" "${output[@]}"; +}; # read stdin to stdout lines +check_parsable_audio_ffprobe() { + # Desc: Checks if ffprobe returns valid audio codec name for file + # Usage: check_parsable_audio_ffprobe [path FILE] + # Version: 0.0.1 + # Input: arg1: file path + # Output: exit code 0 if returns valid codec name; 1 otherwise + # Depends: ffprobe, die() + local file_in ffprobe_out + + if [[ $# -ne 1 ]]; then die "ERROR:Invalid number of args:$#"; fi; + + file_in="$1"; + + # Check if ffprobe detects an audio stream + if ffprobe -v error -select_streams a -show_entries stream=codec_name -of default=nokey=1:noprint_wrappers=1 "$file_in" 1>/dev/null 2>&1; then + return_state="true"; + else + return_state="false"; + fi; + + # Fail if ffprobe returns no result + ffprobe_out="$(ffprobe -v error -select_streams a -show_entries stream=codec_name -of default=nokey=1:noprint_wrappers=1 "$file_in")"; + if [[ -z $ffprobe_out ]]; then + return_state="false"; + fi; + + # Report exit code + if [[ $return_state = "true" ]]; then + return 0; + else + return 1; + fi; +} # Checks if file has valid codec name using ffprobe +get_audio_format() { + # Desc: Gets audio format of file as string + # Usage: get_audio_format arg1 + # Depends: ffprobe + # Version: 0.0.1 + # Input: arg1: input file path + # Output: stdout (if valid audio format) + # exit code 0 if audio file; 1 otherwise + # Example: get_audio_format myvideo.mp4 + # Note: Would return "opus" if full ffprobe report had 'Audio: opus, 48000 Hz, stereo, fltp' + # Note: Not tested with videos containing multiple video streams + # Ref/Attrib: [1] https://stackoverflow.com/questions/5618363/is-there-a-way-to-use-ffmpeg-to-determine-the-encoding-of-a-file-before-transcod + # [2] https://stackoverflow.com/questions/44123532/how-to-find-out-the-file-extension-for-extracting-audio-tracks-with-ffmpeg-and-p#comment88464070_50723126 + local audio_format file_in; + local return_state; + file_in="$1"; + + # Return error exit code if not audio file + ## Return error if ffprobe itself exited on error + if ! ffprobe -v error -select_streams a -show_entries stream=codec_name -of default=nokey=1:noprint_wrappers=1 "$file_in" 1>/dev/null 2>&1; then + return_state="false"; + fi; + + # Get audio format + audio_format="$(ffprobe -v error -select_streams a -show_entries stream=codec_name -of default=nokey=1:noprint_wrappers=1 "$file_in")"; # see [1] + + ## Return error if audio format is incorrectly formatted (e.g. reject if contains spaces) + pattern="^[[:alnum:]]+$"; # alphanumeric string with no spaces + if [[ $audio_format =~ $pattern ]]; then + return_state="true"; + # Report audio format + echo "$audio_format"; + else + return_state="false"; + fi; + + # Report exit code + if [[ $return_state = "true" ]]; then + return 0; + else + return 1; + fi; +} # Get audio format as stdout +checkIsInArray() { + # Desc: Checks if input arg is element in array + # Usage: checkIsInArray arg1 arg2 + # Version: 0.0.1 + # Input: arg1: test string + # arg2: array + # Output: exit code 0 if test string is in array; 1 otherwise + # Example: checkIsInArray "foo" "${myArray[@]}" + # Ref/Attrib: [1] How do I check if variable is an array? https://stackoverflow.com/a/27254437 + # [2] How to pass an array as function argument? https://askubuntu.com/a/674347 + local return_state input arg1 string_test + declare -a arg2 array_test + input=("$@") # See [2] + arg1="${input[0]}"; + arg2=("${input[@]:1}"); + #yell "DEBUG:input:${input[@]}"; + #yell "DEBUG:arg1:${arg1[@]}"; + #yell "DEBUG:arg2:${arg2[@]}"; + + string_test="$arg1"; + array_test=("${arg2[@]}"); + + #yell "DEBUG:string_test:$string_test"; + #yell "DEBUG:$(declare -p array_test)"; + for element in "${array_test[@]}"; do + #yell "DEBUG:element:$element"; + if [[ "$element" =~ ^"$string_test" ]]; then + return_state="true"; + continue; + fi; + done; + + # Report exit code + if [[ $return_state == "true" ]]; then + return 0; + else + return 1; + fi; +} # Check if string is element in array +check_depends() { + if ! command -v ffprobe 1>/dev/random 2>&1; then + die "FATAL:Missing ffprobe."; fi; +}; +main() { + # Input: array: music_codecs ("mp3" "aac" ...) + # stdin + # Output: stdout + # Depends: + # BK-2020-03: read_stdin() 0.0.1 + + # check dependencies + check_depends; + + # iterate through stdin lines + while read -r line; do + # check if file + if [[ ! -f $line ]]; then continue; fi; # reject + + # check if valid codec + if ! check_parsable_audio_ffprobe "$line"; then + continue; fi; # reject + + # Check if desired codec + file_format="$(get_audio_format "$line")"; + if ! checkIsInArray "$file_format" "${music_codecs[@]}"; then + continue; fi; # reject + + # Output line to stdout + printf "%s\n" "$line"; + done < <(read_stdin); +}; # main program + +main "$@" 2>/dev/random; + +# Author: Steven Baltakatei Sandoval +# License: GPLv3+ -- 2.30.2 From 57357d431490ee122e0ea781cd2959d385dec5c5 Mon Sep 17 00:00:00 2001 From: Steven Baltakatei Sandoval Date: Sat, 4 Mar 2023 00:44:18 +0000 Subject: [PATCH 14/16] fix(unitproc/bkshuf):Init Bash PRNG before consuming lines - Note: v0.1.1 -> v0.1.2 --- unitproc/bkshuf | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/unitproc/bkshuf b/unitproc/bkshuf index e41d725..d2d0763 100644 --- a/unitproc/bkshuf +++ b/unitproc/bkshuf @@ -1,7 +1,7 @@ #!/usr/bin/env bash # Desc: Mixes input lines while also preserving some neighbors # Usage: cat file | bkshuf arg1 -# Version 0.1.1 +# Version 0.1.2 # Depends: bc 1.07.1, GNU Coreutils 8.32 (shuf) # Input: var: arg1 initial lines to output @@ -161,6 +161,7 @@ main() { lc_out="0"; # init output line counter if [[ -z "$lc_out_max" ]]; then lc_out_max="$lco"; fi; ip="$(shuf -i0-$(( lco - 1 )) -n1)"; # init input index pointer + RANDOM="$(shuf -i0-32767 -n1)"; # init Bash PRNG n_loop1="0"; #yell "DEBUG:max_blanks:$max_blanks" while [[ $lcr -ge 1 ]] && [[ $lc_out -lt $lc_out_max ]]; do -- 2.30.2 From 62f23c25753db3b522cf9697e2452ae943c56594 Mon Sep 17 00:00:00 2001 From: Steven Baltakatei Sandoval Date: Tue, 7 Mar 2023 21:32:55 +0000 Subject: [PATCH 15/16] feat(unitproc/get_rand_line):Bash script that gets random line --- unitproc/get_rand_line | 38 ++++++++++++++++++++++++++++++++++++++ 1 file changed, 38 insertions(+) create mode 100755 unitproc/get_rand_line diff --git a/unitproc/get_rand_line b/unitproc/get_rand_line new file mode 100755 index 0000000..166d419 --- /dev/null +++ b/unitproc/get_rand_line @@ -0,0 +1,38 @@ +#!/usr/bin/env bash +# Desc: Gets random line from a newline-delimited file +# Usage: get_rand_line [path FILE] +# Version: 0.0.1 + +yell() { echo "$0: $*" >&2; } # print script path and all args to stderr +die() { yell "$*"; exit 111; } # same as yell() but non-zero exit status +must() { "$@" || die "cannot $*"; } # runs args as command, reports args if command fails +get_rand_line() { + # Input: arg1 : file path + # stdout: random line from input file + # Usage: get_rand_line [path FILE] + # Depends: GNU Coreutils 8.32 (shuf) + + # Check if file + if [[ ! -f "$1" ]]; then die "FATAL:Not a file:$1"; fi; + if [[ $# -ne 1 ]]; then die "FATAL:Incorrect argument count:$1"; fi; + + # Get line count + lc="$(wc -l "$1" | awk '{print $1}')"; + + # Calc random line number (zero-indexed) + ln="$(shuf -i 0-"$((lc - 1))" -n1)"; + + n=0; + while read -r line; do + # Check if line is target line + if [[ $n -eq $ln ]]; then + printf "%s\n" "$line"; # output + fi; + ((n++)); + done < "$1"; +}; # Returns random line from newline-delimited file +main() { + get_rand_line "$@"; +}; + +main "$@"; -- 2.30.2 From 8c7da0e29cd807efd368dc51726cbc1561262547 Mon Sep 17 00:00:00 2001 From: Steven Baltakatei Sandoval Date: Tue, 14 Mar 2023 07:55:58 +0000 Subject: [PATCH 16/16] feat(user/ots-git-gpg-wrapper*):Add OpenTimestamps v0.7.0 wrapper scripts - Note: See https://github.com/opentimestamps/opentimestamps-client/blob/cd9ca65cbf7bd70db668cce47241f0feb8ea2186/ots-git-gpg-wrapper.sh - Note: Wrapper scripts are meant to replace the default `ots-git-gpg-wrapper.sh` script included in ots v0.7.0: https://github.com/opentimestamps/opentimestamps-client/blob/cd9ca65cbf7bd70db668cce47241f0feb8ea2186/ots-git-gpg-wrapper.sh --- user/ots-git-gpg-wrapper-wait.sh | 66 ++++++++++++++++++++++++++++++++ user/ots-git-gpg-wrapper.sh | 63 ++++++++++++++++++++++++++++++ 2 files changed, 129 insertions(+) create mode 100755 user/ots-git-gpg-wrapper-wait.sh create mode 100755 user/ots-git-gpg-wrapper.sh diff --git a/user/ots-git-gpg-wrapper-wait.sh b/user/ots-git-gpg-wrapper-wait.sh new file mode 100755 index 0000000..7fc0377 --- /dev/null +++ b/user/ots-git-gpg-wrapper-wait.sh @@ -0,0 +1,66 @@ +#!/usr/bin/env bash +# Desc: Wrapper for ots-git-gpg-wrapper with `--wait` option +# Note: Required because git's gpg.program option doesn't allow you to set +# command line options. +# Usage: git -c gpg.program= commit -S +# Input: file: $HOME/.local/share/ots/jsonrpc_url.txt # first line is JSON RPC +# url for local bitcoin node containing login credentials. Example: +# http://user:35z8deadbeefhrdeadbeef5rk3@192.168.86.1:8332/ +# Depends: ots 0.7.0 (see https://github.com/opentimestamps/opentimestamps-client ) +# Version: 0.0.6 +# Ref/Attrib: [1] “How can I test if a particular alias is defined?”. https://unix.stackexchange.com/a/288513 +# [2] “OpenTimestamps Git Integration”. (2016-10-13). https://github.com/opentimestamps/opentimestamps-client/blob/master/doc/git-integration.md + +declare -a args calendars + +# Specify default calendars +calendars+=("https://finney.calendar.eternitywall.com"); +calendars+=("https://btc.calendar.catallaxy.com"); +calendars+=("https://alice.btc.calendar.opentimestamps.org"); +calendars+=("https://bob.btc.calendar.opentimestamps.org"); + +# Check if gpg is alias. See [1]. +if alias gpg 2>/dev/null; then + ## Get gpg alias command + gpg_cmd="$(type gpg)"; # get raw alias definition + gpg_cmd="${gpg_cmd#*=\'}"; # trim chars before and including first apostrophe + gpg_cmd="${gpg_cmd%\'*}"; # trim chars after and including last apostrophe +else + gpg_cmd="$(which gpg)"; +fi; + +# Assemble args array +## Specify '--wait' option +args+=("--wait"); + +## Specify bitcoin node jsonrpc url if available +path_jsonrpc_url="$HOME/.local/share/ots/jsonrpc_url.txt"; +if [[ -f "$path_jsonrpc_url" ]]; then + jsonrpc_url="$(head -n1 "$path_jsonrpc_url")"; + args+=("--bitcoin-node"); + args+=("$jsonrpc_url"); +fi; + +## Pick random calendar +### Get calendars array size +cal_size="${#calendars[@]}"; +cal_rnd_idx="$(shuf -i 1-"$cal_size" -n 1)"; +cal_rnd_idx="$((cal_rnd_idx - 1))"; # bash array is zero-indexed +url_cal_random="${calendars[$cal_rnd_idx]}"; +args+=("--whitelist"); +args+=("$url_cal_random"); +args+=("--no-default-whitelist"); + +## Specify '--gpg-program' option +args+=("--gpg-program"); +args+=("$gpg_cmd"); + +## Passthrough positional parameters +args+=("--"); # mark end of ots-git-wrapper options +for param in "$@"; do + args+=("$param"); +done; + +# Run command with args +# pseudocode: ots-git-gpg-wrapper --wait $jsonrpc_option $calendar_option --gpg-program $gpg_cmd -- "$@" +ots-git-gpg-wrapper "${args[@]}"; diff --git a/user/ots-git-gpg-wrapper.sh b/user/ots-git-gpg-wrapper.sh new file mode 100755 index 0000000..b61ed95 --- /dev/null +++ b/user/ots-git-gpg-wrapper.sh @@ -0,0 +1,63 @@ +#!/usr/bin/env bash +# Desc: Wrapper for ots-git-gpg-wrapper +# Note: Required because git's gpg.program option doesn't allow you to set +# command line options. +# Usage: git -c gpg.program= commit -S +# Input: file: $HOME/.local/share/ots/jsonrpc_url.txt # first line is JSON RPC +# url for local bitcoin node containing login credentials. Example: +# http://user:35z8deadbeefhrdeadbeef5rk3@192.168.86.1:8332/ +# Depends: ots 0.7.0 (see https://github.com/opentimestamps/opentimestamps-client ) +# Version: 0.0.6 +# Ref/Attrib: [1] “How can I test if a particular alias is defined?”. https://unix.stackexchange.com/a/288513 +# [2] “OpenTimestamps Git Integration”. (2016-10-13). https://github.com/opentimestamps/opentimestamps-client/blob/master/doc/git-integration.md + +declare -a args calendars + +# Specify default calendars +calendars+=("https://finney.calendar.eternitywall.com"); +calendars+=("https://btc.calendar.catallaxy.com"); +calendars+=("https://alice.btc.calendar.opentimestamps.org"); +calendars+=("https://bob.btc.calendar.opentimestamps.org"); + +# Check if gpg is alias. See [1]. +if alias gpg 2>/dev/null; then + ## Get gpg alias command + gpg_cmd="$(type gpg)"; # get raw alias definition + gpg_cmd="${gpg_cmd#*=\'}"; # trim chars before and including first apostrophe + gpg_cmd="${gpg_cmd%\'*}"; # trim chars after and including last apostrophe +else + gpg_cmd="$(which gpg)"; +fi; + +# Assemble args array +## Specify bitcoin node jsonrpc url if available +path_jsonrpc_url="$HOME/.local/share/ots/jsonrpc_url.txt"; +if [[ -f "$path_jsonrpc_url" ]]; then + jsonrpc_url="$(head -n1 "$path_jsonrpc_url")"; + args+=("--bitcoin-node"); + args+=("$jsonrpc_url"); +fi; + +## Pick random calendar +### Get calendars array size +cal_size="${#calendars[@]}"; +cal_rnd_idx="$(shuf -i 1-"$cal_size" -n 1)"; +cal_rnd_idx="$((cal_rnd_idx - 1))"; # bash array is zero-indexed +url_cal_random="${calendars[$cal_rnd_idx]}"; +args+=("--no-default-whitelist"); +args+=("--whitelist"); +args+=("$url_cal_random"); + +## Specify '--gpg-program' option +args+=("--gpg-program"); +args+=("$gpg_cmd"); + +## Passthrough positional parameters +args+=("--"); # mark end of ots-git-wrapper options +for param in "$@"; do + args+=("$param"); +done; + +# Run command with args +# pseudocode: ots-git-gpg-wrapper $jsonrpc_option $calendar_option --gpg-program $gpg_cmd -- "$@" +ots-git-gpg-wrapper "${args[@]}"; -- 2.30.2