Merge branch 'feature/bash-logging-filter/BK-2020-02' of https://zdv2.bktei.com/gitwe...

[EVA-2020-02.git] / doc / time / README.org

#+TITLE: Ninfacyzga-1 Time Tracking
#+AUTHOR: Steven Baltakatei Sandoval
#+EMAIL: baltakatei@gmail.com
* Time Tracking
** About
This document was created by Steven Baltakatei Sandoval on
~2020-07-23T22:27Z~ under a [[https://creativecommons.org/licenses/by-sa/4.0/][Creative Commons BY-SA 4.0 license]]. It
was updated by Steven Baltakatei Sandoval on ~2020-10-16T22:26Z~

** Narrative
The ~ninfacyzga-01~ device is equipped with an Ozzmaker BerryGPS-IMU
module which provides time and location data to ~gpsd~ and ~ntpsec~. The
time is provided by GPS satellites which themselves are
equipped [fn:nasa_20020408_atomicclock] with atomic clocks. This
extremely accurate set of clocks are needed since a GPS receiver
calculates its position in space using a General Relativity
calculation that uses the small variations in the time stamps received
from each satellite. This means that ~gpsd~ may be used to set the
system clock without a need for an internet connection to a default
Debian time server; ~ninfacyzga-01~ can be its own time server.

[fn:nasa_20020408_atomicclock] Title:[[https://science.nasa.gov/science-news/science-at-nasa/2002/08apr_atomicclock/][Tick-Tock Atomic Clock]];
Date:2002-04-08; Website:NASA.gov; [[https://web.archive.org/web/20100429141752/http://science.nasa.gov/science-news/science-at-nasa/2002/08apr_atomicclock/][Archive-link]]; Archive-date:
2010-04-29

** Description
*** Hardware
Ozzmaker BerryGPS-IMU, Version 3 (see [[https://ozzmaker.com/berrygps-berrygps-imu-quick-start-guide/][ref]]).
*** Software
- [[https://tracker.debian.org/pkg/gpsd][~gpsd~]] : A background daemon app capable of interfacing with the
  [[https://ozzmaker.com/berrygps-berrygps-imu-quick-start-guide/][Ozzmaker BerryGPS-IMU]]'s GPS submodule. Installed and initialized by
  ~apt~. Should be installed along with the ~gpsd-clients~
  package. This procedure was developed with ~gpsd~ version

- [[https://tracker.debian.org/pkg/ntpsec][~ntpsec~]] : A security-hardned version of [[https://tracker.debian.org/pkg/ntp][~ntp~]] which is a "[[https://en.wikipedia.org/wiki/Network_Time_Protocol][Network
  Time Protocol]]" app that synchronizes the system clock with local or
  remote time servers or devices. It is capable of extracting time
  data from a GPS signal handled by ~gpsd~. It is also capable of
  using the high precision PPS (pulse-per-second) signal handled by
  ~gpsd~. Installed via ~apt~. This procedure was developed with
  ~ntpsec~ version ~1.1.3~.

** Operating Procedures
*** Initial Startup
**** Perform initial setup.
See [[file:../setup/README.org][Main Setup]] procedure.
**** Install Hardware for time tracking
See [[https://ozzmaker.com/forums/topic/connecting-gps-pps-pin/][this]] Ozzmaker forum topic about connecting the BerryGPS-IMU
~T_PULSE~ pin to GPIO 18.

#+CAPTION: An image showing how to connect the PPS signal from an Ozzmaker BerryGPS-IMU board to a Raspberry Pi Zero W.
#+NAME: fig:PPS_BERRYGPS_RASPIZW
[[../../img/Compact_Stratum_1_NTP_time_server_hardware,_October_2020.jpg]]

Connect the ~T_PULSE~ connection on the BerryGPS-IMU-3 to GPIO pin 18
(ex: with solder and wire) in order to provide the PPS data signal
generated by the BerryGPS-IMU to the Raspberry Pi. Processing of this
data signal is handled by adding a line to ~/boot/config.txt~ in the
next section ("Install Software").

Note: If it is desired to specify a custom GPIO pin besides the one
recommended, see this [[https://raspberryautomation.com/connect-multiple-ds18b20-temperature-sensors-to-a-raspberry-pi/][Raspberry Autom]] article.

**** Install Software for time tracking
The time tracking function can be performed by two programs: ~gpsd~
and ~ntpsec~.

Basically, two things need to happen:

1. ~gpsd~ needs to be pointed towards the correct device files for
   incoming GPS data (in NMEA format) and the PPS signal ("pulse per
   second"; a high precision time signal).

2. ~ntpsec~ needs to be pointed towards the correct local IP addresses
   where ~gpsd~ provides GPS data and the PPS signal.

~gpsd~ then will provide GPS and PPS data to ~ntpsec~ via a "shared
memory driver".

***** Install packages via ~apt~
Run the following command to install the required packages.
: $ sudo apt install gpsd gpsd-clients python-gps pps-tools ntpsec

***** Enable PPS device
Modify the ~/boot/config.txt~ file in order to tell the Raspberry Pi
to expect PPS data on ~BCM 18~ (pin number 12; see [[https://pinout.xyz/][link]]). This is done
by adding the following line to ~/boot/config.txt~ as described on
[[https://ozzmaker.com/forums/topic/problems-with-pps-on-a-pi0w-running-raspian-and-attached-to-a-berrygps-imuv3/][this Ozzmaker page]]:

: dtoverlay=pps-gpio,gpiopin=18

The ~/boot/config.txt~ file can be modified via:

: $ sudo nano /boot/config.txt

PPS data can be confirmed by running:

#+BEGIN_EXAMPLE
$ sudo su -
# ppstest /dev/pps0
trying PPS source "/dev/pps0"
found PPS source "/dev/pps0"
ok, found 1 source(s), now start fetching data...
source 0 - assert 1595708074.003644641, sequence: 219 - clear  0.000000000, sequence: 0
source 0 - assert 1595708075.003709620, sequence: 220 - clear  0.000000000, sequence: 0
source 0 - assert 1595708076.003779580, sequence: 221 - clear  0.000000000, sequence: 0
source 0 - assert 1595708077.003850580, sequence: 222 - clear  0.000000000, sequence: 0
#+END_EXAMPLE

Note: For older Raspberry Pi models, it may be necessary to enable
~pps-gpio~ via modifications to ~/etc/modules~ (see [[https://www.raspberrypi.org/forums/viewtopic.php?p=757747#p757747][link]]).

***** Enable GPS device
The Ozzmaker BerryGPS-IMU makes NMEA sentences available via the
serial "UART" device ~/dev/ttyAMA0~. If bluetooth has not been
disabled, the Raspberry Pi OS automatically creates a software "UART"
device at ~/dev/serial0~. See the "[[file:~/git-OC/ninfacyzga-01/doc/setup/README.org::*Disable%20Bluetooth][Disable Bluetooth]]" section in the
[[file:../setup/README.org][Main Setup]] Initial Startup procedure for instructions on how to
disable bluetooth to free up ~/dev/ttyAMA0~ for use by ~gpsd~.

Note: Older ~ntp~ documentation indicates that ~ntp~ (and presumably
~ntpsec~ might be able to access GPS data (ex: NMEA sentences) without
the aid of ~gpsd~ itself if the data is available at a device named
~/dev/ggp0~. Similarly, the same might apply to PPS data and the
~/dev/gpspps0~ device. However, since the Raspberry Pi OS overlay
system automatically creates ~/dev/pps0~ using the modifications to
~/boot/config.txt~ (described [[*Enable PPS device][elsewhere]] in this procedure).

***** Setup ~gpsd~
See the "[[file:~/git-OC/ninfacyzga-01/doc/location/README.org::*Setup%20~gpsd~][Setup ~gpsd~]]" subsection within the "Initial Startup" section
of the Location Logging [[file:~/git-OC/ninfacyzga-01/doc/location/README.org][~README.org~]] file. There is one additional
change that must be made which is to add a ~/dev/pps0~ item to the
~DEVICES=~ line in ~/etc/default/gpsd~ like so:

: DEVICES="/dev/ttyAMA0 /dev/pps0"

~/dev/ttyAMA0~ is where ~gpsd~ can get NMEA data from the GPS unit.

~/dev/pps0~ is where ~gpsd~ can get a PPS signal.

As an example, the following lines will be present in
~/etc/default/gpsd~ if both location and time tracking are set up:

#+BEGIN_EXAMPLE
START_DAEMON="true"
USBAUTO="false"
DEVICES="/dev/ttyAMA0 /dev/pps0"
GPSD_OPTIONS="-n"
#+END_EXAMPLE

***** Setup ~ntpsec~
The ~ntpsec~ configuration file at ~/etc/ntpsec/ntp.conf~ must be
modified to tell ~ntpsec~ how to retrieve and interpret the GPS and
PPS data provided by ~gpsd~.

In summary, lines need to be added telling ~ntpsec~ how often to poll
certain sources (local or remote) for time information as well as how
to prioritize and label information from each source.

~ntpsec~ uses syntax simplified from that of ~ntp~. Specifically, it
recommends use of the ~refclock~ keyword for configuring local
hardware clocks such as GPS and PPS devices.

Below is an example of lines that may be added to
~/etc/ntpsec/ntp.conf~ in order to ensure that the system clock will always have 

#+BEGIN_EXAMPLE
# Local time sources

# Kernel PPS (precision seconds; driver 22)
refclock pps unit 0 refid kPPS flag3 1 minpoll 4 maxpoll 4 stratum 1

# GPS PPS reference (precision seconds; driver 28; root) (NTP1)
refclock shm unit 1 refid PPS minpoll 4 maxpoll 4 iburst stratum 1

# GPS Serial data reference (coarse time; driver28; root) (NTP0)
refclock shm unit 0 refid GPS prefer time1 0.109 minpoll 4 maxpoll 4 iburst stratum 1

# GPS PPS reference (precision seconds; driver 28; user) (NTP2)
refclock shm unit 2 refid PPSuser minpoll 4 maxpoll 4 iburst stratum 1

# Tolerate outliers (useful for coarse clocks)
tos mindist 0.500
#+END_EXAMPLE

A complete working example of an ~ntp.conf~ file is found in Appendix
A of this document.

Commands useful for examining the shared memory driver used by ~gpsd~
and ~ntpsec~ are:

- ~sudo ntpshmmon~ : Shows live output of data using the shared memory
  driver. ([[https://gpsd.gitlab.io/gpsd/gpsd-time-service-howto.html][ref]])

- ~sudo ipcs -m~ : Show live segments of the shared memory. ([[https://gpsd.gitlab.io/gpsd/gpsd-time-service-howto.html][ref]])

General references for this configuration file are:

- "Raspberry Pi as a Stratum-1 NTP Server - Updating to the final NTP
  configuration file" by David Taylor. [[http://www.satsignal.eu/ntp/Raspberry-Pi-NTP.html#ntp-conf][Link]]. Date: 2020-03-24. Note:
  An example of kernel PPS transfer from ~gpsd~ to ~ntp~ via
  ~127.127.22.0~.

- "GPSD Time Service HOWTO" by Garry E. Miller. [[https://gpsd.gitlab.io/gpsd/gpsd-time-service-howto.html][Link]]. Date:
  2020-06-28. Note: Explains how shared memory segments available at
  ~127.127.28.0~, ~127.127.28.1~, ~127.127.28.2~, etc. work in context
  of an ~ntp~ configuration file.

- "Building a Stratum 1 NTP Server with a Raspberry Pi 4 and Adafruit
  Ultimate GPS Hat". [[http://www.gregledet.net/computers/building-a-stratum-1-ntp-server-with-a-raspberry-pi-4-and-adafruit-ultimate-gps-hat/][Link]]. Date: 2020-02-23. Note: A concise example
  between ~gpsd~ and ~ntp~ setup for a Raspberry Pi 4 using
  ~127.127.28.0~ and ~127.127.28.1~. Recommends use of ~ntpsec~
  instead of ~ntp~.

Expalantions of some ~ntpsec~ [[https://docs.ntpsec.org/latest/comdex.html][commands and options]] used in the
configuration file include:

- ~refclock~ : An ~ntpsec~ keyword not present in ~ntp-4.2.8~ that is
  meant to simplify configuration of reference clocks that use certain
  [[https://docs.ntpsec.org/latest/refclock.html][drivers]] including ~pps~ ([[https://docs.ntpsec.org/latest/driver_pps.html][PPS Clock Discipline]]), ~shm~ ([[https://docs.ntpsec.org/latest/driver_shm.html][Shared Memory
  Driver]]) . In ~ntp-4.2.8~ and earlier, in order to configure a clock,
  instead of ~ntpsec~'s one keyword (~refclock~), two keywords would
  be required (~server~ and ~fudge~).

- ~server [address]~ : Tells ~ntpsec~ the address to query for time
  information. May be IP address or URL. Additional options such as
  ~minpoll~, ~maxpoll~, ~prefer~, and ~true~ may be added.

- ~prefer~ : Indicates the source should be prioritized in some
  way. How exactly the ~ntpsec~ mitigation rules (see [[https://docs.ntpsec.org/latest/prefer.html][ref]]) use the
  ~prefer~ option is complicated but in the context of dealing with a
  GPS source and a PPS source, the GPS source should have a ~prefer~
  option. The PPS source may have a ~prefer~ option but it is not
  necessary; adding ~prefer~ to a PPS source is useful in the corner
  case that the GPS source is less reliable than another source (ex: a
  remote clock specified by ~server~).

- ~... shm ... flag1 1~ : For an ~shm~ driver, tells ~ntpsec~ to
  ignore the default sanity check that discards any time that is more
  than 4-hours off from the system clock. Note that ~ntpsec~ treats
  ~shm flag1 1~ in the opposite manner as ~ntp~ (see [[https://docs.ntpsec.org/latest/driver_shm.html][ref]]).

- ~... shm ... time1 [float]~ : Tells ~ntpsec~ that a latency
  time of ~[float]~ seconds is required for data to travel from the
  address to ~ntpsec~ (ex: a USB hub in the case of a GPS device that
  connects via USB). (see [[https://gpsd.gitlab.io/gpsd/gpsd-time-service-howto.html#_feeding_ntpd_from_gpsd][ref]]) This number of seconds should be
  adjusted to reduce the `offset` time of the address indicated in the
  output of the ~ntpq -pn~ command to near ~0.000~. Note that the
  units of ~offset~ displayed by ~ntpq -pn~ are in milliseconds.

- ~... refid [string]~ : Tells ~ntpsec~ a string used to label the source.

- ~fudge~ : An ~ntp~ keyword that ~ntpsec~ developers do not recommend
  using; ~ntpsec~ documentation for ~fudge~ is widthdrawn. For local
  hardware providing GPS and PPS time information, ~fudge~ can mostly
  be replaced with use of the ~refclock~ keyword instead.

- ~fudge [address]~ : Tells ~ntpsec~ to consider additional options
  when evaluating the accuracy of time from a given address.

- ~fudge [address] ... flag3 1~ : An option that tells ~ntpsec~ to
  enable "kernel PPS discipline" when interpreting the PPS signal at
  the address. (see [[http://doc.ntp.org/4.2.8p7/drivers/driver22.html][ref]]).

- ~stratum 1~ : Tells ~ntpsec~ to label the source as a "stratum 1"
  time server. This means that it is a clock using NTP that is a
  canonical (?) time source. For example, a stratum 0 NTP time server
  would be an atomic clock. A stratum 1 NTP time server could be a
  clock that uses a GPS receiver to synchronize to said atomic clock.

- ~iburst~ : Tells ~ntpsec~ how often to query a source.

- ~fudge [address] ... flag1 [float]~ : See description of ~... shm
  ... time1 [float]~.

- ~tos mindist [float]~ : Tells ~ntpsec~ to increase the tolerance for
  outliers in the clock selection algorithm. Units of ~[float]~ are
  seconds. The default value is ~0.001~ seconds. This [[http://doc.ntp.org/4.2.6/miscopt.html][reference]]
  indicates that it is acceptable to raise this value for "clocks with
  high jitter and a PPS signal". For the device being configured in
  this guide, this is the case. ~ntpsec~ receives a kernel PPS signal
  on ~127.127.22.0~ indicating when each second occurs but without
  information about which second of the day is being ticked. ~ntpsec~
  receives a coarse time from ~127.127.28.0~ supplied by ~gpsd~ that
  is derived from the NMEA sentences generated by the GPS receiver;
  this coarse time (high jitter) lacks the precision of the PPS signal
  since NMEA sentence transmission times are not well-controlled.

****** Disable DHCP services affecting ~ntpsec~
Some processes that use [[https://en.wikipedia.org/wiki/Dynamic_Host_Configuration_Protocol][DHCP]] (Dynamic Host Configuration Protocol) may
cause configuration changes to configuration files of ~ntp~. Since
~ntpsec~ is similar to ~ntp~, it is reasonable to take action to
prevent this.

Remove some files:

: $ sudo rm /etc/dhcp/dhclient-exit-hooks.d/ntp
: $ sudo rm /lib/dhcpcd/dhcpcd-hooks/50-ntp.conf

In the "request" block of ~dhclient.conf~, remove ~dhcp6.sntp-servers~
and ~ntp-servers~.

: $ sudo nano /etc/dhcp/dhclient.conf

For example, the following section of ~dhclient.conf~:

#+BEGIN_EXAMPLE
send host-name = gethostname();
request subnet-mask, broadcast-address, time-offset, routers,
        domain-name, domain-name-servers, domain-search, host-name,
        dhcp6.name-servers, dhcp6.domain-search, dhcp6.fqdn, dhcp6.sntp-servers,
        netbios-name-servers, netbios-scope, interface-mtu,
        rfc3442-classless-static-routes, ntp-servers;
#+END_EXAMPLE

should be changed to:

#+BEGIN_EXAMPLE
send host-name = gethostname();
request subnet-mask, broadcast-address, time-offset, routers,
        domain-name, domain-name-servers, domain-search, host-name,
        dhcp6.name-servers, dhcp6.domain-search, dhcp6.fqdn,
        netbios-name-servers, netbios-scope, interface-mtu,
        rfc3442-classless-static-routes;
#+END_EXAMPLE

***** Disable CPU power saving
Power saving featurs of the Raspberry Pi Zero W may also be disabled
in order to improve accuracy.

****** Configure CPU ~scaling_governor~
If additional precision is required, the PPS signal may be made more
reliable at the cost of increasing CPU power by configuring the CPU to
always run at maximum frequency.[fn:se_20180320_raspicpugov] This
change can be performed by modifying the following file as root:

: /sys/devices/system/cpu/cpu0/cpufreq/scaling_governor

The file should consist of one line. Change

: ondemand

to

: performance

.

This change can be performed via the ~nano~ text editor by running the
following commands:

: $ sudo su -
: # nano /sys/devices/system/cpu/cpu0/cpufreq/scaling_governor

Additionally, in order to prevent the ~raspi-config~ init script from
reverting this text file back to ~ondemand~ after a reboot, this
script must be disabled via:

: $ sudo systemctl disable raspi-config

****** Configure ~/boot/config.txt~
Modify ~/boot/config.txt~ so that it contains these lines in order to
disable power saving functions:

#+BEGIN_EXAMPLE
# Disable power saving
nohz=off
#+END_EXAMPLE

[fn:se_20180320_raspicpugov] Title:[[https://raspberrypi.stackexchange.com/questions/9034/how-to-change-the-default-governor]["How to change the default governor?"]]; Author:[[https://raspberrypi.stackexchange.com/users/5538/goldilocks][goldilocks]]; Date: 2018-03-20; Website:stackexchange.com;

*** Normal Startup
*** Normal Operation
*** Normal Shutdown
*** Unscheduled Shutdown
** Appendix A
*** Example ~ntp.conf~ for ~ntpsec~
Below is an example ~ntp.conf~ file for use with ~ntpsec~.

#+BEGIN_EXAMPLE
# /etc/ntpsec/ntp.conf, configuration for ntpd; see ntp.conf(5) for help

# Drift file
driftfile /var/lib/ntpsec/ntp.drift

# Leap seconds definition provided by tzdata
leapfile /usr/share/zoneinfo/leap-seconds.list

# Log file
#logfile /var/log/ntp.log


# You must create /var/log/ntpsec (owned by ntpsec:ntpsec) to enable logging.
#statsdir /var/log/ntpsec/
#statistics loopstats peerstats clockstats
#filegen loopstats file loopstats type day enable
#filegen peerstats file peerstats type day enable
#filegen clockstats file clockstats type day enable


# Comment this out if you have a refclock and want it to be able to discipline
# the clock by itself (e.g. if the system is not connected to the network).
#tos minclock 4 minsane 3


# Local time sources

# Kernel PPS (precision seconds; driver 22)
refclock pps unit 0 refid kPPS flag3 1 minpoll 4 maxpoll 4 stratum 1

# GPS PPS reference (precision seconds; driver 28; root) (NTP1)
refclock shm unit 1 refid PPS minpoll 4 maxpoll 4 iburst stratum 1

# GPS Serial data reference (coarse time; driver28; root) (NTP0)
refclock shm unit 0 refid GPS prefer time1 0.109 minpoll 4 maxpoll 4 iburst stratum 1

# GPS PPS reference (precision seconds; driver 28; user) (NTP2)
refclock shm unit 2 refid PPSuser minpoll 4 maxpoll 4 iburst stratum 1

# Tolerate outliers (useful for coarse clocks)
tos mindist 0.500


# Specify one or more NTP servers.

# pool.ntp.org maps to about 1000 low-stratum NTP servers.  Your server will
# pick a different set every time it starts up.  Please consider joining the
# pool: <https://www.pool.ntp.org/join.html>
server time.nist.gov iburst
server tick.usno.navy.mil iburst
server 0.us.pool.ntp.org iburst
#server 0.us.pool.ntp.org iburst
#server 0.us.pool.ntp.org iburst
#server 0.us.pool.ntp.org iburst
pool 0.debian.pool.ntp.org iburst
#pool 1.debian.pool.ntp.org iburst
#pool 2.debian.pool.ntp.org iburst
#pool 3.debian.pool.ntp.org iburst


# Access control configuration; see /usr/share/doc/ntpsec-doc/html/accopt.html
# for details.
#
# Note that "restrict" applies to both servers and clients, so a configuration
# that might be intended to block requests from certain clients could also end
# up blocking replies from your own upstream servers.

# By default, exchange time with everybody, but don't allow configuration.
restrict default kod nomodify nopeer noquery limited

# Local users may interrogate the ntp server more closely.
restrict 127.0.0.1
restrict ::1
#+END_EXAMPLE

* Discard

** Transitional ntp to ntpsec config file snippet
# Kernel PPS (precision seconds; driver 22)
#server 127.127.22.0 minpoll 4 maxpoll 4 true
#fudge 127.127.22.0 flag3 1 refid kPPS stratum 1
refclock pps unit 0 refid kPPS flag3 1 minpoll 4 maxpoll 4 stratum 1

# GPS PPS reference (precision seconds; driver 28; root) (NTP1)
#server 127.127.28.1 minpoll 4 maxpoll 4 iburst
#fudge 127.127.28.1 flag1 1 refid PPS
refclock shm unit 1 refid PPS minpoll 4 maxpoll 4 iburst stratum 1

# GPS Serial data reference (coarse time; driver28; root) (NTP0)
#server 127.127.28.0 minpoll 4 maxpoll 4 iburst prefer
#fudge 127.127.28.0 flag1 1 time1 0.109 refid GPS stratum 1
refclock shm unit 0 refid GPS prefer time1 0.109 minpoll 4 maxpoll 4 iburst stratum 1

# GPS PPS reference (precision seconds; driver 28; user) (NTP2)
#server 127.127.28.2 minpoll 4 maxpoll 4 iburst
#fudge 127.127.28.2 flag1 1 refid PPSuser stratum 1
refclock shm unit 2 refid PPSuser minpoll 4 maxpoll 4 iburst stratum 1

# Tolerate outliers (useful for coarse clocks)
tos mindist 0.500