Discussion:
Four hour cycle in GPS NMEA jitter
Add Reply
Trent Piepho
2017-03-20 21:00:20 UTC
Reply
Permalink
Raw Message
Hello time nuts,

I'm working on a custom embedded Linux device, with a custom inertial reference unit, which contains a GPS module. The module is a Telit JN3, which is based on the SiRFSTAR IV I believe. I'd like to use the GPS to sync the Linux system clock. Eventually I'd like to use the PPS signal, which is routed to a FPGA that's part of the CPU, to implement a custom PPS hardware module that I can write a kernel driver for and use the Linux hardpps system. And maybe make that feedback to the CPU's main clock source, since the FPGA also controls that and could create a PLL between the TCXO that serves as the master clock signal and the CPU's source clock.

But first things first. I'm just grabbing the time from NMEA sentences. And there's quite a bit of jitter there! Clearly using the first sentence output by the GPS is critical. I've tried to account for any time delays in the software. I think it's the GPS module that is creating the largest source of jitter. It appears to go in four hour cycles, peaking at 0:00Z, 4:00Z, 8:00Z, etc.

Does this sound like something that one would expect with the NMEA output of a non-timing GPS? Is it related to satellite orbits? Or perhaps is has something to do with the design of the SiRFStar IV?

I'll attach a graph of what I'm seeing. If the attachment doesn't come though it's viewable at https://goo.gl/photos/JtYfJCpRSZb3hCnM8.

Methodology for the graph:
System clock is left free running and not disciplined, after an initial one time set based on the GPS time.
On each NMEA GGA sentence, sent at 1 Hz, the system clock's offset from the NMEA timestamp is measured.
Each minute, the mean, std.dev, min and max are found for the last 60 offset samples. This is plotted on the graph.
Any outlier samples, defined as more than 3 sigma from the previous mean, are also plotted.
Concurrently, the chrony NTP daemon is running and monitoring the IT dept's NTP server, but NOT being used to set the local system clock.
Once a minute, the system clock's offset to chrony's idea of the NTP server's clock is also measured. Chrony is using an algorithm based on median filtering to get its idea of the NTP server's clock.
The NTP server is just a windows domain controller synced to the internet NTP pool and far from a precision source.
Chris Albertson
2017-03-20 22:19:25 UTC
Reply
Permalink
Raw Message
Post by Trent Piepho
Does this sound like something that one would expect with the NMEA output of a non-timing GPS? Is it related to satellite orbits? Or perhaps is has something to do with the design of the SiRFStar IV?
Remember the phone based time service? "At the tome time time will be
.... BEEP" With a GPS the NEMA sentence take the place of the spoken
words on the phone. The NEMA specification allows the sentences to
up to one second "off". That said most GPS receivers do MUCH better
than the NEMA spec but you should never count on non-specified
performance in a professional design. The PPS is of course doing the
job of the BEEP on the old phone system. Set you watch based on the
beep, not NEMA.

If you have an FPGA available then you could significantly improve
system time keeping. Currently the PPS interrupts the CPU to
snapshot internal counter. Unpredictable interrupt latency lifts NTP
timekeeping to about 1 or 2 microseconds but is the counter snap
shooting could be moved out to FPGA hardware there would be no unknown
latency and you could get NTP to break a "magic" 1uS barrier. I've
only hear of 1 uS being broken with hardware. You would actually
not ned to write much software to make this happen, just move the
counter outside the CPU to the FPGA and you about have it.


Chris Albertson
Redondo Beach, California
_______________________________________________
time-nuts mailing list -- time-***@febo.com
To unsubscribe, go to https://www.febo.com/cgi-bin/mailman/listinfo/time-nuts
and follow the instructions there.
Gary E. Miller
2017-03-21 00:06:54 UTC
Reply
Permalink
Raw Message
Yo Chris!

On Mon, 20 Mar 2017 15:19:25 -0700
I've only hear of 1 uS being broken with hardware.
A Raspberry Pi can get down to a Standard Deviation of about 350 nano seconds
using NTPsec..

https://blog.ntpsec.org/2017/02/01/heat-it-up.html


RGDS
GARY
---------------------------------------------------------------------------
Gary E. Miller Rellim 109 NW Wilmington Ave., Suite E, Bend, OR 97703
***@rellim.com Tel:+1 541 382 8588

Veritas liberabit vos. -- Quid est veritas?
"If you can’t measure it, you can’t improve it." - Lord Kelvin
_______________________________________________
time-nuts mailing list -- time-***@febo.com
To unsubscribe, go to https://www.febo.com/cgi-bin/mailman/listinfo/time-nuts
and follow the instructions there.
Chris Albertson
2017-03-21 05:22:48 UTC
Reply
Permalink
Raw Message
Wow. 350 nS. I had not been following this for the last few years. It
seems the ARM is a simply CPU with much more predictable interrupts timing.


One might ask way you'd need such good internal timing. What got me into
this years ago was scientific data acquisition. I wanted to time stamp the
data. So I had a Linux based PC connect periodically to the Internet
using a dial-up phone modem and run NTP. It worked well enough. A GPS
receiver at $10,000 each was out of the question.
Post by Gary E. Miller
Yo Chris!
On Mon, 20 Mar 2017 15:19:25 -0700
I've only hear of 1 uS being broken with hardware.
A Raspberry Pi can get down to a Standard Deviation of about 350 nano seconds
using NTPsec..
https://blog.ntpsec.org/2017/02/01/heat-it-up.html
RGDS
GARY
------------------------------------------------------------
---------------
Gary E. Miller Rellim 109 NW Wilmington Ave., Suite E, Bend, OR 97703
Veritas liberabit vos. -- Quid est veritas?
"If you can’t measure it, you can’t improve it." - Lord Kelvin
_______________________________________________
To unsubscribe, go to https://www.febo.com/cgi-bin/
mailman/listinfo/time-nuts
and follow the instructions there.
--
Chris Albertson
Redondo Beach, California
_______________________________________________
time-nuts mailing list -- time-***@febo.com
To unsubscribe, go to https://www.febo.com/cgi-bin/mailman/listinfo/time-nuts
and follow the instructions there.
Bob Camp
2017-03-20 22:14:20 UTC
Reply
Permalink
Raw Message
Hi


NMEA sentences are not the best thing to use for timing. If you *do* decide to use them, configure the
receiver so that one and only one sentence comes out. Any time you have more than one, you run the risk
of collision in the serial buffer on the part. Next thing to do is to pick the shortest useful sentence you
can find. In some cases you only have one option.

Next up is to be sure that your PC is set up so that there is minimal lag in serial processing. That may not
be as simple as you might think. All modern OS’s head off to do “interesting things” from time to time. The
usual approach is to kill just about anything that *might* create an issue.

Once you have all that, you start pruning the outliers and fitting to the center of the data. Since you don’t have
a super duper clock on your motherboard, you are fitting both the randomness of the clock and the GPS. The
answer is to go slow and look at data over a lot of samples.

The other answer to all this is to use a GPS with a PPS out. Feed that into some sort of capture process and
go from there. The result is likely to improve things by at least a couple orders of magnitude.

Bob
Post by Trent Piepho
Hello time nuts,
I'm working on a custom embedded Linux device, with a custom inertial reference unit, which contains a GPS module. The module is a Telit JN3, which is based on the SiRFSTAR IV I believe. I'd like to use the GPS to sync the Linux system clock. Eventually I'd like to use the PPS signal, which is routed to a FPGA that's part of the CPU, to implement a custom PPS hardware module that I can write a kernel driver for and use the Linux hardpps system. And maybe make that feedback to the CPU's main clock source, since the FPGA also controls that and could create a PLL between the TCXO that serves as the master clock signal and the CPU's source clock.
But first things first. I'm just grabbing the time from NMEA sentences. And there's quite a bit of jitter there! Clearly using the first sentence output by the GPS is critical. I've tried to account for any time delays in the software. I think it's the GPS module that is creating the largest source of jitter. It appears to go in four hour cycles, peaking at 0:00Z, 4:00Z, 8:00Z, etc.
Does this sound like something that one would expect with the NMEA output of a non-timing GPS? Is it related to satellite orbits? Or perhaps is has something to do with the design of the SiRFStar IV?
I'll attach a graph of what I'm seeing. If the attachment doesn't come though it's viewable at https://goo.gl/photos/JtYfJCpRSZb3hCnM8.
System clock is left free running and not disciplined, after an initial one time set based on the GPS time.
On each NMEA GGA sentence, sent at 1 Hz, the system clock's offset from the NMEA timestamp is measured.
Each minute, the mean, std.dev, min and max are found for the last 60 offset samples. This is plotted on the graph.
Any outlier samples, defined as more than 3 sigma from the previous mean, are also plotted.
Concurrently, the chrony NTP daemon is running and monitoring the IT dept's NTP server, but NOT being used to set the local system clock.
Once a minute, the system clock's offset to chrony's idea of the NTP server's clock is also measured. Chrony is using an algorithm based on median filtering to get its idea of the NTP server's clock.
The NTP server is just a windows domain controller synced to the internet NTP pool and far from a precision source.
<plot1.png>_______________________________________________
To unsubscribe, go to https://www.febo.com/cgi-bin/mailman/listinfo/time-nuts
and follow the instructions there.
_______________________________________________
time-nuts mailing list -- time-***@febo.com
To unsubscribe, go to https://www.febo.com/cgi-bin/mailman/listinfo/time-nuts
and follow the instructions there.
Kiwi Geoff
2017-03-20 23:43:58 UTC
Reply
Permalink
Raw Message
Hi Trent,
Post by Trent Piepho
But first things first. I'm just grabbing the time from NMEA sentences.
And there's quite a bit of jitter there! Clearly using the first sentence
output by the GPS is critical. I've tried to account for any time delays in
the software. I think it's the GPS module that is creating the largest
source of jitter. It appears to go in four hour cycles, peaking at 0:00Z,
4:00Z, 8:00Z, etc.
From my observations on various GPS receivers, the 'time" that the
NMEA data is transmitted can be highly variable.

For example, here is a (24 hour) graph from my Garmin 18x (firmware
v3.6) where a plot (thanks to Hal) shows the start time of the NMEA
sentence from the time of the GPS 1PPS edge.

Loading Image...

David Taylor explains more about this Garmin firmware variation here:

http://www.satsignal.eu/ntp/Garmin-GSP18x-LVC-firmware-issue.htm

Where an earlier version of the 18x, the latency could be so looooong
that it actually gave the timestamp for the wrong second !

So yes, beware of using the "transmit time" of NMEA sentences, many
GPS receivers appear to have a cuppa tea, a scone and solve a Sudoko
before telling us the time !

Regards, Kiwi Geoff (Christchurch , New Zealand).
_______________________________________________
time-nuts mailing list -- time-***@febo.com
To unsubscribe, go to https://www.febo.com/cgi-bin/mailman/listinfo/time-nuts
and follow the instructions there.
Trent Piepho
2017-03-21 20:52:23 UTC
Reply
Permalink
Raw Message
Thanks to all who responded. Yes, I know PPS is the way get a more
accurate timestamps. That is the plan, but it takes more time to write
FPGA programs. The surprise is not that there is considerable jitter on
the NMEA output, but rather why does the jitter have patterns in it?

It seems significant enough that someone who had used NMEA for time
would have noticed it before. Unless it is somehow unique to me, but
how would that be?
Post by Kiwi Geoff
For example, here is a (24 hour) graph from my Garmin 18x (firmware
v3.6) where a plot (thanks to Hal) shows the start time of the NMEA
sentence from the time of the GPS 1PPS edge.
I've tried to duplicate that graph to some extent, plotting NMEA
sentence offset from system clock. Attached and also at
https://goo.gl/photos/TBEg27SRqY2EQW3HA

In this plot I've taken the offset from the system clock, fitted it to a
simple f(x)=a+b*x model, then plotted the residuals. I.e., I've taken
out a constant clock drift (of 5.7 ppm). While interesting, there
doesn't appear to be any pattern that synced to every four hours start
at 00 UTC time. It's not a lot different than your plot from the 18x.

However, if I plot not the raw offsets, but the mean and variance over
an interval, we see there is a clear four hour pattern, and it's synced!

https://goo.gl/photos/JZhBbFKFzkBAykti6

Why would a GPS module produce jitter with a pattern like this?
Post by Kiwi Geoff
If you have an FPGA available then you could significantly improve
system time keeping. Currently the PPS interrupts the CPU to
snapshot internal counter. Unpredictable interrupt latency lifts NTP
timekeeping to about 1 or 2 microseconds but is the counter snap
shooting could be moved out to FPGA hardware there would be no unknown
latency and you could get NTP to break a "magic" 1uS barrier. I've
My initial idea for the PPS hardware would be to start a counter in the
FPGA, there is a good 100 MHz clock for this, on the edge of the PPS
signal. The irq handler can use the value of the counter to subtract
out most of the software interrupt latency. Most, since the Linux PPS
framework wants to create the system clock component of the PPS
timestamp pair _after_ the hardware part is generated. There is some
delay and jitter in how long it takes the CPU to create the system
timestamp after I have created the latency compensated PPS timestamp.
Bob Camp
2017-03-21 22:18:46 UTC
Reply
Permalink
Raw Message
HI
Post by Trent Piepho
Thanks to all who responded. Yes, I know PPS is the way get a more
accurate timestamps. That is the plan, but it takes more time to write
FPGA programs. The surprise is not that there is considerable jitter on
the NMEA output, but rather why does the jitter have patterns in it?
It seems significant enough that someone who had used NMEA for time
would have noticed it before. Unless it is somehow unique to me, but
how would that be?
The simple answer is that we *do* see it and that’s why we use PPS and not NMEA.
Arrival times on NMEA sentences varying over a 100 ms region is not at
all unusual. Getting them out “exactly on time” is not a priority compared to the
other tasks overworked CPU in the module needs to perform.

Bob
Post by Trent Piepho
Post by Kiwi Geoff
For example, here is a (24 hour) graph from my Garmin 18x (firmware
v3.6) where a plot (thanks to Hal) shows the start time of the NMEA
sentence from the time of the GPS 1PPS edge.
I've tried to duplicate that graph to some extent, plotting NMEA
sentence offset from system clock. Attached and also at
https://goo.gl/photos/TBEg27SRqY2EQW3HA
In this plot I've taken the offset from the system clock, fitted it to a
simple f(x)=a+b*x model, then plotted the residuals. I.e., I've taken
out a constant clock drift (of 5.7 ppm). While interesting, there
doesn't appear to be any pattern that synced to every four hours start
at 00 UTC time. It's not a lot different than your plot from the 18x.
However, if I plot not the raw offsets, but the mean and variance over
an interval, we see there is a clear four hour pattern, and it's synced!
https://goo.gl/photos/JZhBbFKFzkBAykti6
Why would a GPS module produce jitter with a pattern like this?
Post by Kiwi Geoff
If you have an FPGA available then you could significantly improve
system time keeping. Currently the PPS interrupts the CPU to
snapshot internal counter. Unpredictable interrupt latency lifts NTP
timekeeping to about 1 or 2 microseconds but is the counter snap
shooting could be moved out to FPGA hardware there would be no unknown
latency and you could get NTP to break a "magic" 1uS barrier. I've
My initial idea for the PPS hardware would be to start a counter in the
FPGA, there is a good 100 MHz clock for this, on the edge of the PPS
signal. The irq handler can use the value of the counter to subtract
out most of the software interrupt latency. Most, since the Linux PPS
framework wants to create the system clock component of the PPS
timestamp pair _after_ the hardware part is generated. There is some
delay and jitter in how long it takes the CPU to create the system
timestamp after I have created the latency compensated PPS timestamp.
<gpsoffset.png><jitter3.png>_______________________________________________
To unsubscribe, go to https://www.febo.com/cgi-bin/mailman/listinfo/time-nuts
and follow the instructions there.
_______________________________________________
time-nuts mailing list -- time-***@febo.com
To unsubscribe, go to https://www.febo.com/cgi-bin/mailman/listinfo/time-nuts
and follow the instructions there.
Kiwi Geoff
2017-03-21 23:24:16 UTC
Reply
Permalink
Raw Message
Post by Trent Piepho
https://goo.gl/photos/JZhBbFKFzkBAykti6
Why would a GPS module produce jitter with a pattern like this?
Trent, I must admit I have not seen such a four hour "spike" before in
the NMEA latency, however a clue may be that the GPS ephemeris
(broadcast by the SV's) orbit description is valid only for a "4 Hour
Period" - before the SUN and Moon and other factors make it too
imprecise.

I did a Google search for

telit ephemeris gps "4 hour"

and found that the TELIT appears to have some algorithm that enhances
somehow the ephemeris. Maybe your 4 hour spike (which appears to be
for 4 seconds each time) is evidence of your TELIT doing some extra
homework involving the ephemeris - just a guess on my part.

Regards, Geoff (Christchurch , New Zealand)
_______________________________________________
time-nuts mailing list -- time-***@febo.com
To unsubscribe, go to https://www.febo.com/cgi-bin/mailman/listinfo/time-nuts
and follow the instructions there.
Kiwi Geoff
2017-03-22 19:28:39 UTC
Reply
Permalink
Raw Message
Post by Trent Piepho
https://goo.gl/photos/JZhBbFKFzkBAykti6
Why would a GPS module produce jitter with a pattern like this?
Trent, I decided to R.T.F.M. (read the fantastic manual ;-)

It looks like that in your Telit module, there is a mode called

-----------------------------------------------
Client Generated Extended Ephemeris (CGEE)

CGEE data is always generated for a prediction interval of three days:
- Consists of 18 blocks of 4-hour EE data blocks
- Updated when a newly visible satellite is acquired
- Updated when new broadcast ephemeris is received from a tracked
satellite and the current
EE data block is nearing expiration
- On average, it takes 1.2 seconds per satellite for the receiver to
calculate CGEE
-----------------------------------------------

So I think that is the cause of your 4 Hour NMEA latency issues, when
your Telit module is creating a new EE (Extended Ephemeris) data set.

From a quick glance at the manual, it looks like this mode can be
turned off with:

AT$GPSIFIX=0

That "should" remove your 4 hour NMEA latency issues (fingers crossed).

Regards, Geoff ( Christchurch , New Zealand )
_______________________________________________
time-nuts mailing list -- time-***@febo.com
To unsubscribe, go to https://www.febo.com/cgi-bin/mailman/listinfo/time-nuts
and follow the instructions there.
Trent Piepho
2017-03-24 21:08:33 UTC
Reply
Permalink
Raw Message
Post by Kiwi Geoff
Post by Trent Piepho
https://goo.gl/photos/JZhBbFKFzkBAykti6
Why would a GPS module produce jitter with a pattern like this?
Trent, I decided to R.T.F.M. (read the fantastic manual ;-)
It looks like that in your Telit module, there is a mode called
-----------------------------------------------
Client Generated Extended Ephemeris (CGEE)
That sounds very promising! Your FM is better than my FM, the "JN3
Hardware User Guide" revision 2, which has nothing in it about extended
ephemeris mode.
Post by Kiwi Geoff
- Consists of 18 blocks of 4-hour EE data blocks
- Updated when a newly visible satellite is acquired
I'm indoors with a poor view of the sky, so I'm be surprised if new
satellites don't come into view throughout the day.
Post by Kiwi Geoff
- Updated when new broadcast ephemeris is received from a tracked
satellite and the current EE data block is nearing expiration
That explains the peak every four hours, assuming the EE data blocks are
synced to start at UTC (n*4):00 and not device power on.
Post by Kiwi Geoff
- On average, it takes 1.2 seconds per satellite for the receiver to
calculate CGEE
That would easily explain the few outliers at the 4 hour mark. But
std.dev. sure appears to increase for about 2 hours up to the 4 hour
mark, then rests back to a lower value for about 2 hours before
increasing again. See attached graphs. It's even more apparent when I
overlay each hour period on the same X axis.

That doesn't seem to fit the description of CGEE mode, but then again we
don't really know the algorithms used in the GPS receiver. It's
possible that a calculation involving a 4 hour EE data table takes
longer, and thus produces more jitter in competing threads like NMEA
output, when the computation is using the end of the interval vs the
beginning. E.g., a linear search for a line in the table to use.
Post by Kiwi Geoff
Post by Trent Piepho
From a quick glance at the manual, it looks like this mode can be
AT$GPSIFIX=0
Unfortunately, the GPS's design as part of the IRU means I can't write
to it :(. I'd really like to try that and see what happens.

Obviously the proper fix is to get PPS working and I've known that all
along. I'm really more interested in the WHY of the four hour cycle. I
think turning off CGEE will be the only way to know if that's the sole
explanation, or if there is more at play.
Trent Piepho
2017-08-12 00:24:55 UTC
Reply
Permalink
Raw Message
Obviously the proper fix is to get PPS working and I've known that all along.
Trent, you might want to test the 1PPS on the JN3 given that:

From the Telit JN3 Product Description_r5


-------------------------------------------------------------


4.3. Time Mark Pulse (1PPS)


A 1PPS time mark pulse is provided as an output with a width of 200ms.
This signal has not
been verified or characterized for all operational conditions.


I finally got around to using the PPS output of the Telit JN3. It is indeed a great deal better than the NMEA output.

How much better? I'm limited in that I don't have a precision time signal to compare against. But comparing the PPS phase offset against the system clock at least gives the ability to compare NMEA vs PPS to whatever level of precision the frequency stability of the system clock over a 1 second tau provides.

I can say that if you are using a Linux Kernel PPS driver such as the GPIO PPS or UART PPS, then the jitter introduced on the Linux side is far greater than jitter in the PPS itself from the Telit JN3. At least on my processor. AFAIK, the kernel PPS system is the "best" time source one can use in Linux other than perhaps PTP/IEE1588. So if you're trying to get good time in a Linux system, focusing on the GPS module itself isn't the place to expend effort. Reduce the jitter on the Linux side in how you measure that PPS signal.

I designed the system to trigger logic in an on-SoC FPGA that starts a 50 MHz counter on the PPS edge, which can be read by the Linux kernel driver to subtract out the software interrupt latency. This ends up being a huge win. Standard deviation of the PPS phase offset vs the disciplined system clock is 8.441µs if one were to use just the interrupt timestamps. With the FPGA counter canceling the IRQ latency, this drops to 126 ns.

See density plot of the uncorrected vs irq latency corrected phase offsets of the PPS. Notice the X axis log scale. google link<https://photos.google.com/share/AF1QipPYw4jiJU6V7JOSTFqyGqEZjaDBFAQvbTDpJSyuqtcgwyUu3oDZT6ML72EF8PJRNQ?key=dVhQMlF3RWgzblVRVUpWOW5ORjYzbFZRS0lKbF9R>

[cid:***@kymetacorp.com]

And for what it's worth, the terrible performance of the NMEA output is not present in the PPS. Notice the different scale for PPS offset vs NMEA offset.
[cid:***@kymetacorp.com]

Mark Sims
2017-03-21 01:54:10 UTC
Reply
Permalink
Raw Message
I did a lot of work in Lady Heather to add timing message arrival time analysis capability. Heather has a "set the system clock to receiver time" function that is intended to be used on systems without access to something like NTP. By knowing the arrival time of the last byte of the timing message, Heather can set the system clock more accurately. You can specify the message arrival time offset of the message time code or Heather uses a pre-computed typical value for the receiver.

Heather has a mode that can analyze and calculate the message timing offset for any receiver / cpu combination. During the analysis it plots the message arrival time offset in milliseconds, the jitter in the message arrival time, and ADEV/HDEV/MDEV and TDEV of those values. When you exit message analysis mode it analyzes a histogram of the arrival time data to calculate an "optimum" arrival time offset to use for the system + receiver.

Most timing receivers that can operate in binary mode have rather consistent arrival times. Some NMEA receivers can be quite good and others are rather problematic. Some receivers have rather consistent time for long periods and then jump to a new "stable" value. Also, doing other things on the system during the analysis can cause spikes in the data.

Attached is a table of the arrival time offsets and standard deviations of several receivers.
Loading...