Time-stamp correction of magnetic observatory data acquired during unavailability of time-synchronization

. During magnetic observatory data acquisition, the data time stamp is kept synchronized with a precise source of time. This is usually done using a GPS-controlled pulse per second (PPS) signal. For some observatories located in remote areas or where internet restrictions are enforced, only the magnetometer data are transmitted, limiting the capabil-ities of monitoring the acquisition operations. The magnetic observatory in Lanzhou (LZH), China, experienced an unnoticed interruption of the GPS PPS starting 7 March 2013. The data logger clock drifted slowly in time: in 6 months a lag of 27 s was accumulated. After a reboot on 2 April 2014 the drift became faster, − 2 s per day, before the GPS PPS could be restored on 8 July 2014. To estimate the time lags that LZH time series had accumulated, we compared it with data from other observatories located in East Asia. A synchronization algorithm was developed. Natural sources providing synchronous events could be used as markers to obtain the time lag between the observatories. The analysis of slices of 1 h of 1 s data at arbitrary UTC allowed estimating time lags with an

Abstract. During magnetic observatory data acquisition, the data time stamp is kept synchronized with a precise source of time. This is usually done using a GPS-controlled pulse per second (PPS) signal. For some observatories located in remote areas or where internet restrictions are enforced, only the magnetometer data are transmitted, limiting the capabilities of monitoring the acquisition operations. The magnetic observatory in Lanzhou (LZH), China, experienced an unnoticed interruption of the GPS PPS starting 7 March 2013. The data logger clock drifted slowly in time: in 6 months a lag of 27 s was accumulated. After a reboot on 2 April 2014 the drift became faster, −2 s per day, before the GPS PPS could be restored on 8 July 2014. To estimate the time lags that LZH time series had accumulated, we compared it with data from other observatories located in East Asia. A synchronization algorithm was developed. Natural sources providing synchronous events could be used as markers to obtain the time lag between the observatories. The analysis of slices of 1 h of 1 s data at arbitrary UTC allowed estimating time lags with an uncertainty of ∼ 11 s, revealing the correct trends of LZH time drift. A precise estimation of the time lag was obtained by comparing data from co-located instruments controlled by an independent PPS. In this case, it was possible to take advantage of spikes and local noise that constituted precise time markers. It was therefore possible to determine a correction to apply to LZH time stamps to correct the data files and produce reliable 1 min averaged definitive magnetic data.

Introduction
The Lanzhou Geomagnetic Observatory provides continuous observation of the Earth magnetic field. It is one of the oldest magnetic observatories of China (Yang, 2007) established during the International Geophysical Year initiatives in 1959. It was modernized in 1998, when a collaboration started between the China Earthquake Administration and the Institut de Physique du Globe de Paris (IPGP) (France), which provided new equipment and ensures data processing. Since 2002, this observatory is a part of the International Real-time Magnetic Observatory Network (INTER-MAGNET) (Love and Chulliat, 2013). Its International Association of Geomagnetism and Aeronomy (IAGA) code is LZH and it provides definitive data of 1 min averages of each magnetic component. From 2009 it has also produced 1 s averaged data. The Lanzhou observatory hosts also additional acquisition systems where other magnetometers are usually run in parallel to the main instruments. Absolute measurements are performed by the local staff of the observatory twice per week (Changjiang and Zhang, 2011), while subsequent data processing and production of quasi-definitive (Peltier and Chulliat, 2010) and definitive (INTERMAGNET Operations Committee and Executive Council, 2012) data is done in France. Due to local regulations, the data are transmitted from the observatory with a delay of 1 day and operations on the acquisition system are possible only on-site.
The magnetic instruments include a VM391 three-axis and homocentric fluxgate magnetometer providing 1 s vector data (Chulliat et al., 2009) and a GSM90 scalar magnetometer providing 5 s data. Both are controlled by a data logger running on an Acrosser AR-ES0631 fanless embedded system, using specifically designed software.
Published by Copernicus Publications on behalf of the European Geosciences Union. The acquisition system used for recording LZH data from the VM391 and GSM90 magnetometers includes a GPS receiver that provides a pulse per second (PPS) signal for precise time stamping of the acquired data. Like all recent computers, the data logger is equipped with a material clock: it includes a 64-bit counter that starts when the system is switched on and computes incremental values C i . Its frequency of increment depends on a quartz oscillator that has a nominal frequency of F counter = 1.19318 MHz. A virtual clock is also created to provide the UTC time t now when needed. When a time stamp needs to be generated, the value of the time is obtained from where t sync is the UTC time provided by the GPS at the emission of its PPS when the data logger performs its synchronization. At t sync the data logger counter recorded the value C sync , and records a value C now at the current epoch.

GPS synchronization and correction of oscillator frequency drift
A GPS antenna is installed on the roof of the observatory, connected to a GPS receiver that provides a PPS signal to the data logger via a RS232 link. The width of the PPS signal can be configured between a few microseconds and a few milliseconds. After every PPS emission, the GPS receiver provides also the complete date in UTC hours through the same link. This time stamp pertains to its previous PPS and thus it corresponds to an integer number of seconds. This is also the desired time for obtaining magnetometer readings. Since the frequency of the quartz oscillator depends on its temperature, it is necessary to keep track of the drift of the computed time t now in order to keep the time stamp of the data logger within an acceptable error (INTERMAGNET Operations Committee and Executive Council, 2012). In order to do that, the data logger regularly acquires a new value t sync provided by the GPS PPS and compares it with t now computed using Eq. (1): This error t is then used to correct the frequency F counter used to compute t now to maintain t = 0. The value C sync is also updated to the counter value at the time of synchronization. This process is performed three times per hour when the PPS signal is available, at minutes 15, 30 and 45, all at 0 s. In the case of a failure of the PPS signal, the data logger uses the last values of F counter and C sync that were obtained at the last t sync . A message is issued in the observatory log file to indicate failure of the synchronization. These values are kept in the memory of the data logger but are lost when a reboot of the system becomes necessary.

Verification of time synchronization between different instruments
When we noticed that time synchronization using GPS PPS was unavailable for LZH data, we first decided to use data readily available at IPGP or on INTERMAGNET to see if we could get a reasonable estimate of the time-stamp error of the recorded data. We first selected observatories on the same longitudinal sector as Lanzhou: the nearest observatory available is the one at Phu-Thuy (PHU) in Vietnam at nearly 1700 km distance. We decided to use a few observatories to inter-compare their time series, selecting other observatories within 3500 km distance from Lanzhou. We selected also Da Lat (DLT) observatory in Vietnam, Cheongyang (CYG) observatory in Korea and Kakioka (KAK) observatory in Japan. The details of positions and distances from Lanzhou are shown in Table 1. The farthest observatory, KAK, has a longitude distance of 36 • , which corresponds to more than 2 h delay in the occurrence of the magnetic field diurnal variation, but it is the only one providing a complete time series during the whole period of analysis. For the synchronization process we used variational data, i.e. data that were not manually processed to remove spikes and artefacts. In particular, at Lanzhou observatory, quite frequent magnetic perturbations are observed of various durations, from a few seconds up to a couple of minutes. The longest are due to nearby road traffic and to geophysical experiments running on the same site. Whenever magnetic pulsations were recorded simultaneously at the various observatories, these signals were used to evaluate the time lag between the various time series. The lag is defined as lag = t LZH − t r , where t LZH is the time stamp of LZH data and t r is the time stamp in any observatory used as reference. Figure 1 shows an example of magnetogram recorded on 6 July 2014, just before the GPS receiver was reestablished for LZH data logger. To compare the data from distant locations, each magnetic component time series was first standardized over 1 day. The top panel of Fig. 1 shows that the diurnal variation exhibits different trends at each observatory, since the solar quiet (Sq) current characteristics depend on the magnetic latitude of the observatories. On that day, around 11:00 UTC, a fast increase in the X component appears simultaneously in all observatories, lasting about 20 min. This synchronous event is seen earlier in the LZH time series, indicating that the data logger clock was running faster than at the other sites. This kind of ramp is however not usable to estimate a time lag with a needed precision of the order of 1 s: it is too long and it has a total duration that is not exactly the same at all sites. This event was followed by magnetic pulsations producing numerous oscillations with periods of a few minutes, clearly seen at all sites.
The bottom panel of Fig. 1 shows the data recorded between 11:30 and 12:30 UTC. These time series have been processed to allow further analysis: first each time series in this window has been detrended and standardized. Then, a  (Thebault et al., 2015), geographical coordinates and distance between Lanzhou and the other observatories.
Observatory Geomagnetic Geomagnetic Geographic Geographic Distance latitude longitude latitude longitude  polynomial of order 4 has been fitted to the standardized data and removed. Finally, a Tukey window has been applied to force the edges of the time series to be close to 0. In this figure, a very similar pattern of wave activity is seen at all observatories and a good synchronization is obtained in all distant sites. It can be clearly observed that the LZH time series was incorrectly labelled and preceding the others by nearly 3 min.
To obtain the estimation of the time lag, the filtered time series of each measured component at all observatories have been cross-correlated in pairs. An example for that same day, during local night, is shown in Fig. 2. The cross-correlation with the second acquisition system available at Lanzhou is also shown in this figure. The cross-correlation curves show that it is easier to estimate the lags between time series using the X and Z components, because they present quite sharp peaks. Observatories with reliable time stamp show a crosscorrelation lag near to 0 s, but it can sometimes exceed 10 s, if the cross-correlation peak is wide. Data from all observatories correlated with LZH time series agree to estimate the lag between 162 and 196 s.
From this first analysis, it appeared clearly possible to use distant observatories for verifying the data synchronization, but the precision is not sufficient for the purpose of correcting the data time stamps. We computed estimates of the time lag for each day between January 2013 and July 2014, obtaining coherent trends for all pairs of observatories. Figure 3 shows the time lags obtained using Kakioka observatory data for each measured component. The time lag estimation is very noisy on Z and Y components and the lower sampling of F at Lanzhou produces a curve that is more spread than for the X component. Nevertheless, the variation of the time lag during the year is evident. Changing UTC of analysis or the length of the cross-correlation time window strongly affects the spreading of the resulting lags. The hours around noon are the ones where the lags are obtained with lower noise on the X component. During the period between 1 January and 7 March 2013, when the GPS synchronization of LZH observatory was still operational, the cross-correlation of 1 h X values around 05:30 UTC resulted in an average time lag of −3 s with a standard deviation of 11 s. A full set of figures and tables of statistical values for lags computed at each UTC is provided in the Supplement. The analysis of the evolution of the lags through the whole period reveals that two different trends are observed: a slow variation during 2013 and up to April 2014 and fast variations from April to July 2014, after the data logger was rebooted and the correction for the oscillator frequency was lost. It was therefore decided to use the data of the second acquisition system available in Lanzhou inside the same building for computing the time lags suitable for correcting the time stamps. Two different sensors were deployed on the second acquisition system, one in 2013 and another in 2014, making it possible to generate a complete dataset for comparison. The resulting curves, shown in Fig. 4, present coherent lag values for all the three components X, Y , Z, and the F component is more noisy due to the lower sampling rate of the scalar magnetometer. In particular, the Z component exhibits a very small dispersion of data and just a few outliers: it is the component where the amplitude of the spikes occurring in this observatory is larger. These short spikes, lasting 3 to 5 s, are local phenomena that can be used to precisely estimate the lag between the two time series since they produce a very narrow peak in the cross-correlation curve (see Fig. 2).

Correction of data time stamp
After computing all the time lags, it was decided that only the period up to 2 April 2014 was suitable for time-stamp correction, since the clock drift was very slow during that time. Data for the period between 2 April and 8 July 2014, when the time drift was of 2 s per day, will not be published as definitive data. A new set of corrected LZH 1 s data files was then generated using the computed time lags based on the second instrument available in Lanzhou observatory. A single daily correction value was used since the lags were nearly constant during 1 day. This correction value was calculated for 12:00 UT, following a smooth hyperbolic tangent function fitted to the calculated delays. This choice was possible since the drift of the clock was always The corrected 1 s data files were averaged to compute 1 min data files following the INTERMAGNET recommendations for data filtering (INTERMAGNET Operations Committee and Executive Council, 2012). Lastly, the baseline processing allowed to generate corrected 1 min definitive data for LZH observatory.

Discussion
The time synchronization of LZH VM391 and GSM90 instruments was lost on 7 March 2013, but it was noticed only 1 year later since the drift of the data logger clock was very slow. At that time, the acquired data had accumulated a lag of 28 s compared to UTC time (Fig. 4). This drift is relatively low, thanks to the correction of the oscillator frequency that is done by the data logger. During the first 2 months, only 2 s of lag were accumulated. In the period between June and September the lag increased at a faster rate and reached 24 s. Afterwards the lag increased more slowly, to reach a stable level of 27 s in December 2013. It remained at this level up to mid-March 2014, when a slow increase started again. 28 s lag was reached just before the data logger reboot on 2 April 2014. The most significant part of the lag was accumulated during the summer months, when the temperature of the data logger was the highest (Fig. 5). The automatic monitoring of the LZH acquisition system include measurements of the temperature of some components, but not the oscillator temperature. The closest available temperature is the one of the energy card and we analysed its evolution to understand if it could reveal useful information to interpret the variation of the time lag. Though a correlation between the quartz frequency and the temperature of the data logger is expected, it might be non-linear. The temperature of the energy card follows a seasonal trend. It appears that the difference between this temperature and its value at the time when F counter was last estimated played an important role in the clock drift. Only when this temperature difference was exceeding 5 • C did the clock drift at a higher pace. When the temperature returned below this 5 • C difference, the clock nearly stopped drifting. At the reboot of the data logger on 2 April 2014, the clock correction parameters were erased from its memory and were no longer available. It was not possible for the data logger to compute a new correction, since the PPS was not in operation due to the GPS failure. The data logger clock started drifting with a constant rate of about 2 s per day. Additional reboots were done afterwards, and at every reboot the clock counter started at a different value producing the saw-tooth behaviour that can be observed in the lags shown in Fig. 4. To avoid having similar issues in the future, some options are possible: -Use a data logger with a temperature-compensated crystal oscillator (TCXO).
-Improve the software for clock correction to save the corrections to the quartz frequency so that they are available even after a reboot of the data logger and possibly also include temperature correction.
-Improve the monitoring tools of the observatory so that similar failures could be detected more easily. It should be pointed out that this particular situation occurred because, at that time, the log files were not routinely transmitted along with the data files.
LZH monitoring data are now regularly transmitted to IPGP and checked to prevent future occurrences of unnoticed time synchronization unavailability.

Conclusions
The GPS time synchronization of LZH magnetic observatory was lost on 7 March 2013. Over 1 year, the time-stamp attribute to the acquired data had accumulated a lag of 28 s compared to UTC time. This drift is low, thanks to the correction of the oscillator frequency that is done by the data logger. It is possible to confidently correct the time stamps of the 1 s acquisitions to produce 1 min definitive data, the official IN-TERMAGNET products. It has been proven by comparing the time series of one observatory at mid-latitude with the time series of observatories within 3500 km range that it is possible to detect the correct trend of time drift. This comparison is more effective during hours when there is low diurnal variation and is better identified on the magnetic X component. This is an effective way to verify the stability of clocks in un-manned acquisition systems that cannot be monitored in real time. To be able to construct a precise time correction function it is preferable to use another acquisition system located in the same premises, like in Lanzhou observatory. In this case, all spikes caused by local activities, that are usually removed from magnetic definitive data, provide short signals that facilitate obtaining a precise time correction. In the case of Lanzhou, the vertical component of the magnetic field is the one most affected by spikes and could be used to correct the synchronization of the time series. The time stamps for the whole period between March 2013 and 8 April 2014 were corrected and averaged 1 min values definitive data could be produced.
Data availability. Magnetic observatory data used for this paper are available at the Bureau Central de Magnetisme Terrestre (BCMT) website (www.bcmt.fr) or on BCMT -Magnetic databank https://doi.org/10.18715/BCMT.MAG.DEF and on INTER-MAGNET website (www.intermagnet.org). Definitive 1 min data of LZH observatory will also be included in INTERMAGNET annual DVDs, available upon request at intermagnet@ipgp.fr.
Author contributions. PC analysed the data, produced the corrected files and wrote the article; BH identified the problem and produced definitive data; KT and XL developed the LZH observatory acquisition system and interpreted the instrument response; VL validated the methodology; and CJX processed the data. PC prepared the paper with contributions from all co-authors.
Competing interests. The authors declare that they have no conflict of interest.
Special issue statement. This article is part of the special issue "The Earth's magnetic field: measurements, data, and applications from ground observations (ANGEO/GI inter-journal SI)". It is a result of the XVIIth IAGA Workshop on Geomagnetic Observatory Instruments, Data Acquisition and Processing, Dourbes, Belgium, 4-10 September 2016.