Noise in raw data from magnetic observatories

In spite of significant progress in the development of new devices for magnetic measurements, mathematical and computational technologies for data processing and means of communication, the quality of magnetic data accessible through the data centres (for example, World Data Centres or INTERMAGNET) still largely depends on the actual conditions in which observation of the Earth’s magnetic field is performed at observatories. Processing of raw data of magnetic measurements by observatory staff plays an important role. It includes effective identification of noise and elimination of its influence on final data. In this paper, on the basis of the experience gained during long-term magnetic monitoring carried out at the observatories of IKIR FEB RAS (Russia) and CSIR-NGRI (India), we present a review of methods commonly encountered in actual practice for noise identification and the possibility of reducing noise influence.

hundreds of meters from the pavilions. Russian observatories CPS, MGD, PET and KHB were built in the 60s according to the requirements for complex magnetic-ionospheric stations in the USSR, with a wide range of geophysical observations. In the result, there are other observation systems in the immediate vicinity of magnetic pavilions, which are potential sources of interference, for example, ionosondes for vertical sounding of the ionosphere. Moreover, due to the limited area, there are some facilities such as garages with heavy machinery, wells, electricity power equipment etc. near the observatories which may influence the magnetic measurements.   Extreme climate conditions at IKIR observatories are of great importance. For MGD and KHB observatories, sharply continental climate with seasonal changes of temperature from -40 to +30ºC is ordinary. CPS observatory is located in the High Arctic zone with very hard climate conditions. PET observatory is characterized by abundance of precipitation, the level of snow in winter is up to 2 m. These conditions require special and expensive means to provide the required temperature conditions in magnetic pavilions. Special machinery is necessary to clear the ways to the pavilions from snowthat may also affect the quality of measurements. We should mention such features as stability of external power supply that is difficult to provide at the observatories remote from densely populated areas, as well as ground connection quality problems due to the nature of ground (at Cape Schmidt and Magadan). For Indian observatories climate conditions are also a significant problem, high heat during the year and the rainy season with high humidity.

Figure 1. Location of magnetic observatories of IKIR FEB RAS (CPS, KHB, MGD and PET) and CSIR-NGRI (HYB and CPL). Isolines show horizontal component H (green lines) and declination D (red lines) according to IGRF12
model. Yellow triangles are the INTERMAGNET observatories, red stars are non-INTERMAGNET observatories.
3 Description and classification of noise in raw data

"Regular-random" noises
In order to develop a mechanism to deal with noises effectively, it is necessary to recognize their nature and to classify them as regular and random, frequent and rare, etc. Generally, regular noise is the result of technical problems with measuring equipment, or interference from other devices operating in the vicinity of a magnetometer. An indicative example of regular noise is interference in the magnetic data that arises when ionosondes are operating. This is quite a common problem at integrated remote observatories, where it is physically and organizationally impossible to distance magnetic and ionospheric measurements far enough. variations of the field were eliminated according to the data from other magnetometers installed in the same pavilion: Z component variation was removed using the data of the fluxgate device FRG-601 (Fig. 2b), F(scal) variation -was removed using the data from scalar magnetometer GSM-90, F(var) was calculated from the FGE variations and the corresponding baseline values (Fig. 2a). The authors have also observed similar interference in the raw data from Novosibirsk (NVS, fluxgate magnetometer LEMI-008) and Yakutsk (YAK, FGE magnetometer and Magdalog recorder) observatories. The distance to ionosondes did not exceed 200-300 m. In all cases, the interference was recorded by variation fluxgate magnetometers. At the same time, in the records of the other fluxgate devices located in immediate proximity, such interference from the ionosonde was not observed, for example, in the results of the fluxgate magnetometer FRG-601, the data of which were used to remove natural geomagnetic variations (it is illustrated by the difference between the data obtained by FGE and FRG magnetometers shown in Fig. 2b  synchronized with UTC. However, small shifts are observed, which are associated with a particular mode of sounding and the range of operating frequencies of the ionosonde.
Regularity and accurate synchronization of such interference with UTC makes it easier to identify them and to make a decision about the removal or correction. Certainly, this applies to relatively short noise signals, usually to pulses. A special software module was developed and integrated into the software package for raw magnetic data processing at MGD and PET observatories. The module generates a temporary mask, daily array of 0 and 1, where 0 corresponds to the measurements that include noises. Parameters of the "0-1" sequence are set in a special text file that contains the periodicity of sounding and the shifts of the beginning and the end of a removed interval with respect to the beginning of a sounding session for each day.
During the processing of the raw data, the mask is applied to the original daily time series and the data marked by "0" are filled with special values NaN (Not-a-Number). An example of such clearing is shown in Fig. 2c.
Interference between closely spaced magnetometers may be a source of regular noise. It is a well-known that proton magnetometers can cause interference either by the generation of additional external magnetic fields during polarization of the proton rich liquid, or via direct influence of the DC current powering the proton (or Overhauser effect) magnetometers, which is modulated by the periodicity of measurements. For example, the effect up to 0.2 nT is expected at distance of about 5 m from proton magnetometer (Auster et al., 2008). Vector magnetometers using proton sensors in the coil systems (dIdD GEM System;QMLab), besides the effects during polarization, also produce significant additional magnetic fields affecting the measurements of devices located in vicinity. Figure 3a and b shows manifestation of the dIdD magnetometer operation at Cape Schmidt observatory in the records of dH and dD variations from fluxgate magnetometer MAGDAS. Due to the polar specifics (necessity of heating and the absence of additional pavilions) the both devices are installed on one pillar at a distance of about 2 meters, almost on the same meridian. MAGDAS measurement frequency is 1 Hz. Oscillations in a range up to 2 nT with a period that is multiple of dIdD measurement periodicity (2.5 s) are observed.
There is also noise in the record of vertical component. The mechanism of influence of dIdD on MAGDAS measuring process is complex, it is associated with timer stabilites (noise amplitude "floats" over time) and practically it cannot be reliably corrected by a software during the postprocessing. Therefore, the only effective way to avoid this interference is to distance the magnetometers from each other. Figure 3c shows the second example of the proton magnetometer influence on another device at Paratunka observatory. The fluxgate variation magnetometer FGE signal (measuring frequency is 2 Hz) is modulated by the operation of GSM-90 Overhauser magnetometer with a measuring rate of 5 s, the range of noise in the vertical component Z is up to 1-2 nT (in other components it is less than 0.5 nT). Since the sensor GSM-90 is located at a distance of about 4 meters from the FGE sensor, the impact through additional magnetic fields during polarization is hardly probable. Presumably, the interaction takes place at the hardware or communication level, because the devices are connected by a single datalogger Magdalog. The 7 5 10 15 20 25 same noises are also observed on a similar set of magnetometers at Magadan observatory. We cannot solve this problem technically. However, the synchronicity of measurements by two instruments (on one datalogger with a single timer) has allowed us to implement a software clearing. From the fluxgate magnetometer data we select only those, which occurred during the frequency measurement of proton sensor precession, and fragments during polarization are removed.
Unfortunately, due to a little synchronicity instability of measurements by two devices, only a few samples from the 5second cycle of FGE can be reliably distinguished (for reliability actually only one sample is chosen, which is shown in Fig.   3c by "o"), i.e. forced tenfold data loss takes place. In minute values, calculated in accordance with INTERMAGNET standard, normally distributed noise can (in special cases) be removed by filtering. However, the fact that noise is often asymmetrical creates at least two problems on minute intervals: (a) when averaging the noise with asymmetric signals, the obtained mean values are also biased. For the noise shown in Fig.   3c with magnitude up to -1 nT, the average estimate will be systematically biased downward by about 0.5 nT, this value is quite significant in relation to current requirements for long-term magnetic measurements; (b) if during the absolute observations a reading at the zero position of DIflux magnetometer fluxgate sensor coincides with noise in the variometer, than a difference between absolute and variation measurement arises, i.e. accuracy of baseline values is decreased, and in the worst cases a systematic errors in final values of the total field vector arise.

Pulse noise
A generalized view of possible shapes of structured noise is presented, for example, in Lopez-de-Lacalle (2016, Figure 1).
Pulses (spikes) are perhaps one of the most common types of signals in magnetic records, which with a sufficient probability are not related with natural processes. Spike is interpreted as relatively short signal with a significant amplitude (duration is less than a few seconds or several measurements, if magnetometer has a measurement frequency from ones to tenths of Hz), with well-defined sharp leading and back edges similar in amplitude. If one of the edges is weak or is absent, then we can talk about the jump. All these properties of pulses can be used for their detection and removal. We can also note that pulses with duration of one measurement cycle are the most likely to be associated with hardware problems or interferences from nearby sources. The amplitude is also an important characteristic. Pulses with amplitude in units, tens and more of nanoteslas also have a low probability of being caused by natural sources. Figure 4a shows an example of a daily record of the field total intensity F recorded at Cape Schmidt observatory by Overhauser magnetometer POS-1, the measurement periodicity is 3 s. For CPS observatory, problems with the stability of the power supply and the quality of grounding are known, to which POS-1 is sensitive enough. On the record, outliers with the amplitude up to 1500 nT can be seen (over 90 events). The duration of these pulses does not exceed one measurement, i.e. they have sharp edges with almost equal magnitude of leading and back fronts. Therefore, it is not difficult to identify and to locate them. The simplest algorithm of allocation of such signals by amplitude, based on the proximity of the pulse edges and its isolation, works quite effectively. The result of the program implementation of the algorithm is shown in Fig. 4b. It should be noted that in the Overhauser magnetometers POS-1, which are quite widely distributed at magnetic observatories, each record is accompanied by the estimation of measurement quality using a special parameter QMC (Quality Measurement Criterion), which value is related to the quality of the precession signal by which frequency the F is calculated, as well as qualitative estimates of measurement conditions such as signal-to-noise ratio, the duration of the It is clear, that noise in the record shown in Fig. 4a does not represent a problem for program processing, and in case of smaller quantity they can be processed manually. However, difficulties arise in the case of more irregular shape of noiseoutliers, when they can not be considered as narrow isolated pulses. The examples are the noises described above, arising during ionosonde operation ( Fig. 2a and b), which are often extended in time and can have multimode structure. Efficiency of their processing is provided by the strict repeatability. Figure 5 shows the daily record of dH, dD, dZ variations, obtained at Hyderabad observatory using the fluxgate magnetometer FGE, measurement frequency is 2 Hz. As it can be clearly seen, there is irregular and frequent noise in the vertical component in the form of outliers (spikes) with amplitude of more than 5 nT. The observatory is located on the territory of CSIR-NGRI Institute, within the city. The most probable reason of this noise is the metro line (above-ground) passing at a distance of 200-300 m to the south from the observatory. Figure 5c shows a 1.5-minute fragment for dZ, which shows that outliers have sufficiently definite and stable structure, a sharp leading edge and falling exponentially back edge, the total pulse duration is up to 3-6 samples, i.e. about 1-3 s. The algorithm applied at HYB observatory to detect and remove the noise caused by metro line is based on its structural stability and works as follows: when a change of dZ between neighboring measurements (in absolute value) exceeding a given threshold is found, three subsequent measurements are discarded. It is clear that this algorithm creates risks: (a) natural signals with sharp edges, for example, during magnetic disturbances, may be discarded; (b) noise may be removed not completely, if its duration is longer than given one; (c) noise can be missed, if due to a small shift its leading edge takes 2 samples. However, to calculate the minute values of the magnetic field variations, that may be enough. Figure 5d shows the results of the algorithm application during real processing of raw HYB data (threshold value dZ/dt = 1 nT/0.5 s was used).
An algorithm based on continuous wavelet transform showed higher efficiency in detection of such spikes. For the first time it was proposed in the paper (Zhizhikina et al., 2016). This algorithm includes the following main operations: (1)  The algorithm efficiency is determined by the wavelet transform sensitivity to sharp changes of function values, amplitudes of wavelet coefficients significantly increase in the areas containing local features in the form of sharp peaks (Daubechies, 1992). The geomagnetic variations and noise are dependent from location of the observatory. Therefore, preliminary tuning of the algorithm parameters is required for the selected observatory. Currently, the algorithm is adapted for middle latitude Paratunka observatory and for equatorial Hyderabad observatory. The values of the parameters were defined for criterion of the absence of false detections, using selected data. The effectiveness of the algorithm was estimated for quiet and disturbed magnetic field, the spikes detected by the experienced magnetologist were considered as reference. Table 2 shows the results of the estimation of the algorithm effectiveness for Hyderabad observatory. The results show high reliability of the wavelet-based algorithm to detect the main part of spikes: about 99% pulses with high amplitude (> 3 nT) are isolated. But pulses with small amplitude (<0.5 nT), which are 25% from all pulses, can not be reliably detected by magnetologists. This restricts the optimization of the algorithm.

Jumps
In a certain sense, the jumps in the results of measurements can be considered as pulse noise described above, but with a continuous interval of record between the leading and the back edges or in the absence of back edge. In case of sharp edges and sufficient amplitude of a jump, it is not difficult to identify it. However, unlike spikes, in practice such jumps are quite rare in raw magnetic data (for example, Fig. 2). Changes of magnetic record level with gentle edges or with edges that contain other noises with considerable amplitude are predominantly observed. The reasons of such jumps are technical operations with equipment, change of the magnetic state in the pavilion with the magnetometer or near it, change of instrument parameters, etc.
An example of noise in the form of jumps caused by the magnetic environment changes near the pavilions at Paratunka observatory is shown in Fig. 6. The effect is visible in the field total intensity F, which occurred during removal and reinstallation of casing pipes in a well of 80 m in depth, located approximately at a distance of 100 m to the south from the magnetic pavilions. These operations were carried out within six hours, heavy machinery was used (truck crane, bulldozer).
In general the effect does not exceed 1 nT, but it stands out well in the difference between the records of the two scalar magnetometers, located at different distances from the well (dIdD is the closest, POS-1 is the most remote, the distance between them is 30 m). Each removal of a pipe section from the well causes a jump of the magnetic field gradient between (b) such signals can be identified reliably only in difference data obtained by spaced magnetometers; (c) practically, the only way to detect such signals is by an experienced, trained magnetologist, largely based on the additional information about measurement conditions. In relation to the example given in Fig. 6, we can assert that only a magnetologist ( jumps in the variations and knowing that works on the well were carried out at that time. Similar anthropogenic disturbances are practically not corrected and in most cases the record including noises is just removed. Figure 7 shows an example of jumps in the daily record of the field total intensity F, obtained by dIdD GSM-19FD and POS-1 magnetometers at Cape Schmidt observatory. It can be seen in Fig. 7c, that the measurement difference of these two Overhauser magnetometers contains jumps with an amplitude of several nT. However, noises arise only in the dIdD record  In this example, our interest is not in the cause or mechanisms of noise, but in the possibility of its identification and correction. Since the amplitude is fairly significant, edges are sharp, and measurements are not burdened by this noise, the posed problem can probably be effectively solved by software tools. However, in this case, processing is performed by a magnetologist, who estimates the size and the location of the jump using F(dIdI)-F(POS) plot, followed by program correction of F(dIdD) and, if necessary, removal of unreliable data at the time of the jump, if it had a significant duration. It should be noted that the jumps, after which the record level is changed and retained for a long time (several days or longer, for example, after magnetometer reinstallation), are recorded in the baseline values of variational instruments and they are eliminated during calculation of the total field vector using the standard measurement technology at magnetic observatories (see also some remarks in Sect. 4). Of course, such a method works with rather frequent absolute observations and usually additionally requires compensation of the jump in the part of the daily data due to the fact that according to the standards, the baselines are taken as constants within a day. In any case, the undisturbed magnetic records are needed as references.

Bay-like noise
Bays are a common type of noise at magnetic observatories. They are often the result of changes in the magnetic field near the magnetometer due to moving objects with magnetic effect, for example, a car, a person with instruments, etc. In case of such noise, shapes of signals in the field components are specific and connected among themselves. Nevertheless, this noise, if its amplitude does not reach extreme values, is hardly distinguishable from natural variations. The possible ways of identification are comparison with the data obtained by other magnetometers (gradiometer principle) and analysis of the information about events at an observatory (logging of such events is the direct responsibility of the observatory and its staff). duration is about 30-40 s, the amplitude is up to 20-30 nT, their time shift is clearly defined. We may also note the dependence mentioned above between the form of a signal in different field components. The identification of a signal as the noise was made by a magnetologist by the shape of a signal, time shift and by comparing it with similar signals that have been earlier observed (in this case there was no information about the source of the noise at the observatory). After localization, the noise was removed by a magnetologist during raw data processing.
One more example is shown in Fig. 9. A bulldozer cleared the way to magnetic pavilions from snow at Khabarovsk observatory. Field variations were recored by the quartz magnetometer CAIS. Noise duration is about one minute, amplitude is maximum in the vertical component (up to 10 nT). It would be difficult for the staff of the observatory to identify the signal in Fig. 9 as noise without the information about the works conducted near the pavilions, because according the signal parameters, it is rather close to natural geomagnetic variations (an exception is possible only for Z-component, since its natural variations on this day did not exceed several of nT). Just like in the previous case, the noise was removed by a magnetologist during raw data processing.

Lower panel: dZ (MAGDAS, upper curve), dF (POS-1, middle curve with dots) and dF (dIdD, lower curve).
MAGDAS and dIdD magnetometers are located at a distance of about 30 m from the POS-1.

Random-like noise
Man-made disturbances, which are expressed as additional random noise at the background of original useful signal will be understood as random noise. This notion generalizes a very big class of noises which are often difficult to classify. In many cases these noise is not localized in frequency and/or time domain. They may have a hardware origin or may be connected with real field noisiness from external sources at the observatory. Almost always the problems of these noises are solved either technically (fine adjustement of magnetometers, improvement of grounding, power supply, etc.), or organizationally (moving of a measurement point, replacement of a magnetometer by another one which is less sensitive to noise, etc.). It is hardly probable to find effective methodological and program approaches.
Identification of hardware noise in data is possible if we compare the results of measurements by different magnetometers or if we change the operating modes of a device if it is the only one at the observatory and if it is possible by the specification of a devices. The increased noise of the magnetic field at the observatory (as a result of total impact of many factors) can be identified, for example, if we make measurements by the same magnetometer at the observatory and in the place with obviously low noise. In general, assessment of the background noise is a labor-consuming task including a research aspect and very often it has not effective practical results. The problem becomes more complicated due to the fact that in many cases it is almost impossible to distinguish the noise background from a natural signal which is a subject of scientific research, for example, seismomagnetic effects. Fig. 10 and 11, as an example, show the random noise which arises in the results of magnetic measurements in case of problems with power supply. Figure 10 shows the results of daily measurements of the field total intensity F, made at Khabarovsk observatory using scalar Overhauser magnetometer POS-1. There was a general power outage at the observatory and in the nearby settlements at about 08:50 UT. The measurements were continued using an autonomous power supply system at the observatory. At about 12 UT the external power supply was restored. On the record obtained by POS-1 noise with the amplitude up to 1 nT was recorded. It lasted for more than a day. For a descriptive graphical representation, natural geomagnetic variations were excluded using the data from another scalar magnetometer , in the measurements of which the noise caused by power failure did not manifest (Fig. 10a). Fig. 10b shows the behavior of QMC parameter, which estimates the quality of At about 02:45UT a failure of a diesel generator occurred and emergency scheme of power supply with external battery package as a source of voltage and disconnection of all powerful devices has been activated, the standard power supply was restored at about 11:55UT. It is shown in Fig. 11a that pulses with amplitude up to 10-20 nT and random noise up to 5 nT appeared during the operation of the emergency power system in two measurement channels of dIdD (with additional fields in coil systems D and I, see, for example dIdD Instruction Manual (2010)). For illustration purposes, Fig. 11b shows the record after eliminating the low-frequency variations. Since the amplitude of the noise is significant, it can be easily identified during the analysis. In both examples given above, the only way to exclude noise is to remove all data fragments which contain noise. It is impossible to restore the original useful signal with the acceptable quality.

Critical and weak noises
This criterion is important due to the fact that it determines the extent of noise influence in magnetic data on the results of researches which are carried out using these data. When a researcher uses the final data from observatories published in some way, in most cases he has no information in what conditions the measurements were performed, what procedures were applied during preliminary processing (except for the cases when these procedures are prescribed by standards), etc. Thus, the responsibility for the quality of the data provided to scientific community is very high and it completely lies on an observatory.
Let us consider a "classic" case. An observatory of INTERMAGNET network obtains primary 1-second data of dHs, dDs, performed with these data. That makes it necessary to identify these critical noises and to remove them during the primary data processing. It entails secondary problems, such as gaps in original data, which are used to calculate published minute values. In spite of INTERMAGNET recommendation, that mean values should be calculated in accordance with the 90% availability rule, such criteria are difficult to define and sometimes they are not applicable at all.

Noise with known and unknown sources
If it is known that a signal, which is suspected as a noise, is the result of some reasons which are not associated with natural variations of magnetic field, then it is rather a powerful argument to remove the signal. At the same time, however, there are some fine points: (a) noise removal leads to the gaps in published data, therefore scientist in many cases before application of methods of analysis is forced to fill these gaps with dummy data, i.e. the data, calculated from available dataset using some interpolation method. Errors of calculation results arising due to the filling may be comparable or even greater than the errors that occur due to the noise which was not deleted; (b) problems which are solved using the final data should be defined. In some cases, signals identified by the criterion of origin as noise can be of independent scientific interest. For example, in practice of observatory measurements, a fictitious "seismomagnetic effect" is well-known, when we observe oscillations in the data of magnetometers with a suspended system for compensation of sensor inclinations or in the records of induction magnetometers. These oscillations arise when a seismic wave from a near or strong earthquake is passing the place where the magnetometer is installed. As an example, figure 12 shows the records of magnetic field variations at IKIR FEB RAS observatories . On these fragments the effect of a strong earthquake with the magnitude of 8.3 which occurred on 24 May 2013 in the Sea of Okhotsk at the depth of about 600 km is illustrated. The following magnetometers were applied: dIdD GSM-19FD (Dm and Im modules with additional fields of coil system) at the CPS and PET, fluxgate FGE (H channel) at MGD, digital magnetometer with Bobrov's quartz sensors (H channel) at KHB. It is clear that the earthquake is manifested well at long distances, the magnetometer data based on different measurement principles. For the tasks of studying the variability of the magnetic field, the signals shown in Fig. 12 are noise and they should be removed. However, if, for example, seismomagnetic effects are investigated, then the recorded "fictitious" signals in Fig. 12 would be a good bench mark to estimate the passage of a seismic wave in the area where magnetometers are installed. Unfortunately, it is necessary to note that in some cases, the researchers of seismic effects in the magnetic field even do not ask what type of a magnetometer is, the data from which they use. If a signal is suspected as a noise, but its structure does not allow us to recognize it as a noise, and there is no additional information about a possible source, then it is very complicated for a magnetologist to make a decision. In most cases, these signals are not removed, that creates risks of reducing the quality of observatory data.

Possible methods for noise removal from raw magnetic data
In the previous section we classified noises, which are encountered the most frequently in actual magnetic measurements at the observatories of IKIR FEB RAS and CSIR-NGR, and illustrated them by some characteristic examples. Naturally, the description and the samples are quite limited, since the variety of noises is extremely large.
It is clear that identification of noise is the solution only of a part of the problem. The second part is to choose an effective method of further work with this noise. Unfortunately, the choice of possibilities is very small. Mainly a fragment is simply removed from a record that results in a gap, which can be filled by the most suitable "dummy" data, or if the structure of the noise is recognized, it is removed and the original useful signal remains.
Earlier, in the description of pulses (see Fig. 2 manually by a magnetologist. There are different approaches to implement these procedures. If a file with raw data has text format (for example, POS-1 User manual (2004) and dIdD Instruction Manual (2010), it is possible to remove the unreliable records manually. In this case, during the further work with the corrected file, there will be gaps not only in the measured magnetic values, but also in a timestamp, that is not always convenient if we need a uniform time grid. Another possible approach is to create an intermediate file, usually in text format, in which noisy data are manually or semi-automatically replaced by the values that indicate unreliable measurements, for example, "99999". This option is particularly useful if the initial measurements are recorded in files in binary format, e.g., MAGDAS-A Installation Manual (2005). In this case special converter software is usually used, including those which allow an operator to encode the required data as unreliable.
In the software for magnetologists developed in IKIR FEB RAS, which is based on MATLAB and Octave mathematical packages (Khomutov, 2016), the third approach is used. It is based on the following methodical principles: (a) original files are always used in processing of raw ( Maximally simplified text format is used, that reduces the probability of errors during manual typing and speeds up the file reading. The example shows the information, used to remove the noise caused by a nearby earthquake on 30 January 2016 in the data from FGE magnetometer. In the processing of measurement results of 30 January 2016, D values from 03:25:37(=03.4269) to 03:34:46(=03.5795)UT, which are noisy due to the mechanical influence of a seismic wave on the suspended sensor, will be filled with a special symbol NaN and will be excluded from the further processing. The boundaries of intervals with noise are defined by a magnetologist in interactive mode by record plots for the corresponding components and with required scaling that is provided by convenient interface with graphics in MATLAB and Octave. Noise in other components is removed similarly. The effect of the described earthquake and the results of applying the clearing procedure are shown in Considerably a more difficult situation arises in the attempt to restore an original signal, i.e. removal only of noise from the measurement results. It is impossible in the most cases in actual working practice of the observatory due to the unknown structure of a useful signal and the unknown structure of noise. Analysis of a concrete situation is individual, and often grows almost to a scientific research. However, in some simple cases the problem can be solved. These situations include, for example, noise, leading to jumps of record level, sometimes followed by restoration also in the form of a jump. It is clear that in this case the noise is a constant addition to the useful signal, and after its subtraction, the initial undisturbed signal will be restored. The main problem is the correct estimation of the value of the noise (magnitude of a jump) and the necessity to be sure that noise has constant value in the analyzed interval of the record.
In practice, identification of a jump and estimation of its parameters (time and magnitude) is performed by a magnetologist, usually in interactive mode of work with magnetic record graphs. If the jump was very rapid and time of a transitional process is comparable with the duration of the interval between samples, and generally faster than possible natural variations in the magnetic field, then the jump parameters can be estimated visually with sufficient reliability. If transitional process is 23 5 10 15 20 Geosci. Instrum. Method. Data Syst. Discuss., doi:10.5194/gi-2017-10, 2017 Manuscript under review for journal Geosci. Instrum. Method. Data Syst. Discussion started: 10 May 2017 c Author(s) 2017. CC-BY 3.0 License. long enough, then field natural variations can significantly affect the accuracy of the estimation. In this case, it may be useful to compare the analyzed record with measurement results obtained by another magnetometer, because natural geomagnetic variation will be excluded from the difference between two records, and the jump will be manifested as two permanent levels of the difference. However, in this case, there are also a lot of limitations, for example, the nature of the cause of the jump. If the jump is caused by a change in the magnetic field, the source of this change is fairly close to the place of measurements, and the compared magnetometers are located closely (usually in the same or in neighboring pavilions), then noise effect will be manifested in the data of both devices and it will be difficult to make any reliable estimations. If the reason of the jump is of technical origin or its impact on the supporting magnetometer is small, it is possible to estimate the required parameters.
The situation described above is partly similar to that which occurs in case of elimination of long-term changes of the level of magnetic record from variation magnetometer, using absolute observations. Figure 14 shows an example of compensation of jumps that arose in the records of FGE magnetometer at Paratunka observatory after the earthquake on 30 January 2016 (see. Fig. 13). Regular absolute magnetic observations and dense series of obtained baseline values allow us to eliminate effectively the effect of such jumps in the series of full values of the magnetic field components, i.e., for example, the total values of the declination D = D0+dD since 31 January 2016 will be free of the jump effects in case of earthquake.

Conclusions
The review of noises in raw magnetic data and some methods of its identification and removal given above represent only a part of the real situation with which a magnetologist deal at an observatory when processing the measurement results.
However, even with such a volume it is possible to draw the following important conclusions: 1) in most cases, the correct identification of noise can be performed only according to the estimates made by an expert, a magnetologist of the observatory, who uses raw (original) results of measurements and all available additional information about measurement conditions; 2) the most important source of information about noise is the comparison of measurement results obtained by different magnetometers, including those using different measuring principles, as well as careful monitoring of the environment at the observatory; 3) automatic program identification and correction of noise have an auxiliary nature and are principally an interactive tools to help a magnetologist in data processing.
All this indicates the fact that data processing should be carried out by a qualified magnetologists directly at a workplace, i.e. at the observatory, and include the whole set of requirements (Jankowski and Sucksdorff, 1996;Nechaev, 2006;INTERMAGNET Tech. Ref. Manual, 2012). The similar opinion is presented by Linthe et al. (2012). Unfortunately, at the present time, many observatories, especially newly created and located in remote areas, have problems with staff and their qualification. In these cases it seems reasonable to create centers for collecting raw magnetic data, where full cycle of data processing would be performed. Examples of such centers are BGS with a center in Edinburgh (http://www.geomag.bgs.ac.uk/data_service/space_weather/current_conditions.html) or GC RAS (http://geomag.gcras.ru/) which collects and processes the data from the Russian magnetic observatories. At the same time, a part of the problems, including those connected with incorrect noise processing can be solved ineffectively.