Data flow of spectral UV measurements at Sodankylä and Jokioinen

The data flow involved in a long-term continuous solar spectral UV irradiance monitoring program is investigated and structured to provide an overall view on the multiphase process from data acquisition to the final products. The program employing Brewer spectrophotometers as measuring instruments is maintained by the Finnish Meteorological Institute (FMI) ever since the 1990s at two sites in Finland: Sodankylä (67 N) and Jokioinen (61 N). It is built upon rigorous operation routines, processing procedures, and tools for quality control (QC) and quality analysis (QA) under continuous development and evaluation. Three distinct levels of data emerge, each after certain phase in the data flow: Level 0 denoting raw data, Level 1 meaning calibrated data processed in near-real time, and Level 2 comprising of postprocessed data corrected for all distinguishable errors and known inaccuracies. The final products disseminated to the users are demonstrated to result from a process with a multitude of separate steps, each required in the production of high-quality data on solar UV radiation at the Earth’s surface.


Introduction
The Brewer spectrophotometer (Brewer) (Bais et al. 1996;Brewer, 1973) is originally degined to measure total ozone, but has also been developed to measure the spectral UV irradiance and sulfur dioxide (SO2). At present there are over 220 instruments set up by research institutes all over the world (http://kippzonen-brewer.com/). These instruments form an important network for monitoring changes in the total ozone column and, e.g., are used as validation measurements for satellite retrievals. The Finnish instruments were set up in 1990 and 1995, in Sodankylä and Jokioinen, respectively, to respond to the need to monitor total ozone and UV radiation after the discorvery of the Arctic ozone loss. Nowadays, these spectral UV time series of over twenty years are unique and among the longest measured in the Arctic. The homogenized time series have been used for several international studies related to Arctic ozone loss (e.g., Bernhard et al. 2013;Manney et al. 2011;Knudsen et al. 1998), satellite data validation (Hassinen et al. 2008), biological (e.g., Lappalainen et al. 2010;Martz et al. 2009), material (Heikkilä 2014) and health research (Kazantzidis et al. 2009).
The high dynamical range of UV radiation reaching the surface of the Earth sets challenges to the instruments, which are designed to monitor both the short UV-B wavelengths (290-315 nm) and the longer UV-A wavelengths (315-400 nm). Also the Brewer is a versatile and an extremely complex instrument, with many intermediate steps and corrections in the processing chain from data acquisition to final data dissemination. High quality data can only be ensured after careful characterization of the instrument, correction of known measurement errors and careful quality control (QC) and quality assurance (QA). (Seckmeyer et al. 2001;Garane et al. 2006;Lakkala et al. 2008;Webb et al. 2003).
In particular, keeping a Brewer absolutely calibrated is difficult Seckmeyer 1999, Webb et al. 1998). International campaigns are organized to evaluate the calibration and measurement procedures performed by different Brewers and institutes. The difficulty of the absolute calibration was seen in the last European Brewer comparison organized by the COST 1207 project in El Arenosil, Spain. There, 6 Brewers from 18 differed more than 10% from the reference, when using the calibration provided by the operator. During the comparison campaign, each instrument was recalibrated against a common calibration lamp. The Geosci. Instrum. Method. Data Syst. Discuss., doi:10.5194/gi-2015-42, 2016 Manuscript under review for journal Geosci. Instrum. Method. Data Syst. Published: 18 January 2016 c Author(s) 2016. CC-BY 3.0 License. calibration procedure was performed by only one operator. The results showed that the difference between the two calibrations could be even more than 20%. When using the calibration based on the common lamp, the difference between most of the Brewers diminished to be within ± 6% (Julian Gröbner, personal communication). The remaining difference could result from different data correction and data processing procedures, e.g., differences in the way to take into account the temperature dependence and the angular response of the instrument.
To enable Brewer data from around the world to be comparable, it is necessary to very carefully document the traceability of the calibration and how the data has been processed.
Careful documentation should be part of routine QC/QA procedures at each site. This allows anyone to audit all steps which have been taken before delivering the data, and allows changes to be made in post-processing without starting everything from the beginning. This paper documents the steps that are involved in the acquisition, processing, storage, and dissemination of data from the Finnish Brewers, located in Sodankylä and Jokioinen. The observatory at Jokioinen is in the process of being shut down, and the spectral UV measurements have been moved to Helsinki. Thus, this paper also serves as a historical description of the Jokioinen measurements. A detailed description will be given of the process flow from the raw photon counts to the calibrated spectral UV irradiances and UV products.
We also describe the quality control and quality assurance systems that are used to ensure valid output. In a companion paper (Mäkelä et al. 2015, this issue) we describe how the calibration and homogenization of the data is made. In another companion paper (Heikkilä et al. 2015, this issue) we describe how the data are further processed in the EUVDB database.

The Jokioinen station (Brewer #107)
The meteorological observatory in Jokioinen (60.82°N, 23.50°E) is at an altitude of 107 m above sea level. The observatory is located in a rural area surrounded by fields and mainly coniferous forest. The ground is covered by snow most of the time during December-March.
Temperatures can range from -20C to +30C. The observatory will be shut down in the near future and the Brewer was moved to Helsinki in November 2015. The Brewer was located on the roof of the sounding station (see Figure 2 and Figure 3).
The FMI acquired the current Mk III Brewer #107 in the observatory in Jokioinen in 1995.
Brewer #107 has a double monochromator. It collects UV radiation with a hemispherical field-of-view through a PTFE diffuser enclosed in a quartz dome. It originally operated in the wavelength range 286.5-363 nm, but the optics were changed in April 1997, and its wavelength range since then is 286.5-365-nm. The width of the slit function is 0.59 nm at The stability of the Brewer is monitored by measuring an internal lamp (typically about 2-5 times per day, see Figure 5) at the six wavelengths used for total ozone retrieval. The information about the stability is used for the post processing of total ozone measurements, during which the effect of changes in the instrument can be corrected. In addition, every three weeks a more extensive stability check is made using external 50 W lamps. Then, the whole UV wavelength range is measured, and possible drifts in the spectral response of the Brewer can be detected and corrected afterwards during the post processing of UV spectra.

Processing of spectral data
The spectral processing is done using custom-made software, which is based on the original software provided by the manufacturer, mostly perl and shell scripts. In addition to the nearreal-time measurements, every morning the data from the previous day is reprocessed and checked, and all relevant databases updated. The algorithms have been described in detail by Lakkala et al. (2008), and only key points will be summarized here (see Figure 6).
Scans are performed from small to larger wavelengths. The Brewer also returns the dark current for each scan. The total scanning time is about 4 minutes for Brewer #037 and 5 minutes for Brewers #107 and #214 due to the larger wavelength range. The noise spikes are removed based on the method of Meinander et al. (2003). The dark current count is subtracted from measured counts. The dead time is measured daily, and the data are corrected using an iteration of an exponential function including the number of counts and the dead time values.
The stray light is calculated as the average of all counts below 292 nm (#107 and #214) or 293 nm (#037). Since #107 and #214 have a double monochromator, the stray light counts are small, while the Sodankylä Brewer #037 has a single monochromator, and the stray light counts are larger (Bais et al. 1996).
The counts are then converted to irradiances by dividing the counts by the daily response. The determination of the daily response is briefly described in Lakkala et al. (2008) and is covered in more detail in the companion paper (Mäkelä et al. 2015, this issue). The temperature and cosine corrections are then made to the spectral irradiances. Geosci. Instrum. Method. Data Syst. Discuss., doi:10.5194/gi-2015-42, 2016 Manuscript under review for journal Geosci. Instrum. Method. Data Syst. Published: 18 January 2016 c Author(s) 2016. CC-BY 3.0 License.
Data are uploaded to the following databases.
• The IDEAS database for quick and long-term quality control: every five minutes • The FMI climate database (UVI index) every time a new UVI measurement is made. • The EUBREWNET database for collaboration with the international Brewer community within the COST 1207 project: every 20 minutes (http://rbcce.aemet.es/eubrewnet) • The database of the FMI-Arctic Research Centre (http://litdb.fmi.fi/): Sodankylä data, once a year • The European UV database (http://uv.fmi.fi/uvdb/): once a year

IDEAS: Real-time QA and monitoring
In 2015 a new software has been introduced to facilitate both quick and long term quality control of data, and to improve the potential of the Brewers to work as real-time operational devices. IDEAS is a tool for cheking that the Brewer is functioning correctly. The Brewer itself makes several check measurements during the day, and these measured parameters are used to monitor the stability of the instrument. The IDEAS software is also used e.g. to calculate the daily mean of total ozone, which can be directly submitted to databases. Every measurement of the Brewer and process of the operating software is recorded in addition to appropriate data files. These are stored as socalled B-files, and updated to the server in which IDEAS is running.

Annual quality assurance
The responsivity times series of the Brewers are recalculated typically once a year. During the process, the drifts of the instruments or the calibration lamps are taken into account. The spectra are recalculated with the methods described in a companion paper (Mäkelä et al 2015, this issue). If necessary, a wavelength correction is made using the SHICRIVM algorithm (Slaper et al 1995). The dose rates are compared with reconstructed UV, model calculations of clear sky UV and global radiation, global radiation and broad band UV data in order to distinguish erroneous measurements. In addition each spectrum is checked by eye and bad measurements are excluded.

Conclusions
The FMI has operated Brewer spectrophotometers since the 1990's in two locations, Jokioinen and Sodankylä. During that time, FMI has implemented all corrections and improvements that have been identified by the Brewer community. The outputs are used to calculate multiple UV products which require spectral data. A special new focus is being put on real-time quality control, with the IDEAS software allowing warnings of malfunctions to be sent to operators very rapidly. Although the measurements at the Jokioinen station has      19