Automatic segmentation and classification of seven-segment display digits on auroral images

. In this paper we describe a new and fully automatic method for segmenting and classifying digits in seven-segment displays. The method is applied to a dataset consisting of about 7 million auroral all-sky images taken during the time period of 1973–1997 at camera stations centred around Sodankylä observatory in northern Finland. In each image there is a clock display for the date and time together with the reﬂection of the whole night sky through a spherical mirror. The digitised ﬁlm images of the night sky contain valuable scientiﬁc information but are impractical to use without an automatic method for extracting the date–time from the display. We describe the implementation and the results of such a method in detail in this paper.


Introduction
Aurora light is a result of interactions between charged particles and atmospheric atoms and molecules.Guided by the Earth's magnetic field, charged particles of magnetospheric and solar origin deposit their excess energy in circular zones around the magnetic poles.On the night side of the Earth, this so-called auroral oval is typically observed at magnetic latitudes of about 65-75 • (Nevanlinna and Pulkkinen, 2001), but its width and brightness vary with solar activity and processes in near-Earth space (Nevanlinna and Pulkkinen, 1998;Partamies et al., 2014).Analysis of the structural and temporal changes in the aurora thus provides information on processes in the magnetospheric source region of the precipitating particles as well as information on the terrestrial re-sponses to space weather events.The auroral emissions are produced at about 100-200 km above the ground, and the most intense emission is typically an atomic oxygen emission line at 557.7 nm (green, often visible to human observers).
Systematic ground-based imaging of the aurora by the Finnish Meteorological Institute started in 1973, and for two solar cycles  a total of eight all-sky colour film cameras were operated at stations across Finland and at Hornsund, Svalbard.Cameras of the same type were operated by other institutes during the same time period in four other locations: Tromsø and Andøya (Norway), Kiruna (Sweden) and Georg Foster (Antarctica).Typically a 20 s exposure was taken by each camera once per minute.Towards the end of the 20th century the film cameras were replaced with digital imagers and incorporated into the Magnetometers-Ionospheric Radars-All-sky Cameras Large Experiment (MIRACLE) network (Syrjäsuo et al., 1998).These new instruments provide data in a format which can be readily analysed using computer programs.To extend the digital auroral image database of the MIRACLE network of all-sky cameras (ASCs) (Sangalli et al., 2011;Rao et al., 2014), the data from the film cameras were recorded to VHS tapes for easier access and later digitised into MPEG video format.In this paper we document an automatic number recognition method developed for reading the digits in an LED clock display visible in the digitised film camera images.This method has been applied to data from 1973-1997 from the seven Finnish camera stations and the one in Hornsund, Svalbard.The process is important because it allows us Published by Copernicus Publications on behalf of the European Geosciences Union.
to add two full solar cycles to the length of the digital image time series, which starts in 1996.We aim to make the film camera data accessible to the scientific community.
Seven-segment displays are found on calculators, digital watches, and audio and video equipment (Bonačić et al., 2015).Automatic means of segmenting the displays and recognition of the digits would enable automation of many laborious tasks.The recognition of segmented handwritten digits has been widely studied (Deng, 2015;Oliveira et al., 2002).The methods found in the literature work exceptionally well for segmented datasets such as the Mixed National Institute of Standards and Technology (MNIST) dataset (Deng, 2015).Existing recognition algorithms for classifying digits typically perform with about 99 % accuracy with hard-to-read hand-written digits (Salakhutdinov and Hinton, 2007).These algorithms are applicable to sevensegment displays as well.
With the advent of computational power some effort has recently been put into detecting numbers in digital displays.Some of the problems associated with segmenting numbers in digital displays are low contrast, high contrast, bleeding and specular reflections (Tekin et al., 2011).Long time series consist of vast amounts of data, which makes it impractical to have a training set containing half of the dataset.The reason for using automatic methods in the first place is the fact that the labelling of the data is costly and laborious.
Segmentation and specifically fully automatic segmentation algorithms are built on a priori knowledge about the dataset.The simplest ones include plenty of experimentally obtained threshold values such as average intensity in a specific region (Sezgin and Sankur, 2004).Threshold-based methods work well on small homogeneous sets but not on large datasets, mainly because obtaining threshold values experimentally is time-consuming and a fixed parameter often poorly describes the entire dataset.Consider an LED screen for a single hour of images.During this hour the intensity of the screen can be steadily 20 % brighter than the background.It is then easy to find a threshold value that makes the algorithm work with almost 100 % accuracy.However, the assumption that the intensity is above this threshold value is probably not true for the next hour or day, especially during full moon or foggy conditions.
In addition to solving the segmentation problem, the a priori knowledge has to be formulated in mathematical terms and discretised to make it computable.The computational complexity and system runtime of the algorithm are important to enable fast testing and data extraction.
To address these challenges, we describe a new method that 1. incorporates the a priori knowledge of the display geometry as a template; 2. detects if the image has a display with digits; 3. finds the midpoints of the digits in the display and extracts the digits; 4. determines the scale and the intensity of the digits and normalises them; 5. uses a combination of principal component analysis (PCA) and k-nearest neighbours (KNN) techniques to classify accurately the individual numbers with only ∼ 0.1 % of the dataset as a training set; 6. has small memory requirements and processes 24 images s −1 on modest hardware in 2015; 7. works well with noisy and low-resolution images.

Data description
Our dataset consists of about 6 700 000 images similar to those in Figs. 1, 4, 5 and 6.The data were collected in 1973-1997 with a total of eight all-sky colour film cameras that were operated at stations across Finland and at Hornsund, Svalbard.The full list of film camera stations together with their years of operation is given in Table 1.The key features are the roughly continuous imaging at a few Lapland stations: Kevo, Kilpisjärvi, Muonio and Sodankylä, which together provide an almost uninterrupted time series of auroral observations for 25 years in a seasonal sense.The image cadence is 1 min and the exposure time was 20 s.The cameras were standing outside in all weather conditions, which means that the snow or rain blurred or covered the digits at times.The imaging was started and stopped manually by station managers, which led to daylight contamination of some of the images.The images were recorded to film, later copied to VHS for easier access and then digitised to MPEG videos which were split to individual JPG images.Some of the digitisation steps have caused artefacts in the final JPG images.A single image consists of the date-time display which is the subject of this article, a mirror image of the night sky and calibration lights in the lower left and right corners as well as in the middle of the mirror.The date-time display consists of 11 digits for the decade (1), year (2), day of the year (3-5), hour (6-7), minute (8-9) and second (10)(11).Since the image cadence is 1 min, we limit the automatic number recognition to the first nine digits only.The location, p1, of the leftmost digit (a number for the decade) and the distances between the consecutive digits in the display, d 1 , d 2 , d 3 , . .., d 10 (shown in Fig. 2), determine the orientation and the scale of the image.In order to classify the digits in the date-time display, the locations of the individual digits must be found.
The relative intensity of the lights in the digit display with respect to the background varies as a result of changing imaging conditions in the outside environment, as can be seen from  display in an image was chosen as a basis of the automatic segmentation algorithm.Since our dataset is unlabelled and unsorted, no assumptions can be made on the continuity of the data.The manually examined data for estimating the false-positive rate of the binary classification algorithm also indicated that there are discontinuities on timescales of hours, days and months.Therefore feature tracking methods common in video processing are not feasible.In some of the images extracted from the digitised films the clock display is not visible at all; for example the camera mirror and the display may be covered in snow or saturated by the light of the full moon.A binary classifier for determining if the image has a time display is needed.Only ∼ 50 % of all images had a visible and humanreadable display.
In the following sections our display template refers to a collection of manually calculated values from a single representative image.These include the approximate size of the digits (in pixels), the pattern of relative distances and approx-Table 1. List of film camera stations together with their years of operation (Nevanlinna and Pulkkinen, 2001) Hornsund 1980-1988, 1994-1998Kevo 1973-1997Kilpisjärvi 1978-1997Ivalo 1973-1979Muonio 1973-1984, 1987-1997Sodankylä 1973-1995Oulu 1973-1979Hankasalmi 1978-1982 imate size of the digit plate, all calculated from the template image (Fig. 1).

Problem description
The purpose of the algorithm described here is to accurately predict the values of the first nine digits in the image.The problem is formulated as follows: we begin with image A (Fig. 4) with dimensions of 576 × 720 pixels.Each pixel of the image has three colour components (R: red; G: green; B: blue).Since the colours in the image are not calibrated, we are only interested in the intensity of the pixels (Y ).
The task is then to find the labels for the first nine digits in the image.These labels can be represented by the label vector L = {l 1 , l 2 , l 3 , l 4 , l 5 , l 6 , l 7 , l 8 , l 9 }.Given the matrix (image) A we want to estimate labels in such a way that the classification error E = P (L = L), where L = estimate for the label (1) is minimised.This problem can be divided into subproblems that can be solved independently.In our method we divide the problem into three phases:

Phase II
In a real valued matrix D corresponding to the pixels of the date-time display in the image, we find position vectors p 1 , p 2 , . ..p 11 which describe the midpoints of the digits in screen coordinates.

Phase III
Subsets of pixels corresponding to each digit are extracted from the matrix A when position vectors p i , the width of the digits w and the width-to-height ratio are known from the template and defined by the above criteria.Feature vectors are constructed from the pixels of individual digits.A classifier then assigns a label (value) to each extracted digit based on this feature vector.The flow chart of the data conversion process is summarised in Fig. 3.

Methods and analysis
In this section we describe in detail the three phase process used for segmenting and detecting the clock display digits in the images.Phases I and II are more task-specific and incorporate more a priori information than phase III, which could be changed to a more suitable multiclass classification algorithm depending on the particular dataset.

Phase I: binary classification
The purpose of phase I is to determine if the image has digits or not.The optimal binary classifier should have both 0 % false-negative probability (classifying images with digits as images not having digits) and 0 % false-positive probability (classifying images not having digits as having digits).It was decided that it is most important to minimise the former probability in order not to miss useful data, and therefore a moderately strict classifier is needed.
The classification in both phases I and II is based on four assumptions: 1. the relative distances and shapes of the numbers follow the template of the display within the range defined by the phase II criteria in Sect.3; 2. the 11 highest local maxima in the column-summed intensities (Eq. 3 below) correspond to the midpoints of the digits; 3. the display and the numbers are approximately aligned horizontally; 4. the number of digits is 11.With these assumptions the task of determining if the image has 11 digits is a task of finding the pattern of 11 intensity maxima in the image.
www.geosci-instrum-method-data-syst.net/5/305/2016/Geosci.Instrum.Method.Data Syst., 5, 305-314, 2016  The first step in phase I is calculating the intensity for each pixel as the Y component of the CIE 1931 colour space (Hill et al. , 1997): (2) Before a binary classification is done, the image is medianfiltered.The median filter removes some noise from the original image; the difference can be seen by comparing the column-summed intensities (Eq. 3) of the original image (Fig. 7) and the median-filtered image (Fig. 8).The actual classification works as follows: 1. Consider each row of the image as a one-dimensional signal.
3. Take 10 rows at a time.
4. Sum the values of the 10 rows to get a new signal.
5. Normalise this signal by calculating the mean and subtracting it from the signal.6. Save the first 16 non-zero values to a list.
7. If the distances between the highest 11 of the 16 follow the pattern slsslslsls, where s stands for short and l for long, then the signal is valid and the digit pattern has been found.
8. If the signal was valid, mark it to a result array as valid.
9. Go to the row i +1 and repeat steps 3-8 until row n−10, where n is the number of rows.
10.If the number of valid rows is 10 or greater, then the image is classified as having digits.
11.The digit display location in the y direction is calculated as the mean location of the rows that matched the pattern.

Phase II: midpoint detection
The purpose of phase II is to extract the width, height and location of the midpoints of the individual digits.The assumptions made in phase I apply also here.Additional assumptions are as follows:  1.The number of peaks in the filtered column-summed intensities is exactly 11 (the number of digits).
2. The width-to-height ratio of the digits is 0.7; i.e. the width = 0.7 × height.5.The distance between the midpoint of the rightmost digit and the midpoint of the leftmost digit is greater than 100 pixels.
In phase II we calculate column-summed intensities of the display by summing over y axis values of the intensity matrix (image) D: (3) The column-summed intensities of images A, B and C (shown in Figs. 4, 5 and 6) can be seen in Fig. 9.
In phase II the following steps are done in order: 1. Raise every intensity value to the second power; see Fig. 10.
3. Set values below the mean to 0; see Fig. 11.
4. Estimate a common width for all digits as the mean width of the third-, fourth-and fifth-largest widths of all digits.
5. Apply a closing operator (a morphological operator filling real data gaps) with a sliding window, using the estimated common width as a parameter; see Fig. 12.
7. Find the location of the midpoints in the vertical direction: (a) sum the x values around the midpoints; (b) set values below the mean to 0; (c) estimate the midpoint in the vertical direction as the midpoint of the first non-zero region in the rowsummed intensities.
The automatically segmented and normalised images corresponding to original Figs. 4, 5 and 6 can be seen in Figs. 13, 14 and 15 respectively.
2. Zero rejection rate: all feature vectors will be assigned a label.
3. The predicted dates and times are assumed to be reasonable within the above-mentioned restrictions; i.e. there is no additional checking for dates like 70 388 27 52 (yr day h min).

Results and discussion
The algorithm presented here was developed in order to automatically read the date-time metadata for a 25-year time series of images of aurora.Four measures were used to determine the performance of the algorithm: 1.The recognition error, i.e. the number of correctly classified consecutive digits in images which include a Geosci.Instrum.Method.Data Syst., 5, 305-314, 2016 www.geosci-instrum-method-data-syst.net/5/305/2016/clock display.Since labelling low-quality digits has some ambiguity, lower and upper limits are given.The lower limit corresponds to labelling all unclear digits (to a human observer) as wrongly classified, and the upper limit corresponds to labelling digits as correct if the algorithm's classification seems like the best guess.This step was performed by all three authors as human experts visually examining the images.
2. The probability that an image has digits if it was classified as having digits.
3. An estimate of the amount of useful data missed by the algorithm.This step is done by visually inspecting the data.
4. The approximate system runtime per image.
The success of the recognition was defined as correct segmentation and classification of the first n digits in the display.This measure is based on the fact that the first digits are more significant in determining the time stamp of the image.The recognition error was estimated by selecting a random sample of 1000 images from the 3 000 000 images that were classified as having digits and comparing the computer predictions to the original unprocessed images; see Table 2.
The probability of a false-positive result was estimated from the same random sample as the recognition rate.For estimating the probability of a false-negative result, a considerably larger dataset consisting of several winters of rejected data from different decades was manually examined.Of this sample only 26 954 (7.46 %) images had digits which were human-readable.Most of the rejected images that had human-readable numbers contained contaminating light: moon reflection, sunrise or sunset, or bright aurora.Other common reasons for rejection include problems with the LED display (too bright, too dim or unevenly bright) and artefacts resulting from the digitisation process (extra stripes of light, badly synchronised video etc.), which make the search routine for the clock display pattern fail.Based on visual inspection of the classified images, the algorithm clearly captures most of the data that include human-readable numbers in the clock display.Only exceptionally bright aurora may saturate the display.Assessing the exact number of images including aurora in the detected and non-detected data would require a separate study on detection of the aurora which is beyond the detection of the clock display digits of this study.A summary of the assessment of the algorithm is shown in Table 3.
From a computational point of view the algorithm performed quickly; the whole dataset was processed over one weekend.Measurements of the total runtime are given in Table 4.Most of the software was written in the Python language using the SciPy library (Oliphant, 2007), but the most time-consuming parts were written in C in order to improve performance.The computation speed is mostly limited by the read and write speeds of the computer disks.Since the images are processed individually, the program should be easily parallelisable by dividing the image set between multiple storage locations and starting separate processes at each location.The computation speed was estimated using a mid-2014 MacBook Pro Retina computer with 2.6 GHz dual-core Intel Core i5 processor and 8 GB of 1600 MHz DDR3L onboard memory with a 256 GB PCIe-SSD hard drive.

Conclusions
In this work we describe a general method for processing images of varying quality using pattern matching and other signal-processing techniques in order to automatically determine the digits displayed in a seven-segment display.The algorithm performs at or above the level of human recognition.It was able to recognise almost all of the digits of good quality, while becoming increasingly uncertain with decreasing quality of the images.Since it is hard for human experts to classify the low resolution and blurred digits, it is hard to evaluate further improvement of the classification.Good results were obtained with respect to both classification error and runtime.
Even though the method was tailored for a specific problem, it is applicable to other related segmentation problems as well.The algorithm could be used to process similar old data stored at other research institutes with minor modifications, such as adjusting the template describing the digit display.Based on the results of this study, the digitisation of photographic film containing old data is a crucial process worth doing well in order to keep the data quality as high as possible.
The algorithm could be improved by considering the rotation of the images, since only small rotations (±5 • ) are handled by the current program.One way to further improve the method would be to parametrise the various constants and optimise the parameters with respect to error rates and detection rates.
After applying the automatic number detection described here to the LED clock display contained in the auroral images, and appending additional metadata describing the station location, the digitised auroral images can be compiled into a format similar to the modern digital data format for scientific analysis.

Data availability
The current version of the data is available upon request.Please contact the authors for further information.

Figure 1 .
Figure 1.Image used for calculating template values, such as the distances between digits.

Figure 2 .
Figure 2. Location of the leftmost digit (p 1 = p 1 x, p 1 y) and the distances d1, . . .d10 between the digits in the display.

Figure 3 .
Figure 3. Flow chart of the classification process.Each individual image is converted to greyscale and median-filtered before finding and extracting the clock display digits.Detected digits are scaled in size and normalised in intensity prior to the number classification.

Figure 4 .
Figure 4. Image A: a high-quality image with human-readable digits at the top of the image.Prediction for the numbers in the clock display: 79 005 22 12.The bright round objects in the lower left and right corners and in the middle of the image are calibration lights.

Figure 5 .
Figure 5. Image B: a fair-quality image with approximately humanreadable digits.Prediction: 90 314 03 36.The bright round object on the right-hand side of the image is the moon.

Figure 7 .
Figure 7.The column-summed intensities of the clock display intensities of images A, B and C (Figs. 4, 5 and 6).

Figure 8 .
Figure 8.The column-summed intensities of the clock displays of images A, B and C (Figs. 4, 5 and 6) after median filtering.

Figure 9 .
Figure 9.The column-summed intensities of the clock displays of images A, B and C (Figs. 4, 5 and 6) after extracting the clock display from the full image.

Figure 10 .
Figure 10.The column-summed intensities for images A, B and C (Figs. 4, 5 and 6) after squaring the pixel intensities.

Figure 11 .
Figure 11.Average filtered column-summed intensities of the previous figure.

Figure 12 .
Figure 12.The column-summed intensities of images A, B and C (Figs. 4, 5 and 6) after applying a closing operator.

Figure 13 .
Figure 13.The extracted, scaled and normalised digits of image A (Fig. 4).The scaling is necessary for obtaining feature vectors of a constant size (384), and the intensity normalisation improves classification accuracy.Prediction: 79 005 22 12.

Table 2 .
Number of images with at least n correctly classified digits out of nine.The sample of 1000 images were manually viewed by all authors independently.

Table 3 .
True positives vs. false-negative error rates in detecting the clock display of all digitised images.

Table 4 .
Runtimes for the digit detection algorithm using a mid-2014 MacBook Pro computer.