2.1

Super-resolution and compressive sensing

The concept of super-resolution is based on the possibility of obtaining a final image with a higher resolution than the nominal resolution of the chosen detector, thanks to the use of an SLM with a number of elements NxN times greater than the elements of the detector. A single pixel on the detector is associated with each NxN group of SLM elements. The image focused on the SLM is modulated through appropriate binary coding masks. Each pixel of the detector thus integrates the light coming from groups of NxN elements (therefore with N times higher resolution). The samples thus acquired allow to obtain, once transmitted to the ground and with appropriate reconstruction algorithms, an image at the same resolution as that on the focal plane at the SLM. If NxN modulations are carried out during the acquisition, there is no data compression, but if the number of modulations is lower, a decrease in the number of samples to be sent to the ground is inherently obtained (Compressive Sensing).

The theory of Compressive Sensing (CS) exploits the a priori knowledge on the sparsity characteristics of the data in order to digitally acquire it using much fewer samples of the ones envisaged Nyquist-Shannon^[5]. In general, it is always possible to find a domain where the data is sparse, the one of acquisition or eventually a transformed domain (DCt, wavelet). Following CS theory, several algorithms have been developed to obtain good quality images starting from acquisitions performed in compressive mode. Moreover, instruments have been implemented as well, by exploiting CS theory: an example of these is the single Pixel camera implemented by the Rice Institute^[6]. Following the example of such prototype, the interest in such kind of technology grew exponentially, leading to the development of several prototypes that demonstrated both the CS and the possibility of obtaining super-resolution in various application sectors^[6]. Nevertheless, up to now very few examples of prototypes addressed the aerospace sector^[7]. On the other hand, in recent years ESA has funded some studies to investigate the potential of CS for the construction of optical payloads based on this approach[7]^[7]-[10]. Regarding super-resolution, some prototypes have been developed to demonstrate its feasibility^[11]-[13]. In Mahalanobis, R., Shilling, R., Murphy, R. and Muise, R., “Recent results of medium wave infrared compressive sensing,” Appl. Opt. 53, 8060-8070 (2014).^[7], the set-up is sensitive in the MIR frequencies, while in Dumas, J. P., Lodhi, M. A., Bajwa, W. U. and Pierce, M. C., “From modeling to hardware: an experimental evaluation of image plane and Fourier plane coded compressive optical imaging,” Opt. Express 25, 29472-29491 (2017).^[13] the prototype operates in the visible and in Chen, H., Asif, M. S., Sankaranarayanan, A. C. and Veeraraghavan, A., “FPA-CS: Focal plane array-based compressive imaging in short-wave infrared,” 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2358-2366 (2015).^[11] and Mahalanobis, R., Shilling, R., Muise, R. and Neifeld, M. A., “High-resolution imaging using a translating coded aperture,” Opt. Eng. 56(8), 084106 (2017).^[12] in the SWIR.

2.2

Scientific and technological challenges

The study of a multispectral imaging spectrometer operating in the MIR region (3-5 µm) with a high GSD represents an opportunity of great interest for the scientific community, both from an application and a technological point of view. Indeed, this study will allow to evaluate the potential of spectroscopic methodologies for a better characterization and understanding of high temperature phenomena, helping to answer the crucial questions related to the changes occurring on Earth. In particular, the SISSI study may bring significant contributions to the following Scientific Challenges^[14]:

Challenge A2 - Interaction between the atmosphere and Earth’s surface involving natural and anthropogenic feedback processes for weather, energy and atmospheric composition. Plumes are produced by the continuous degassing of magma and represent an indicator of volcanic activity. As the magma rises up the pipe, the pressure decreases and the gases are released into the atmosphere, dragging fragments of magma. Plumes are a turbulent mixture of gases, solid particles and drops, continuously emitted at high temperatures from summit craters, fumarolic fields or during eruptive episodes. The spectral range in the MIR allows the recognition of spectral absorptions related to volcanic activity; in particular, the SO₂ band at 4.0 µm in the MIR could be studied by SISSI also during the non-eruptive degassing phases thanks to the high GSD. For similar reasons (high GSD of the bands in the range 3-5 µm) SISSI would allow the study of the absorption characteristics of CO₂, CO and CH₄ and therefore to observe small emissions during different volcanic phases, with particular reference to the bands (4.8 µm, 4.6 µm and 3.3 µm).

Challenge L1 - Natural processes and human activities and their interactions on the land surface. Thanks to the high GSD, SISSI can lead to the recognition of active fires and FRP (Fire Radiative Power) with a spatial precision level of the order of ten meters, maintaining the radiometric dynamics high compared to most space sensors available today. Regarding the effects of fire on atmospheric chemistry and climate, real-time FRP data provided by e.g. MODIS, or geostationary systems are already used in many scientific studies on fire emissions, and in pre-operational systems such as GMES Atmospheric Core Service prototype However, due to the large number of small fires and low FRP values typical of the most common fires, the aerosol and tracking gas emissions resulting from these calculations will almost certainly be significantly underestimated.

Challenge G1 - Physical processes associated with volcanoes, earthquakes, tsunamis and landslides. In order to better assess the natural hazards, Volcanic thermal modelling and precursor analysis. The gas emissions from the fumarole fields or degassing plumes emitted from the summit craters could be studied with high resolution multispectral images acquired in the MIR. At the moment only ASTER with 0.3 K NEDT on 5 channels TIR only allows temperature/emissivity separation and, especially with time series data, allows the detection of low temperature geothermal anomalies of ~ 6-10 K (SNR ~ 20-30) on areas of about 100 m in diameter. The loss of heat during the placement of a lava flow is its most salient feature and it modulates the whole process of setting up. The spectral radiation emitted by hot volcanic sources such as molten lava or high-temperature gas emissions reaches its maximum in the MIR region (approximately 3 µm). MIR is adequate where the High Temperature Events (HTE) radiation is much higher than both the reflected solar radiation and the background. In order to separate small HTEs from the thermal background and estimate HTE parameters, a mission such as SISSI would produce MIR data capable of filling this observation gap and improving understanding of the thermal properties of melted materials.

As far as the technological aspects are concerned, both the European Community and ESA and other European national agencies have funded various studies for thermal imaging missions with resolution <100 m over the past decade. These include the FUEGO program, aimed primarily at monitoring fires, with the aim of converging into a low-cost operational concept suitable for a microsatellite mission or as a secondary payload of a larger system. MISTIGRI (MicroSatellite for Thermal InfraRed Ground Surface Imaging) and THIRSTY are studies carried out nationally by CNES for applications in the agricultural sector. To date, however, the operational capabilities of European Earth Observation do not include a high-resolution space mission in the MIR-TIR. Ultimately, the present study is in synergy with many of the studies planned both at national, European and international level - yet maintaining a completely complementary character – and it focuses attention on a particularly critical aspect of the missions in the thermal infrared, the ground spatial resolution.

3.1

The concept

The operating principle of the SISSI instrument is inspired by that of the single pixel camera^[6]. A CS instrument’s architecture is implemented - in its simplest form – by using a light modulation element (Spatial Light Modulator - SLM) and a single pixel detector on which the light is concentrated through a lens (condenser). Each acquisition (measurement) is given by the product - element by element - between each element of the encoding binary pattern applied to the modulator and the corresponding element of the image focused on it, and the subsequent integration carried out by the condenser on the single-pixel detector. Based on the sparseness of the data to be acquired, it is possible to reconstruct the initial image with sufficient quality by making a number of measurements equal to or less than half of the pixels of the image to be reconstructed, thus merging the acquisition and compression phases into a single step. Similarly, the key elements (Figure 1) of the optical module of the SISSI instrument are:

Figure 1:

Functional block diagram of the optical module of the SISSI payload.

With respect to the single pixel camera architecture, in this case there is a parallel acquisition of the images of MxM groups of adjacent micro-mirrors of the SLM. Each group (macro-pixel) consists of NxN micro-mirrors suitably encoded by a binary mask, different for each frame. Each of the MxM elements of the detector acquires the signal coming from one of the MxM groups of NxN micro-mirrors. Each detector pixel as a whole subtends an area of the scene on the ground corresponding to a macropixel on the SLM (Figure 2). Each SLM macropixel is made up of NxN micropixels, each of which corresponds to a micromirror of the SLM. The NxN factor represents the super-resolution factor of the instrument. Each micropixel subtends to the ground a portion of the scene corresponding to the super-resolute GSD.

Figure 2:

Working principle of the super-resolution of SISSI optical instrument: macro-pixel, micro-pixel e GSD.

The acquisition of the same scene in several spectral bands (data cube) is obtained thanks to the presence of spectral filters on the detector, arranged along the across-track dimension of the scene on several contiguous lines of the detector, and to the apparent movement of the scene in the direction along-track through the field of view (push-broom acquisition without slit). In this way, thanks to the movement of the platform, the spatially resolved scene (with a GSD equal to that determined by the micro-pixel) crosses each spectral band on the detector, allowing - line by line - the reconstruction of the spatially resolved image in each spectral band.

In the SISSI instrument, thanks to the super-resolution concept, we aim at achieving a super-resolution factor equal to 4x4. A single pixel of the detector will acquire the portion of the scene corresponding to a macropixel on the SLM, that is, an area of the DMD consisting of 4x4 micro-mirrors (each micromirror corresponds to a micro-pixel). The super-resolution of the scene (resolution at the micropixel level) is achieved by acquiring a series of measurements with different spatial encodings of each macropixel, spatially integrated by each element of the detector. The series of measurements acquired by applying a different encoding mask for each measurement at the macropixel level is finally used to reconstruct - using an ad-hoc algorithm - an image of the scene with a GSD equal to that of the projection on the ground of the dimensions of the single micropixel with a super-resolution factor equal to 4x4.

3.2

Acquisition mode

The SISSI instrument is a spectrometer that operates in push-broom acquisition mode without slit (Figure 3), i.e. the light that reaches the 2-D detector comes from the entire scene framed by the instrument’s field of view. Thanks to a series of filters, deposited in correspondence with a contiguous set of detector lines, at each acquisition each portion of the scene will be acquired exclusively in the corresponding spectral band. The SISSI instrument then acquires spatially resolved images both across and along-track, with groups of lines detected in different spectral bands, depending on their position on the sensor (Figure 3b).

Figure 3:

SISSI acquisition mode: (a) Traditional spectrally resolved push-broom acquisition, and (b) spectrally resolved push-broom acquisition in different spectral bands by means of optical filters placed on the detector.

Each spectral band is made up of r lines. The scene seen by the sensor, during the forward movement of the platform, slides on the filters, allowing the acquisition of the same in all the expected bands. The 3D data (spectrally and spatially resolved) is no longer comparable to a cube as in the case of a traditional push-broom instrument (Figure 3a). Instead, to reconstruct an entire scene in each spectral band (as in the push-broom traditional imagers), the data must be reorganized by extracting from each acquisition the data referring to a number r of rows for each spectral band and that are relative to the same scene (Figure 3b).

3.3

Observational requirements

In this first stage of the project, we outlined the SISSI payload’s observation requirements starting from the analysis of the end user requirements and a first iteration on the observation constraints posed by the instrument’s acquisition method. The observation requirements for the SISSI payload are summarized in Table 1.

Table 1.

Main requirements of the SISSI payload.

3.4

Optical section of the payload

Based on the observational requirements outlined from the analysis of the end user requirements and a market survey aimed at identifying the COTS elements available, the most suitable optical elements were identified (Table 2).

Table 2.

Main technical specifications of the SISSI payload.

Starting from the observational requirements and the devised main technical specifications, we did a preliminary optical design of the instrument’s optical section (Figure 4). The study outlined as, in both collection and focusing optics, the use of reflective optical elements was a priority. This was due to the need to minimize the effect of chromatic aberration and, at the same time, to maximize the transmittance of the optical system. Actually, one of the main critical aspects of achromatic optical systems in the MIR spectral region is the absorption due to glass. Consequently, decreasing the thickness of any dielectrics maximizes the transmittance of the optical system. The design of the optical section in its preliminary version as a whole has provided good results in terms of enclosed energy per detector pixel (equal to about 85%) and a total transmittance of about 30% (DMD efficiency excluded). In the next steps, the optical design will be further refined in order to increase the overall transmittance of the system.

Figure 4.

3D optical CAD of the SISSI instrument.

3.5

Electronic section of the payload

The study aims also at outlining the architecture of the electronic section of the payload. This will be made of the following main logic modules:

These, however, should be regarded as logic modules rather than physical ones: for example, it has been verified that an ideal physical subdivision would lead to physically separated sections between the power section, the digital one and the analogue one. Under this respect, the HCC and HDP modules will be provided on a single board.

HPU module must supply the main power lines starting from the primary 28Vdc provided by the spacecraft. We are only talking about main lines because the very low voltage power supplies necessary for the various subassemblies, both to limit voltage drops and to decouple the various components as much as possible, should be generated close to the devices themselves that will use them. For this reason, each module must in any case have its power management section.

HPE module has the function of managing the interfaces towards the DMD and the detector with the relative cooler. The transmission of the binary configuration patterns to the DMD will be carried out by the FPGA through high frequency (400 MHz) LVDS buses. The interface with the detector, on the other hand, will be based on a Camera Link bus.

HDP module has the task of acquiring data from the sensor and sending configuration information to the DMD. Presently, after a market survey, it is under consideration to implement the module starting from a Xilinx radiation tolerant FPGA.

HCC module is in charge of the management of the communication towards the spacecraft and the general control of the instrument. Presently, we are considering to use a DSP from Texas Instruments to play this role. The choice of the component was based above all on the objective of allocating a substantial availability of processing capability to support that available at the FPGA level.

The activity carried out up to now made it possible to define the logical architecture of the section and to identify the main problems and construction risks. The physical architecture will be defined in the next project’s phase, in accordance with the results of the detailed design of the other sections of the payload.

3.6

Reconstruction algorithms

There are different types of reconstruction algorithms that can be used for CS^[15]-[17]. We can mainly distinguish them into two categories: traditional and based on deep learning. For this project, we have chosen as the representative of the former the algorithm that minimizes the Total Variation (TV) pseudo-norm, while for the latter the convolutional neural network called ISTA-Net (ISTA-Net + in its improved version); the latter implements at each phase the Iterative Shrinkage Thresholding Algorithm, which is a traditional method.

A traditional solution to solve the image reconstruction problem consists in minimizing the Total Variation (TV) pseudo-norm^[18]. This approach is particularly useful when dealing with natural images that have compressible gradients, or images such that most of the energy is enclosed in very few gradients. In the TV formulation, the reconstruction problem can be formulated as:

where X represents the two-dimensional image while x its vectorized version and the isotropic TV is defined as:

Reconstruction by TV is fast and widely used and thus it can be regarded as a typical traditional method.

A different approach is offered by the methods based on deep learning, such as the algorithm known as ISTA-Net. This method, and its improved version ISTA-Net +, are particularly suitable for working with natural images. The authors’ idea is to map the updating steps of the traditional ISTA algorithm on a deep neural network architecture in which there is a fixed number of phases that correspond to the iteration in the classical algorithm. In the ISTA-Net algorithm, a general nonlinear transformation is used to sparsify the images. The optimization in ISTA-Net + is due to the fact that natural images are more compressible than other and this feature is here fully exploited.

The results obtained up to now showed that ISTA-Net significantly exceeds the performances achieved by the other algorithms, both traditional and based on deep learning, for each CS ratio considered, where CS ratio means the ratio between the number of measurements acquired and the number of total pixels n. Also, with regard to computational time, the performances of ISTA-Net and ISTA-Net + are definitely better^[19].

Since the ISTA-Net neural network provides an excellent compromise between reconstruction accuracy and flexibility, the framework, in its improved version ISTA-Net +, was chosen as the starting point for the development of deep learning-based reconstruction algorithms for this study.