LIRIS FLIGHT DATABASE AND ITS USE TOWARD NONCOOPERATIVE RENDEZVOUS

ESA£s ¦fth and last Automated Transfer Vehicle, ATV Georges LemaŠtre, tested new rendezvous technology before docking with the International Space Station (ISS) in August 2014. The technology demonstration called Laser Infrared Imaging Sensors (LIRIS) provides an unseen view of the ISS. During Georges LemaŠtre£s rendezvous, LIRIS sensors, composed of two infrared cameras, one visible camera, and a scanning LIDAR (Light Detection and Ranging), were turned on two and a half hours and 3500 m from the Space Station. All sensors worked as expected and a large amount of data was recorded and stored within ATV-5£s cargo hold before being returned to Earth with the Soyuz §ight 38S in September 2014. As a part of the LIRIS post§ight activities, the information gathered by all sensors is collected inside a §ight database together with the reference ATV trajectory and attitude estimated by ATV main navigation sensors. Although decoupled from the ATV main computer, the LIRIS data were carefully synchronized with ATV guidance, navigation, and control (GNC) data. Hence, the LIRIS database can be used to assess the performance of various image processing algorithms to provide range and line-of-sight (LoS) navigation at long/medium range but also 6 degree-of-freedom (DoF) navigation at short range. The database also contains information related to the overall ATV position with respect to Earth and the Sun direction within ATV frame such that the e ̈ect of the environment on the sensors can also be investigated. This paper introduces the structure of the LIRIS database and provides some example of applications to increase the technology readiness level of noncooperative rendezvous.


INTRODUCTION
The ATV has been servicing the ISS regularly since 2008.The 5th and ¦nal §ight took place in 2014 and this was the last European opportunity for performing in §ight experiments related to Rendezvous and Docking (RVD).Following a similar approach that the ATV Rendezvous Predevelopment (ARP) Project [1] that §ew three §ight demos on Space Shuttles, a proposal for performing an in- §ight experiment of new navigation sensors was submitted two years prior to the ATV ¦nal §ight and implemented within less than 18 months by Airbus Defence and Space and its subcontractors.
The ATV rendezvous is fully cooperative with the target, but future mission will also need rendezvous with noncooperative targets.Hence, the interest to §y and test in a known environment is a di¨erent sensor suite for both cooperative and noncooperative rendezvous.The behavior of this new sensor suite could then be compared to the measurements provided by the ATV navigation sensors that have been characterized through the series of ¦ve §ights [2].The proposed sensor suite was composed of infrared camera that could detect the target at long-range while being entirely passive and being able to provide measurements even in challenging illumination conditions, such as eclipse or Sun within the ¦eld of view, and a three-dimensional (3D) LIDAR to provide measurements at shorter range, typically from 3.5 km (Fig. 1).The LIRIS demonstration embarked two infrared cameras and one visible camera provided by SODERN and a 3D scanning LIDAR [3,4] provided by Jena-Optronik on the front cone of ATV-5 as seen in Fig. 2. The sensors suite was integrated on ATV with minimum interfaces (electrical and few commands and monitoring, in addition to mechanical and thermal aspects) and dedicated recorders were used to store and retrieve data at a later stage.The implementation and operations on ATV-5 have already been described in [5,6].This paper shows how the data collected by LIRIS sensors were carefully calibrated and synchronized with ATV own navigation sensors in order to create a §ight database that can be used for future studies.

OVERVIEW OF LIRIS DEMONSTRATION
The LIRIS demonstration was activated twice during the ATV-5 mission.The ¦rst activation on August 8, 2014 involved only the cameras during a speci¦c ISS Fly-Under manoeuvre that was designed to allow the observation of ISS at long range while enforcing the safety constraints with respect to ISS (see dashed trajectory in Fig. 3).Such observation would not have been possible during the nominal ATV rendezvous since the spacecraft is then kept in a speci¦c attitude that maximizes the power generation from its solar panels and, hence, ISS would  be out of the cameras ¦eld of view.The §y-under manoeuvre allowed to view ISS with cameras from a range of about 70 km up to 8.8 km.
The second activation occurred during ATV-5 rendezvous on August 12, 2014 from a station-keeping point referred as S2 (see zoom in Fig. 3) located 3500 m behind ISS and 100 m above.Indeed, from that point on, it was possible to orient ATV front cone toward ISS while ensuring su©cient energy storage on ATV batteries to cope with any potential docking contingency situation.At roughly 3500 m, all the cameras but also the LIDAR started recording ATV approach to ISS up to docking which lasted about 2 h.
After dismounting of the recorders by ISS crew, they have been recovered by the manufacturers following Soyuz 38 return in September 2014.More than 18 000 pictures have been recorded per camera and about 7 500 scans of the LIDAR (Table 1), meaning more than 72 GB have been recorded during LIRIS demonstration.
Then, all the images and LIDAR data collected during the LIRIS demonstration have then been organized in a dedicated database together with all relevant information to be able to exploit the data.This database is the end-product of the LIRIS demonstration to serve future investigations in vision-based navigation and associated sensors development.It has been developed by Airbus Defence and Space as a part of LIRIS post §ight activities.

LIRIS DATABASE
The LIRIS database is organized in a set of ¦les in a directory tree called LIRIS-DB (Fig. 4) for LIRIS §ight database.It includes, in particular: LIRIS-1 Camera images for CAMIR1, CAMIR2, and CAMVIS (raw data); LIRIS-2 LIDAR data (preprocessed data); estimated ATV-ISS relative state vector vs. time for reference; and ¦xed data: sensor position/alignment, camera distortion calibration laws, etc.
For the Fly-under §ight phase, the ¦le organization is the same as for the rendezvous, except it does not contain any LIDAR data.The directory tree of the database is presented in Fig. 4.
The content of the ¦les in general is explained in the following subsections.

Figure 4
The LIRIS database content and structure description

Camera Data
The Camera Images folder contains all camera images acquired, stored and downloaded after the mission.One folder is dedicated to ATV-5 rendezvous (RDV) and one is dedicated to Fly-Under (FU).These sets of images are grouped in a unique subdirectory, but sorted by camera type (CAMVIS, CAMIR1, and CAMIR2) and §ight phase (RDV, FU).The image format is TIFF and their characteristics are as follows: CAMIR1/ CAMIR2: 640 × 480 pixel, ADC 14 bits per pixel.The size of one infrared image is 603 KB (uncompressed); and CAMVIS: 1360 × 1024 pixel, ADC 14 bits per pixel.The size of one visible image is 2.66 MB (uncompressed).
The images are in the camera measurement reference frames.The correspondence between the camera frames and ATV reference frame can be deduced by the de¦nition of di¨erent frames.The raw images can be corrected using the distortion laws provided by SODERN for each camera and applica-ble for all images of one camera type.Each image ¦le contains one raw image taken by a camera at a speci¦c time.The image ¦lename contains the image date at LIRIS recorder level (CAMREC) and has the following format: camera name camrec time .tif,e. g., IR1 2014-08-08 21-10-35-6.tif.In order to associate an image with another LIRIS-DB information (e. g., ATV-ISS state vector, mission event, etc.), the user shall use a speci¦c ¦le giving the time correspondence of the image ¦lename and ATV reference time.This ¦le is called camera name RDV/FU .txt,for which reason there are 6 synchronization ¦les (one per camera and per §ight phase).They contain the list of the image name ¦les, the corresponding time stamp relative to a reference initial time T 0 with an accuracy better than 0.17 s, and the corresponding GMT (Greenwich Mean Time) with an accuracy better than 0.011 s relative to ATV time.This shows that the jitter on all images is well inside one sampling time T s = 1 s of the closed loop control system.

LIDAR Data
All LIDAR data acquired, stored, and downloaded after the mission are grouped in a unique subdirectory relative to the §ight phase (RDV).The LIDAR data ¦le name contains the scan number, the scan mode identi¦er, and the date of the data postprocessing after §ight.The scienti¦c data are split in several ¦les, 1 ¦le containing 1 scan.The content of each scan ¦le is the following: 1 header containing scan description, scan number, and LIDAR mode; and a set of data for each return of the scan containing: time relative to T 0 ; counter of target returns; range; azimuth; elevation; and amplitude.
Each ¦le corresponds to a scan taken by the LIDAR at a speci¦c time.The scan duration depends on the scan mode.Each return within a scan is dated separately with respect to the ATV reference time with an accuracy better than 1 ms.The LIDAR data have been processed and corrected by Jena-Optronik with regards to ¦eld of view distortion and they are given relatively to the LIDAR measurement reference frame.The correspondence between the LIDAR frame and ATV reference frames can be deduced by the de¦nition of the di¨erent frames (Fig. 5).

Time and Synchronization
Since the LIRIS demonstration was integrated with limited interfaces to ATV, it was not connected to the main data exchange bus and, hence, each experiment was running its own time during the §ight.In order to be able to correctly interpret the data collected, it was then important to correlate with a good accuracy the LIRIS time and the ATV time.
To do so, the recording commanding sequence sent by ATV was recorded at experiment level with its own time.The di¨erent time delays at ATV commanding level and experiment level have been estimated prior §ight (Fig. 6).Furthermore, the relative drift of the experiment clock with respect to the ATV time was also evaluated and considered in the post §ight synchronization of the respective times.Finally, for each camera, a time ¦le is provided within the LIRIS database to give the ATV time associated to each set of data.This ATV time is also provided as a function of GMT in a separate ¦le to have an absolute time reference.For LIDAR data, the time has been postprocessed and corrected directly into the data ¦les during the synchronization process.

Frames and Calibration
As explained above, the LIRIS sensors data are provided in their respective measurement frames.To be able to exploit these data with respect to ATV own navigation data, it is, therefore, important to know precisely the position and the orientation of the sensors as mounted on ATV.In order to obtain an accurate knowledge of each sensor orientation within ATV reference frame, a speci¦c Figure 7 The LIRIS calibration tests pre §ight calibration test was performed in the frame of acceptance tests prior to §ight with all LIRIS sensors but also ATV sensors, i. e., the ViDeoMeter (VDM) and the TeleGonioMeter (TGM), looking at the same target.This ground target has a T-shape and is composed of laser retrore §ectors similar to the ones installed on ISS service module for ATV ¦nal approach (Fig. 7).
The data of all sensors were collected for a target located at di¨erent distances from ATV.Based on these data, it was possible to estimate precisely the position and orientation of each sensor measurement frame in the ATV axis.The resulting rotation matrix and translation vector to transform the coordinates of a given point from the sensor frame to the ATV reference frame are provided in the database for each sensor.

ATV GNC Trajectory State Vector Data
During the §ight, the ATV state vector was stored for later use on the ground via telemetry.This will serve as a calibrated and well §ight proven reference.The state vector is the estimate from the GNC based upon using the sensors suite that has been used for all previous ATV §ights (Fig. 8).
A replay of the §ight allowed to generate data at 1 Hz.A relative state vector between ATV LIRIS frame and ISS LIRIS frame is computed (frames with origins at the respective docking ports).The relative attitude information is issued from gyrometers, star tracker, ISS attitude, and VDM at short range.The position and velocity information are issued from relative GPS (global navigation system) navigation between ATV and ISS at long range and VDM at short range.To correlate LIRIS ¦ndings to external events, additional parameters are given like the Sun position or absolute position and velocity of the ATV LIRIS frame with respect to the J2000 frame (absolute GPS information).The full set of parameters included in the database are given in Table 2.
These trajectories (one for §y-under and one for rendezvous) can be used as a reference for comparison with LIRIS raw data or for evaluating the performance of image processing and navigation algorithms using LIRIS data.

UTILIZATION OF DATABASE
Obviously, the use of the database can be for di¨erent purposes like: §ight quality investigation of images from di¨erent technologies; distortion, contrast and §ight illumination issues; use as a real source for image processing to obtain kinematic measures; input to GNC state estimation ¦lter design, etc.
Because the data of each sensor can be used to perform relative navigation without any need for speci¦c ISS data apart its geometry, LIRIS database could be used to estimate the performance of sensors in a noncooperative scenario.The following examples are provided to illustrate some of the di¨erent potential uses. the corresponding GNC state vector can then be found in the ¦le on ATV-ISS states vs. time (Fig. 9); and the transformation from cameras measurements frame to ATV reference frame shall ¦nally be utilized in order to perform relative state vector determination.
The next step for using images for estimation purposes and prior to that image processing will be to apply the distortion correction.The correction has to be applied at pixel level.
This process needs to be repeated for each pixel in an image to build one with distortion correction.An example of a correction is shown in Fig. 9 together with a set of ATV reference data that can be associated to each image through the database.camera detector, at 70 km, the apparent ISS size is just comparable to one-pixel ¦eld of view.Even if the capability to observe the ISS at such range has been demonstrated by LIRIS, the capability to isolate and detect without ambiguity ISS from the overall ¦eld of view remains to be investigated.In particular, the impact of the camera noise and the variation in illumination and range on the ISS signal received by each camera should be further studied.
Figure 10 shows the signal associated to ISS on the infrared camera #1 during the Fly-Under at a relative distance of about 9 km.At such distance, ISS illuminates more than one pixel and the associated signal can clearly be distinguished from the surrounding background noise on the camera.

LIDAR Range Comparison to Automated Transfer Vehicle Reference Trajectory
At medium range, the envisioned navigation use for such sensor suite would be to obtain range and LoS information from LIDAR measurements.Using LIRIS database, it is possible to compare the range measurements obtained by the 3D scanning LIDAR with the range derived from the ATV reference trajectory.The LIDAR range of the target is considered here as the minimum range of all target returns in a given scan corrected by the position of the LIDAR in the ATV frame.The obtained LIDAR range is then compared to the ISS range derived from ATV reference trajectory (see Fig. 9).This example is only an approximation to illustrate the use of the database since the geometry of the ISS docking port is complex and some structures such as the high-gain antenna are sticking out from the ISS reference plane.As can be seen in Fig. 11, most of the LIDAR range data are within 24 m of the ATV GNC reference trajectory with only a few outliers above 6 m of di¨erence.Such range di¨erences are in any case within the accuracy of the ATV GNC reference trajectory at such distance.
A more detailed and accurate modeling could be performed using LIRIS database to derive LIDAR performances.

Figure 11
The ATV-ISS range comparison between GNC reference trajectory (1) and LIDAR data (2)

CONCLUDING REMARKS
The LIRIS technology demonstration successfully §ew a new suite of sensors composed of two infrared cameras, one visible camera, and a 3D scanning LIDAR during the last servicing mission of the ATV to ISS.Such sensors could be used for future exploration or servicing missions in either a cooperative or noncooperative rendezvous.Data collected during LIRIS demonstration, from a range of 70 km up to docking, has been gathered in a database that can be used to assess the in- §ight behavior of these types of sensors.The database contains also the reference relative trajectory §own during the ATV Fly-Under and rendezvous based on ATV own navigation sensors.The careful calibration and synchronization of the LIRIS sensors with the ATV reference allow using the database to also estimate the potential performance of image processing and navigation algorithms based on data coming from cameras or LIDAR.

Figure 1
Figure 1 Proposed sensor suite and its potential navigation use as a function of range

Figure 2
Figure 2 ATV-5 with the LIRIS sensors prior docking to ISS

Figure 3
Figure 3 The LIRIS activation pro¦le during Fly-Under and Rendezvous along ATV-5 trajectory with respect to ISS: 1 ¡ boost; and 2 ¡ slew

Figure 5
Figure 5 (a) Visible camera image ( c SODERN) and (b) 3D LIDAR image ( c Jena-Optronik) taken at a range of about 20 m from ISS Service Module docking port

Figure 6
Figure 6 Schematic illustrating the principles for the dates correlation between ATV time and LIRIS cameras experiment (CMU ¡ CPU/Memory board Unit)

Table 1
The LIRIS data collected during ATV-5 mission

Table 2
Parameters available within ATV reference state