• unlimited access with print and download
    $ 37 00
  • read full document, no print or download, expires after 72 hours
    $ 4 99
More info
Unlimited access including download and printing, plus availability for reading and annotating in your in your Udini library.
  • Access to this article in your Udini library for 72 hours from purchase.
  • The article will not be available for download or print.
  • Upgrade to the full version of this document at a reduced price.
  • Your trial access payment is credited when purchasing the full version.
Buy
Continue searching

Vision-based navigation and mapping for flight in gps-denied environments

Dissertation
Author: Allen D. Wu
Abstract:
Traditionally, the task of determining aircraft position and attitude for automatic control has been handled by the combination of an inertial measurement unit (IMU) with a Global Positioning System (GPS) receiver. In this configuration, accelerations and angular rates from the IMU can be integrated forward in time, and position updates from the GPS can be used to bound the errors that result from this integration. However, reliance on the reception of GPS signals places artificial constraints on aircraft such as small unmanned aerial vehicles (UAVs) that are otherwise physically capable of operation in indoor, cluttered, or adversarial environments. Therefore, this work investigates methods for incorporating a monocular vision sensor into a standard avionics suite. Vision sensors possess the potential to extract information about the surrounding environment and determine the locations of features or points of interest. Having mapped out landmarks in an unknown environment, subsequent observations by the vision sensor can in turn be used to resolve aircraft position and orientation while continuing to map out new features. An extended Kalman filter framework for performing the tasks of vision-based mapping and navigation is presented. Feature points are detected in each image using a Harris corner detector, and these feature measurements are corresponded from frame to frame using a statistical Z-test. When GPS is available, sequential observations of a single landmark point allow the point's location in inertial space to be estimated. When GPS is not available, landmarks that have been sufficiently triangulated can be used for estimating vehicle position and attitude. Simulation and real-time flight test results for vision-based mapping and navigation are presented to demonstrate feasibility in real-time applications. These methods are then integrated into a practical framework for flight in GPS-denied environments and verified through the autonomous flight of a UAV during a loss-of-GPS scenario. The methodology is also extended to the application of vehicles equipped with stereo vision systems. This framework enables aircraft capable of hovering in place to maintain a bounded pose estimate indefinitely without drift during a GPS outage.

T ABLE OF CONTENTS DEDICATION...................................iii ACKNOWLEDGEMENTS............................iv LIST OF TABLES.................................ix LIST OF FIGURES................................x SUMMARY.....................................xiv I INTRODUCTION..............................1 1.1 Motivation...............................2 1.2 Literature Review...........................4 1.2.1 Summary of Specific Works..................7 1.3 Summary of Contributions.......................10 1.4 Outline of Thesis............................11 II BACKGROUND...............................12 2.1 Relating 3D Position to 2D Camera Images.............12 2.1.1 The Basic Pinhole Camera Model...............12 2.1.2 Camera Calibration and Lens Distortion...........13 2.2 Reference Frames............................15 2.3 Overview of the Extended Kalman Filter...............16 2.3.1 Extended Kalman Filter Prediction..............16 2.3.2 Extended Kalman Filter Correction..............16 2.3.3 The Correspondence Problem.................18 2.4 Image Processing............................19 2.4.1 Detecting Feature Points....................19 2.4.2 Stereo Vision..........................20 III ESTIMATING LANDMARK POSITIONS GIVEN VEHICLE STATES 24 3.1 Extended Kalman Filter Formulation.................24 vi

3.1.1 Process Model.........................24 3.1.2 Measurement Model......................25 3.1.3 Initializing Points Located on Ground Plane.........26 3.1.4 Initialization of Covariance Matrix for Mapping.......27 3.2 Simulation Results for Vision-Based Mapping............28 3.3 Generic Initialization of Database Points...............30 IV ESTIMATING VEHICLE STATES GIVEN FEATURE LOCATIONS.38 4.1 Extended Kalman Filter Formulation.................38 4.1.1 Process Model.........................39 4.1.2 Measurement Model......................40 4.2 Application to Indoor Flight......................41 4.2.1 Platform and Hardware Description.............42 4.2.2 Comparison of Method to Motion Capture System.....46 V SIMULATION AND FLIGHT TEST RESULTS.............54 5.1 Modifications for Combining the Estimators.............55 5.2 Simulation Results...........................56 5.3 Flight Test Results...........................60 5.4 Extension to Stereo Vision.......................62 5.4.1 Simulation Results.......................64 5.5 Discussion of Method.........................68 5.5.1 Tuning the Statistical Point Correspondence.........68 5.5.2 Implementing the Estimators.................68 5.5.3 Camera Calibration......................69 5.5.4 Image Processing........................70 5.5.5 Computational Complexity..................70 VI CONCLUSION................................72 6.1 Contributions of Thesis........................73 6.1.1 Monocular Vision-Based Mapping..............73 vii

6.1.2 Initialization of Points in the Mapping Filter........73 6.1.3 Vision-Aided Inertial Navigation...............74 6.1.4 Framework for Flight in GPS-Denied Environments.....74 6.2 Future Work..............................75 APPENDIX A ATTITUDE ERROR STATE REPRESENTATION....77 APPENDIX B DATA NORMALIZATION FOR DLT TRIANGULATION 81 REFERENCES...................................84 viii

LIST OF TABLES 1 Initialized landmark coordinates in simulation using DLT.......36 2 Effect on disparity in pixels for varying baseline separation distance between cameras and different ranges to target for a stereo rig.The cameras in this have a 320×240 pixel resolution and a 55 degree field of view...................................64 ix

LIST OF FIGURES 1 Small unmanned aerial vehicles,such as the Hornet UAV shown above, are often not capable of carrying large computer systems........3 2 The FCS20 is a small autopilot system with a DSP and FPGA along with integrated sensors including an IMU,GPS,and pressure sensor.4 3 Camera perspective projection model used for relating 3D position to position in 2D images.The point (A,B,C) is projected onto the camera image plane to the point (b,c)..................13 4 An example of the effect of distortion caused by a camera lens.The image on the left is the image obtained directly from a camera with a wide angle lens whereas the right image shows the undistorted version of the image................................14 5 Geometry for a calibrated stereo rig with a baseline distance b between the two cameras with identical focal lengths.A landmark at a distance r from the stereo rig provides a disparity of D = X l − X r in the horizontal location between the left and right images.........21 6 The output of the stereo image processor froma synthetically generated scene.Shown in the left image are the locations of the feature points as found in the left camera of the stereo rig.The right image displays the disparity map of the pixels in the left camera to represent the distance to the points................................22 7 Sample rectified images fromonboard a rotorcraft UAV using functions from OpenCV.Rectified stereo images simplify the computation of the disparity map for a stereo rig.......................23 8 Vectors used in describing the EKF measurement model........25 9 By assuming all landmarks to lie on the ground,points in the map- ping database can be initialized by finding the intersection of the first observation ray with the ground plane..................26 10 Sample images from the simulation of the mapping estimator.Upper left shows the trajectory of the helicopter as it flies around the target points.The upper right window shows the image viewed by the simu- lated camera.The lower left shows the image processing results of the corner detector.The output of the estimated locations are replicated in the simulated camera image as red spheres in the lower right window.29 11 Estimate errors for the fifth estimated point.This point is located at [-6,5.5,0] ft................................30 x

12 Progression of the feature estimates in North,East,and down axes for the mapping estimator.The initial estimates are obtained by project- ing the initial measurements onto a plane at 20 ft altitude whereas the actual points lie on the ground at 0 ft altitude..............31 13 Visualization for the feature point tracks when initializing the 3D iner- tial location of landmarks using multiple observations for triangulation. The blue lines show the locations of the last 20 measurements for the associated feature point..........................36 14 Visualization for the feature point tracks when initializing the 3D iner- tial location of landmarks using multiple observations for triangulation. The blue lines show the locations of the last 20 measurements for the associated feature point..........................37 15 Ducted fan UAV used for demonstrating the vision-aided inertial nav- igation algorithm.Tethers are mounted to the side of the aircraft to prevent the aircraft from striking the ground during initial testing...42 16 Avionics block diagram for the ducted fan UAV.Low-level sensor and actuator interfacing and the feedback control are performed on the aircraft’s ARMmicrocontroller,whereas the more computationally in- tensive image processing and EKF are handled by the GCS laptop. The Vicon motion capture system is for comparison purposes to assess estimator performance...........................43 17 Architecture for control of the ducted fan UAV.The outer loop is responsible for tracking position whereas the inner loop regulates atti- tude of the aircraft.The outer loop commands roll and pitch attitude angles to the inner loop in order to achieve the desired translational motion...................................44 18 The experimental setup used for testing the localization and mapping algorithms.The Vicon system consists of the infrared cameras shown in the background.The vision sensor and the IMU are on the black vehicle marked with the reflective markers.A sample target is shown on the ground with black rectangles against a white background....46 19 Sample output image from the framegrabber and image processing. The detected features are marked by the green crosses.........47 20 Position North for the vehicle when using vision and IMU only for vehicle pose estimation.The Vicon values in this plot have been bias shifted by 0.425 ft.............................48 21 Position East for the vehicle when using vision and IMUonly for vehicle pose estimation.The Vicon values in this plot have been bias shifted by -0.175 ft.................................49 xi

22 Position Down for the vehicle when using vision and IMU only for vehicle pose estimation.The Vicon values in this plot have been bias shifted by 0.25 ft..............................49 23 Roll attitude for the vehicle when using vision and IMU only for vehicle pose estimation...............................50 24 Pitch attitude for the vehicle when using vision and IMU only for vehicle pose estimation...........................50 25 Yaw attitude for the vehicle when using vision and IMUonly for vehicle pose estimation.The Vicon values in this plot have been bias shifted by -5.7 degrees...............................51 26 Number of points used in estimation.This depends on the number of feature points in the camera’s field of view,and which points the state estimator expects to be in the field of view................51 27 Placing the image processing targets on the ground (configuration A) provided markedly better results than when the target was placed up- right on a wall (configuration B).This is because when the camera is looking down at the target,then the gravity vector assists in determin- ing the relative pose of the vehicle....................52 28 Multiple viewpoints from a single camera of a target on the ground. This illustrates the effect of varying light conditions on the feature point detector since the water lying in the ditch causes reflections at specific angles of incidence with respect to the Sun.Several false fea- tures are generated by these reflections..................55 29 Demonstration of the removal of points fromthe database using sample images from a previously recorded flight.The points generated from light reflections off water in a nearby ditch are removed after they have not been seen for several frames......................56 30 The estimated,actual,and commanded positions for the aircraft during a loss-of-GPS scenario expressed in North,East,and altitude coordi- nates.At t = 555 sec,the GPS is ignored from the navigation solution so that only the IMU and monocular vision sensor are being used for determining the location of the helicopter for closed-loop control...58 31 The error between the estimated and actual position of the helicopter during the simulation run.........................59 32 The GTMax rotorcraft UAV is a modified Yamaha R-Max helicopter equipped with custom avionics.For testing the vision-based algo- rithms,a machine vision progressive scan camera is used for acquiring images.Two onboard computers divide up the tasks of image process- ing and estimation.............................60 xii

33 Block diagramillustrating the layout of the dual computers on the GT- Max.Image processing is performed on a secondary computer whereas a primary flight computer handles the estimation and control for the aircraft...................................60 34 The estimated,actual,and commanded positions for the GTMax dur- ing a flight-tested loss-of-GPS scenario expressed in North,East,and altitude coordinates.At t = 96 sec,the GPS is ignored from the navi- gation solution so that only the IMU and monocular vision sensor are being used for determining the location of the helicopter for closed-loop control...................................62 35 Simulation setup used for testing the vision-based mapping and nav- igation filter.A checkerboard pattern is on the face of a building as a sample target.The helicopter UAV hovers in front of the target to map it and then switches over to vision-aided inertial navigation...65 36 A sample output from the stereo image processing taken during a sim- ulation run.The left window shows the locations of the feature points found fromthe corner detector,and the right window shows a grayscale representation of the computed disparity map..............65 37 The estimated,actual,and commanded positions for the aircraft during a loss-of-GPS scenario with stereo vision expressed in North,East,and altitude coordinates.At t = 166 sec,the GPS is ignored from the navigation solution so that only the IMU and stereo vision sensor are being used for determining the pose of the helicopter for closed-loop control...................................66 38 The error between the estimated and actual position of the helicopter during the simulation run using stereo vision..............67 39 An attitude error quaternion can be used to express the incremental difference between a tracked reference body frame and the actual body frame for the vehicle............................77 xiii

SUMMAR Y Traditionally,the task of determining aircraft position and attitude for automatic control has been handled by the combination of an inertial measurement unit (IMU) with a Global Positioning System (GPS) receiver.In this configuration,accelerations and angular rates from the IMU can be integrated forward in time,and position up- dates from the GPS can be used to bound the errors that result from this integration. However,reliance on the reception of GPS signals places artificial constraints on air- craft such as small unmanned aerial vehicles (UAVs) that are otherwise physically capable of operation in indoor,cluttered,or adversarial environments. Therefore,this work investigates methods for incorporating a monocular vision sensor into a standard avionics suite.Vision sensors possess the potential to ex- tract information about the surrounding environment and determine the locations of features or points of interest.Having mapped out landmarks in an unknown envi- ronment,subsequent observations by the vision sensor can in turn be used to resolve aircraft position and orientation while continuing to map out new features. An extended Kalman filter framework for performing the tasks of vision-based mapping and navigation is presented.Feature points are detected in each image using a Harris corner detector,and these feature measurements are corresponded from frame to frame using a statistical Z-test.When GPS is available,sequential observations of a single landmark point allow the point’s location in inertial space to be estimated.When GPS is not available,landmarks that have been sufficiently triangulated can be used for estimating vehicle position and attitude. xiv

Sim ulation and real-time flight test results for vision-based mapping and naviga- tion are presented to demonstrate feasibility in real-time applications.These methods are then integrated into a practical framework for flight in GPS-denied environments and verified through the autonomous flight of a UAV during a loss-of-GPS scenario. The methodology is also extended to the application of vehicles equipped with stereo vision systems.This framework enables aircraft capable of hovering in place to main- tain a bounded pose estimate indefinitely without drift during a GPS outage. xv

CHAPTER I INTRODUCTION This thesis focuses on the application of vision sensors to unmanned aerial vehicles (UAVs) for the tasks of navigation and environment mapping.Navigation or local- ization is the problem of figuring out where in space a vehicle is actually located and how it is oriented.This information can then be used for the closed-loop control of an aircraft through an autopilot system or for guiding a vehicle from one destination to another.Mapping refers to the problemof determining the relative locations of points or structures of interest in the surrounding environment which can be used for obsta- cle detection and avoidance or relative navigation.In this work,a method for how a monocular vision sensor can be fused with and inertial measurement unit (IMU) in an extended Kalman filter framework for navigation and mapping is proposed.Here, a Harris corner detector extracts feature points from each captured image,and a statistical Z-test is used to correspond features from frame to frame.Results demon- strating the application to the autonomous flight of a UAV in a loss-of-GPS scenario are presented to validate the method. The framework presented here consists of a vision-based mapping phase and a vision-aided inertial navigation portion.During nominal operations of a UAV when GPS is available,vision-based mapping can be performed to estimate the locations of landmark points in the surrounding environment.These landmarks are mapped out so that they can in turn be used for estimating the pose of the aircraft should GPS be lost.In the event of GPS outage,vision-aided inertial navigation using observations of the mapped out landmark locations to estimate the position and attitude of the aircraft.For aircraft that are capable of stationary hover,such as rotorcraft,the 1

framew ork allows the vehicle to maintain hover indefinitely with a bounded drift in the state estimate when GPS is unavailable. 1.1 Motivation The combination of an IMU with a Global Positioning System (GPS) receiver has typically been used to determine the position and attitude for an aircraft.In this con- figuration,accelerations and angular rates from the accelerometers and gyroscopes of the IMU can be integrated forward in time,and position updates from the GPS can be used to bound the errors that result from the integration.This solution to the navigation problem makes aircraft prone to certain modes of failure due to their re- liance on the reception of external signals from the GPS network.GPS signals can suffer from obstructions or multipath in cluttered environments,and the reception of these signals can furthermore be jammed or otherwise denied.Similarly,the task of mapping the surrounding environment is commonly approached by using ranging sensors to scan areas of interest.However,these sensors typically rely on the emission and reception of a signal to determine range which can be undesirable if the vehicle needs to remain undetected.Range sensors capable of functioning over longer dis- tances also tend to be large and bulky limiting their applications to smaller aerial vehicles. Vision sensors have demonstrated immense potential for application to localiza- tion and mapping since they provide data about the surrounding environment and simultaneously allow for the possibility of inferring information about vehicle motion from these images.However,the majority of results presented in these areas have been applied to ground robots where size and payload considerations are often not a limitation.Ground robots also have the option to stop motion to safely process information and plan out the next move,whereas aerial vehicles operating in 3D space could crash without pose updates during that time.Over recent years,it has 2

b een proposed that adding an IMU to a vision system could help to assist with these algorithms because the inclusion of inertial sensors allows for the prediction of cam- era motion from frame to frame and also helps with resolving scale ambiguity.A navigation system that is capable of functioning with only a combination of inertial and vision sensors would be a fully self-contained one that would also not be prone to jamming or detection. Figure 1:Small unmanned aerial vehicles,such as the Hornet UAV shown above, are often not capable of carrying large computer systems. There are currently several small UAVs that could benefit from the integration of vision sensors for navigation and mapping purposes.Many of these vehicles in fact already include camera systems since small UAVs are often used for aerial imaging and surveillance.One such example is the Hornet UAV shown in 1 which is capable of close-quarters flying in urban environments.Operating in urban canyons means that this type of vehicle could potentially be prone to GPS dropouts due to obstructions or poor performance from multipath.This small rotorcraft UAV has a miniature autopilot systemthat was developed at Georgia Tech called the Flight Control System 20 (FCS20) [13] as shown in Figure 2.The FCS20 consists of a processor board 3

and a sensor board.The processor board contains a signal processor (DSP) and a field-programmable gate array (FPGA).The DSP is a high rate processor that is responsible for performing primary guidance,navigation,and control algorithms. Meanwhile,the FPGA performs the low level hardware interfacing with its numerous digital inputs and outputs and its ability to perform parallel operations.The sensor board has a three axis IMU with accelerometers and gyroscopes,a GPS,and absolute and differential pressure sensors.The FCS20 also has the ability to grab digital images from a high resolution CMOS image sensor.The combination of the camera interface and the FPGA’s high rate and parallel processing capabilities make this a potential platform for vision-based algorithms for small-scale UAVs. Figure 2:The FCS20 is a small autopilot system with a DSP and FPGA along with integrated sensors including an IMU,GPS,and pressure sensor. 1.2 Literature Review Many researchers have been investigating the use of vision for localization and map- ping onboard UAVs.Some of the initial work in the area started out with the use of stereo vision systems [31].Stereo systems have seen much success in the areas of vision-based control for ground vehicles,so it seems a natural extension to apply them for use in aerial vehicles [39],[34],[45].Depth information for a given stereo frame 4

can be calculated fromthe disparity between two images.Given the baseline distance between two aligned cameras,the difference in horizontal pixel location of a point in both the left and right images allows the distance to the point to be computed.How- ever,stereo vision systems can be complex due to the amount of image data that needs to be processed as well as the careful camera calibrations and hardware syn- chronization that needs to be performed.Stereo systems also rely on a minimum baseline distance between the two cameras to provide enough disparity between the two images for adequate ranging information,which may not be practical for many small UAVs. Some of the earlier results in the area have also included the use of vision to assist in the landing of aerial vehicles such as in [53],[40],[49],and [48].This is a great use for vision since landing is really a relative navigation problem,especially on mobile platforms such as aircraft carriers or other moving vehicles [5].Addressing the landing problem also typically means that some knowledge of the environment is available beforehand since the vehicle is usually returning to its original point of deployment. Using the known coordinates or size of targets simplifies the use of monocular vision systems since distance information is implicit in the relative positioning or sizing of features [50],[41].Researchers have also tried constructing customtargets specifically designed for these applications that allow full pose of the aircraft to be determined from a single observation [60]. Others have tried to take advantage of structured environments for vision-based navigation and mapping with aerial vehicles.If some knowledge of a given envi- ronment is available beforehand,then the vehicle can look for more complex shapes than just point features.Most work that used knowledge of the environment’s struc- ture has used image processing that would look for lines or squares,both of which are abundant in urban and man-made environments [12].The use of complex fea- tures tends to make the measurements more robust to noise when compared to point 5

features. However,the image processing also tends to be more computationally in- tensive,and the requirement that these features be available and abundant enough can be problematic. An alternative to the use of specific structures in a given environment is to use point features in an environment [46],[14].Feature points have the advantage that most general environments are capable of providing a rich set of features since point features represent extrema on a very local scale.However,the use of feature points with a monocular vision system for the purposes of navigation and mapping can be difficult for several reasons.First off,observation of a point feature by a single camera only provides the relative bearing to the target.This is because the range information is lost from the projection of the point in 3D space onto the 2D image sensor plane. Furthermore,point features are more sensitive to image noise and distortion because less information is available in each feature point when compared to more complex features.This also makes it more difficult to track or correspond features from frame to frame.These are the challenges that need to be addressed and will be investigated in this work.Researchers have also applied vision sensors to the obstacle detection and avoidance problems [11],[52],[54].Some have also integrated the trajectory generation of the vehicle together with the visual sensing to optimize the motion of the vehicle for observations from differing viewpoints,such as in [63] and [8],in order to overcome the lack of range information available from monocular vision systems. Some have investigated vision-based simultaneous localization and mapping (SLAM) for aerial vehicles [32],[10],[64],[56],[25] [44].In the SLAM problem,both naviga- tion and mapping are performed at the same time so that the vehicle needs to figure out where it is while trying to figure out where the features in space are.The work presented here takes a step back from this problem and instead just looks at the indi- vidual problems of localization and mapping separately since there have been limited real-time hardware results in these areas alone.Vision-only algorithms have also been 6

in vestigated for the autonomous operation of aerial vehicles such as in [47] and [42]. Optical flow is another vision-based measurement that has been utilized.The optical flow of features from frame to frame is the displacement of a given feature point in the image plane.This represents a velocity type measurement that depends on the relative distance between the camera and the target landmark point.Optical flow has been investigated for aiding with navigation,such as in [59],[30],[55],and [18] as well as for obstacle detection and avoidance [61],[43],[51],[19],[23],and [22]. 1.2.1 Summary of Specific Works There is extensive literature covering the use of vision for mapping and navigation for aerial vehicles currently available.This section summarizes a handful of related works that have already been published to provide a general idea of progress that has been made in the area. One of the earlier works to appear in the area of vision-based navigation for UAVs was Amidi [4] which presented the development of a visual odometer for flying an autonomous helicopter over flat ground.The visual odometer used a stereo pair of cameras pointed at the ground together with angular rate measurements from gyroscopes to determine the position of the aircraft.The method involved first finding a template,and then tracking the template from frame to frame.Since a stereo pair was used,the relative location of the template with respect to the aircraft could be found from the range information.Position information was backed out from the template track by first removing the vehicle’s rotation by using the gyroscopes,and then using the displacement of the template in each image.Flight results using a constrained testbed were presented in addition to results from an outdoor helicopter. Koch et al.have presented work where a downwards looking camera for the closed- loop flight of a helicopter UAV when GPS is lost in [33] which builds upon work done in [7] with the same flight platform.A Lucas-Kanade feature point tracker was used 7

to detect and correspond features in each image.The location of the landmarks in the environment were found by assuming them to lie on the ground plane.When GPS is lost,then points that have been located can be used for localizing the aircraft in the absence of GPS.Real-time flight test results demonstrated their work. In [35],Langelaan used an unscented Kalman filter for simultaneously estimating vehicle states as well as landmark locations in inertial space.A Mahalonobis norm was used as a statistical correspondence for data association from frame-to-frame for each estimated landmark.New landmarks were initialized in the filter by performing a projection onto the ground plane.Simulation results for a UAV navigating through a 2D environment were presented. In [38],Madison et al.presented an EKF design where the vehicle states were being estimated as well as the inertial locations of features in 3Dspace.Their research presented a method which would allow a vehicle with a traditional GPS-aided inertial navigation system(INS) to fly along and estimate the locations of features,and then in the absence of GPS,use these features for aiding the INS.Features were selected and tracked using a Lucas-Kanade feature tracker.A few different methods for initializing the estimated locations of tracked features were implemented.The authors opted for a method of motion stereo where multiple observations of a new feature are first taken, and then the 3–D location of the point is computed using a least-squares solution to find the intersection of the observation rays.Simulation results for a vehicle that momentarily loses GPS were provided.Authors in [3] later built upon this work to fly a quadrotor vehicle autonomously in real-time using offboard processing.A reactive vision-based obstacle avoidance scheme was also incorporated into the system. The authors in [58] implemented a Harris feature detector along with a random sample consensus (RANSAC) algorithm for the correspondence of features in an FPGA.In this work,Fowers et al.used the FPGA to also read in digital images from a digital CMOS camera.This was used for providing drift corrections to an INS 8

to stabilize a quad-rotor vehicle for short-term hover in an indoor flight environment. By assuming that the the vehicle remains relatively level,the RANSAC algorithm was used to provide estimates of the translation,yaw rotation,and change in scale. RANSAC is a model fitting algorithm where a collection of points are fitted against a model,and the points are sorted into groups of inliers and outliers.These estimated values were then used to provide drift correction measurements to the integration of the IMU. Frietsch et al.in [17] presented an application of vision to the hovering and landing of a quad-rotor UAV.These authors presented a method for estimating the above ground level altitude without the need for ranging sensors by utilizing the optical flow from a vision sensor in conjunction with a barometric altimeter.The proposed method used a Lucas-Kanade tracker to perform the detection and tracking of features from frame to frame.A RANSAC algorithm was then used to estimate a homography that relates points on the ground plane as viewed from two different perspectives.To figure out the distance of the ground plane from the vehicle,the net optical flow of the scene was combined with readings from a barometric altimeter. Since this work was intended for the applications of hovering and landing,it was assumed that the camera’s motion was dominated by rotation and that the scene was planar.Simulation results of the method were provided. Watanabe presented a method for 3Dobstacle reconstruction froma 2Dmonocular vision sensor in [62].By using image processing to detect line segments in each frame,an EKF was used to estimate the locations of the lines in 3D space assuming knowledge of the vehicle’s position and attitude.Data association of lines from frame to frame was handles using a statistical Z-test correspondence.Simulation results for obstacle detection and terrain mapping were provided. In [65],Webb et al.presented an implicit extended Kalman filter that used the epipolar constraint of features from frame to frame to estimate vehicle states.The 9

Full document contains 105 pages
Abstract: Traditionally, the task of determining aircraft position and attitude for automatic control has been handled by the combination of an inertial measurement unit (IMU) with a Global Positioning System (GPS) receiver. In this configuration, accelerations and angular rates from the IMU can be integrated forward in time, and position updates from the GPS can be used to bound the errors that result from this integration. However, reliance on the reception of GPS signals places artificial constraints on aircraft such as small unmanned aerial vehicles (UAVs) that are otherwise physically capable of operation in indoor, cluttered, or adversarial environments. Therefore, this work investigates methods for incorporating a monocular vision sensor into a standard avionics suite. Vision sensors possess the potential to extract information about the surrounding environment and determine the locations of features or points of interest. Having mapped out landmarks in an unknown environment, subsequent observations by the vision sensor can in turn be used to resolve aircraft position and orientation while continuing to map out new features. An extended Kalman filter framework for performing the tasks of vision-based mapping and navigation is presented. Feature points are detected in each image using a Harris corner detector, and these feature measurements are corresponded from frame to frame using a statistical Z-test. When GPS is available, sequential observations of a single landmark point allow the point's location in inertial space to be estimated. When GPS is not available, landmarks that have been sufficiently triangulated can be used for estimating vehicle position and attitude. Simulation and real-time flight test results for vision-based mapping and navigation are presented to demonstrate feasibility in real-time applications. These methods are then integrated into a practical framework for flight in GPS-denied environments and verified through the autonomous flight of a UAV during a loss-of-GPS scenario. The methodology is also extended to the application of vehicles equipped with stereo vision systems. This framework enables aircraft capable of hovering in place to maintain a bounded pose estimate indefinitely without drift during a GPS outage.