I tried to combine some ideas building from our OpenTopography project with nascent ML collaborations with Dr. Zhiang Chen. Thanks for their contributions.
Opportunities associated with AI/ML are really exciting, but there is a lot to think about for infrastructure, research, and education.
Our OpenTopography project was honored at this year's AGU with the Open Science Recognition prize: "For outstanding contributions in cyberinfrastructure, data management, training, and outreach associated with open-access high-resolution topography." It is a great honor and nice reognition for more than 15 years of work by our team. Huge thanks to Roman DiBiase for leading the nomination and for the letter writers Mike Oskin, Paola Passalacqua, and Josh Roering.
Chelsea Scott and Chris Crosby made a nice presentation summarizing our efforts. The recording is here: link.
It was also nice to meet the other winner, Tasha Snow and appreciate her efforts. In particular, I really appreciated her articulation of Open Science values (image from her presentation):
Understanding and interpreting rapidly changing Earth surface processes across a template of a rapidly urbanizing and increasingly connected world is a major challenge. Our ability to observe and measure features on the Earth surface is increasing in quality, resolution, and temporal repeat. Thus, we have an opportunity to move understanding beyond the assumption of steadiness. It is clear that many surface phenomena occur rapidly (e.g., wildfires and the subsequent drainage network response, volcanic eruptions, earthquakes, mass movements, etc.). Each is part of a cascade of precursory and subsequent processes.
Revolutions in Earth observing and the connectedness of humanity (e.g., internet of things, social media) provide a major opportunity to characterize surface process events across the world. But they also provide a great data discovery, integration, and analysis challenge. How to bring the disparate observations into a common and quantitative 4D framework so they can be examined, and rates of change measured? For example, satellite imagery such as that available from Planet.com provide a daily view of the Earth’s surface at <5m per pixel. This temporal and spatial resolution enables us to observe many phenomena at or near the spatial and temporal scales at which critical processes operate. Discovering these data is relatively straightforward, but their rapid integration with data for context, as well as with ground observations is difficult and time consuming. Integrating the synoptic view from the space-based platform with the typically less intentional but ubiquitous eyewitness views from social media posts or public image sharing platforms can provide essential ground truth, detailed observations of phenomena, and an indication of the human experience of the event.
For many events, a generic workflow can be imagined:
Build on the well curated contextual geospatial data (landcover/land use, topography, imagery, 3D structures) to include (with proper geo- and temporal referencing) near real time Earth observations and compute if necessary derived products of interest (e.g., NDVI for vegetation health).
Discover, locate in time and space, assess for veracity, and examine user contributed or freely posted images and videos from the ground (and maybe from UAS).
Use the high geodetic accuracy of the framework to measure changes using space-based, airborne, and ground-based data. These changes may be spectral or 3D. Use the measurements to contribute to process-based models.
Present predicted conditions (e.g., hazard maps and forecasts) potentially in a updating cycle defined by subsequent additional observations.
Educate both in the short term (explain the event) and long term (enhance science and engineering literacy).
Several uses cases are evident:
Wildfires: Map forest health using space imaging (NDVI) before, during, and after wildfire seasons. Much commercial space imaging is well configured for measuring vegetation vigor. Exposure of the built environment to the fires (and to subsequent debris flows) would be easy to explore, and hazard maps easily visualized. Including user contributed or freely posted images and videos from the ground (and maybe from UAS) during the fires and after would provide a sense of the detailed processes and phenomena.
Debris flows in mixed wild-agricultural-urban environments: Flooding, especially by heavy sediment-laden flows, are hazardous and their conveyance highly sensitive to the complex 3D near surface environment which many include natural and built structures. Observations of them include larger watershed scale activation and evolution during storm events (space and airborne observations potentially combined with very high resolution 3D data from as built urban models and with on ground experiences from mobile phone picture and video). Flow simulations are available and may be useful for forecasting hazardous conditions, and also maybe updated and calibrated with detailed observations.
Tsunami inundation in complex coastal environments: The 2011 Tohoku Japan earthquake and tsunami showed the very complex and rapid large scale interaction of the rising waters and the coastal Japanese environment. These were observed by some airborne and many haphazard ground-based views. Integrating those observations, and georeferencing imagery to help to measure inundation depths and flow velocities could be done with value for fluid dynamics simulations as well as for tsunamic education for coastal communities.
ETC.
I wanted to capture this text that I contributed to a recent proposal and stash it here.
This is part of some ideas that I have been working on with capstone students in our capstone class. Here is a link to a presentation: LINK.
Of course, one of the really nice things that the OpenTopography Community Dataspace publishing of one's data allows is to mint a DOI. That DOI allows then for a data citation. I have added a new part of my CV that has a section on data publication. Here is an example citation style: Arrowsmith, J R., DiMaggio, E. N., Garello, G. I., Villmoare, B. and LediGeraru Research Project (2018): Photogrammetric model of a portion of the LeeAdoyta Basin, Afar, Ethiopia (point cloud [122M points], orthophoto [2cm/pix], and DEM [25 cm/pix]). Distributed by OpenTopography. AccessedOctober 23, 2018. https://doi.org/10.5069/G95X271W.
The conclusions are useful to highlight as well:
We are part of a revolution in 3 and 4 D data collection and analysis
Additional needs for the community include
Optimized data acquisition strategies
Low cost and high performance computation of point clouds and models
Efficient and accurate georeferencing
High quality differencing for change detection
Bring the tools and data into the (outdoor) classroom; need more curriculum (c.f. GETSI - GEodesy Tools for Societal Issues (UNAVCO) at https://serc.carleton.edu/getsi/)
OpenTopography Community Dataspace
Great opportunity to expand the impact of emerging topography through improved access
Services and existing community of users
Community engagement & best practices
Please join us and start sharing your models and ideas for how to improve
Video to images. The main generic challenge for SfM from video is to extract the video frames and prepare them for the SfM. The SfM part is no different from what my group has been doing for a while with Agisoft Photoscan. I used MATLAB to do the video processing. The script is here: readplanetvid.m. The main code bits include:
PlanetObj = VideoReader(videoname); %make a video object from an MP4 file
vidWidth = PlanetObj.Width; %get the width
vidHeight = PlanetObj.Height; %and height
mov = struct('cdata',zeros(vidHeight,vidWidth,3,'uint8'),...
'colormap',[]); %set up a MATLAB structure to contain the video
k = 1;
while hasFrame(PlanetObj)
mov(k).cdata = readFrame(PlanetObj); %pull out the frames one at a time from the MP4 object and put them in the mov
k = k+1;
end
step = floor(k/number_of_frames) %determine how many frames to skip each time to get the desired number
for i = 1:step:(k-1)
framepart = sprintf('_frame_%06d.png', i);
filename = strcat(foldername,'/',projectname,framepart);
imwrite(mov(i).cdata, filename) %easy to write the frame out as a png file
end
Satellite video from Terra Bella. I have been watching the hi resolution satellite activity with great interest. Skybox had a few relatively high res (approx 1 m ground resolution) visible and near IR satellites with video capability. They were bought by Terra Bella (google) and then now are owned by Planet (who were just visiting us on the ASU campus last week and with whom we are building some collaborations). Some of the Terra Bella imagery is available on youtube. I grabbed one video of the Usak Mine in Turkey (used real player to convert youtube to mp4):
You can really see the parallax as the satellite moves over (not to mention the activity of the vehicles).
I ran my script on the mp4 and extracted 100 png frames. Here is an example:
I ran the files through the Agisoft Photoscan sequence of alignment (high), build dense cloud (medium), build mesh (medium), and build texture (medium). Here are a few screen captures of the result:
You can see the model and the camera positions. They are in the roughly correct arc, and relatively far away, but they should be much farther (orbit is approx 450 km).
Nice looking textured mesh. It is distorted, but not too bad, all things considered!
What did we learn? We learned that the SfM from video is doable (see a future post from my backyard and phone video). Here is the Photoscan report on the Usak project. The geometry that is computed from the satellite video is not bad. Agisoft Photoscan does a pretty good job. We cannot get under the hood very easily to see more about the processing. I think that someone who knows more about computer vision than me would be able to comment as to the performance. I think that the main issue is probably the relatively low angular variation for the model.
Planet Team (2017). Planet Application Program Interface: In Space for Life on Earth. San Francisco, CA. https://api.planet.com.
