University of California
Informatics and GIS Program

IGIS Blog

Working with Climate Data - Workshop

Have you ever wanted to work with climate change data?  Last week I took a workshop from the UC Berkeley Geospatial Innovation Facility (GIF) titled "Working with Climate Data (link).  This workshop showed how to work with climate data hosted by the GIF in their Climate App (Cal-Adapt) and other climate data hosted as NetCDF datasets.  Through this workshop we learned how to use jupyter notebooks and the python language to query and analyze these types of data.  The workshop also highlighted the use of NetCDF data within ArcGIS Pro and its new multidimensional charting tools.

I recommend this training and other trainings held at the GIF.  I look forward to applying these new tools in my work with IGIS.

The image below is the Growing Degree Days calculation for 2013 with a 10 degree C base, that I was able to develop with techniques used at this workshop.

Posted on Thursday, May 16, 2019 at 12:09 PM
Focus Area Tags: Natural Resources

IGIS Recognized for Drone Data Management Tools at Geography Conference

At the recent conference of the American Association of Geographers (AAG) in Washington DC, IGIS was recognized for its work in developing open source data management tools for drone data. The AAG is the largest association of geographers, GIS specialists, environmental scientists in the world, and its annual conference attracts over 8,500 people and over 6,900 presentations. IGIS researchers Andy Lyons, Jacob Flanagan, and Sean Hogan won second place in a poster competition sponsored by the AAG Remote Sensing Specialty Group and photogrammetry company Pix4D.

Data management is not the most exciting aspect of working with drones, but it's an essential component when using drones for research mapping. In drone mapping, the drone flies a crisscross pattern over the study area, taking pictures at regular internals. When the flight is over, you have to get the images off the drone's camera, and feed them into photogrammetry software that stitches them together. The software also removes all the overlap, corrects for perspective and distortion, and generate 3D models of the terrain. An enormous amount of data is involved at each step of the process.

Data management involves protocols and file transfer utilities to help you organize data, assess quality, create backups, find and retrieve files, and reproduce workflows. Photogrammetry programs are brilliant at stitching photos into beautiful high resolution imagery, but they don't offer many tools for managing data before or after the processing. This is the gap IGIS is addressing.

IGIS has been working on data management tools for the past couple of years to help us manage the dozens of drone projects we conduct each year on behalf of UC researchers. In keeping with our public mission and research innovation focus, we use open source programming platforms such as Python and R and share our tools on GitHub so anyone can use them.

The poster presented at the AAG conference showcases 5 data management projects. The foundation of all our projects is a directory structure and file naming protocol that encompasses all the types of data in a drone project, including images, ground control points, GIS layers, documentation, intermediate files, and final outputs. Building upon this, the uavimg package for R creates offline HTML catalogs and maps of images, allow for quality control checks in the field and a master catalog to help an analyst find the right set of images for a project. The IGIS Drive Monitor, written in Python, is designed to run on a laptop in the field and automates the process even further by monitoring a USB drive and copying files into the right place automatically. The IGIS Pix4D Controller, also written in Python, runs on a server and automates the next step of the process, launching the Pix4D stitching software when a new set of images is detected, creating a new project, and initiating the stitching process. Finally, the IGIS Drone Data Management Logbook, still being developed, combines functionality of the previous utilities with additional visualization tools for quality control.

Our drone data management utilities are still under development, but they are available currently available for testing (see GitHub links in the poster). If you'd like to be a beta tester, please let us know!). Later this year, we plan on holding a webinar describing how to use these tools and invite feedback from users. Open source software is designed for collaboration, and our ultimate hope is to collaborate with other drone users and programmers facing similar needs. In the meantime, it's great to be recognized by leading experts in the field.

 

Posted on Friday, April 19, 2019 at 8:01 AM
Tags: data management (1), drones (19), IGIS (61)

Maggi's Sabbatical in China in the spring...

Sabbatical report April 2019

I've been on sabbatical now for a few months, and it's time to report. I've been working on updating all my course materials: slides, reading and labs, for the fall. This has been a blast, and a lot of work! Especially the labs. We are finally moving to ArcGIS Pro, people! It's been scary, but thanks to some excellent on-line resources, including this list of excellent tutorials from Jarlath O'Neil-Dunne and from ESRI (Getting started with Pro) we are making progress. Shane Feirer and Robert Johnson from #IGIS are helping here too, and we'll likely be using some of the new material in IGIS workshops soon. 

Currently, I am in China, visiting former PhD student Dr Qinghua Guo and my “grandstudent” Dr Yanjun Su at their lab set in the bucolic Institute of Botany northwest of Beijing (just outside the 5th ring, for those of you in the know). It has been a blast. I came to catch up on all the excellent UAV, lidar, remote sensing, and modeling work going on in the Digital Ecosystem Lab at the Institute of Botany (part of the Chinese Academy of Sciences). These students are serious Data Scientists: they are working on key spatial problems and remote sensing data problems using ML, classification, spatio-temporal algorithms, data fusion tools. They routinely work with lidar, hyperspectral, multispectral and field data, and focus on leaf-scale to landscape-scale processes. One of the big experiments they are working on uses a new instrument, dubbed “Crop3D”. It is a huge frame installed over an ag field with a movable sensor dock. The field is about 30m x 15m, and the sensor can move to cover the entire field. Here is my summary in graphic form:

This season's experiment focuses on mapping corn plant phenotypes using hyperspectral, RGB, and lidar data by classifying leaf-scale metrics such as leaf angle and branching angles, along with spectral indices. VERY COOL STUFF. I am eager to hear more about the results of the experiment and see what is yet to come. 

I gave a couple of talks, one on “big” (serious air quotes here) data and ecology (to the Institute of Botany at the Chinese Academy of Sciences) and one on UAVs (to the Institute of Geographical Sciences and Natural Resources Research, CAS). In both I highlighted all the excellent work done by students and staff in my various research and outreach groups. In the first I focused on our Lidar work in the Sierra Nevada (with Qinghua Guo, Yanjun Su, Marek Jakubowski); the VTM work and FAIR data (with Kelly Easterday); and UAV/water stress (with Kelly again plus Sean Hogan and Jacob Flanagan). In the second talk I got to gush about all the IGIS work we are doing across our “Living Laboratories” in California. We have flown ~30 missions (total 25 km2) on and around the network of research properties in California (see the panel below for some examples). I talked about the recent CNN work with Ovidiu Csillik; the BORR water stress experiment with Kelly, Sean and Jacob; the fire recovery work at Hopland with Shane Feirer and the rest of the IGIS crew; and the outreach we do like DroneCamps. I also talked about UAV Grand Challenges: Scaling, Sampling, and Synergies. Those ideas are for another post. 

My hosts took great care of me: Showing me the sites, making sure I tried all the regional delicacies, and indulging me in my usual blather. Below are some pics of us on our adventures, including in the bus on our way to a distant portion of the Great Wall. Walking the Wall was: 1) awesome (in the real sense of the word – it really is mind-blowing); 2) STEEP (calves were screaming at the end of the day); and 3) windy. Plus there are snakes. I was told that there are other sections of the Wall that are called “Wild Wall” which I think is extremely cool. And speaking of walls, GOT starts again this weekend. China in springtime is BURSTING with flowers. And being housed at the Institute of Botany means all of them are on show in a concentrated area. Finally, you can get all over this huge country on trains. Trains that go really fast (220 MPH), and are on time, and are comfortable! I went to Shanghai (800+ miles away) for the weekend by train! My current joke: “In China, it takes 4 hours to get from Beijing to Shanghai. In California, it takes 4 hours to get from Berkeley to Sacramento.” (Thanks Dad!)

Off to Tokyo. But not before a final panel of pics that remind me of this trip: Technology, Art, Food, Flowers, Shopping, History. Here in China, Red = Happiness + GoodLuck, not the Cardinal.

Posted on Monday, April 15, 2019 at 3:57 PM

Esri Developer Summit 2019

Dev Summit 2019

The Esri Developer Summit in Palm Springs was, as usual, a very informative look into the resources Esri provides for those who want to take their software to the next level. Although this year's conference was a bit light on major announcements (they're probably saving all the good stuff for the User Conference in July) there were still a few interesting takeaways:

  • ArcGIS Notebooks are continuing the trend of tight integration of Python with the ArcGIS suite. While currently available through ArcGIS Enterprise portal, the next release of ArcGIS Pro is going to support Python Notebooks directly in the desktop environment.
  • Web apps are becoming lighter, more efficient and more mobile-friendly. Progressive web app standards are making it easier to develop functional, responsive web apps rather than needing to delve into the more complicated world of native apps.
  • ArcGIS Pro is finally going to support publishing to standalone servers in the next release! Esri is framing this as a tool for transitioning to Portal servers, but those of us that still use standalone servers will take it.
  • ArcMap really has become obsolete. While Esri is still pledging to support it for "years to come", all new functionality will only be available in Pro.

All in all, the Dev Summit is a great resource for keeping up to date on all the cutting-edge technology that Esri has to offer. It's a great learning experience and I look forward to attending again in the future.

Posted on Tuesday, April 9, 2019 at 9:11 AM
Tags: ArcGIS Pro (4), Dev Summit (2), ESRI (13), IGIS (61), Python (2)

Salinas Ag Tech Summit

Ag Tech Summit

2019.03.26 - 27 in Salinas, California - The Epicenter of AgTech

The Ag Tech Summit at Hartnell College showcased people's passion for agriculture and the ability to integrate new technologies for improved quality and production. It was a chance for growers, researchers and technology providers to connect and reimagine an better future for propagating industry, providing to our world population and everyone's concern with making food healthier and more organic.

Some Topics covered included:

  • Remote Sensing
  • IPM Approaches
  • Managing Virus Spread
  • Automation for SGMA Groundwater Management
  • Traceability for Managing Food Safety Risk
  • Big Data in Ag

[Below Image: Morning Keynote - Bob Whitaker, Produce Marketing Association]

Remote Sensing

The remote sensing panel included researchers and industry from companies like Ceres Imaging, AirSpace Integration and CSU Monterey Bay. It was insightful to hear how RS, especially those using drones as a platform, have been used in the Agriculture sector and in what ways Ag is benefiting from it.

Traceability for Managing Food Safety Risk

New systems are being explored and adoptions from companies like UPS that help track food found at the supermarket back to its origin. This is useful in cases of food related outbreaks which can be mitigated its impact on the consumer and the industry. This panel explained that even a simple note taking application in field has high reward for targeting reasons for outbreak.

Big Data in Ag

This panel covered how big data can speed up agronomic decisions through analysis of traceability systems, remote sensing and other technology in correlation with market analysis. A very notable company was Project Athena that is using artificial intelligence to analyze big data and turn it into information.

Gabe Youtsey - UC ANR

Gabe attended the Ag Tech Summit as a keynote speaker and panel member. During his keynote, he spoke about the progress of VINE and UC ANR's commitment in improving California's Ag. He was part of the "How California is Strategizing to Meet the 2020 Workforce Needs" panel sitting in with other members from Cal Poly, CSU Monterey Bay and Ca Community College Chncellor's Office.

[Below Image: Afternoon Keynote - Gabe Youtsey, Chief Innovation Officer, UC ANR]

Given all these outlooks, I'm looking forward to 2020's Ag Tech Summit.

Posted on Monday, April 1, 2019 at 1:15 PM

Next 5 stories | Last story

 
E-mail
 
Webmaster Email: stfeirer@ucanr.edu