Posts Tagged: Big Data
Today was a great day at the ESRI User conference. I attended several sessions covering the use of raster and vector data in big data analyses, the use of python and r for data science, and the use of arcpy to create and modify maps in ArcGIS Pro.
Speaking of big data and arcgis, over the past 5 years I have watched ESRI develop their software from a desktop environment to a distributed computer platform capable of analyzing millions of spatial features in minutes and hours instead of days or weeks. These tools are now accessible to the average users of arcgis without the need to understand the underlying big data frameworks and software. We now have the ability to install tools such as GeoAnalytics Server, the Spatial Temporal Datastore, and ArcGIS Pro and take advantage of big data analytics within ArcGIS Pro. By using these tools we do not have to know how to setup and manage tools like apache spark, elasticsearch, or other tools.
The above tools are used to store and analyze vector data. To analyze multidimensional rasters, we have the ability to use mosaic datasets to store and analyze multiple raster datasets and multiple raster formats. Over the past years I underestimated the capability and the utility of mosaic datasets. These datasets have the ability to calculate custom raster functions on the fly. These datasets can also be shared via ArcGIS Server and ArcGIS Online. I look forward to utilizing the datasets in the future.
One thing to note at the ESRI User Conference is that like last year all of the presentations that I have seen by ESRI staff this year have been on ArcGIS Pro. The ArcGIS Pro software is getting more powerful and ESRI is building more functionality into this new software platform.
I cannot wait to see what day 4 brings tomorrow.
A Unique Data Science Summit
Yesterday, several of us in the IGIS Program participated remotely in a very interesting summit on data science in agriculture. The summit was sponsored by the National Institute of Food and Agriculture (NIFA), which is the funding arm of the US Department of Agriculture (USDA). The goal of the summit was to hear examples of how data collection systems and analytics are playing a transformative role in agriculture, in order to help USDA develop an investment strategy for the next phase of their data science grant program. USDA has been funding innovative big data projects for some time, and will soon be rolling out a new initiative called FACT (Food and Agriculture Cyberinformatics and Tools Initiative).
It was exciting to hear the presentations about how rapid advancements in data collection systems, processing, and analytics are changing agriculture across the US and overseas. From sensor systems that support precision farming, to a new generation of genomics studies, to smarter production models and decision support systems, innovation is happening everywhere. The recorded presentations are online.
What Should USDA Fund?
NIFA is actively soliciting input from experts in the field about funding priorities, and have set up an online forum where people can provide feedback and vote for ideas. The forum is centered around six questions that were also discussed in breakout groups at yesterday's summit. The questions ask what are the most promising opportunities for:
- data-driven advances in agriculture and the food-production systems?
- enhancing cross-sector advances in data applications?
- data-driven advances to address societal well-being and consumer demands?
- to address challenges of various facets of data management and application?
- to ensure future generations of data expertise?
- big data in communications, property rights, and communities?
Data Science in ANR
ANR Farm Advisors and Specialists have been exploring similar questions for years. To name just a couple of examples, the Precision Agriculture workgroup has been developing methods to measure and manage for in-field variability. ANR has also sponsored several apps-for-ag hackathons, including one they hosted this past summer in collaboration with the State Fair. Here at IGIS, we teach workshops on geospatial data analysis, data management, and remote sensing with drones. We also maintain ANR's network of Flux towers, and have digitized historical records from ANR's network of Research and Extension Centers.
What do YOU think?
Many people think data analytics will be the engine for the next revolution in agriculture - what do you think the priority areas should be? NIFA is soliciting input through their Ideas Engine through the end of October. Take this unique opportunity to help shape the future of agricultural data science by letting your voice be heard!
I recently attended the Pacific Research Platform Science Engagement Workshop hosted by CITRIS and Calit2 at UC Merced. The PRP is a "data freeway" currently being developed by researchers at UC San Diego and UC Berkeley to connect the major research universities and other research institutions on the west coast with the goal of sharing large datasets and computational resources at speeds of 10-100 Gbps.
The focus of the workshop were a number of digital archaeology projects that were either already using the PRP architecture or had the potential to do so. It was quite interesting to see the efforts being undertaken to preserve at-risk heritage sites using drone imagery, 3D scans of artifacts and 360° virtual reality imagery. All of these techniques produce massive datasets (often several terabytes) which require extensive post-processing and therefore are exactly the type of projects that the PRP was designed for.
After the presentations, we headed over to UC Merced's new WAVE (Wide-Area Visualization Environment) facility to check out some of the virtual reality imagery that had been discussed. The WAVE consists of twenty 4k 3D monitors arranged in a parabolic curve to create an immersive VR environment. We saw several incredibly detailed image sets of archaeological sites in Greece, Egypt, Saudi Arabia and Belize that had been processed and were being served through the PRP architecture. The effect was quite impressive, it really was the next best thing to being at the actual site.
All in all, this was a real eye-opening workshop that gave a compelling picture of the future of sharing and visualization of Big Data.
Day 3 of the ESRI User Conference, new tools, new story maps, and new ways to work with data.
New Tools, ESRI is supporting new tools with the python and R programming languages. With python they have integrated the ability to easily use 3rd party libraries within ArcGIS by integrating conda into the upcoming release of ArcGIS Pro 1.3 and they have also made it possible to use python to manage ArcGIS online content with the Python API. With R, ESRi has released a ArcGIS R Bridge that allows for the use use or esri data sets in R and the easy use or results from your R analyzes in ArcGIS.
New Story Maps, at the user conference last year, ESRI highlighted a new story map style called the cascade story map. I found out yesterday that they have developed an app builder for this new style of story map and they have also released another style called a crowdsource story map. I also reached out to the developers of story maps today and found out they are developing a new template, they are going to share this new template with us. I cannot wait to see how these storymaps will be used by UCANR in the coming months / year.
New ways to work with data, ESRI has developed new ways to work with data, these data may include Big Data or Multi-dimensional Data. In the case of Multi-dimensional Data they have highlighted new tools to work with netcdf data, but they also showed how that are using existing tools within ArcGIS to work with Multi-dimensional Data. These tools start by importing Multi-dimensional Data into raster mosaics and they using the full suite of ArcGIS tools on these data structures. When it comes to Big Data, they have created a new suite or tools and capabilities within ArcGIS that will allow us to perform big data analysis directly within ArcGIS. Multi-dimensional Data can be used now with ArcGIS and Big Data Analytics will be available in the coming months.
I look forward to seeing what the 4th day will bring.