Optimising Agri-Tech with Data Integration

How FME Transformed Agri-EPI's Data Workflows

Introduction

Navigating the complexity of data-driven agriculture, Agri-EPI Centre stands at the forefront of revolutionising farming practices through cutting-edge technology. Born out of a need to promote sustainable and profitable food production, Agri-EPI utilises innovative solutions including robotics, autonomous systems, and data analytics. However, the vast and diverse data ecosystem that underpins their operations: integrating diverse data from a myriad of sources into a coherent, actionable catalogue, presented a significant challenge. Recognising the critical role of effective data management, Agri-EPI partnered with Tensing to leverage Safe Software’s FME Platform for optimised data integration.

A little about Agri-EPI Centre

Agricultural Engineering Precision Innovation Centre (Agri-EPI Centre) uses innovative technologies to improve agricultural efficiency and precision, helping to promote food production practices that help to mitigate the impacts of climate change. Since its founding in 2015, Agri-EPI Centre has been helping to transform, drive growth and support innovative solutions that help farmers and agri-food businesses become more sustainable and profitable.

Their work is founded on providing scientifically robust and commercially viable solutions to improve productivity and sustainability within the food production system. This is achieved through the use of cutting-edge technologies, robotics, autonomous solutions, and data to make the harvest as consistent as possible. Agri-EPI Centre has a range of key collaborators from Farmers, Food suppliers and the Government through to Engineers, Scientists, Investment companies and Start-ups.

The Agri-Tech Effect: Mitigating Climate Change

Agri-EPI is developing technologies in a range of areas to help mitigate climate change effects, these include:

• Efficient Resource Management: Through the use of precision irrigation and nutrient management, water consumption and nutrient runoff can be drastically reduced. Sensors and precision technology play a vital role in monitoring and regulating resource usage.
• Digital Tools & Data Analytics: By employing remote sensing, drones, and sensors, crop performance monitoring can be enhanced. This allows for early detection of pests and diseases and strengthens traceability in the supply chain.
• Soil Monitoring: Constant tracking of soil moisture, temperature, pH, and nutrient levels using precision technology enables the optimisation of irrigation and fertilisation methods, leading to healthier crops and lower resource usage.
• Precision Farming: Data from a range of sources are utilised for precise farming and adaptive management, resulting in improved yields, lower costs, and a smaller environmental footprint.
• Harvest Management: The monitoring of crop growth, yield, and quality improves harvest planning and post-harvest management, thereby reducing waste and optimising the use of resources.
• Multi-spectral imaging: Using UAVs at the farm scale helps to estimate water stress and evaluate micro-climates and temperature variation of specific crops over time, with the hope of improving output.

The Challenge

The world of agriculture and agri-tech is vast, reaching far beyond the fields and into a network of collaborators, each with distinct data requirements. Every collaborator has a vital part to play, each with their own unique data needs. Yet the diversity of data sources, formats, and systems, along with the challenge of integrating these into a coherent data catalogue required an innovative solution.

One of the critical functions of Agri-EPI is to collate data from a variety of sources such as IoT (Internet of Things) sensors, Farm Management Information Systems (FMIS), and other database platforms spread across a network of associated farms.

Manual data collection, data storage, and processing were labour-intensive, error-prone, and ultimately time and resource-consuming. There was a pressing need for Agri-EPI to create a range of automated ETL (Extract, Transform, Load) data workflows to expedite data collection from these disparate sources, validate and ensure robust data quality, and prevent data loss through centralising the data storage and creating a robust backup solution.

Solution: The FME Platform

Recognising the need for a platform that could handle data diversity with efficiency and flexibility, Agri-EPI reviewed the functionality of several technologies including Agrimetrics, Esri ArcGIS, ArcGIS Velocity, Databricks, Elastic, FME, GDAL and GeoServer. Each of these offered distinct capabilities, from data sharing and AI tools (Agrimetrics), map creation and data management (ArcGIS), data analytics (Databricks), full-text search (Elastic), geospatial data translation (GDAL), to data sharing via open standards (GeoServer), and real-time data processing (ArcGIS Velocity).

The preferred solution however, quickly surfaced as Safe Software’s FME Platform, comprising FME Form (previously FME Desktop) and FME Flow (previously called FME Server), because it offered combined capabilities from many of the other assessed solutions under one offering. FME is a data integration platform that simplifies spatial data translation and automates custom integration workflows. It can handle hundreds of data formats and its ability to integrate disparate systems, makes it easier to share and analyse data across different platforms. Its capabilities also reach beyond spatial data, making it a powerful tool for various data integration challenges in the Agri-tech sector.

FME Form is where workflows (known as workspaces) are created and tested that convert and transform data. FME Form, is based on a visual programming paradigm and provides a visual interface where you can build data workflows by adding data readers, data transformers and data writers using a drag-and-drop, no-code approach.

FME Flow is the server-side component that enables the automation of workflows. These workflows can be triggered by ‘events’ such as new files being added to a directory or FTP site, an email arriving with an attachment or a record in a database receiving an update. FME Flow also provides self-service solutions through Apps that can be made accessible to agricultural partners who may wish to access specific data, or upload content for inclusion in the database for analysis. This is all possible through a web front-end without the need for code.

FME offers compatibility with over 450 spatial and non-spatial data formats, whilst its support for large-scale data processing principally led it to being selected by Agri-EPI. FME can also be deployed flexibly, offering potential cost-effectiveness in licensing and platform hosting costs. The recommended product was the FME Desktop Database Edition, recently superseded by FME Form, fulfilling Agri-EPI's core requirements while integrating easily with their current platform.

The Role of Tensing: Orchestrating Data Integration using FME

Agri-EPI selected Tensing to support them to create a data integration solution based on FME. Four initial data sources were chosen by Agri-EPI for Tensing to develop a set of FME Workspaces to extract data from and load centrally to an open-source PostgreSQL database. Following their development, Tensing subsequently configured FME Flow and deployed the FME workspaces to it to orchestrate the processing transactions. With the system now in place FME Flow integrates and centralises data on a daily cadence into a data warehouse from the following data sources:

1) Richies Beef Monitor: Used to integrate cattle weight data, thereby allowing better management of livestock. This web dashboard does not offer an API interface, so the FME Workspace developed is required to authenticate and then ‘scrape’ the data directly from the web page. The solution is flexible enough to be able to iterate through a range of URLs to extract the full dataset.

The Ritchie Beef Monitor is a smart water trough that weighs cattle via tag scanning, relaying data to an online platform.

2) Fullwood: Real-time dairy milking data is integrated into the system, enhancing dairy production efficiency and sustainability. The system produces CSV output files regularly, FME Flow retrieves these files and loads only the incremental changes to the database.

3) UNIFORM Agri: The integration of herd health and treatment data allows for better monitoring of animal health and improved treatment plans, thereby enhancing the overall welfare of livestock. FME connects directly to the associated Firebird backend database and synchronises any differences (deltas).

4) IrriMAX: A network of 70 high-precision soil sensors across 20 farms provided crucial data on soil moisture, salinity, and temperature, enabling real-time monitoring for precision irrigation. IrriMAX Live exposes the sensor data via an API and FME uses the HTTPCaller transformer to retrieve the data.

 

IrriMAX captured dataset, displaying soil water content at various depths.

Results

Every day, FME Flow automations now gather thousands of new data points from the four different systems. This data is centralised in a PostgreSQL database for analysis and can be easily filtered, queried and exported in a format such as CSV when required. The system has proven its robustness already as it collected all missed data following a recent server downtime period due to relocation.

This process has saved the team countless hours each month in data collation and has plugged any potential gaps in data recording. It has also helped avoid potential data duplication and ensured the most recent data is readily available to the team.

The in-house technical team have begun adapting the developed workspaces to compile data from additional sources, such as SQL Server and other third-party APIs. The self-documenting FME Workspaces have smoothed the transition by guiding the adaptation of data workflows for these other sources.

Conclusion: Laying the Foundation

By working in partnership with Tensing to create a range of FME data workflows, Agri-EPI has begun to build an evolving system for the collection, processing, and analysis of diverse data across its network of associated farms. This project has helped Agri-EPI in their goal of integrating and making accessible the wide range of datasets they are currently collecting through a disparate array of technical innovations.

The first steps have helped set the foundation for preparing for future data pipelines and delivery formats and methods. The initial integration of different systems and the automation of data collection, act as a useful framework to evolve and build more complex data workflows and automations moving forward.

Future plans: More formats and integrations to come

Agri-EPI has recently started collecting multi-spectral imaging and Light Detection and Ranging (LiDAR) data from their farms, using Unmanned aerial vehicles (UAVs). Until recently this was only possible for large-scale mapping projects but with the accessibility of UAVs, undertaking analysis of orchards and vineyards is now possible. Fortunately, FME is capable of managing and transforming point cloud LiDAR and raster data efficiently. We look forward to putting FME through its paces with these new data integration challenges in the future and creating actionable data-driven insights for Agri-EPI and their diverse collaborators.

Inspired by Agri-EPI's success story and want to write one of your own? Reach out to us today to get started on your journey towards data integration.

 

 

This might also interest you