The project will be conducted over 36 months.
The planned tasks - comprising six separate Work Packages (WPs) - will be conducted during the fruit crop growing season. Milestones are provided for each task and include a short description where applicable. A separate Project Annex comprises additional information on the consortium's expertise, cited references, and supporting letters.
The following Six (6) Work Packages or WPs form the basis of this project.
WP1: Project Management:
This WP is designed to oversee the day-to-day functioning and management of all the other WPs. This will include the overall management, organisation of meetings, project updates, raising awareness about the project for its duration and upcoming tasks. The lead partner (Zhaw) will also maintain contacts with all the partners via email, and will be in charge of updating partners about project progress and changes as well as organizing project meetings.
The WP’s objective is to provide the partners with well-organized information and to remind and update in time for upcoming tasks, milestones and deadlines.
Overall WP1 does not lead to a deliverable, but provides the basis to manage the project work and deliverables.
WP2: Competitive Sticky Trap Evaluation per Crop:
Based on the findings of attractiveness in SWD, D. melanogaster, and closely related species, different sticky trap variants in relation to crop will be evaluated in Switzerland. Traps will be kept as simple as possible but will be adapted and/or equipped in color, shape and bait. The most promising trap variants will be co-evaluated in selected crops in The Netherlands and the UK. Sticky traps will be monitored with a hand-held camera for further analysis in WP3. Depending on the possibility to enhance contrast between trap color and insects we will consider the use of either multispectral or hyperspectral cameras.
Task 2.1: Color-choice experiments.
First the project will measure and compare the UV (Ultraviolet) and VIS (Visible) spectral reflectance of the traps and the attractive fruit crops. Colors comparable to fruit feed colors will be used in color/color and color/fruit-choice experiments under laboratory conditions.
Task 2.2: Bait and glue-choice experiments.
Since the results from pre-experiments suggested a potential repellent activity of glues the project will examine the attractiveness of available glues such as Tangle Trap (Andermatt Biocontrol, Switzerland) and will evaluate the possibilities to add single molecule substances such as IPA or mixtures (wine or vinegar) to the glue. The different combinations will be ranked based on choice experiments.
Task 2.3: Impact of trap shape.
Based on the results of Tasks 2.1 and 2.2 the project will add the level of trap shape. With these results the three traps performing best under controlled conditions will be selected and will be further analyzed in the field (Task2.4).
Task 2.4: In-field trap performance.
The best performing traps selected in Tasks 2.1-2.3 will be evaluated in field trials in majorly affected crops: strawberry, cherry, raspberry, blueberry, and grapes. The traps will be imaged with different cameras such as the hyperspectral frame camera Rikola (Rikola Ltd. Oulu, Finland) available at WUR or a simple tri-band or RGB cameras such as the Canon S110 NIR providing Green, Red and NIR band data and the Canon G9X RGB (both from Sensefly, Switzerland, available at ZHAW), respectively, to provide the basis for WP3. Through interchange with the progress of WPs 3 and 4 camera selection will be adapted to the most informative and drone applicable equipment.
The aim of WP2 is to evaluate the different variants of sticky traps always keeping in mind to develop an attractive trap which is easy to photograph.
WP2 will identify the most attractive sticky trap for SWD. The results of the experiments will lead to at least one peer reviewed publication.
WP3: Image-Based Fully Automated Detection and Differentiation of SWD:
Imagery from WP2 will be manually annotated by experts.
Multi-/hyperspectral imagery data will allow training of machine learning-based algorithms in a high-dimensional feature space. This will allow selection of the most informative spectral bands and hence improve the accuracy of image segmentation. The data will then be used to train a computer-vision algorithm for automatic SWD detection.
Task 3.1: On-trap insect identification.
SWD is a small insect which we aim to detect on traps placed in natural environments. Since the traps will be somewhat - but not highly - specific the target insect has to be discriminated from bycatch (other insects). The authors of this project are not currently aware of a functioning algorithm or computer program available to achieve this. Initially the training images acquired in WP2 Task 2.4 will be adopted for manual data labelling of SWD. The identified training areas will be investigated using multivariate approaches (e.g. Principal Component Analysis, PCA; Minimum Noise Fraction, MNF) to characterize and identify unique spectral and geometric features of SWD for the camera types included in this project. This will also provide background and define relevant spectral bands and their associated width and the required spatial resolution to be adopted for the field-observed images.
Task 3.2: Autonomous target insect counting.
Based on previous studies in literature and expertise of the project team an autonomous approach for target insect counting will be developed. This means that individual images acquired from the UAV-based platform will be automatically classified and the number of SWD individuals will be counted using automated computer-vision algorithms. The performance of regularly applied algorithms like watershed algorithm segmentation and Mahalanobis distance classification will be compared to more advanced approaches like deep learning approaches (e.g., convolutional neural networks) which can take advantage of the hyperspectral dimension of the images. In the current approach developed in this project, images will be processed on a separate storage server which means that processing capability is unlimited. The Machine Vision software developed will be tested with the captured imagery to optimize the extraction of the required information from the UAV-based aerial photographs. This will involve identifying (i) the image processing tools and techniques required to pre-process, enhance, and identify the fruit flies (D. suzukii) in the remotely acquired images of the insect traps; (ii) to generate an insect count, and (iii) ultimately, customization of the DIP software to automatically extract the required information from the imagery to provide the basis for development of the spatially-based decision-support and planning information system. The resolution of the UAV imagery will depend on the choice of camera, the focal length of the camera lens, and the flying height of the platform above the surface. As the UAV will be positioned directly over the trap at any one location, stationary, and at the desired height above the trap to optimize the image detail recorded, image blurring will be limited.
The first objective of WP3 is to explore unique spectral and geometric features for identification of SWD; and the second to develop and evaluate autonomous target insect counting algorithm for UAV-based multi/hyperspectral images.
WP3 will deliver an implemented and tested algorithm for detection of SWD from UAV-based multi/hyperspectral images. The results of the experiments will lead to at least one peer-reviewed publication.
WP4: Airborne Image Acquisition Under Open Field Conditions:
A small autonomous multi-rotor UAV carrying a digital aerial camera/sensor, gimbal, RTK GPS, and using a First Person View (FPV) system will be used to position the camera to capture digital color aerial imagery and data from the trap sites. Using a pre-programmed flight planner with the traps identified as waypoints, automated drone trap-hopping will be undertaken with image capture at a small distance above the trap position to ensure high-resolution (sub-mm) imagery of each trap. The image data will then be transferred directly to cloud-based storage for subsequent processing and analysis with the aid of DIP software.
Task 4.1: Specification of the UAV Technology (hardware and software) to undertake the aerial photographic survey of the sticky traps placed at known spatial locations in the field.
This will involve choosing a suitable off-the-shelf RTF (Ready To Fly) aerial UAV platform that can easily be modified for purpose and flown easily at a low-altitude above the plant canopy of choice with the aid of a suitable waypoint-based autopilot to allow for the selection of a number of location-based aerial photographs. The UAV platform will be a small quad-copter carrying an RTK GPS (Global Positioning System) unit with cm positional locational accuracy and a small format, high-resolution lightweight photographic camera. As part of the research work automatic upload and storage of the imagery captured with the camera system on the UAV will be necessary ready for the use of DIP software to extract the information from the photographs e.g. identity (ID) of insect and a count (Number) of the insects for each class. The specific software for target insect identification and counting on the field derived images will be developed in house at WUR (WP3).
Task 4.2: Airborne data acquisition.
Following selection of the UAV hardware and software for the proposed work (autopilot/image processing), the system specified for use will need to be repeat flight-tested in order to fine-tune the data semi-automated collection process. This will involve testing: (a) the UAV platform flight capability and RTK GPS locational accuracy, (b) the waypoint-based UAV autopilot, (c) the capability to stop the drone in transit over the study area and to collect aerial photographic-based information at a low altitude over the insect trap, (d) the optimum height for aerial photographic capture, and (e) the successful upload of the aerial photographic imagery from the UAV to cloud-based storage, access, and processing.
Task 4.3: UAV-based image capture of traps in the field.
The final stage of this part of the research project work will involve the testing and fine-tuning of the system in the field at different locations as a proof of concept stage, and will involve a series of training flights to be conducted (see Figure 1 in Annex Document).
The first objective of this WP is to specify the UAV and associated hardware and software technology comprising the system to undertake the aerial overflights and data/imagery acquisition (T4.1). The second objective is to test the proposed system and to acquire test data and imagery over the traps as input to the information extraction stage of the project (T4.2). Finally, the third objective is to undertake a proof of concept of the system (T4.3).
The deliverables from this WP will be (a) to develop and test a UAV designed to acquire high resolution data and imagery of insect traps in the field, and (b) to acquire aerial data and imagery for information extraction.
WP5: Preparing the Implementation for DSS Application – Threshold Definition for Impacted Crops:
WP5 integrates the findings of WPs 2-4 and therefore all partners will be involved. Plant phenology during SWD invasion and PPP retention times are crucial for decision-making. Depending on the phenological state of the specific crop, predicted weather conditions until harvest and estimated losses/costs for sorting, this information will lead to three possibilities for decision-making: (1) continue with surveillance, (2) treat with PPP, or (3) harvest. Data output will be streamlined for incorporation into available DSS such as Agrometeo.ch - a widely used decision-support system for cherry, grape and other fruit growers in Switzerland. Additionally, the UAV-collected data is by nature geo-referenced and could be used in a GIS-based DSS giving farmers the possibility to incorporate data directly into their precision farming platforms.
Task 5.1: Data complementation and integration.
Where available the acquired insect counts will be complemented with crop-specific actively collected data on phenology, PPP application possibilities, allowing threshold definition and a decision support. Such data is available e.g. for grapevines in Switzerland at www.agrometeo.ch, in southern Germany and Austria at www.vitimeteo.de.at. For those crops without active data collection, thresholds will be defined based on published literature and agricultural and horticultural extension services or outreach programs who recommend PPP application.
Task 5.2: DSS implementation.
The imagery data and threshold definitions as well as the output of T5.1 will be prepared exemplarily for the implementation in running DSS such as the Swiss Agrometeo and comparable platforms. Close to market solutions will be invented for farmers of crops without existing DSS.
In WP5 we aim to integrate the image-based insect count data with biological and environmental relevant data for threshold definition and decision-making. The thresholds derived will be used to generate data that will be prepared for implementing the existing DSS or, if not available, for easy-to-use close to market applications.
The first task of WP5 will deliver applicable thresholds of SWD numbers per trap and perimeter a report on which will be published in peer reviewed publication(s). The second task will provide solutions to data preparation in a suitable format to incorporate into existing DSS or push-pull services for farmers.
WP6: Dissemination of Results:
WP6 manages the dissemination of the results to different end users and target groups i.e. mainly scientists in the respective research fields, agronomists, consultants and fruit producers. To meet the needs of each of the groups, different channels for dissemination will be used such as the www, social media e.g. Facebook and Twitter, popular and scientific journals and magazines, workshops for producers, inter-/national conferences etc.
Task 6.1: Dissemination through the WWW
We will establish a project homepage and social-media presence to present information to a wider community, e.g. Facebook, Twitter
Task 6.2: Grower Communications.
Presenting and communicating results to growers at their meetings and via their newsletters/magazines, e.g. The Commercial Grower, Schweizer Bauer.
Task 6.3: Scientific Communication.
This task aims at publishing at least 3 peer-reviewed papers such as (1) the trap selection/evaluation experiments, (2) the automatic identification and counting of the insects on the trap caught under field conditions and (3) the automated airborne data collection and data preparation for DSS and end -user in journals as e.g. International Journal of Remote Sensing.
This WP will distribute the findings of the whole project to different end-user groups who are either involved in the research or impacted by the pest itself.
Deliverables: WP6 will deliver a project homepage, an actively managed social-media presence, reports and talks to growers and fruit producers, as well as to the scientific community.