Evaluating Calibration and Spectral Variable Selection Methods for Predicting Three Soil Nutrients Using Vis-NIR Spectroscopy

Hyperspectral signals in the soil: Plant-soil hydraulic connection and disequilibrium as mechanisms of drought tolerance and rapid recovery

Predicting soil water status remotely is appealing due to its low cost and large-scale application. During drought, plants can disconnect from the soil, causing disequilibrium between soil and plant water potentials at pre-dawn. The impact of this disequilibrium on plant drought response and recovery is not well understood, potentially complicating soil water status predictions from plant spectral reflectance. This study aimed to quantify drought-induced disequilibrium, evaluate plant responses and recovery, and determine the potential for predicting soil water status from plant spectral reflectance. Two species were tested: sweet corn (Zea mays), which disconnected from the soil during intense drought, and peanut (Arachis hypogaea), which did not. Sweet corn's hydraulic disconnection led to an extended 'hydrated' phase, but its recovery was slower than peanut's, which remained connected to the soil even at lower water potentials (-5 MPa). Leaf hyperspectral reflectance successfully predicted the soil water status of peanut consistently, but only until disequilibrium occurred in sweet corn. Our results reveal different hydraulic strategies for plants coping with extreme drought and provide the first example of using spectral reflectance to quantify rhizosphere water status, emphasizing the need for species-specific considerations in soil water status predictions from canopy reflectance.

Monitoring of volatile fatty acids during anaerobic digestion of olive pomace by means of a hand held near infrared spectrometer

The accumulation of volatile fatty acids (VFAs) over anaerobic digestion (AD) leads to malfunctioning of industrial reactors, hence decreasing biogas production. Real-time monitoring of VFAs is a challenge due to the complexity and high cost of current methods for their quantification. For this reason, this research evaluated the application of near infrared (NIR) spectroscopy to quantify volatile fatty acids as a tool for AD reactors monitoring. To do that, 129 samples from various AD reactors fed with olive oil pomace were taken and their NIR spectra were acquired with a hand-held spectrometer. After performing grid search, three spectral variable selection methods, namely competitive adaptive reweighted sampling, uninformative variable elimination (UVE) and successive projections algorithm, were assayed before developing PLRS models to correlate the NIR light transmittance through the samples at the wavelengths selected by those methods with their VFAs concentrations. UVE led to the best performance for all the VFAs assayed. Thus, R2 of validation of UVE-PLSR models for acetic, propionic, butyric, valeric and total VFAs were 0.895, 0.622, 0.866, 0.898 and 0.871, respectively. The predictive model for total VFAs achieved the highest accuracy (RMSEV = 539.5 mg/L), explained by the correlation between the light absorption at the wavelengths selected by UVE and the chemical characteristics of VFAs. All in all, the prediction errors achieved suggest that a portable near infrared spectrometer can be used for monitoring VFAs in AD processes.

A novel near infrared spectroscopy analytical strategy for soil nutrients detection based on the DBO-SVR method

Soil is the basis of agricultural production and accessing accurate information on soil nutrients is essential. Traditional methods of soil composition detection, which are based on chemical analysis, are characterized by being costly and polluting. Spectroscopic analysis has proven to be a rapid, non-destructive and effective technique for predicting soil properties in general and potassium, phosphorus and organic matter in particular. However, previous research on soils has rarely combined optimization algorithms with machine learning techniques, which has led to suboptimal model accuracy and convergence speed. In this study, a total of 184 soil samples were collected from three cities of Linhai, Yueqing and Longyou County, Zhejiang Province, China. After measuring pH values, alkali-hydrolyzable nitrogen (SAN), available phosphorus (SAP), available potassium (SAK) and soil organic matter (SOM) contents, along with their corresponding spectroscopic measurements, nine pretreatment methods and their combinations are adopted. A novel assessment model, integrating support vector machine and dung beetle optimization algorithm (DBO-SVR), is proposed to predict pH values and SAN, SAP, SAK, SOM content. Meanwhile, the DBO algorithm is compared with three mainstream optimization algorithms (particle swarm optimization (PSO), whale optimization algorithm (WOA) and grey wolf optimizer (GWO)). Results showed that the DBO-SVR model was shown best performance with Rp, RMSEP and RPD of 0.9842, 0.1306, 5.6485 respectively for prediction of pH value, with Rp, RMSEP and RPD of 0.8802, 15.0574 mg/kg and 2.0508, respectively for assessment of SAN content, with Rp, RMSEP and RPD of 0.9790, 12.8298 mg/kg, and 4.5132, respectively for assessment of SAP content, with Rp, RMSEP and RPD of 0.8677, 22.5107 mg/kg, and 1.9546, respectively for assessment of SAK content, and with Rp, RMSEP and RPD of 0.9273, 2.6427g/kg , and 2.1821, respectively for assessment of SOM content. This study demonstrates that the combination of near-infrared (NIR) spectroscopy and the DBO-SVR algorithm is capable of predicting soil nutrient composition with greater accuracy and efficiency.

A critical systematic review on spectral-based soil nutrient prediction using machine learning

The United Nations (UN) emphasizes the pivotal role of sustainable agriculture in addressing persistent starvation and working towards zero hunger by 2030 through global development. Intensive agricultural practices have adversely impacted soil quality, necessitating soil nutrient analysis for enhancing farm productivity and environmental sustainability. Researchers increasingly turn to Artificial Intelligence (AI) techniques to improve crop yield estimation and optimize soil nutrition management. This study reviews 155 papers published from 2014 to 2024, assessing the use of machine learning (ML) and deep learning (DL) in predicting soil nutrients. It highlights the potential of hyperspectral and multispectral sensors, which enable precise nutrient identification through spectral analysis across multiple bands. The study underscores the importance of feature selection techniques to improve model performance by eliminating redundant spectral bands with weak correlations to targeted nutrients. Additionally, the use of spectral indices, derived from mathematical ratios of spectral bands based on absorption spectra, is examined for its effectiveness in accurately predicting soil nutrient levels. By evaluating various performance measures and datasets related to soil nutrient prediction, this paper offers comprehensive insights into the applicability of AI techniques in optimizing soil nutrition management. The insights gained from this review can inform future research and policy decisions to achieve global development goals and promote environmental sustainability.

Prediction of cyanidin 3-rutinoside content in Michelia crassipes based on near-infrared spectroscopic techniques

Currently the determination of cyanidin 3-rutinoside content in plant petals usually requires chemical assays or high performance liquid chromatography (HPLC), which are time-consuming and laborious. In this study, we aimed to develop a low-cost, high-throughput method to predict cyanidin 3-rutinoside content, and developed a cyanidin 3-rutinoside prediction model using near-infrared (NIR) spectroscopy combined with partial least squares regression (PLSR). We collected spectral data from Michelia crassipes (Magnoliaceae) tepals and used five different preprocessing methods and four variable selection algorithms to calibrate the PLSR model to determine the best prediction model. The results showed that (1) the PLSR model built by combining the blockScale (BS) preprocessing method and the Significance multivariate correlation (sMC) algorithm performed the best; (2) The model has a reliable prediction ability, with a coefficient of determination (R2) of 0.72, a root mean square error (RMSE) of 1.04%, and a residual prediction deviation (RPD) of 2.06. The model can be effectively used to predict the cyanidin 3-rutinoside content of the perianth slices of M. crassipes, providing an efficient method for the rapid determination of cyanidin 3-rutinoside content.

Predicting Soil Organic Matter, Available Nitrogen, Available Phosphorus and Available Potassium in a Black Soil Using a Nearby Hyperspectral Sensor System

Black soils, which play an important role in agricultural production and food security, are well known for their relatively high content of soil organic matter (SOM). SOM has a significant impact on the sustainability of farmland and provides nutrients for plants. Hyperspectral imaging (HSI) in the visible and near-infrared region has shown the potential to detect soil nutrient levels in the laboratory. However, using portable spectrometers directly in the field remains challenging due to variations in soil moisture (SM). The current study used spectral data captured by a handheld spectrometer outdoors to predict SOM, available nitrogen (AN), available phosphorus (AP) and available potassium (AK) with different SM levels. Partial least squares regression (PLSR) models were established to compare the predictive performance of air-dried soil samples with SMs around 20%, 30% and 40%. The results showed that the model established using dry sample data had the best performance (RMSE = 4.47 g/kg) for the prediction of SOM, followed by AN (RMSE = 20.92 mg/kg) and AK (RMSE = 22.67 mg/kg). The AP was better predicted by the model based on 30% SM (RMSE = 8.04 mg/kg). In general, model performance deteriorated with an increase in SM, except for the case of AP. Feature wavelengths for predicting four kinds of soil properties were recommended based on variable importance in the projection (VIP), which offered useful guidance for the development of portable hyperspectral sensors based on discrete wavebands to reduce cost and save time for on-site data collection.

Prediction of leaf nitrogen in sugarcane (Saccharum spp.) by Vis-NIR-SWIR spectroradiometry

Nitrogen is one of the essential nutrients for the production of agricultural crops, participating in a complex interaction among soil, plant and the atmosphere. Therefore, its monitoring is important both economically and environmentally. The aim of this work was to estimate the leaf nitrogen contents in sugarcane from hyperspectral reflectance data during different vegetative stages of the plant. The assessments were performed from an experiment designed in completely randomized blocks, with increasing nitrogen doses (0, 60, 120 and 180 kg ha-1). The acquisition of the spectral data occurred at different stages of crop development (67, 99, 144, 164, 200, 228, 255 and 313 days after cutting; DAC). In the laboratory, the hyperspectral responses of the leaves and the Leaf Nitrogen Contents (LNC) were obtained. The hyperspectral data and the LNC values were used to generate spectral models employing the technique of Partial Least Squares Regression (PLSR) Analysis, also with the calculation of the spectral bands of greatest relevance, by the Variable Importance in Projection (VIP). In general, the increase in LNC promoted a smaller reflectance in all wavelengths in the visible (400-680 nm). Acceptable models were obtained (R2 > 0.70 and RMSE <1.41 g kg-1), the most robust of which were those generated from spectra in the visible (400-680 nm) and red-edge (680-750 nm), with values of R2 > 0.81 and RMSE <1.24 g kg-1. An independent validation, leave-one-date-out cross validation (LOOCV), was performed using data from other collections, which confirmed the robustness and the possibility of LNC prediction in new data sets, derived, for instance, from samplings subsequent to the period of study.

Soil carbon content prediction using multi-source data feature fusion of deep learning based on spectral and hyperspectral images

Visible near-infrared reflectance spectroscopy (VNIR) and hyperspectral images (HSI) have their respective advantages in soil carbon content prediction, and the effective fusion of VNIR and HSI is of great significance for improving the prediction accuracy. But the contribution difference analysis of multiple features in the multi-source data is inadequate, and there is a lack of in-depth research on the contribution difference analysis of artificial feature and deep learning feature. In order to solve the problem, soil carbon content prediction methods based on VNIR and HSI multi-source data feature fusion are proposed. The multi-source data fusion network under the attention mechanism and the multi-source data fusion network with artificial feature are designed. For the multi-source data fusion network based on the attention mechanism, the information are fused through the attention mechanism according to the contribution difference of each feature. For the other network, artificial feature are introduced to fuse multi-source data. The results show that multi-source data fusion network based on the attention mechanism can improve the prediction accuracy of soil carbon content, and multi-source data fusion network combined with artificial feature has better prediction effect. Compared with two single-source data from the VNIR and HSI, the relative percent deviation of Neilu, Aoshan Bay and Jiaozhou Bay based on multi-source data fusion network combined with artificial feature are increased by 56.81% and 149.18%, 24.28% and 43.96%, 31.16% and 28.73% respectively. This study can effectively solve the problem of the deep fusion of multiple features in the soil carbon content prediction by VNIR and HSI, so as to improve the accuracy and stability of soil carbon content prediction, promote the application and development of soil carbon content prediction in spectral and hyperspectral image, and provide technical support for the study of carbon cycle and the carbon sink.

Vis-NIR Spectroscopy Combined with GAN Data Augmentation for Predicting Soil Nutrients in Degraded Alpine Meadows on the Qinghai-Tibet Plateau

Soil nutrients play vital roles in vegetation growth and are a key indicator of land degradation. Accurate, rapid, and non-destructive measurement of the soil nutrient content is important for ecological conservation, degradation monitoring, and precision farming. Currently, visible and near-infrared (Vis-NIR) spectroscopy allows for rapid and non-destructive monitoring of soil nutrients. However, the performance of Vis-NIR inversion models is extremely dependent on the number of samples. Limited samples may lead to low prediction accuracy of the models. Therefore, modeling and prediction based on a small sample size remain a challenge. This study proposes a method for the simultaneous augmentation of soil spectral and nutrient data (total nitrogen (TN), soil organic matter (SOM), total potassium oxide (TK₂O), and total phosphorus pentoxide (TP₂O₅)) using a generative adversarial network (GAN). The sample augmentation range and the level of accuracy improvement were also analyzed. First, 42 soil samples were collected from the pika disturbance area on the QTP. The collected soils were measured in the laboratory for Vis-NIR and TN, SOM, TK₂O, and TP₂O₅ data. A GAN was then used to augment the soil spectral and nutrient data simultaneously. Finally, the effect of adding different numbers of generative samples to the training set on the predictive performance of a convolutional neural network (CNN) was analyzed and compared with another data augmentation method (extended multiplicative signal augmentation, EMSA). The results showed that a GAN can generate data very similar to real data and with better diversity. A total of 15, 30, 60, 120, and 240 generative samples (GAN and EMSA) were randomly selected from 300 generative samples to be included in the real data to train the CNN model. The model performance first improved and then deteriorated, and the GAN was more effective than EMSA. Further shortening the interval for adding GAN data revealed that the optimal ranges were 30-40, 50-60, 30-35, and 25-35 for TK₂O, TN, TP₂O₅, and SOM, respectively, and the validation set accuracy was maximized in these ranges. Therefore, the above method can compensate to some extent for insufficient samples in the hyperspectral prediction of soil nutrients, and can quickly and accurately estimate the content of soil TK₂O, TN, TP₂O₅, and SOM.

Evaluation of Near-Infrared Reflectance and Transflectance Sensing System for Predicting Manure Nutrients

Livestock manure is widely applied onto agriculture soil to fertilize crops and increase soil fertility. However, it is difficult to provide real-time manure nutrient data based on traditional lab analyses during application. Manure sensing using near-infrared (NIR) spectroscopy is an innovative, rapid, and cost-effective technique for inline analysis of animal manure. This study investigated a NIR sensing system with reflectance and transflectance modes to predict N speciation in dairy cow manure using a spiking method. In this study, 20 dairy cow manure samples were collected and spiked to achieve four levels of ammoniacal nitrogen (NH4-N) and organic nitrogen (Org-N) concentrations that resulted in 100 samples in each spiking group. All samples were scanned and analyzed using a NIR system with reflectance and transflectance sensor configurations. NIR calibration models were developed using partial least square regression analysis for NH4-N, Org-N, total solid (TS), ash, and particle size (PS). Coefficient of determination (R2) and root mean square error (RMSE) were selected to evaluate the models. A transflectance probe with a 1 mm path length had the best performance for analyzing manure constituents among three path lengths. Reflectance mode improved the calibration accuracy for NH4-N and Org-N, whereas transflectance mode improved the model predictability for TS, ash, and PS. Reflectance provided good prediction for NH4-N (R2 = 0.83; RMSE = 0.65 mg mL−1) and approximate predictions for Org-N (R2 = 0.66; RMSE = 1.18 mg mL−1). Transflectance was excellent for TS predictions (R2 = 0.97), and provided good quantitative predictions for ash and approximate predictions for PS. The correlations between the accuracy of NH4-N and Org-N calibration models and other manure parameters were not observed indicating the predictions of N contents were not affected by TS, ash, and PS.

Spectral Index for Mapping Topsoil Organic Matter Content Based on ZY1-02D Satellite Hyperspectral Data in Jiangsu Province, China

Estimation of soil organic matter content (SOMC) is essential for soil quality evaluation. Compared with traditional multispectral remote sensing for SOMC mapping, the distribution of SOMC in a certain area can be obtained quickly by using hyperspectral remote sensing data. The Advanced Hyper-Spectral Imager (AHSI) onboard the ZY1-02D satellite can simultaneously obtain spectral information in 166 bands from visible (400 nm) to shortwave infrared (2500 nm), providing an important data source for SOMC mapping. In this study, SOMC-related spectral indices (SIs) suitable for this satellite were analyzed and evaluated in Shuyang County, Jiangsu Province. A series of SIs were constructed for the bare soil and vegetation-covered (mainly rice crops and tree seedlings) areas by combining spectral transformations (such as reciprocal and square root) and dual-band index formulas (such as ratio and difference), respectively. The optimal SIs were determined based on Pearson’s correlation coefficient (ρ) and satellite data quality, and applied to SOMC level mapping and estimation. The results show that: (1) The SI with the highest ρ in the bare soil area is the ratio index of original reflectance at 654 and 679 nm (OR-RI(654, 679)), whereas the SI in the vegetation area is the square root of the difference between the reciprocal reflectance at 551 and 1998 nm (V-RR-DSI(551, 1998)); (2) the spatial distribution trend of regional SOMC results obtained by linear regression models of OR-RI(654, 679) and V-RR-DSI(551, 1998) is consistent with the samples; (3) based on the optimal SIs, support vector machine and tree ensembles were used to predict the SOMC of bare soil and vegetation-covered areas of Shuyang County, respectively. The determination coefficient of the soil–vegetation combined prediction results is 0.775, the root mean square error is 3.72 g/kg, and the residual prediction deviation is 2.12. The results show that the proposed SIs for ZY1-02D satellite hyperspectral data are of great potential for SOMC mapping.