Non-Contact Heart Rate and Blood Pressure Estimations from Video Analysis and Machine Learning Modelling Applied to Food Sensory Responses: A Case Study for Chocolate

Figure 1. Data gathering set-up showing: (a) data acquisition system displaying: (1) the tablet with the Bio-sensory application for video acquisition, (2) oscillometric HR and blood pressure monitor and (3) finger sensor controlled by (4) an Arduino® board, and (b) diagram of the protocol for the data and video acquisition.

Figure 2. Flow diagram showing the data acquisition, analysis and outputs of the five methods presented to measure HR: (1) oscillometric monitor, (2) raw video analysis (RVA), (3) pre-processing using the Eulerian Magnification Algorithm (EMA), followed by the magnified video analysis (MVA) (4) FaceReader™ software, and (5) finger sensor. The diagram also depicts the inputs and targets/outputs used to develop the machine learning models. Abbreviations: ANN = Artificial Neural Networks, BPM = beats per minute.

Figure 3. Diagrams of the machine learning models developed where: (a) shows Model 1 for regression fitting with a two-layer feedforward network and tan-sigmoid function in the hidden layer and a linear transfer function in the output layer, in which a total of 18 inputs and three targets/outputs used with ten hidden neurons, and (b) Model 2 using the shallow neural network time-series prediction with a series-parallel architecture using a tan-sigmoid function in the hidden layer and a linear transfer function in the output layer, nine inputs, two delays, two targets/outputs and five hidden neurons. Abbreviations: SD = standard deviation, w = weights and b = biases.

Figure 4. Correlation between the HR measured using the oscillometric monitor (x-axis) and the video analysis using FaceReader™ (y-axis) in beats per minute (BPM). R2 = determination coefficient.

Figure 5. Correlations between heart rate (HR) measured using the oscillometric monitor (x-axis) and video analysis using Matlab® and the pre-processed videos with the Eulerian Magnification algorithms (MVA) (y-axis) in beats per minute (BPM). The figure shows the results of the videos analyzed using the luminosity changes in (a) the red channel, (b) the green channel and (c) the blue channel. R2 = determination coefficient.

Figure 6. Correlations between heart rate (HR) measured using the oscillometric monitor (x-axis) and video analysis using the raw video HR algorithm (RVA) (y-axis) in beats per minute (BPM). The figure shows the results of the videos analyzed using the luminosity changes in (a) the red channel, (b) the green channel and (c) the blue channel. R2 = determination coefficient.

Figure 7. The artificial neural network regression overall model (Model 1) showing a high correlation coefficient (R = 0.85), where the x-axis represents the observed (targets) and the y-axis represents the predicted (outputs) values of heart rate (HR) and blood pressure (BP) measured with the oscillometric monitor. All values were normalized in a scale from −1 to 1.

Figure 8. Response from the shallow neural network time-series (SNNTS) model (Model 2) where: (a) shows the regression model with an overall high correlation (R = 0.97), where the x-axis represents the targets (observed) and the y-axis depicts the outputs (predicted) values of the raw signal of the luminosity changes and HR measured using the finger sensor. All values were normalized in a scale from −1 to 1; (b) shows the graph with the response output for the time series for every 1 second comparing the targets and outputs where x-axis represents the time (s) and the y-axis the normalized (−1 to 1) response values of raw signal of the luminosity changes and real HR (HR).

Figure 9. Correlation matrix of a chocolate sensory test showing the significant correlations (p < 0.05) in which the blue side of the color scale (−1 to 1) represent the positive and the yellow side the negative correlations. Abbreviations: (i) heart rate (HR), (ii) systolic pressure (SD), (iii) diastolic pressure (DP), (iv) liking of cocoa aroma (ACocoa), (v) liking of sweetness (Tsweet), (vi) sweetness intensity with just about right (JAR) test (JARSweet), (vii) liking of bitterness (TBitt), (viii) JAR of bitterness intensity, (ix) liking of hardness (Hard), (x) liking of smoothness (Smooth), (xi) overall liking, and (xi) purchase intention (PurInt).