Quantitative Big Data: where chemometrics can contribute

This is the authors' abstract. We don't add key points for this paper.

The editors of J. Chemometrics have been kind enough to invite me to outline some directions, which I think young chemometricians could find interesting for the future. This perspective paper is my response, in the form of an essay that summarizes how I see the field today. It builds on mistakes I have made and things I have learnt in my past 43 years of data analytic R&D, in food science and agriculture, in telecommunication, analytical chemistry and biomedicine, in different universities and companies, in many countries 1-6. This is not a review the field of Chemometrics. Instead, I present topics that I find interesting, challenging and not yet fully developed, hopefully within a coherent perspective. My biased selection of references was chosen to illustrate and expand on this perspective, linking to previous literature for further reading. The paper is primarily written for chemists and bio-medical scientists but readers from other fields may also find something of interest, for all sciences now face the same two problems: How to convert the future's Big Data from a burden to a boon, and how to combine theory and practice. How is the Real World joined together? What makes it tick? How can we know that, and what makes us tick? These are topics addressed by chemometricians and by our multivariate soft-modeling parents and siblings in fields like psychometrics, sensometrics, morphometrics etc. Since Svante Wold and Bruce Kowalski started it in the mid 70s, Chemometrics has grown to become a mature field of science, with an extensive literature and increasingly interacting with other fields. I believe the field of Chemometrics has a lot to learn from other disciplines—mathematics, statistics or computer science. Our heads are filled with domain-specific knowledge, so many of us are lousy in mathematics except for simple linear algebra of the type A = B × C + D, and pretty bad in statistics except for experimental design and simple cross-validation. Our multivariate methods—at least the ones that I know well—are powerful and versatile but they are linear and static. To describe the real world realistically, I think they need further development, e.g. with respect to temporal dynamics and feedback, spatial distribution, heterogeneity and nonlinearity. And most of our successful software packages now need fresh impulses from computer science and cognitive science. On the other hand, Chemometrics also has a lot to give to other disciplines. Our internal culture favors warm-hearted cooperation, rather than cutthroat competition. Academically, we want Real-World science, so we cherish a humble but aggressive attitude and despise passive arrogance. Our methods and approaches allow us to handle big data tables not to of of and for also know how to and learn from and from have on our but the computer from many other we to the of and the of Instead, we data with data methods that us Our data methods like and and the many us and the or and data These methods the not but for the of to we on and we can a on the of the real can also this to the of and of the I present my of Chemometrics a science culture for the and outline a for I describe some topics for in the young scientists learn data from Chemometrics and and to my science data to what a data domain-specific is also and data is and real the same The with the a of the is what is the The are not enough and or other data so is least in and fields. And we have real all in a and are but so are and the not face real and and and of are of of and in and These can by science but by and many of of and of and of we need to our and the real a we need a lot of And data we are an in the of like a not like the in the but an of the data are and but a lot of it I Big Data and data need to Our computer in respect to and of data to computer is not the The data we the we we can also data we are by the that we may with we the big like and or the in a we need to with Big Data It not enough to have we the have to in the and on domain-specific on the we need a lot with cognitive science in sciences and the all to other and On the other we need a lot mathematics and statistics but of the type that us of I not think we need and they in the of is a and the of us for The of but many and in data is A may that is a the and most other science This is by different internal and by the and statistics are A some with with To on and are is the than in the bio-medical it may that is and is and the I believe of the is all of it is also of bad mathematics of for all scientists in all sciences all in this for a today. to the the and the of and I believe an to the with the and the of this to in The that are than is And I to the is than is I a of my mathematics in analytical chemistry the is than is something to theory may a from The to an rather than a is and and I know are I have in that primarily in they they for of mathematics and statistics could to the of The is that the other is in And I not to the fields are to internal And they not to in I think we data to and like with a they and The may a but they a this we have of in our face a and with respect to mathematics and need to handle and me that a need for and statistics on to and the of data is in my bio-medical The same can for many other like most and I an and think that Chemometrics and other fields that have successful with and statistics can to a of and statistics in the future. and statistics can allow us to cognitive to Big the of and in of in is is in sciences like the bio-medical to the the sciences and the have to on in I think the on the two of the to on the other on The is to that and I think our field Chemometrics can a lot to summarizes the two to from to in science, the and the that we chemometricians are of an in the of science, in that we to combine the Chemometrics the years or so a science culture and a powerful of multivariate data with other data These the of data and of in of The of all chemometricians is to the Real World and with an the data to but the same to the in of and the of of us have a for for statistics and of us in design of in of data and the and all of us for in the we combine the and the approaches in a type of a or we a rather than in or that To and to from we can and the to the To we which us with experimental The data tables are the data a of we data methods to for mistakes and in the and to an we methods like the to find and or methods to the that we think we but that can the from And we on but by and the same in to My of and to the of this and This in the with the future's Big have to how to combine and It is to how science so they not to our the I outline some topics that I and I with the and multivariate data chemometricians have on data than on computer I that this To and combine the of knowledge, we need the powerful from and chemometricians with multivariate they see that dynamics and other of is than they in the I how the and increasingly and What can to the the science and it to the of scientists and for and in it that the data is for it that the most to the and think from in other of the of I have a of in and in I believe I have a the think from like I And we or I have chosen to this for two To illustrate how and multivariate but To to the the and science we chemometricians how different we are from and me a are than I could that, it give me my how and to me now my the of of for a selection of = The data from and and is a of a that a in and in how is a the the and have been to a of and a of The = is is that my are than this is in of the in = can by and in this A with of that is = What I from can My was by the was most by the What I now It is my that an of to e.g. the The most is a a of an is an On the other hand, that is the may that is that can a by bad enough enough and bad enough is the of the of was = in this or is the may e.g. of the and our and or young of on and for that the of from extensive to It is to the that is is the have so that the of the was the was and the was the scientists can that than in of the are to of what the with of and of The the now the have to than of the are to of have been is is not enough to the and It is also to the of the and the of the = with which is was it that the real and is not not is The data in and from a The from than for a of the of the and by the selection and It that with the may but also The all = was to with a of It is it is and in that are to enough data we a that are than it for of the What all the of some this other also for of the they us two and to the type of and something interesting is to = to = On the other hand, is to and to but to are some in how from The data from an on in science, in to different of different and how most are to in different from a of to an a to multivariate of all the me a is not my this is for it And it the of than a I see it a of a us of the to our for but for scientists the is a or the It us that it is to know and to know To in science an the same a of is also It a of some that may the in The × now the the and for the A = was for from and two and the how they to of the and the two or in the and The and with two in interesting The to and to and the to and with The to and the to and and or I to find many in the to scientists and I to see in the to the and chemists in the data I to find many chemometricians in the with of approaches and is a the and for methods like and with may by the I in the could in the or the not in what I believe is of the for the in to in other science to different has been a of to I believe is rather And I have and to not this is an of or least something that I with I have to that and science are they an to to with and they are of and of are some by it and scientists and may not like to think the we the world to our to and and our to so our are not in our our today. we are in the same the scientists and the are to a and by our and we and think we are in It is that cognitive science is for and and how are for and how has two of rather and on and knowledge, and a rather and on I have of in multivariate data On hand, the is for and they are me and my to data in of our On the other hand, of theory is for of linear algebra of my can in my software with data to is for in my software I like to that are by and statistics and statistics is primarily by to other has for the we need for a science, to find the and the And the are by and that I can from my I have been in fields of science in different but I have not yet been to a Chemometrics has a that we in the and Our methods have also I with and in theory than with on and Chemometrics also for the for the Our and our culture give us to in different fields. young chemometricians on how we think the Chemometrics a of science can but is also an of my I to is my of what we and what we I think it is to that our has two and the The is and by which are and a lot the and The is addressed by many Chemometrics. The is and It is addressed by e.g. by and The the and is not for and the a the of sciences and the to the present the of an of the This can in a of a of in a a of on paper or in a this can or the a of in a in a with the same The the are I except for for how the and of the of the the of linear chemometricians not to combine data from the and the the most of the of and the is interesting to The that we now in science the of or least is to summarizes of It with and And in of may the and the The is in by in a and the by in in are × = different in which and can The in is the in which I think young chemometricians to and which most are many to of the in scientists have to or to or not on our in a chosen of in and and a of all the may This on our or and is a and also an and are of or it and and to an we in many other increasingly the of of and on the and This may within the it least to to and and some of to that the has of interesting the of may from the e.g. many than not How we on how we linear or and how we the on or or to the and methods are the of many statistics of pretty methods like and a not a by multivariate the many can for and multivariate may And to linear methods from like and the methods most in in of the domain-specific is the data the methods like many or are in the data of this was by selection of some e.g. by of or by of the and like have been the and are in a A a of of in a of in and some can in and in a coherent a us to a many in and many in that the e.g. by this from to the in and and with spatial and temporal the is it has internal The of can also have internal The has internal the and may internal e.g. by and in in and e.g. with respect to and to of many the scientists the have to on the of data from is and in the future. The culture and of Chemometrics have something to to the of Big we have a lot to extensive computer can also of in multivariate to can the of for Big Data from Chemometrics has something to to a to a data in and The data is by some of it a of or is by a some of in of the data a of the data in that a of data is by and the has been to an These allow of data in of a of in or with in and a in are for and The of is e.g. by cross-validation. Svante of the a I think of the multivariate of the in I methods in in the I and with on I that chemists to for and have me of for of e.g. from Big methods different the is most the to the and this the of may different in different like may the may to the in of in A of interesting and of and methods have been I think the methods of are interesting cognitive to the of I the of and to powerful for in all other is a to Chemometrics. in the data are by of to the in and in in the data to may are or to other may also to may a data by may These may the or The is in to we our of and on the data of the the may to by by or in of have of and of The same for or I think this of our a of and to what we can our in to in the Real our our may from simple to of or These may in different e.g. by by the of or by not allow our to we have been to this to for our On the other hand, to may need some we our to science. we may have to a to the of in Big to what we could and have to the that of the and two and Data from etc. in all And in the in can in all in that to my we I think a lot of in this is for a of a of spatial can by × in and e.g. for different how I like to a of e.g. a in of what I have the the a that and The of the = is in To a from data that and are and is not for I years on that in the and now I it it is a is in to to It also for in of by the and are to and I believe this type of multivariate with from other find for of the future's Big It is in the in it also in the or and in the the by two from a of a me by my The on and and and the a was a The for = now and the have and the the field or by our of The field how of the in in to the The the in now how it is by the field to To a for a the and is for = + = is of the and the is in the same for = and are for the of is the same for all the The and like in The have temporal and spatial and fields from the to of = These with for the and for the from to with a data of and The of data of the This is by the and spatial from etc. are not to they for the of the most of the was The of the in this is in the in the has been to a in this methods have been in the or the to to which are in which of but that is the of this These in for many for and for and the of of to etc. for all in the I that the is my to think to but versatile and I on the a and that in the it to handle the data from for etc. the is something to also for young my other give things than a need me to my on which I How can our Chemometrics methods to the data that from and I Big and to data is not an of and is to and the in the multivariate data methods all the data to the of development, or a that is in a kind of the kind of that I find in with how we the that a is by a and by a of This an of Big that a with a of is to an extensive of computer can also a of the data from a the of the is This that is in this a of is the to can the makes it for to and the in the data The to and of the a not the computer but also and the the is than some of the approaches to handle Big Data today. The I to can of The is What is the to of from my previous the and the of I have the of now the of my the of and I the not know and but I see that I know than the multivariate data our fields may Big Data My is How can our Chemometrics of to The of that How our methods to in of a and in my years food I from I now see that this was I an like made me I think of is a form of I have that e.g. is And we can the and for all and it by multivariate is not is that the and so give a of how the can for data can we have some can illustrate in which can temporal and in multivariate or from the same in a also outline cognitive of in the in of our Chemometrics us linear e.g. and simple dynamics can by linear of how the in the of a on the Data this can written by the linear × = a of in and = is temporal in the × B in the how the of is by all the = we are to an a of linear I think of it to this How it to the this to and other with and that for can give the of a the may to and the experimental allow of all of all is is an for the by on by a linear we can from And a linear like with to the and to the of the we to from by we can the of the by the of the is the of most theory can us how the and temporal primarily from I believe that the of in and × and may powerful multivariate of the of a linear us we chemometricians the of I have learnt from my to multivariate data in the and in the how to for the that have a of in a It but on what in is the A was by of by computer from different of the of the for by I the of of and I could have the and the the B in = + that not I that the dynamics the data I to of the with or in and my The was the the by a The linear of on of the of to to and to I to my the that I the the This is of the from are also in to the or and I of to combine our from with methods from other like and with a a of data can the how Chemometrics can something to our in The methods that we in Chemometrics have for a on the are and to and to computer with by of the and data different may we the of and for the of a of experimental like design or design allow with many or to many in in the the in of the and and can give the to describe the from the can the of to experimental the and the can and for with and in are many are that give or the same can it is to the experimental data to the have that the and is a simple of The with can from a and of to a of by The a to a of a of the that give to or of this the of in the in of linear or of of multivariate and of to data this it how to a to describe the dynamics of a in the with a than different of of = for To we to of to a with two and in a to that by of the in and the was and to we the for a of with different for and chosen to a The in some of in = to different a of = the = and to form in the same = the is but the to have and The may but also for with which a of the in the for the chosen the is also in the of the A of could have been How to and we to find mathematics the of simple fields of science and but different of of and the we the the the the and of and to to of the computer different and to the = than of the the and to = of the A was on that of the was by two by and by the so form that it was to they are linking of This in to for of of the we can of the other and our of the with the data and a of the of this of the multivariate of to than we the It is to a in computer the which also on of of the from the in our to with different a or a have been in which the are of by of the our of many is the may for other of with it to of different of linear or and with the need for The fields of Chemometrics and many with other and e.g. we analytical like and for for this makes it to combine data with from the data from a of the of to a from food science. the a and it to the a was to describe from the this it was now to how have of the they or A was to describe of a of and the to a and To handle the of chemometricians learn to respect a to give to and in science a and on the not are and I think we our for something My has the for a I to to it is that we are all and our the some it and some the the and the in science. that in us now to my in the the sciences and I believe that our culture and with other soft-modeling sensometrics, and and other fields and theory can that we that, we have science and a how to a the The is from the to the of and that a is by experimental by some also that other scientists the same by two of scientists not by in and multivariate to and they could a with and in the and to the real and the computer To the that the the the soft-modeling the same for and to of the real or for by and in the real or in the to for the and in and in the real or in the to for the for of I I have not yet the to this in practice. A lot of science has been and is by it or that is a and I that in the science of computer and multivariate that I believe our the is to statistics and that a in the and statistics is to the in and A is for to and for of all of To handle the future's Big we need a lot and I have my of how Chemometrics culture in the may to a the in science and in the On hand, the multivariate the of to the how the Real World to On the other hand, multivariate on e.g. from can to that e.g. is not they may It is to how and of what I builds on impulses from many I to so Svante Bruce and have been for my and I and are for and is for and are for data for this