Answers are grounded in this paper and cite their sources.
In today's business environment, enterprises are increasingly under pressure to process the vast amount of data produced everyday within enterprises. One method is to focus on the business intelligence (BI) applications and increasing the commercial added-value through such business analytics activities. Term weighting scheme, which has been used to convert the documents as vectors in the term space, is a vital task in enterprise Information Retrieval (IR), text categorisation, text analytics, etc. When determining term weight in a document, the traditional TF-IDF scheme sets weight value for the term considering only its occurrence frequency within the document and in the entire set of documents, which leads to some meaningful terms that cannot get the appropriate weight. In this article, we propose a new term weighting scheme called Term Frequency – Function of Document Frequency (TF-FDF) to address this issue. Instead of using monotonically decreasing function such as Inverse Document Frequency, FDF presents a convex function that dynamically adjusts weights according to the significance of the words in a document set. This function can be manually tuned based on the distribution of the most meaningful words which semantically represent the document set. Our experiments show that the TF-FDF can achieve higher value of Normalised Discounted Cumulative Gain in IR than that of TF-IDF and its variants, and improving the accuracy of relevance ranking of the IR results. Keywords: enterprise information retrievalterm weighting schemeterm frequencyfunction of document frequencyrelevance ranking Acknowledgements This work is supported by a grant from the State Key Laboratory of Software Development Environment, Beihang University (No. SKLSDE-2009ZX-12), the Changjiang Scholar Program, Chinese Ministry of Education, National Natural Science Foundation of China 70971005 and Key Project of The National Social Science Fund Grant No. 11AZD096.
No takes yet. Share an insight, caveat, or question.
Zhang et al. (2012) studied this question.
Synapse has enriched 10 related papers. Consider them for comparative context: