semantic search research papers

semantic search engine Recently Published Documents

Total documents.

• Latest Documents
• Most Cited Documents
• Contributed Authors
• Related Sources
• Related Keywords

Question Answering Systems for Covid-19

Abstract In the present scenario COVID-19 pandemic has ruined the entire world. This situation motivates the researchers to resolve the query raised by the people around the world in an efficient manner. However, less number of resources available in order to gain the information and knowledge about COVID-19 arises a need to evaluate the existing Question Answering (QA) systems on COVID-19. In this paper, we compare the various QA systems available in order to answer the questions raised by the people like doctors, medical researchers etc. related to corona virus. QA systems process the queries submitted in natural language to find the best relevant answer among all the candidate answers for the COVID-19 related questions. These systems utilize the text mining and information retrieval on COVID-19 literature. This paper describes the survey of QA systems-CovidQA, CAiRE (Center for Artificial Intelligence Research)-COVID system, CO-search semantic search engine, COVIDASK, RECORD (Research Engine for COVID Open Research Dataset) available for COVID-19. All these QA systems are also compared in terms of their significant parameters-like Precision at rank 1 (P@1), Recall at rank 3(R@3), Mean Reciprocal Rank(MRR), F1-Score, Exact Match(EM), Mean Average Precision, Score metric etc.; on which efficiency of these systems relies.

Dug: A Semantic Search Engine Leveraging Peer-Reviewed Literature to Span Biomedical Data Repositories

As the number of public data resources continues to proliferate, identifying relevant datasets across heterogenous repositories is becoming critical to answering scientific questions. To help researchers navigate this data landscape, we developed Dug: a semantic search tool for biomedical datasets that utilizes evidence-based relationships from curated knowledge graphs to find relevant datasets and explain why those results are returned. Developed through the National Heart, Lung, and Blood Institute's (NHLBI) BioData Catalyst ecosystem, Dug can index more than 15,911 study variables from public datasets in just over 39 minutes. On a manually curated search dataset, Dug's mean recall (total relevant results/total results) of 0.79 outperformed default Elasticsearch's mean recall of 0.76. When using synonyms or related concepts as search queries, Dug's (0.28) far outperforms Elasticsearch (0.1) in terms of mean recall. Dug is freely available at https://github.com/helxplatform/dug, and an example Dug deployment is also available for use at https://helx.renci.org/ui.

A Vertical Semantic Search Engine In Electric Power Metering Domain

Fenix: a semantic search engine based on an ontology and a model trained with machine learning to support research.

This paper presents the final results of the research project that aimed to build a Semantic Search Engine that uses an Ontology and a model trained with Machine Learning to support the semantic search of research projects of the System of Research from the University of Nariño. For the construction of FENIX, as this Engine is called, it was used a methodology that includes the stages: appropriation of knowledge, installation and configuration of tools, libraries and technologies, collection, extraction and preparation of research projects, design and development of the Semantic Search Engine. The main results of the work were three: a) the complete construction of the Ontology with classes, object properties (predicates), data properties (attributes) and individuals (instances) in Protegé, SPARQL queries with Apache Jena Fuseki and the respective coding with Owlready2 using Jupyter Notebook with Python within the virtual environment of anaconda; b) the successful training of the model for which Machine Learning algorithms and specifically Natural Language Processing algorithms were used such as: SpaCy, NLTK, Word2vec and Doc2vec, this was also done in Jupyter Notebook with Python within the virtual environment of anaconda and with Elasticsearch; and c) the creation of FENIX managing and unifying the queries for the Ontology and for the Machine Learning model. The tests showed that FENIX was successful in all the searches that were carried out because its results were satisfactory.

COVID-19 preVIEW: Semantic Search to Explore COVID-19 Research Preprints

During the current COVID-19 pandemic, the rapid availability of profound information is crucial in order to derive information about diagnosis, disease trajectory, treatment or to adapt the rules of conduct in public. The increased importance of preprints for COVID-19 research initiated the design of the preprint search engine preVIEW. Conceptually, it is a lightweight semantic search engine focusing on easy inclusion of specialized COVID-19 textual collections and provides a user friendly web interface for semantic information retrieval. In order to support semantic search functionality, we integrated a text mining workflow for indexing with relevant terminologies. Currently, diseases, human genes and SARS-CoV-2 proteins are annotated, and more will be added in future. The system integrates collections from several different preprint servers that are used in the biomedical domain to publish non-peer-reviewed work, thereby enabling one central access point for the users. In addition, our service offers facet searching, export functionality and an API access. COVID-19 preVIEW is publicly available at https://preview.zbmed.de.

A Proposed Framework for Building Semantic Search Engine with Map-Reduce

Strings and things: a semantic search engine for news quotes using named entity recognition, user queries for semantic search engine, the technique of different semantic search engines.

Semantic Search is a search technique that improves looking precision through perception the reason of the search and the contextual magnitude of phrases as they show up in the searchable statistics space, whether or not on the net to generate greater applicable result. We spotlight right here about Semantic Search, Semantic Web and talk about about exceptional kind of Semantic search engine and variations between key-word base search and Semantic Search and the benefit of Semantic Search. We additionally provide a short overview of the records of semantic search and its function scope in the world.

Optimization of Information Retrieval Algorithm for Digital Library Based on Semantic Search Engine

Export citation format, share document.

A new look of the Semantic Web

Research areas.

Data Management

Data Mining and Modeling

Learn more about how we conduct our research

We maintain a portfolio of research projects, providing individuals and teams the freedom to emphasize specific types of work.

Analyze research papers at superhuman speed

Search for research papers, get one sentence abstract summaries, select relevant papers and search for more like them, extract details from papers into an organized table.

Find themes and concepts across many papers

Don't just take our word for it.

Tons of features to speed up your research

Upload your own pdfs, orient with a quick summary, view sources for every answer, ask questions to papers, research for the machine intelligence age, pick a plan that's right for you, get in touch, enterprise and institutions, common questions. great answers., how do researchers use elicit.

Over 2 million researchers have used Elicit. Researchers commonly use Elicit to:

• Speed up literature review
• Find papers they couldn’t find elsewhere
• Automate systematic reviews and meta-analyses
• Learn about a new domain

Elicit tends to work best for empirical domains that involve experiments and concrete results. This type of research is common in biomedicine and machine learning.

What is Elicit not a good fit for?

Elicit does not currently answer questions or surface information that is not written about in an academic paper. It tends to work less well for identifying facts (e.g. "How many cars were sold in Malaysia last year?") and in theoretical or non-empirical domains.

What types of data can Elicit search over?

Elicit searches across 125 million academic papers from the Semantic Scholar corpus, which covers all academic disciplines. When you extract data from papers in Elicit, Elicit will use the full text if available or the abstract if not.

How accurate are the answers in Elicit?

A good rule of thumb is to assume that around 90% of the information you see in Elicit is accurate. While we do our best to increase accuracy without skyrocketing costs, it’s very important for you to check the work in Elicit closely. We try to make this easier for you by identifying all of the sources for information generated with language models.

How can you get in contact with the team?

You can email us at [email protected] or post in our Slack community ! We log and incorporate all user comments, and will do our best to reply to every inquiry as soon as possible.

What happens to papers uploaded to Elicit?

When you upload papers to analyze in Elicit, those papers will remain private to you and will not be shared with anyone else.

How accurate is Elicit?

Training our models on specific tasks, searching over academic papers, making it easy to double-check answers, save time, think more. try elicit for free..

Thank you for visiting nature.com. You are using a browser version with limited support for CSS. To obtain the best experience, we recommend you use a more up to date browser (or turn off compatibility mode in Internet Explorer). In the meantime, to ensure continued support, we are displaying the site without styles and JavaScript.

• View all journals
• Explore content
• About the journal
• Publish with us
• Sign up for alerts
• Published: 11 November 2016

AI science search engines expand their reach

• Nicola Jones  

Nature ( 2016 ) Cite this article

2242 Accesses

3 Citations

259 Altmetric

Metrics details

• Computer science
• Neuroscience
• Research data

Semantic Scholar triples in size and Microsoft Academic's relaunch impresses researchers.

A free AI-based scholarly search engine that aims to outdo Google Scholar is expanding its corpus of papers to cover some 10 million research articles in computer science and neuroscience, its creators announced on 11 November. Since its launch last year , it has been joined by several other AI-based academic search engines, most notably a relaunched effort from computing giant Microsoft.

Semantic Scholar , from the non-profit Allen Institute for Artificial Intelligence (AI2) in Seattle, Washington, unveiled its new format at the Society for Neuroscience annual meeting in San Diego. Some scientists who were given an early view of the site are impressed. “This is a game changer,” says Andrew Huberman, a neurobiologist at Stanford University, California. “It leads you through what is otherwise a pretty dense jungle of information.”

The search engine first launched in November 2015, promising to sort and rank academic papers using a more sophisticated understanding of their content and context. The popular Google Scholar has access to about 200 million documents and can scan articles that are behind paywalls, but it searches merely by keywords. By contrast, Semantic Scholar can, for example, assess which citations to a paper are most meaningful, and rank papers by how quickly citations are rising — a measure of how ‘hot’ they are.

When first launched, Semantic Scholar was restricted to 3 million papers in the field of computer science. Thanks in part to a collaboration with AI2’s sister organization, the Allen Institute for Brain Science, the site has now added millions more papers and new filters catering specifically for neurology and medicine; these filters enable searches based, for example, on which part of the brain part of the brain or cell type a paper investigates, which model organisms were studied and what methodologies were used. Next year, AI2 aims to index all of PubMed and expand to all the medical sciences, says chief executive Oren Etzioni.

“The one I still use the most is Google Scholar,” says Jose Manuel Gómez-Pérez, who works on semantic searching for the software company Expert System in Madrid. “But there is a lot of potential here.”

Microsoft’s revival

Semantic Scholar is not the only AI-based search engine around, however. Computing giant Microsoft quietly released its own AI scholarly search tool, Microsoft Academic , to the public this May, replacing its predecessor, Microsoft Academic Search, which the company stopped adding to in 2012.

Microsoft’s academic search algorithms and data are available for researchers through an application programming interface (API) and the Open Academic Society , a partnership between Microsoft Research, AI2 and others. “The more people working on this the better,” says Kuansan Wang, who is in charge of Microsoft's effort. He says that Semantic Scholar is going deeper into natural-language processing — that is, understanding the meaning of full sentences in papers and queries — but that Microsoft’s tool, which is powered by the semantic search capabilities of the firm's web-search engine Bing, covers more ground, with 160 million publications.

Like Semantic Scholar, Microsoft Academic provides useful (if less extensive) filters, including by author, journal or field of study. And it compiles a leaderboard of most-influential scientists in each subdiscipline. These are the people with the most ‘important’ publications in the field, judged by a recursive algorithm (freely available) that judges papers as important if they are cited by other important papers. The top neuroscientist for the past six months, according to Microsoft Academic, is Clifford Jack of the Mayo Clinic, in Rochester, Minnesota.

Other scholars say that they are impressed by Microsoft’s effort. The search engine is getting close to combining the advantages of Google Scholar’s massive scope with the more-structured results of subscription bibliometric databases such as Scopus and the Web of Science, says Anne-Wil Harzing, who studies science metrics at Middlesex University, UK, and has analysed the new product . “The Microsoft Academic phoenix is undeniably growing wings,” she says. Microsoft Research says it is working on a personalizable version — where users can sign in so that Microsoft can bring applicable new papers to their attention or notify them of citations to their own work — by early next year.

Other companies and academic institutions are also developing AI-driven software to delve more deeply into content found online. The Max Planck Institute for Informatics, based in Saarbrücken, Germany, for example, is developing an engine called DeepLife specifically for the health and life sciences. “These are research prototypes rather than sustainable long-term efforts,” says Etzioni.

In the long term, AI2 aims to create a system that will answer science questions, propose new experimental designs or throw up useful hypotheses. “In 20 years’ time, AI will be able to read — and more importantly, understand — scientific text,” Etzioni says.

You can also search for this author in PubMed   Google Scholar

Related links

Related links in nature research.

Artificial-intelligence institute launches free science search engine 2015-Nov-02

Google Scholar pioneer on search engine’s future 2014-Nov-07

How to tame the flood of literature 2014-Sep-03

Online collaboration: Scientists and the social network 2014-Aug-13

Computer science: The learning machines 2014-Jan-08

Related external links

Rights and permissions.

Reprints and permissions

About this article

Cite this article.

Jones, N. AI science search engines expand their reach. Nature (2016). https://doi.org/10.1038/nature.2016.20964

Download citation

Published : 11 November 2016

DOI : https://doi.org/10.1038/nature.2016.20964

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

This article is cited by

A qualitative study of large-scale recommendation algorithms for biomedical knowledge bases.

• Tsahi Hayat
• Sam Molyneux

International Journal on Digital Libraries (2021)

Shalosh B. Ekhad: a computer credit for mathematicians

• Jacqueline Eviston-Putsch

Scientometrics (2020)

Quick links

• Explore articles by subject
• Guide to authors
• Editorial policies

Sign up for the Nature Briefing: AI and Robotics newsletter — what matters in AI and robotics research, free to your inbox weekly.

Subscribe to the PwC Newsletter

Join the community, search results, compressed indexes for fast search of semantic data.

1 code implementation • 16 Apr 2019

The sheer increase in volume of RDF data demands efficient solutions for the triple indexing problem, that is devising a compressed data structure to compactly represent RDF triples by guaranteeing, at the same time, fast pattern matching operations.

Embedding-based Retrieval in Facebook Search

2 code implementations • 20 Jun 2020

In this paper, we discuss the techniques for applying EBR to a Facebook Search system.

The Semantic Knowledge Graph: A compact, auto-generated model for real-time traversal and ranking of any relationship within a domain

5 code implementations • 2 Sep 2016

This provides numerous benefits: the knowledge graph can be built automatically from a real-world corpus of data, new nodes - along with their combined edges - can be instantly materialized from any arbitrary combination of preexisting nodes (using set operations), and a full model of the semantic relationships between all entities within a domain can be represented and dynamically traversed using a highly compact representation of the graph.

Supervised Topical Key Phrase Extraction of News Stories using Crowdsourcing, Light Filtering and Co-reference Normalization

1 code implementation • LREC 2012

We also experimented with 2 forms of document pre-processing that we call light filtering and co-reference normalization.

COIL: Revisit Exact Lexical Match in Information Retrieval with Contextualized Inverted List

1 code implementation • NAACL 2021

Classical information retrieval systems such as BM25 rely on exact lexical match and carry out search efficiently with inverted list index.

Efficient Neural Ranking using Forward Indexes

1 code implementation • 12 Oct 2021

In this paper, we propose the Fast-Forward index -- a simple vector forward index that facilitates ranking documents using interpolation of lexical and semantic scores -- as a replacement for contextual re-rankers and dense indexes based on nearest neighbor search.

Approximate Nearest Neighbor Search with Window Filters

1 code implementation • 1 Feb 2024

We define and investigate the problem of $\textit{c-approximate window search}$: approximate nearest neighbor search where each point in the dataset has a numeric label, and the goal is to find nearest neighbors to queries within arbitrary label ranges.

Model-enhanced Vector Index

1 code implementation • NeurIPS 2023

We empirically show that our model achieves better performance on the commonly used academic benchmarks MSMARCO Passage and Natural Questions, with comparable serving latency to dense retrieval solutions.

SpaDE: Improving Sparse Representations using a Dual Document Encoder for First-stage Retrieval

2 code implementations • 13 Sep 2022

Sparse document representations have been widely used to retrieve relevant documents via exact lexical matching.

Bootstrapping meaning through listening: Unsupervised learning of spoken sentence embeddings

1 code implementation • 23 Oct 2022

Inducing semantic representations directly from speech signals is a highly challenging task but has many useful applications in speech mining and spoken language understanding.

Building a Better Search Engine for Semantic Scholar

Sergey Feldman

Sergey Feldman is a Senior Applied Research Scientist at AI2 in Seattle, focused on natural language processing and machine learning. Follow Sergey on Twitter .

2020 is the year of search for Semantic Scholar , a free, AI-powered research tool for scientific literature, based at the Allen Institute for AI. One of our biggest endeavors this year is to improve the relevance of our search engine, and my mission beginning at the start of the year was to figure out how to use about 3 years of search log data to build a better search ranker.

Ultimately, we ended up with a search engine that provides more relevant results to our users, but at the outset I underestimated the complexity of getting machine learning to work well for search. “No problem,” I thought to myself, “I can just do the following and succeed thoroughly in 3 weeks”:

• Get all of the search logs.
• Do some feature engineering.
• Train, validate, and test a great machine learning model.

Although this is what seems to be established practice in the search engine literature, many of the experiences and insights from the hands-on work of actually making search engines work is often not published for competitive reasons. Because AI2 is focused on AI for the common good, we make a lot of our technology and research open and free to use. In this post, I’ll provide a “tell-all” account of why the above process was not as simple as we had hoped, and detail the following problems and their solutions:

• The data is absolutely filthy and requires careful understanding and filtering.
• Many features improve performance during model development but cause bizarre and unwanted behavior when used in practice.
• Training a model is all well and good, but choosing the correct hyperparameters isn’t as simple as optimizing nDCG on a held-out test set.
• The best-trained model still makes some bizarre mistakes, and posthoc correction is needed to fix them.
• Elasticsearch is complex, and hard to get right.

Along with this blog post and in the spirit of openness, we are also releasing the complete Semantic Scholar search reranker model that is currently running on www.semanticscholar.org, as well as all of the artifacts you need to do your own reranking. Check it out here: https://github.com/allenai/s2search

Search Ranker Overview

Let me start by briefly describing the high-level search architecture at Semantic Scholar. When one issues a search on Semantic Scholar, the following steps occur:

• Your search query goes to Elasticsearch (we have almost ~190M papers indexed).
• The top results (we use 1000 currently) are reranked by a machine learning ranker.

We have recently improved both (1) and (2), but this blog post is primarily about the work done on (2). The model we used was a LightGBM ranker with a LambdaRank objective. It’s very fast to train, fast to evaluate, and easy to deploy at scale. It’s true that deep learning has the potential to provide better performance, but the model twiddling, slow training (compared to LightGBM), and slower inference are all points against it.

The data has to be structured as follows. Given a query q, ordered results set R = [r_1, r_2, …, r_M], and number of clicks per result C = [c_1, c_2, …, c_M], we feed the following input/output pairs as training data into LightGBM:

f (q, r_1), c_1 f (q, r_2), c_2 … f (q, r_m), c_m

Where f is a featurization function. We have up to m rows per query, and LightGBM optimizes a model such that if c_i > c_j then model( f (q, r_i)) > model( f (q, r_j)) for as much of the training data as possible.

One technical point here is that you need to correct for position bias by weighting each training sample by the inverse propensity score of its position. We computed the propensity scores by running a random position swap experiment on the search engine results page.

Feature engineering and hyper-parameter optimization are critical components to making this all work. We’ll return to those later, but first I’ll discuss the training data and its difficulties.

More Data, More Problems

Machine learning wisdom 101 says that “the more data the better,” but this is an oversimplification. The data has to be relevant , and it’s helpful to remove irrelevant data. We ended up needing to remove about one-third of our data that didn’t satisfy a heuristic “does it make sense” filter.

What does this mean? Let’s say the query is Aerosol and Surface Stability of SARS-CoV-2 as Compared with SARS-CoV-1 and the search engine results page (SERP) returns with these papers:

• Aerosol and Surface Stability of SARS-CoV-2 as Compared with SARS-CoV-1
• The proximal origin of SARS-CoV-2
• SARS-CoV-2 Viral Load in Upper Respiratory Specimens of Infected Patients

We would expect that the click would be on position (1), but in this hypothetical data it’s actually on position (2). The user clicked on a paper that isn’t an exact match to their query. There are sensible reasons for this behavior (e.g. the user has already read the paper and/or wanted to find related papers), but to the machine learning model this behavior will look like noise unless we have features that allow it to correctly infer the underlying reasons for this behavior (e.g. features based on what was clicked in previous searches). The current architecture does not personalize search results based on a user’s history, so this kind of training data makes learning more difficult. There is of course a tradeoff between data size and noise — you can have more data that’s noisy or less data that’s cleaner, and it is the latter that worked better for this problem.

Another example: let’s say the user searches for deep learning , and the search engine results page comes back with papers with these years and citations:

• Year = 1990, Citations = 15000
• Year = 2000, Citations = 10000
• Year = 2015, Citations = 5000

And now the click is on position (2). For the sake of argument, let’s say that all 3 papers are equally “about” deep learning; i.e. they have the phrase deep learning appearing in the title/abstract/venue the same number of times. Setting aside topicality, we believe that academic paper importance is driven by both recency and citation count, and here the user has clicked on neither the most recent paper nor the most cited. This is a bit of a straw man example, e.g., if number (3) had zero citations then many readers might prefer number (2) to be ranked first. Nevertheless, taking the above two examples as a guide, the filters used to remove “nonsensical” data checked the following conditions for a given triple (q, R, C):

• Are all of the clicked papers more cited than the unclicked papers?
• Are all of the clicked papers more recent than the unclicked papers?
• Are all of the clicked papers more textually matched for the query in the title?
• Are all of the clicked papers more textually matched for the query in the author field?
• Are all of the clicked papers more textually matched for the query in the venue field?

I require that an acceptable training example satisfy at least one of these 5 conditions. Each condition is satisfied when all of the clicked papers have a higher value (citation number, recency, fraction of match) than the maximum value among the unclicked. You might note that abstract is not in the above list; including or excluding it didn’t make any practical difference.

As mentioned above, this kind of filter removes about one-third of all (query, results) pairs, and provides about a 10% to 15% improvement in our final evaluation metric, which is described in more detail in a later section. Note that this filtering occurs after suspected bot traffic has already been removed.

Feature Engineering Challenges

We generated a feature vector for each (query, result) pair, and there were 22 features in total. The first version of the featurizer produced 90 features, but most of these were useless or harmful, once again confirming the hard-won wisdom that machine learning algorithms often work better when you do some of the work for them.

The most important features involve finding the longest subsets of the query text within the paper’s title, abstract, venue, and year fields. To do so, we generate all possible ngrams up to length 7 from the query, and perform a regex search inside each of the paper’s fields. Once we have the matches, we can compute a variety of features. Here is the final list of features grouped by paper field.

• title_fraction_of_query_matched_in_text
• title_mean_of_log_probs
• title_sum_of_log_probs*match_lens
• abstract_fraction_of_query_matched_in_text
• abstract_mean_of_log_probs
• abstract_sum_of_log_probs*match_lens
• abstract_is_available
• venue_fraction_of_query_matched_in_text
• venue_mean_of_log_probs
• venue_sum_of_log_probs*match_lens
• sum_matched_authors_len_divided_by_query_len
• max_matched_authors_len_divided_by_query_len
• author_match_distance_from_ends
• paper_year_is_in_query
• paper_oldness
• paper_n_citations
• paper_n_key_citations
• paper_n_citations_divided_by_oldness
• fraction_of_unquoted_query_matched_across_all_fields
• sum_log_prob_of_unquoted_unmatched_unigrams
• fraction_of_quoted_query_matched_across_all_fields
• sum_log_prob_of_quoted_unmatched_unigrams

A few of these features require further explanation. Visit the appendix at the end of this post for more detail. All of the featurization happens here if you want the gory details.

To get a sense of how important all of these features are, below is the SHAP value plot for the model that is currently running in production.

In case you haven’t seen SHAP plots before, they’re a little tricky to read. The SHAP value for sample i and feature j is a number that tells you, roughly, “for this sample i , how much does this feature j contribute to the final model score.” For our ranking model, a higher score means the paper should be ranked closer to the top. Each dot on the SHAP plot is a particular (query, result) click pair sample. The color corresponds to that feature’s value in the original feature space. For example, we see that the title_fraction_of_query_matched_in_text feature is at the top, meaning it is the feature that has the largest sum of the (absolute) SHAP values. It goes from blue on the left (low feature values close to 0) to red on the right (high feature values close to 1), meaning that the model has learned a roughly linear relationship between how much of the query was matched in the title and the ranking of the paper. The more the better, as one might expect.

A few other observations:

• A lot of the relationships look monotonic, and that’s because they approximately are: LightGBM lets you specify univariate monotonicity of each feature, meaning that if all other features are held constant, the output score must go up in a monotonic way if the feature goes up/down (up and down can be specified).
• Knowing both how much of the query is matched and the log probabilities of the matches is important and not redundant.
• The model learned that recent papers are better than older papers, even though there was no monotonicity constraint on this feature (the only feature without such a constraint). Academic search users like recent papers, as one might expect!
• When the color is gray, this means the feature is missing — LightGBM can handle missing features natively, which is a great bonus.
• Venue features look very unimportant, but this is only because a small fraction of searches are venue-oriented. These features should not be removed.

As you might expect, there are many small details about these features that are important to get right. It’s beyond the scope of this blog post to go into those details here, but if you’ve ever done feature engineering you’ll know the drill:

• Design/tweak features.
• Train models.
• Do error analysis.
• Notice bizarre behavior that you don’t like.
• Go back to (1) and adjust.

Nowadays, it’s more common to do this cycle except replacing (1) with “design/tweak neural network architecture” and also add “see if models train at all” as an extra step between (1) and (2).

Evaluation Problems

Another infallible dogma of machine learning is the training, validation/development, and test split. It’s extremely important, easy to get wrong, and there are complex variants of it (one of my favorite topics ). The basic statement of this idea is:

• Train on the training data.
• Use the validation/development data to choose a model variant (this includes hyperparameters).
• Estimate generalization performance on the test set.
• Don’t use the test set for anything else ever .

This is important, but is often impractical outside of academic publication because the test data you have available isn’t a good reflection of the “real” in-production test data. This is particularly true for the case when you want to train a search model.

To understand why, let’s compare/contrast the training data with the “real” test data. The training data is collected as follows:

• A user issues a query.
• Some existing system (Elasticsearch + existing reranker) returns the first page of results.
• The user looks at results from top to bottom (probably). They may click on some of the results. They may or may not see every result on this page. Some users go on to the second page of the results, but most don’t.

Thus, the training data has 10 or maybe 20 or 30 results per query. During production, on the other hand, the model must rerank the top 1000 results fetched by Elasticsearch. Again, the training data is only the top handful of documents chosen by an already existing reranker, and the test data is 1000 documents chosen by Elasticsearch. The naive approach here is to take your search logs data, slice it up into training, validation, and test, and go through the process of engineering a good set of (features, hyperparameters). But there is no good reason to think that optimizing on training-like data will mean that you have good performance on the “true” task as they are quite different. More concretely, if we make a model that is good at reordering the top 10 results from a previous reranker, that does not mean this model will be good at reranking 1000 results from ElasticSearch. The bottom 900 candidates were never part of the training data, likely don’t look like the top 100, and thus reranking all 1000 is simply not the same task as reranking the top 10 or 20.

And indeed this is a problem in practice. The first model pipeline I put together used held-out nDCG for model selection, and the “best” model from this procedure made bizarre errors and was unusable. Qualitatively, it looked as if “good” nDCG models and “bad” nDCG models were not that different from each other — both were bad. We needed another evaluation set that was closer to the production environment, and a big thanks to AI2 CEO Oren Etzioni for suggesting the pith of the idea that I will describe next.

Counterintuitively, the evaluation set we ended up using was not based on user clicks at all. Instead, we sampled 250 queries at random from real user queries, and broke down each query into its component parts. For example if the query is soderland etzioni emnlp open ie information extraction 2011 , its components are:

• Authors: etzioni, soderland
• Venue: emnlp
• Text: open ie, information extraction

This kind of breakdown was done by hand. We then issued this query to the previous Semantic Scholar search (S2), Google Scholar (GS), Microsoft Academic Graph (MAG), etc, and looked at how many results at the top satisfied all of the components of the search (e.g. authors, venues, year, text match). For this example, let’s say that S2 had 2 results, GS had 2 results, and MAG had 3 results that satisfied all of the components. We would take 3 (the largest of these), and require that the top 3 results for this query must satisfy all of its component criteria (bullet points above). Here is an example paper that satisfies all of the components for this example. It is by both Etzioni and Soderland, published in EMNLP, in 2011, and contains the exact ngrams “open IE” and “information extraction.”

In addition to the author/venue/year/text components above, we also checked for citation ordering (high to low) and recency ordering (more recent to less recent). To get a “pass” for a particular query, the reranker model’s top results must match all of the components (as in the above example), and respect either citation order OR recency ordering. Otherwise, the model fails. There is potential to make a finer-grained evaluation here, but an all-or-nothing approach worked.

This process wasn’t fast (2–3 days of work for two people), but at the end we had 250 queries broken down into component parts, a target number of results per query, and code to evaluate what fraction of the 250 queries were satisfied by any proposed model.

Hill-climbing on this metric proved to be significantly more fruitful for two reasons:

• It is more correlated with user-perceived quality of the search engine.
• Each “fail” comes with explanations of what components are not satisfied. For example, the authors are not matched and the citation/recency ordering is not respected.

Once we had this evaluation metric worked out, the hyperparameter optimization became sensible, and feature engineering significantly faster. When I began model development, this evaluation metric was about 0.7, and the final model had a score of 0.93 on this particular set of 250 queries. I don’t have a sense of the metric variance with respect to the choice of 250 queries, but my hunch is that if we continued model development with an entirely new set of 250 queries the model would likely be further improved.

Posthoc Correction

Even the best model sometimes made foolish-seeming ranking choices because that’s the nature of machine learning models. Many such errors are fixed with simple rule-based posthoc correction. Here’s a partial list of posthoc corrections to the model scores:

• Quoted matches are above non-quoted matches, and more quoted matches are above fewer quoted matches.
• Exact year match results are moved to the top.
• For queries that are full author names (like Isabel Cachola ), results by that author are moved to the top.
• Results where all of the unigrams from the query are matched are moved to the top.

You can see the posthoc correction in the code here .

Bayesian A/B Test Results

We ran an A/B test for a few weeks to assess the new reranker performance. Below is the result when looking at (average) total number of clicks per issued query:

This tells us that people click about 8% more often on the search results page. But do they click on higher position results? We can check that by looking at the maximum reciprocal rank clicked per query. If there is no click, a maximum value of 0 is assigned.

The answer is yes — the maximum reciprocal rank of the clicks went up by about 9%! For a more detailed sense of the click position changes here are histograms of the highest/maximum click position for control and test:

This histogram excludes non-clicks, and shows that most of the improvement occurred in positions 2, followed by position 3, and position 1.

Why not do all this in Elasticsearch?

The following two sections were written by Tyler Murray , search engineering lead at Semantic Scholar.

Elasticsearch provides a robust set of tools and integrations that enable even the most advanced users to construct a wide range of matching, scoring, and reranking clauses. Though powerful, these features can often prove confounding and at worst nonsensical when combined across many fields / clauses / filters. The effort involved in debugging and adjusting boosts, filters, rescoring can quickly become untenable for the more complex search use cases.

LTR tends to be the preferred approach for search teams that want to go from hand tuned weighting and rescoring to a search system trained on real-world user behavior. When implementing LTR vs another approach the following pros and cons arise:

• Rescoring occurs ”natively” within the query lifecycle in Elasticsearch.
• Avoids or minimizes network cost associated with “passing” candidates for reranking.
• Keeps technology within a tighter orbit to the primary storage engine.
• Uptime of the search technology is isolated to the cluster, not spread across services.
• Plugin architecture requires a Java binary to run in the Elasticsearch JVM.
• Iteration speed can be quite sluggish when modifying, deploying, testing due to the need for a complete rolling restart of the cluster. Especially true with larger clusters. (>5 TB)
• Though Java maintains an active and mature ecosystem, most cutting edge machine learning and AI technologies currently live in the Python world.
• In a space where gathering judgements is difficult and/or just not possible at scale, the LTR method becomes difficult to train effectively.
• Limited flexibility in testing ranking algorithms side by side in an A/B test without running multiple ranking plugins in the same cluster.

As we looked to the future of how we wanted to test and deploy ranking changes the cons of the Elasticsearch plugin approach greatly outweighed the pros along two main axes; first, iteration and testing speed as it is paramount to our approach of launching user focused improvements. Offline measurement is critical in sanity testing various models as we iterate, but the final measure will always be how the model performs in the wild. With the plugin architecture provided by Elasticsearch iteration and testing becomes quite tedious and time consuming. Secondly, the powerful toolchain enabled through the Python ecosystem outweighed any short term latency regressions. The flexibility in integrating a wide range of language models and existing machine learning technologies has proven fruitful in addressing a wide range of relevance issues. To translate these solutions back into the Java ecosystem would become a non-trivial effort. In summary, Elasticsearch provides a strong base for building robust search experiences, but as the need to handle more complex relevance issues along with a greater speed of iteration our need to look outside of the Elasticsearch ecosystem became more clear to us.

Tuning the Candidate Query in Elasticsearch

Getting the right mix of filtering, scoring, and rescoring clauses proved more difficult than expected. This was due in part to working from an existing baseline used to power a plugin based ranking model, but also due to some issues with the index mapping. Some do’s and don’ts to help guide others along their journey:

• Allow the documents you’re searching against to become bloated with anything but the necessary fields and analyzers for search ranking. If you’re using the same index to search and hydrate records for display you may want to consider whether multiple indices/clusters are necessary. Smaller documents = faster searches, and becomes increasingly more important as your data footprint grows.
• Use many multi_match queries as these are slow and prove to generate scores for documents that are difficult to reason about.
• Perform function_score type queries on very large result sets without fairly aggressive filters or considering whether this function can be performed in a rescore clause.
• Use script_score clauses, they’re slow and can easily introduce memory leaks in JVM. Just don’t do it.
• Ignore the handling of stopwords in your indices/fields. They make a huge difference in scoring, especially so with natural language queries where a high number of terms and stopword usage is common. Always consider the common terms (<= v7.3) query type mentioned below or a stopword filter in your mapping.
• Use field_name.* syntax in filters or matching clauses as this incurs some non-trivial overhead and is almost never what you want. Be explicit about which fields/analyzers you are matching against.
• Consider using common terms queries with a cutoff frequency if you don’t want to filter stopwords from your search fields. This was what pushed us over the edge in getting a candidate selection query that performed well enough to launch.
• Consider using copy_to during indexing to build a single concatenated field in places where you want to boost documents that match multiple terms in multiple fields. We recommend this approach anywhere you are considering a multi_match query.
• Use query_string type queries if your use case allows for it. IMO these are the most powerful queries in the ES toolbox and allow for a huge amount of flexibility and tuning.
• Consider using a rescore clause as it improves performance of potentially costly operations and allows the use of weighting matches with constant scores. This proved helpful in generating scores that we could reason about.
• Field_value_factor scoring in either your primary search clause or in a rescore clause can prove incredibly useful. We consider highly cited documents to be of a higher relevance and thus use this tool to boost those documents accordingly.
• Read the documentation on minimum_should_match carefully, and then read it a few more times. The behavior is circumstantial and acts differently depending on the context of the use.

Conclusion and Acknowledgments

The new search is live on semanticscholar.org , and we think it’s a big improvement! Give it a try and provide us some feedback by emailing [email protected] .

The code is also available for you to scrutinize and use. Feedback is welcome.

This entire process took about 5 months, and would have been impossible without the help of a good portion of the Semantic Scholar team. In particular, I’d like to thank Doug Downey and Daniel King for tirelessly brainstorming with me, looking at countless prototype model results, and telling me how they were still broken but in new and interesting ways. I’d also like to thank Madeleine van Zuylen for all of the wonderful annotation work she did on this project, and Hamed Zamani for helpful discussions. Thanks as well to the engineers who took my code and magically made it work in production.

Appendix: Details About Features

• *_fraction_of_query_matched_in_text — What fraction of the query was matched in this particular field?
• log_prob refers to a language model probability of the actual match. For example, if the query is deep learning for sentiment analysis , and the phrase sentiment analysis is the match, we can compute its log probability in a fast, low-overhead language model to get a sense of the degree of surprise . The intuition is that we not only want to know how much of the query was matched in a particular field, we also want to know if the matched text is interesting. The lower the probability of the match, the more interesting it should be. E.g. “preponderance of the viral load” is a much more surprising 4-gram than “they went to the store”. *_mean_of_log_probs is the average log probability of the matches within the field. We used KenLM as our language model instead of something BERT-like — it’s lightning fast which means we can call it dozens of times for each feature and are still able to featurize quickly-enough for running the Python code in production. (Big thanks to Doug Downey for suggesting this feature type and KenLM.)
• *_sum_of_log_probs*match_lens — Taking the mean log probability doesn’t provide any information about whether a match happens more than once. The sum benefits papers where the query text is matched multiple times. This is mostly relevant for the abstract.
• sum_matched_authors_len_divided_by_query_len — This is similar to the matches in title, abstract, and venue, but the matching is done one at a time for each of the paper authors. This feature has some additional trickery whereby we care more about last name matches than first and middle name matches, but not in an absolute way. You might run into some search results where papers with middle name matches are ranked above those with last name matches. This is a feature improvement TODO.
• max_matched_authors_len_divided_by_query_len — The sum gives you some idea of how much of the author field you matched overall, and the max tells you what the largest single author match is. Intuitively if you searched for Sergey Feldman , one paper may be by (Sergey Patel, Roberta Feldman) and another is by (Sergey Feldman, Maya Gupta), the second match is much better. The max feature allows the model to learn that.
• author_match_distance_from_ends — Some papers have 300 authors and you’re much more likely to get author matches purely by chance. Here we tell the model where the author match is. If you matched the first or last author, this feature is 0 (and the model learns that smaller numbers are important). If you match author 150 out of 300, the feature is 150 (large values are learned to be bad). An earlier version of the feature was simply len(paper_authors), but the model learned to penalize many-author papers too harshly.
• fraction_of_*quoted_query_matched_across_all_fields — Although we have fractions of matches for each paper field, it’s helpful to know how much of the query was matched when unioned across all fields so the model doesn’t have to try to learn how to add.
• sum_log_prob_of_unquoted_unmatched_unigrams — The log probabilities of the unigrams that were left unmatched in this paper. Here the model can figure out how to penalize incomplete matches. E.g. if you search for deep learning for earthworm identification the model may only find papers that don’t have the word deep OR don’t have the word earthworm . It will probably downrank matches that exclude highly surprising terms like earthworm assuming citation and recency are comparable.

Follow @allen_ai and @semanticscholar on Twitter, and subscribe to the AI2 Newsletter to say current on news and research coming out of AI2.

Written by Sergey Feldman

Lead Applied Research Scientist @ allenai.org | Head of AI @ Alongside.care

Text to speech

• Edit on GitHub

Open-Source Search Engine with Apache Lucene / Solr

Integrated research tools for easier searching, monitoring, analytics, discovery & text mining of heterogenous & large document sets & news with free software on your own server, search engine(fulltext search).

Easy full text search in multiple data sources and many different file formats: Just enter a search query (which can include powerful search operators ) and navigate through the results.

Thesaurus & Grammar(Semantic search)

Based on a thesaurus the multilingual semantic search engine will find synonyms, hyponyms and aliases , too. Using heuristics for grammar rules like stemming it finds other word forms, too.

Interactive filters(Faceted search)

Easy navigation through many results with interactive filters (faceted search) which aggregates an overview over and interactive filters for (meta) data like authors, organizations, persons, places, dates, products, tags or document types.

Exploration, browsing & preview(Exploratory search)

Explore your data or search results with an overview of aggregated search results by different facets with named entities (i.e. file paths, tags, persons, locations, organisations or products) , while browsing with comfortable navigation through search results or document sets. View previews (i.e. PDF, extracted Text, Table rows or Images). Analyze or review document sets by preview, extracted text or wordlists for textmining .

Collaborative annotation & tagging (Social search & collaborative filtering)

Tag your documents with keywords, categories, names or text notes that are not included in the original content to find them better later (document management & knowledge management) or in other research or search contexts or to be able to filter annotated or tagged documents by interactive filters (faceted search).

Or evaluate, value or assess or filter documents (i.e. for validation or collaborative filtering).

Datavisualization (Dataviz)

Visualizing data like document dates as trend charts or text analysis for example as word clouds , connections and networks in visual graph view or view results with geodata as interactive maps .

Monitoring: Alerts & Watchlists (Newsfeeds)

Stay informed via watchlists for news alerts from media monitoring or activity streams of new or changed documents on file shares: Subscribe searches and filters as RSS-Newsfeed and get notifications when there are changed or new documents, news or search results for your keywords, search context or filter.

Supports different file formats

No matter if structured data like databases, tables or spreadsheets or unstructured data like text documents , E-Mails or even scanned legacy documents: Search in many different formats and content types (text files, Word and other Microsoft Office documents or OpenOffice documents, Excel or LibreOffice Calc tables, PDF, E-Mail, CSV, doc, images, photos, pictures, JPG, TIFF, videos and many other file formats ).

Supports multiple data sources

Find all your data at one place: Search in many different data sources like files and folders, file server, file shares , databases , websites, Content Management Systems, RSS-Feeds and many more.

The Connectors and Importers of the Extract Transform Load (ETL) framework for Data Integration connects and combines multiple data sources and as integrated document analysis and data enrichment framework it enhances the data with the analysis results of diverse analytics tools.

Automatic text recognition

Optical character recognition (OCR) or automatic text recognition for images and text content stored in graphical format like scanned legacy documents, screenshots or photographed documents in the form of image files or embedded in PDF files.

Open-Source enterprise search and information retrieval technology based on interoperable open standards

Mobile (responsive design).

Open Semantic Search can not only be used with every desktop (Linux, Windows or Mac) or web browser. With its responsive design and open standards like HTML5 it is possible to search with tablets, smartphones and other mobiles.

Metadata management (RDF)

Structure your research, investigation, navigation, document sets, collections, metadata forms or notes in a Semantic Wiki, Drupal or another content management system (CMS) or with an innovative annotation framework with taxonomies and custom fields for tagging documents, annotations, linking relationships, mapping and structured notes. So you integrate powerful and flexible metadata management or annotation tools using interoperable open standards like Resource Description Framework (RDF) and Simple Knowledge Organization System ( SKOS ).

Filesystem monitoring

Using file monitoring , new or changed files are indexed within seconds without frequent recrawls (which is not possible often if many files). Colleagues are able to find new data immediately without (often forgotten) uploads to a data or document management system (DMS) or filling out a data registration form for each new or changed document or dataset in a data management system, data registry or digital asset management (DAM) system.

An official website of the United States government

Here’s how you know

Official websites use .gov A .gov website belongs to an official government organization in the United States.

Secure .gov websites use HTTPS A lock ( Lock A locked padlock ) or https:// means you’ve safely connected to the .gov website. Share sensitive information only on official, secure websites.

https://www.nist.gov/metis/semantic-assets

Semantic Assets

Semantic Assets are digital resources that have been enhanced via metadata. METIS will rely on a controlled vocabulary, a taxonomy, and an ontology to create the metadata needed for Semantic Assets.

A semantic asset is a digital resource (e.g., research paper, dataset, software, model) that has been enriched with metadata and semantic annotations that are designed to be machine-readable to make it easier for users to find resources, understand their meaning and relationships, and combine them in their research.

METIS provides access to semantically enriched research products through the use of the NIST Extensible Resource Data Model (NERDm), a JSON-LD-formatted metadata schema used by the NIST Public Data Repository (PDR) and Science Data Portal to describe data resources available from NIST. NERDm is defined using JSON Schema and is designed using best practices and standards-oriented representations, enabling efficient discovery and integration of research products that adheres to Findability, Accessibility, Interoperability, and Reusability (FAIR) principles. This allows for seamless exploration, combination, and analysis of data from diverse sources.

The METIS project is currently working to enhance its semantic assets in three ways:

METIS Controlled Vocabulary

Metis taxonomy, metis ontology.

These three items are intended to address perceived current metadata gaps for the METIS use cases.

The efficient organization and search of large resource collections rely heavily on the availability of standardized terms for annotation. However, identifying and defining these terms can be an arduous task that spans multiple years, requiring significant input from domain experts. To expedite this process, our approach leverages terms from diverse sources and employs large language models (specifically, gpt-turbo-instruct and ChatGPT 3.5) to generate a substantial collection of approximately 12,000 proto-definitions. 

Our primary objective is to facilitate the vocabulary creation process by providing domain experts with an extensive collection of plausible candidate terms for their selection, review, editing, and refinement. This approach enables domain experts to focus on higher-level tasks such as reviewing, validating, and fine-tuning the generated terms, rather than starting from scratch. Our goal is to accelerate the community development of a standardized vocabulary, thereby enhancing the discoverability, accessibility, and usability of large resource collections for various CHIPS stakeholders.

Encouraged by our initial results, we have started the process of developing the Controlled Vocabulary Curation System (CVCS). This web-based application will allow domain experts to generate proto-definitions for terms of their choice and to manage the term review and curation process.

We have started developing a taxonomy to categorize and organize METIS resources. The METIS taxonomy will benefit from the METIS Controlled Vocabulary to provide definitions for items in the taxonomy.

We are also working towards the development of an ontology to serve as a formal logical model that will provide representations of concepts, relationships, and axioms, to facilitate machine-readable understanding and reasoning about resources. The development of the METIS ontology will benefit from our work on the controlled vocabulary and taxonomy.

Related Semantic Areas

The METIS project's focus on semantic assets and its current efforts with controlled vocabularies, and topic taxonomies are closely related to other areas in the realm of semantics:

• Knowledge Graphs : The creation of comprehensive knowledge graphs that interconnect research products, datasets, software, and models, enabling advanced discovery and integration capabilities.
• Linked Data : Implementing linked data principles to enable the seamless integration of resources with external datasets, promoting a web of interconnected knowledge.
• Artificial Intelligence (AI) and Machine Learning (ML) : Leveraging AI/ML techniques to enhance semantic asset creation, vocabulary development, and topic taxonomy refinement, as well as improving search, recommendation, and analytics capabilities.

These related areas will inform and shape the evolution of METIS's semantic assets, ultimately enhancing the overall discoverability, accessibility, and usability of research products for CHIPS stakeholders.

• DOI: 10.52152/3922
• Corpus ID: 271513549

Research on Residential Interior Design and Energy Saving Optimization with Sustainable Low-carbon Development

• An Wang , Juanfen Wang
• Published in RE&amp;PQJ 21 July 2024
• Environmental Science, Engineering
• RE&amp;PQJ

Related Papers

Showing 1 through 3 of 0 Related Papers