Photo credit: http://www.flickr.com/photos/doug88888/4535174486
As you may or may not know, WebLearn’s search tool has been completely rewritten. This rewrite has introduced a number of powerful new features that have been hitherto hidden from the user.
As part of the funding given to OUCS / IT Services by the Blavatnik School of Government in order to develop a world-class iPad ‘Learning App’ for their Master in Public Policy course, the WebLearn team reimplemented all the complicated plumbing within the search tool. (For those that are interested, we now use Apache Solr as a front-end to Apache Lucene .)
On the surface, the Search tool behaves exactly as normal, however, it is now possible to supply advanced queries which will allow a more relevant list of results to be returned. Here is a handy guide to these new features. A lot of this guide is taken from the Apache Lucene help pages.
Terms
A query is broken up into terms and operators. There are two types of terms: Single Terms and Phrases.
A Single Term is a single word such as “hip” or “priest”.
A Phrase is a group of words surrounded by double quotes such as “hip priest”
Multiple terms can be combined together with Boolean operators to form a more complex query (see below).
The search tool will also perform stemming.
Wikipedia tells us that “in linguistic morphology and information retrieval, stemming is the process for reducing inflected (or sometimes derived) words to their stem, base or root form—generally a written word form”.
Wikipedia cites the following examples: “a stemming algorithm reduces the words ‘fishing’, ‘fished’, ‘fish’, and ‘fisher’ to the root word, ‘fish’. On the other hand, ‘argue’, ‘argued’, ‘argues’, ‘arguing’, and ‘argus’ reduce to the stem ‘argu’ (illustrating the case where the stem is not itself a word or root) but ‘argument’ and ‘arguments’ reduce to the stem ‘argument’.
So that’s clear then!
Fields
A faceted search query of the following form can be made
<field>:<argument> [<optional-search-term(s)>]
The following fields are available
tool- specify the tool to search, if no tool field is supplied than all tools are searched. The following are valid tools, none of the other tools have their contents indexed so will not show up in search results:
forumswikiemail (aka ‘Email Archive’ tool)content (aka ‘Resources’)announcementchatsite (aka ‘Home’ tool)
title – search just the title, this could be the title of an email message or a documentcontents – search just the contents i.e. not title or description etc.siteid – search just the stated site. The site ID is found in the URL of a Resource, it is accessible via the Edit Details page:
Site ID is part of the URL of a resource
As an example, if you want to find the document entitled “The Right Way” which contains the text “don’t go this way”, you can enter:
title:"The Right Way" AND contents:go
Note: The field is only valid for the term that it directly precedes, so the query
title:The Right Way
Will only find “The” in the title field. It will find “Right” and “Way” in other areas.
The query
tool:forums "Mark E Smith"
will search the forums tool only for mentions of everybody’s favourite northern curmudgeon. The query
tool:content "Mark E Smith"
will just search Resources. The query
tool:content OR tool:forums "Mark E Smith"
will search both tools. See later for an explanation of Boolean operators.
Term Modifiers
WebLearn search supports modifying query terms to provide a wide range of searching options.
Wildcard Searches
You can make single and multiple character wildcard searches within single terms (not within phrase queries).
To perform a single character wildcard search use the “?” symbol.
To perform a multiple character wildcard search use the “*” symbol.
The single character wildcard search looks for terms that match that with the single character replaced. For example, to search for “text” or “test” you can use the search:
te?t
Multiple character wildcard searches looks for 0 or more characters. For example, to search for test, tests or tester, you can use the search:
test*
You can also use the wildcard searches in the middle of a term.
te*t
Note: You cannot use a * or ? symbol as the first character of a search.
Regular Expression Searches
WebLearn search supports regular expression searches matching a pattern between forward slashes “/”. For example to find documents containing “moat” or “boat”:
/[mb]oat/
Fuzzy Searches
WebLearn search supports fuzzy searches based on Damerau-Levenshtein Distance. To do a fuzzy search use the tilde (twiddle), “~”, symbol at the end of a Single word Term. For example to search for a term similar in spelling to “roam” use the fuzzy search:
roam~
This search will find terms like foam and roams.
An additional (optional) parameter can specify the maximum number of edits allowed. The value is between 0 and 2, For example:
roam~1
The default that is used if the parameter is not given is 2 edit distances.
Proximity Searches
WebLearn search supports finding words are a within a specific distance away. To do a proximity search use the tilde, “~”, symbol at the end of a Phrase. For example to search for a “WebLearn” and “indispensable” within 10 words of each other in a document use the search:
"WebLearn indispensable"~10
Range Searches
Range Queries allow one to match documents whose field(s) values are between the lower and upper bound specified by the Range Query. Range Queries can be inclusive or exclusive of the upper and lower bounds. Sorting is done lexicographically.
title:{Aida TO Carmen}
This will find all documents whose titles are between Aida and Carmen, but not including Aida and Carmen. Inclusive range queries are denoted by square brackets. Exclusive range queries are denoted by curly brackets so if you wanted to include Aida and Carmen then use square brackets:
title:[Aida TO Carmen]
Boosting a Term
WebLearn search provides the relevance level of matching documents based on the terms found. To boost a term use the caret, “^”, symbol with a boost factor (a number) at the end of the term you are searching. The higher the boost factor, the more relevant the term will be.
Boosting allows you to control the relevance of a document by boosting its term. For example, if you are searching for
WebLearn indispensable
and you want the term “WebLearn” to be more relevant boost it using the ^ symbol along with the boost factor next to the term. You would type:
WebLearn^4 indispensable
This will make documents with the term WebLearn appear more relevant. You can also boost Phrase Terms as in the example:
"WebLearn indispensable"^4 "Usage data"
By default, the boost factor is 1. Although the boost factor must be positive, it can be less than 1 (e.g. 0.2)
Boolean Operators
Boolean operators allow terms to be combined through logic operators. WebLearn search supports AND, “+”, OR, NOT and “-” as Boolean operators (Note: Boolean operators must be ALL CAPS).
OR
The OR operator is the default conjunction operator. This means that if there is no Boolean operator between two terms, the OR operator is used. The OR operator links two terms and finds a matching document if either of the terms exist in a document. This is equivalent to a union using sets. The symbol || can be used in place of the word OR.
To search for documents that contain either “WebLearn indispensable” or just “indispensable” use the query:
"WebLearn indispensable" indispensable
or
"WebLearn indispensable" OR indispensable
AND
The AND operator matches documents where both terms exist anywhere in the text of a single document. This is equivalent to an intersection using sets. The symbol && can be used in place of the word AND.
To search for documents that contain “WebLearn indispensable” and “Usage data” use the query:
"WebLearn indispensable" AND "Usage data"
+
The “+” or required operator requires that the term after the “+” symbol exist somewhere in a the field of a single document.
To search for documents that must contain “WebLearn ” and may contain “indispensable” use the query:
+WebLearn indispensable
NOT
The NOT operator excludes documents that contain the term after NOT. This is equivalent to a difference using sets. The symbol ! can be used in place of the word NOT.
To search for documents that contain “WebLearn indispensable” but not “Usage data” use the query:
"WebLearn indispensable" NOT "Usage data"
Note: The NOT operator cannot be used with just one term. For example, the following search will return no results:
NOT "Usage data"
-
The “-” or prohibit operator excludes documents that contain the term after the “-” symbol.
To search for documents that contain WebLearn indispensable” but not “Usage data” use the query:
"WebLearn indispensable" -"Usage data"
Grouping
WebLearn search supports using parentheses to group clauses to form sub queries. This can be very useful if you want to control the boolean logic for a query.
To search for either “WebLearn” or “Sakai” and “indispensable” use the query:
(WebLearn OR Sakai) AND indispensable
Field Grouping
WebLearn search supports using parentheses to group multiple clauses to a single field.
To search for a title that contains both the word “return” and the phrase “pink panther” use the query:
title:(+return +"pink panther")
Escaping Special Characters
WebLearn search supports escaping special characters that are part of the query syntax. The current list special characters are
+ - && || ! ( ) { } [ ] ^ " ~ * ? : \ /
To escape these character use the \ before the character. For example to search for (1+1):2 use the query:
\(1\+1\)\:2
