We are quite happy about the fact that Intervals has been increasing in size and popularity since we first launched it six years ago. One of the “growing pains” we encountered was the challenge of providing a fast and relevant search for our customers as the amount of data in their accounts increased. Intervals’ customers tend to accumulate a lot of data, and getting to past and present tasks, projects, documents and other such data quickly is a top priority. Unfortunately, our search utilities had been under-performing.
In this post…
Designing and developing search utilities is not easy. Creating a fast and relevant search is even more difficult, and is becoming more so as web technology advances. Just look at the number of web sites out there using Google as their site search engine to see what I mean. Indexing web site data and searching against it is typically left to third party services. For our customers, we needed a way to index and search their data in a fast, relevant, and most importantly, secure fashion. One that operated wholly inside of Intervals.
After a lot of time at the whiteboard discussing and researching strategies, we’ve rebuilt the Intervals search from the ground up. Now when you login to Intervals and search for tasks, milestones, project notes, documents or projects, the results will appear much faster and contain much more relevant information.
Whether you are searching for an exact phrase or any combination of keywords, the new and improved Intervals search will find it for you. And now that we’ve overhauled the core functionality driving the Intervals search, we’ll be adding additional functionality in the near future to help you get at your data as quickly as possible.
We want Intervals to grow with your small business. This is just one more way we are striving to make that happen.
A Brief How-to on Search
The new search utility provides a basic keyword search by default. Enter one or more keywords to find the data you seek. When multiple keywords are entered, Intervals will look for data that contains any of those keywords. To enforce a stricter search on keywords, put “AND” between each one to tell the application to look for data that contains all of those keywords. To refine the search results even more, enclose the keywords in quotes to do an exact phrase search. For more granular details on how to use the new search, see our forum post on Search Improvements, Syntax and Tips.
Under the Hood
Happen to be a fellow web developer or database administrator and want the nitty, gritty details on what we did? Keep reading…
Intervals is built on top of a PostgreSQL database, which has it’s own built-in search utility known as Full Text Search. This solution was ideal when we first launched for two reasons. One, it was one of the only options available at the time that would accomplish strict set of requirements. And two, it was native to our development stack in that it was part of the database.
Over time, the default Full Text Search settings became less and less suitable to the type of indexing and searching we were doing. Intervals customer data is in a constant state of flux, requiring indices to undergo constant rebuilding. Search results were slowing down and becoming less relevant as a result of trying to keep up with the dynamic nature of the Intervals web-based application. We faced a crossroads. Overhaul the native PostgreSQL Full Text Search settings or develop something else.
Our team’s expertise lies primarily in web design and development. So rather than train everyone in PostgreSQL administration, we went with a solution that would have a smaller learning curve for our developers — Apache Solr. Solr is blazing fast, robust, open-source, and was easy for our web developers, who already have an extensive knowledge of XML and RESTful/JSON APIs, to pick it up and run with it. Also, Solr is widely used by many prominent web sites and is continually supported by an active open-source community.
Most of our time at the white board was spent designing the new data structures for the search indices. Once that was done, plugging them into Solr was easy. Now we’ve got Solr handling indexing and searching customer data, while leaving PostgreSQL to do what it does best, store, retrieve and maintain the integrity of our data.
Photo credit: josie lynn richards
Is it me or the search functions of the API returns all result no matter what we search for? Any changes to the API?
Yes it was me :D, good job guys!
I’m very happy to see Search get some dedicated attention and the improvements are most welcome!
The next improvement I’d really like to see is the ability to search across all “buckets” (Tasks, Projects, Notes, etc.) at one go. Many times I find myself searching each in turn for the same keywords because some people think Project Notes is the right place to stick X while others prefer Tasks.