Ok I’ve cracked it! The Scout driver is more limited in the query construction it can support.
I’m breaking out to use the Meili client directly and can build up my queries with a lot more flexibility.
Migrating Search to Meilisearch or Similar
I began a partial migration to Algolia for cafe search a while ago. That was a spike that only covered some of the simpler uses.
As part of the work to make location search easier with suburb autocompletion I’d also like to move the suburb database into a search index. The free tier of Algolia is limited and it’d blow away my quota, so I’m looking at Meilisearch (MS). That’d mean the cafe search would move into it as well.
Initial testing was ok, but it seems that MS does not support multiple geo location positions per document in the same way that ElasticSearch does. This would have been extremely useful, as my data model has a one to many relationship between cafes and their individual locations. The search would be easier to build if MS could internally handle that.
If I’m right, I’ll either need to do a hybrid, multi-step search to find cafes within a location, and then another with those ids to handle the main search which might sort by score and other ranking factors.
Going full ElasticSearch is also not that appealing because it’s not supported by Scout, so I’d have to do more custom data mapping, querying and syncing.
Much to consider.
Cafe Search Progress
Oh my god, this feature has been like an iceberg that reaches the bottom of the Mariana Trench.
I thought I had a decent code outline before this Christmas week of development, but there was way more to it.
Location
I didn’t want string based matching of the location search term, because nearby suburbs, and relevant matches would be excluded. The feature uses a maps API to geocode the term into a lat/long position. It then uses the spherical distance formula to return a set of cafes within a reasonable distance. As the database grows I wanted to avoid huge DB table scans, having to calculate the distance for every row in order to get a result.
So to make that more performant I’m first calculating an approximate bounding box to limit the results using indices on the lat/lng columns.
The size of the area to search, and the number of results to expect also depends on the context. For example if you’re in Melbourne CBD you’d expect to get a good number of results in a close radius, and showing cafes 4km wouldn’t be useful. However if you’re in a regional Queensland town with much lower density of specialty coffee the radius might need to expand a lot more to show any useful results.
Searching with Cafe Name
This also complicates the search logic as it affects what might be reasonable search radius when used in combination with a location. The name searches uses a full text index against a separate table help fuzzy match terms (to avoid the user having to match it precisely).
This means some irrelevant cafes might be returned and I don’t want to limit the radius too closely or we might only return the wrong cafe, when the correct one was just a bit further away.
I also discovered the complexity of different location scopes. For example if a user searches for “Foster and Black” in “Brisbane” the geolocation lookup will centre Brisbane as being the centre of the CBD. Foster and Black might have locations in various Brisbane suburbs but they might not appear if those suburbs are too far from the CBD. The users intent would be to include them though.
I’ve attempted to solve this by expanding the radius when a name is included until either the max radius is reached, or we find cafes matching the name.
It’s difficult to know until I have more users and cafes if the search logic will hold up and be robust enough. Long term I might need to migrate to a dedicate search tool like Meilisearch or Elasticsearch if they also support geography.
Progress this Week
I’ve spent about 18 hours this week on this (along with a bit of infrastructure work around servers and build tools) and am pretty happy with the progress.
I have found inconsistencies in the data format being returned to the frontend though (depending on the search criteria). I’ll need to normalize that before building out the frontend Vue components.
I’m also wary of the UX if the user clicks away to a result cafe, then comes back to an empty search state. I might look at implementing InteriaJS so this can happen in an SPA style to preserve state. There could be some edge cases there in partially implementing it into a project that also uses traditional server page loads.
Miscellaneous Problems
This UI uses a location watcher for the browser’s geolocation API feature. I need to geocode that lat/lng too, and I found that in some situations the browser just keeps reporting either the same location, or a very subtly different location again and again.
To avoid thrashing my API endpoint that does the lookup (which itself caches/proxies the external API) I implemented a basic client side cache so that repeat requests for the same position are not sent to the server.
Testing
Unfortunately I need to refactor some of this work to be more testable. The geo lookup code a bit too embedded in other logic, so to test that would result in real external API calls.
I need to abstract it out (already used in two places in the app) and make it mockable.
Next Steps
I think the search logic is passable. I would like to incorporate more algorithmic sorting including the full text search ‘score’ of the cafe name, and other parameters such as rating and relevance to the user’s coffee history. That might have to wait for phase 2 though as it’ll add a lot of time to development otherwise.
The immediate step is to build out the result set frontend components so they look good and are usable. There’s some data massaging to go along with that so that individual cafe locations are combined into the parent cafe rows.
Beta Launch
And as far as I can remember the only other main features I need to launch are:
- User ability to add a new cafe (and for me to review the data and approve)
- Ability to edit a reviewed coffee (e.g. fix a mistake)
Would also really like more robust testing of the search feature.
Cafe Suggestions with Geo Search
I’m keen to make some solid progress on HadCoffee over the Christmas period to hopefully get to an early 2019 v1 launch. This week I was working on the main coffee rating interface which depends on users being able to select or enter the café they visited.
I’ll add the autocomplete lookups for typed cafe names, but I also wanted some basic quick suggestions based on where the user has been before and their current location.
This would save typing for common places and maybe also hint at other nearby cafés the user mightn’t know about.
The frontend part of these feature went pretty smoothly. The Vue component watches for the users location (HTML5 Geolocation) and calls the backend to load these quick suggestions as it’s known. There’s a little debounce action to slow things down as the geolocation API can return updated positions in quick succession as it locks on a more accurate location.
Geographic Search by Proximity
The smooth progress I was making through this feature hit a wall when it came to actually querying cafés by distance though. I was hoping to use MySQL’s ST_Distance_sphere function to let the DB do that work. I’m running MariaDB though, which although it’s advertised as a ‘drop in’ replacement for MySQL does not support this feature ????
I prefer a simpler dev environment (I’m not using Laravel Valet or Docker images) so I didn’t feel like swapping to MySQL for this project. Changing my workflow to use Valet also wasn’t very appealing when I’m otherwise happy with the setup. so I briefly tried migrating to Postgres. I know it’s a great DB, but I haven’t used it before and that’s a big change to have to make to run one type of query.
In the end I’m going with a raw SQL query to help with this. I’ll add a simple bounding box to its parameters first to avoid having to do a table scan of every cafe in the world (once my DB gets to that point ????)
Although it took a windy path this geo search will also provide the basis for the other cafe search features on the site such as the autocomplete (to improve relevance) and the location based search.
$query = "SELECT id, cafe_id, lat, lng, address, locality, city,
( 6371 * acos( cos( radians(:lat) ) * cos( radians( lat ) )
* cos( radians( lng ) - radians(:lng) ) + sin( radians(:lat2) ) * sin(radians(lat)) ) ) AS distance
FROM cafe_locations ";This is a good step towards being able to add café & coffee reviews, however the next big sticking point will be letting users add new cafés as they go.
Ideally I’d like to collect a bit of meta data such as roasters, menu and seating options to help users finding cafés, but I’ll have to see how much data entry users will tolerate. I also need to be aware of how or if I can verify this community sourced data.