Aza Raskin has been experimenting with a mouse-based UI for Ubiquity. Here’s my attempt – the LazyBar. It’s extremely simple. It’s for people who are too lazy to even select something.
Imagine you’re listening to You’re gonna go far, kid by The Offspring on Youtube, what are you most likely to use Ubiquity for? Searching Youtube for more videos by The Offspring? The Wikipedia page for their new album ? Well, LazyBar does that automatically for you. It analyzes the page with OpenCalais and adds the relevant buttons as shown here:
Here’s another scenario when you are on the Wikipedia page for this year’s Oscar nominations. When you hover over the buttons, you see neat Ubiquity previews as well:
Lazybar only works in Ubiquity 0.1.5. Previews don’t work too well yet (see Internals for details).
Click here to install. After you install, execute the command “lazybar-setup” or restart your Firefox to get started. On Macs (due to a small problem with toolbar icons), you won’t see the icons until you restart Firefox.
Is Brad Pitt a programmer?
I must admit that after coding the LazyBar, I’m slightly disappointed at the result because the entities returned by OpenCalais are too broad. The most commonly encountered one is of course “Person” but that doesn’t tell you a whole lot. Is the person popular enough to have a Wikipedia page? Is she an actress? What movies did she act in? Consequently, we’re just returning Google search results (and Wikipedia pages) for every kind of person. But if we integrate with Freebase (as Gray Norton has done here), we can figure out the person’s occupation and return better results:
- Politician - Campaign website, Wikipedia entry, Related news on NYT
- Entrepreneur/Programmer – LinkedIn, Twitter, Facebook, Blog
- Musician – YouTube, Song Lyrics, MySpace, Facebook Fan page, Amazon, iTunes
Integration with Ubiquity core
Eventually, something like this needs to get into Ubiquity. When we open Ubiquity without a selection, no suggestions are given. This is just as inefficient as an empty new tab screen. If we analyze the page and provide some suggestions based on the extracted semantic information, we’ll be making using Ubiquity even quicker. Any microformats found on the page should also be integrated in the same way.
Alternative Interfaces
- Instead of seperate buttons for each command like Flickr Milk, Youtube Milk, would having a single button for Milk with a dropdown menu to select the command be better? For one thing, it would allow the LazyBar to fit in more results.
- The main problem with having a toolbar (besides consuming valuable screen space) is that the urge to do a search or execute another command comes only when you are focused on the content of the webpage. So, if you’re reading this article about Kraak & Smaak, a possible solution to this problem is illustrated using transparent buttons that only appear when you hover over “Kraak & Smaak” (similar to Encouraged Commentary). But how do we do this without being obstructive?
Which UI do you think would be most effective?
Internals
- Currently, I’m taking the full text of the current page and passing it to OpenCalais because OpenCalais can only handle text. The problem with this approach is that it oversimplifies. For example, it’s likely that text in the larger font should have a greater “relevance” score.
- Once OpenCalais returns the results of the analysis in JSON, I order the keywords according to their relevance score. Then, for each keyword, some commands are assigned based on the type of entity:
- “EmailAddress” : ["email"],
- “Movie” : ["youtube", "imdb", "wikipedia"],
- “TVShow” : ["youtube", "imdb", "wikipedia"],
- “SportsEvent” : ["youtube", "wikipedia", "google"],
- “City” : ["map", "weather", "wikipedia"],
- “ProgrammingLanguage” : ["wikipedia"],
- “Organization” : ["wikipedia", "google"],
- “ProgrammingLanguage” : ["wikipedia"],
- “OperatingSystem” : ["wikipedia"],
- “Product” : ["amazon", "flickr"],
- “Technology” : ["wikipedia", "google"],
- “Person” : ["google", "flickr", "wikipedia"]
- MusicAlbum, MusicGroup, TVStation, SportsLeague and SportsGame – YouTube, Wikipedia
- Country, Region, ProvinceOrState, Continent and NaturalFeature – Map, Wikipedia
- EntertainmentAwardEvent, Holiday, MedicalTreatment, MedicalCondition, Organization, FaxNumber, PhoneNumber, RadioProgram, RadioStation, Currency, TVStation, MarketIndex, Facility, Holiday, Organization are only mapped to Google.
- URL, PublishedMedium and IndustryTerm are ignored.
- LazyBar then interfaces with Ubiquity using the internal API to execute and preview the commands. This means we have to deal with a number of inconsistencies. For example, I used the cmd.preview( {}, {text:arg}, mods || {}, elem ) function to set the preview on the elem which in this case is the popup. But later, the preview-change event is only sent to a hard-coded preview element (the preview iframe of Ubiquity) even if elem is specificied. Hence, you find that the previews are sometimes buggy as shown below. They will fixed when 0.2 is released and there’s complete separation between the UI Manager and the UI itself.
- The OpenCalais results are then cached for the session. I stored the JSON object as a preference. Blair McBride promises to give us a better way to store data in Ubiquity.
- The bookmarks toolbar is styled differently from the LazyBar. Overflows are not handled properly in the LazyBar. It was too complicated especially because Ubiquity does not support commands with multiple files. It’s a future TODO.
16 Comments
January 26, 2009 at 2:46 am
Very cool, might want to think about bringing in some of the codebase from Operator (at least the stuff that hasn’t been integrated into Firefox 3) so you can pick up any structured data being encountered on the page as well (people, locations, times). Also, do you think freebase might be a useful source of semantic information in addition to OpenCalais?
January 26, 2009 at 3:21 am
@Alex Yes! Integrating microformats is extremely important. Will look at the Operator code. I’m thinking about supporting microformats with Ubiquity first and then, exposing a API so that LazyBar can make use of it.
Freebase is really extensive (even I’m on it! [1]) and that’s great. OpenCalais only tells us that Brad Pitt is a “Person” but Freebase can tell us even more about him (Person, Film actor, Film producer, Musical Artist, TV Actor, Award Nominee, Award Winner,Influence Node, Celebrity)[2]. Then, as I mentioned in the post, we can use this information to associate “Brad Pitt” with more relevant commands (youtube, imdb, news) rather the generic google & wikipedia. For common folk like me, there’s no point providing a Wikipedia or IMDB button cos Freebase would tell us that I’m not on Wikipedia or IMDB.
OpenCalais is also different from Freebase in that Freebase is a database while OpenCalais analyzes the page without referring to a database backend. Sometimes, on blogs with a lot of comments [3], it claims that “Jim Jeffers Says” is a Person. So, we need to use both.
[1] http://www.freebase.com/view/guid/9202a8c04000641f8000000009025301
[2] http://www.freebase.com/view/en/brad_pitt
[3] http://donttrustthisguy.com/2009/01/04/encouraged-commentary/
January 26, 2009 at 4:54 am
Abi:
Tom Tague from Calais here.
First – very sweet! We love it when people bolt together existing tools with a lot of creativity and make new things happen. This is a great example.
A few suggestions to think about.
Two tier menus: Space on the toolbar is precious – if for a particular entity type you could offer a drop down to launch the presentation around different sources (Wikipedia, Flickr, etc) then you’d have much more space to work with.
Linked data: With the new release of Calais we provide Linked Data assets for about 25 of the entities. For example Company is linked to a Company RDF page that has information about the company and disambiguated links to the appropriate pages of Dbpedia and Freebase. You can then harvest these pages (and their links, and their links….) to construct a nicely formatted summary of the entity.
Hover Over: One way around the toolbar problem is to use hoverover on entities– like our Gnosis Firefox plugin. Downside is that it can be visually unpleasant. Another option is to insert in-stream icons before selected entities.
Popularity Contest: We don’t disambiguate person (yet) – so you might want to basically use a popularity contest (are they on Freebase, etc) to do so. Obviously this will be a hack – but from a likelihood of correct hit basis it’s probably not bad.
SemanticProxy; You might want to take a look at semanticproxy.com. This is a front end to Calais that fetches a page, attempts to clean it and then returns the RDF to you. Might help get rid of some of the HTML noise. Sematicproxy.com also caches results pages so you should see higher performance on frequently accessed pages.
We’re looking forward to the next iteration.
Regards
January 26, 2009 at 7:49 am
@Tom Thanks for the stopping by. The suggestion about the two-tier menus sounds good. Currently, I’m more likely to experiment with the hover over Encouraged Commentary-like overlay in the next iteration. The more I think about it, the more it feels like the way to go. Especially because you don’t lose your focus on the text that you are reading.
About the popularity contest thing, the hack won’t be necessary. According to Will Moffat here [1], “Freebase supports many different namespaces (Wikipedia, IMDB, Musicbrainz etc) and the Freebase Search API has a not-yet-documented option to return these keys directly.” That should work fine for checking if a person is on the website or not.
The API of OpenCalais is pretty large and I haven’t completely explored it yet. I’ll check it out the Linked Data capability in Calais 4.
When I was testing all the semantic services available, I considered using semanticproxy.com. But it seems to work only on certain pages. I get a “Unexpected Error” on many websites including Youtube (which is one of most common use cases). Another problem is that I’ll be passing a URL rather text or HTML. This won’t work for Gmail, Google Reader and anywhere else where you are logged in. Getting the LazyBar to work on all pages was a very big priority, so I decided to use the OpenCalais text API.
[1] https://ubiquity.mozilla.com/trac/ticket/265
January 26, 2009 at 3:13 pm
I really really like this, it just needs to get smarter and better integration (not to mention support for the new ubiquity
)
But I really look forward to trying this once it’s had a little more development.
Also, I’d just like to say how pleased I am that there’s such quick development with ubiquity and various things around it, no sitting for 3 months and seeing no changes at all and wondering if anyone’s still even working on it.
Great work!
January 26, 2009 at 7:06 pm
Hey Abi
Fantastic work! One of the first things I noticed was that there’s just way too much stuff in the “LazyBar” — it loses its sense of connection. Let’s delve into effortless context: we know roughly what the user is looking at based on what’s on screen (i.e., where the viewport is looking). Can you try updating the LazyBar after page scrolls? That way, when I’m reading about something, the LazyBar will always have something that’s targeted and interesting.
January 27, 2009 at 8:06 am
@Aza Agreed. What you suggest should solve the issue to some extent and is probably simple to implement. Once (and if?) it’s possible get only the text/HTML in the viewport, then it’s just about matching the all the keywords with the viewport text.
But I think toolbars are ultimately problematic. They’re too high and they’re above the content. I think most people are really focused on the webpage rather than the chrome (do you ever look at your tab bar when you’re reading a page?). Here’s another hypothesis – because text flows from from top to bottom, when you’re reading something, it’s easier to look down rather than up. So, another possible place to put the results is in a fixed bar at the bottom like what Facebook does (not the statusbar). But even this takes up screen space. I’ll have to think more about this problem. The “hover-over buttons” seems like a better approach compared with any kind of bar.
February 2, 2009 at 1:34 pm
Could you add a hide lazybar command because once you have set it up it there is no other way to remove it other than restarting the browser.
Although it seems a cool thing i first i think it will get annoying after a while because out of all the commands that get generated only a few selections are going to interesting. For instance if a user surfs to tech sites a lot he/she doesn’t want to see commands for html, api, javascript every time.
Maybe there should be a way to ignore certain selections?
February 3, 2009 at 9:25 am
@david You can unsubscribe to the command on about:ubiquity if you want to hide the LazyBar and then, resubscribe whenever you want it again. Yes, we should definitely ignore “certain selections” as you put it. But I think the whole toolbar UI is terrible for this.
I’m working on the overlay concept right now. Expect to be done in a week or two. Thanks for testing this version of the LazyBar!
February 3, 2009 at 10:45 am
@abi is there a reason why the lazybar needs to be unsubscribe to become inactive. I restarted my browser but the lazybar stayed. It’s not the most common way for a command to behave.
I even unchecked the lazybar in the commandlist but it still stayed on top.
I understand it’s a prototype but it seems like a drastic way to stop a command from working.
February 3, 2009 at 7:55 pm
@david The reason for that is that not all Ubiquity “commands” are commands (where you execute the command and something gets done). Ubiquity also allows you to write functions that begin with pageLoad_ and startup_ and it will run them on page load and startup. Unfortunately, as these kinds of “commands” are different from traditional commands, things like the commandlist which are designed for traditional commands might not work as expected. So, as you would expect any toolbar to behave, once you install, you always have the toolbar when Firefox is started up.
You can use this command[1] to hide the LazyBar but it’ll popup up again once you restart (for reasons mentioned above) unless you unsubscribe from it. You can always resubscribe easily and directly from the about:ubiquity page in the feed graveyard.
[1] http://gist.github.com/57911
February 4, 2009 at 10:35 am
@abi thank you to take the time to explain the behaviour.
I’m wrapping my head around the way to make make greasemonkey like commands. But i will ask more questions in the ubiquity google group.
February 5, 2009 at 1:06 pm
Hello –
I installed lazybar and enjoyed fiddling around with it. I don’t think I’d like it permanently installed. I unsubscribed from it, but the bookmark toolbar icons persist. Is there anyway to remove that?
Thanks.
February 19, 2009 at 6:42 am
@Jeremy Unfortunately, there are no uninstall hooks that allow Ubiquity to clean up any changes that it made. But you can do it manually, find your profile folder (http://kb.mozillazine.org/Profile_folder), then go to the Chrome folder within that and delete the userChrome.css file.
June 17, 2009 at 11:21 am
I just found this command and I’m liking it so far. But I think that this command would be even better suited for the newly released jetpack
https://jetpack.mozillalabs.com/
That way you would have more options with the chrome appearance.
June 26, 2009 at 2:18 am
Glad you like it.
I’ve been thinking about various interfaces for the LazyBar. Yes, it might more sense as a Slidebar in Jetpack but I have my doubts about that too. I’ll try implementing a prototype and see if it actually works.