Ubuntu Smart Scopes

A new feature of Ubuntu was discussed today (which is like an announcement but without overhyping it), it is called Smart Scopes and is documented here https://wiki.ubuntu.com/SmartScopes1304Spec go read that first and then I have a video for you to watch.

http://www.youtube.com/watch?feature=player_embedded&v=CBeQur7VBDM

Now go back and read the spec that I told you to read earlier, but all the way to the end this time.

In the video from left to right is Alan Bell (me), David Callé, Jono Bacon, Michael Hall, Roberto Alsina and Stuart Langridge, all discussing this new framework for searching. It is coming soon, to the Ubuntu Raring desktop and then to phone and TV and tablet etc. The objective is to make searching really really effective and helpful to the user, but as with the previous efforts in this direction there will be some concerns around how it is implemented.

In short, Canonical will be running a server much like the existing productsearch.ubuntu.com server which will accept queries and return a bunch of results as json. The current implementation searches Amazon and the Ubuntu One music store and a few other places. The new one will do the same, plus more server-side searches, plus a new feature altogether which is a list of good scope names for the client to search. Your client will now send a list of all locally installed scopes to the server (actually a list of scopes you have added and a list of scopes you have removed or turned off from the standard set) along with your query. The server then returns results it found and wants to put in your dash, plus a subset of the local scopes you sent it, in order, that the server thinks would be good places to hunt for your search term. This means that your client might have 100 or more locally installed search scopes, but the server will advise it which are likely to give good results. Now for the scary bit, once you have looked at the results and perhaps clicked on something then your client pings the server again to tell it which scope produced the most relevant result. This means that the server can learn from this feedback about which scopes produce high quality results for that keyword, and perhaps rank that one a bit higher in future recommendations lists.

  • Lenses are now called master scopes
  • You control each individual scope that you want to search in or not search in, not the master scopes so you will have 100 or so things to turn on or off.
  • You can still have locally installed scopes that search authenticated data sources
  • You could in principal run your own search server if you write one to implement the API and patch the home master scope to look at your own server
  • The server isn’t open source
  • You can’t opt out of the feedback process (without turning off the smart scope altogether – which you can do)
  • If you install a local scope then your client will tell the server the name of that scope
  • Every query to the server is going to include a list of locally installed scope names (100 or so perhaps?)
  • You can focus a search at a particular scope by using a keyword, for example “omlet: chicken house” to only search the Omlet scope and not the chicken stuff master scope.
  • The rather poorly thought out remote-content-search checkbox to disable local scopes from doing online searches remains in place – however you don’t need it as you have per-scope controls.
  • There may be some code quality checks introduced to stop scopes that don’t pay attention to the remote-content-search setting from getting into the Ubuntu distribution. – but you don’t need it.
  • This probably won’t put more adverts on your desktop while you are trying to do work.
  • This is probably a more private way of searching for stuff than googling for it.
  • This won’t be opt-in, all the good stuff in Ubuntu is turned on by default.
  • Your IP address gets logged on the web server logs, but not in the database of the smart scopes application running on the server. The developers working on the smart scopes don’t have access to the web server logs.
  • It would be relatively trivial (I could do it in a day or so if I felt like it) to write a gnome-shell client for this smart scopes server to display the remote results, however doing something with the scope recommendations list would be a bit of a struggle.
  • The home master scope (dash) search box will contain the help text “search your computer and online sources” to make it clear that it isn’t just a local search.

Now to the big question. How much are people going to freak out about this? Well if they read the spec all the way to the end they will see all the stuff that is being collected, how it is aggregated, how much or how little privacy this is costing them and why it is being done for the greater good of having decent search results. The feedback data collection process is likely to be slightly freakout causing. I can see why the developers want this turned on and I can see why it is antisocial to turn it off, like leeching on bittorrent while downloading an Ubuntu iso or whatever. I think they would be wise to have a checkbox in the privacy settings dialogue so that antisocial people can turn this off. I imagine the developers will stick with the current policy that if you want to use smart scopes you have to participate in the feedback process to make it better.

I think we need to do some education around the lack of an applications launcher though. Currently people think that Super + name of application is a replacement for the Gnome 2 applications menu. It isn’t. Super+a + name of application is how to start applications. This is going to focus the search on just applications and will work a lot faster than doing an omniglobaleverywhere search which is what the superkey does by itself.

For me this is a good development overall. The privacy debacle will be solved to my satisfaction when you can locally and personally blacklist scopes. This will mean that I can write a scope without it being co-dependent on all the other online scopes and I don’t have to worry about whether intranet access constitutes internet access. All scopes can simply stop if remote-content-search is set, but nobody needs to set it, the flag will basically just break all searching and be a bit pointless.

30 Comments

  • Jef Spaleta says:

    The effort to document the over the network data collection, and the motivation for each piece is much appreciated.

    I would suggest that designers talk to people using Canonical products inside corporate networks that are established as ubuntu friendly corporate environments and make sure this concept of integrated searching, and data flow in the UI is compatible with standing corporate acceptable use policies.

    And a follow-up on that point, does the search functionality fallback gracefully to something useful using installed scopes if corporate network admins blocks traffic to Canonical’s products server?

    -jef

    • Alan Bell says:

      good question, I think that they are mostly considering individual end users on personal machines. I think scopes and searching are massively more useful to the business environment and it can be shoehorned into that context but clearly it isn’t a primary design consideration. They don’t appear to have a design persona that works in a cube farm!

      • Jef Spaleta says:

        Even for personal devices….on corporate networks… like smartphones and tablets, the are very real issues concerning data leakage that network admins is serious work environments like municipal government, fed government, gov contractors, law and medical offices really really have to worry about.

        Ubiquity is a huge mess for these sectors and if Canonical can find a way to work within the constraints of the data privacy requirements and deliver a UI that is a bit more… empathetic… to data privacy issues, I’d wager its a significant bullet point in their favor. Moreso if they can turn the corner and can build an on-premises or cloud search appliance for corps as part of managed services that replaces their general use products server and can give corporate clients customized search with a lower risk of data leakage to any 3rd parties…including Canonical.

        -jef

  • I’m less freaked out about the privacy concerns and more freaked out that more of Ubuntu is /relying/ on non-free software to function correctly.

    What I find even more concerning is that many people at Canonical don’t even seem to understand that.

    • Alan Bell says:

      I don’t think anything important is relying on this. Mostly it seems like a performance hack to allow hundreds of scopes to coexist, by only searching a few of the more probable ones. It would be interesting to know what the fallback position is if the client is offline or the server is unavailable or slow.

      • Michael Hall says:

        The fallback is to use the “default scopes”, which are those that the Dash will always search, regardless of the smart-scope-service. This will include things like local app, local files, and others that the user should be able to configure for themselves.

    • John says:

      Yeah. The problem is it doesn’t matter what they say about privacy and data collection, if the software is closed we can’t know for sure what is really going on. I never thought they could do this, but it seems Canonical turned the Amazongate into something even worse.

      Open its source and we may talk about it.

      And I don’t Google, I use DuckDuckGo or StartPage, so no, this won’t be more private and secure to me.

  • I appreciated your efforts to query the opt-in and feedback privacy elements, Alan.

    What I’d really like Canonical to do though is offer me the option to trust them. At the moment, they take my trust for granted and I feel like rebelling against them for it. Conversely, Google is very open about asking whether I trust them – so I do. They know every single query I’ve typed into my browser or phone going back years – because they asked me if that was okay, and sold me on the benefits of saying yes.

    I’d like Canonical to offer me the option that I don’t mind if they identify and profile me when using their Dash features. They’re baking in all these backlash privacy features to appease the masses… but if they just asked up front “can we do this”, I bet many, many people like myself will say “yeah, okay”.

    And many won’t – but that’s okay, because now at least now even those guys respect Canonical for asking.

    • Jef Spaleta says:

      Neil,
      I belive the counter argument synopsis is that the action of choosing to use the Unity desktop, the action of installing the Ubuntu desktop product, or the Ubuntu phone product or the tablet product and so on.. is the “I trust you” moment.

      This might not sit well with you as an established Ubuntu user, but jump into your time-machine and travel 20 minutes into the future… the future where the Ubuntu brand is a retail consumer device stalwart. In that future, the trust in Canonical is implicitly assumed as part of the consumer purchase of the Ubuntu powered devices. Not just assumed, but a selling point of the lifestyle brand.
      You either trust Canonical, or you don’t. You either buy into the lifestyle brand or your don’t. Much like how react to the Apple brand. That is the future Canonical wants. It is not a future built with the loyal opposition in mind.

      -jef

      • Fabio Rosa says:

        I guess thats the whole problem.
        Dont try to be apple. If I want apple, I would buy it (and I dont).
        Keep being canonical, and ubuntu.
        Inform and educate people. Technical docs are good for us, but not for the average user. You _MUST_ warn them, and inform them in an acessible way, and _give_ the option of not going this way.
        Thats the microsoft and apple way… And I would like that Ubuntu just did not went the same direction .
        Makes me sad …
        Would realy kill someone to ask , during installation, if people do agree with this, the same way you ask if you can install non free software … Why is that so diferent, why are you being so freaking unflexible about that ! ??
        I just dont get it… really …
        It is the very first time that I feel really disapointed with canonical

        • Alan Bell says:

          There are a lot of things not asked about in Free Software. You don’t get asked to click through a GPL EULA every time an application is installed. It just gets on and does what you asked it to do. This is much the same. If you upgrade or install Ubuntu Raring to get the Smart Scopes feature then you get the Smart Scopes feature without dicking about with unnecessary questions in the installer. I can see why they don’t bother asking in the install, but I did advise them to add a checkbox to allow leechers to opt out of the feedback process.

          • Zygmunt Krynicki says:

            The GPL is not an EULA. You don’t have to agree to it to use any software. Please get this right.

          • Alan Bell says:

            I did get it right, it isn’t an EULA. I said that. We don’t do EULAs because it is a license to copy the software, not a license to use it. My point was we don’t bombard the user with pointless clickthrough legalese and caveats.

          • Fabio Rosa says:

            I do agree that, if you want to use smart scopes, the feedback is mandatory, there is no problem on that.
            The problem is: Smart Scopes _WILL_ send information to the outside …. So, it just makes sense to ask me _IF_ I want smart scopes , or if I just want Local scopes.. The same way yo already ask if I wanna install the mp3 codecs and other non-free software…
            Its the same class of problems…
            Warn for non-free software installation…
            Warn for sending information to your servers.
            Its only a checkbox, you can let it already checked (as you already do with mp3 codecs)… and most people will not notice..
            But, the ones that are sensible to those kind of things, can take action during installation.
            Its about sending a message:
            “See, we built this for you, and its great, but, we need some info from your computer for that. Do you agree?”
            _Everybody_ gets happy … with a single checkbox … !
            Im pretty sure that there are people out there that wants Raring not because of Smart Scopes, but, for everything else that makes Ubuntu an amazing distro…
            Show you care about other people privacy concerns …
            Im pretty sure you _DO_ care… I do trust canonical .. But, there are people waaayy more sensible to those things out there…
            Its all about sending the message …
            And, you are sending the wrong message…

      • I think you’re right, Jef, that they believe that this is the “I trust you” moment, but I believe that’s utterly wrong, on many levels – transparency, privacy, trust to name a few.

        To clarify, I have expectations about Ubuntu formed from 6 years of experience. But when 12.10 came out, and without making it clear anywhere during the install or upgrade, they have assumed my trust.

        I’d likely give it (if they asked), but they assume it, and that’s wrong. And they’re still assuming it. And they refuse to NOT assume it, which is bold, but also in my opinion, very wrong.

      • icewater says:

        “lifestyle brand”. Oh boy…

        So Canonical wants to become Apple, you say?

    • Michael Hall says:

      The primary difference is that we won’t be able to identify or profile you. Your searches aren’t connected to you, we couldn’t tell you if any two searches belonged to the same user, let alone a user’s entire search history. The only data available will be connecting search queries and the name of the scope that provided what the user picked.

      • My point, Michael, is that I’d actually like transparency in this process so that I can GIVE YOU permission to NOT HAVE TO GUESS. In other words, you would know that concurrent searches are from me, and in fact, you’d have a history of everything I searched for AND a profile of what apps I use most.

        But instead Canonical are sneaking these features in, then Stallman and others are having a fit, because it’s not transparent.

        I doesn’t matter that you provide an off switch. For most, the damage will already have been done – they’ll feel their trust has been abused and that will undermine everything you do with this amazing technology.

    • Tor H says:

      Firefox Mobile has the perfect UI solution for this, which I believe Canonical should consider. When you first start typing in the URL bar, it has a yes-no-question “Do you want to show search suggestions from Google?”. You answer, one click, and you’re done. Simple and effective.

      Similarly, the first time you type in the Dash, why not ask that same question?

      I realize Mark Shuttleworth has claimed we already trust Canonical fully because they “have root”, but you know what, an attitude like that does not make me less concerned. Yes, you have root, but the code is open. If you did something bad with your root accessible code, one of the many wiresharking security geeks would notice and there would be an outcry. However, once my IP address with my search query hits your server, no one has that ability to wireshark it except Canonical – and that’s a substantial difference. If I want to use this feature, I now have to trust in code that no one but Canonical can audit, on servers that no one but Canonical audit. Even with proprietary code running on my own machine, I get to see the network requests being made.

      I also found the use of the term “antisocial” a bit tendentious, I hope Canonical doesn’t officially use such language. I agree that it can be immoral to reap the benefits without sharing back, but I’d say that sharing-by-default is equally immoral, since the majority of users won’t even know they have a choice. What would be antisocial would be to gather all that anonymized data and then not share it with the rest of the world in raw form 😉 If it’s not private, it damn well better be one hundred percent public.

      • Alan Bell says:

        I have not seen anyone from Canonical use the word “antisocial” in regard to failing to do the feedback ping to the server, that was my word.

  • Ken Kinder says:

    So, let me get this straight. Canonical believes that there are a multitude of people out there who are dissatisfied with the current general purpose search engines of the Internet, and some random aggregation of stuff like Amazon and a music store, plus whatever they’ve thrown together in the latest iteration, jammed into thumbnails in Dash, will definitely be better than what Google, Bing, and Duckduckgo have been working on for over a decade.

    ಠ_ಠ

  • […] Alan Bell, hacía la presentación de lo que será Smart Scopes, que tendremos pronto en Raring Ringtail y […]

  • Adonis K. says:

    “The server isn’t open source”
    “You can’t opt out of the feedback process (without turning off the smart scope altogether – which you can do)”
    “This probably won’t put more adverts on your desktop while you are trying to do work.”
    “This is probably a more private way of searching for stuff than googling for it.”
    “This won’t be opt-in, all the good stuff in Ubuntu is turned on by default.”

    Cool story bro.

    Keep stuffing that dash as much as you can, eventually people won’t even be able to search their local sources without a 15 min break.

    • Alan Bell says:

      Feel free to use something else, Ubuntu distributes other desktop environments such as Gnome Shell and now Cinnamon. You could also use Debian or Fedora or one of the others if you wanted to.

      • Ken Kinder says:

        I think the consternation is more that, for better or worse, Ubuntu has become the standard-bearer for the Linux desktop. It’s invariably what geeks have told their friends to try, if they want to experience Linux without a lot of hassle. Now that Canonical is treading water not even Microsoft or Apple dare — putting advertising right in the heart of the GUI — we all kind of look like asses because we have to say, “oh, no, we don’t recommend Ubuntu anymore. It broadcasts your keystrokes in your app launcher and uses those to serve you ads.”

        And then we’re supposed to say with a straight face that part of the beauty of Linux is all that freedom and Open Sourcy user control type stuff?

        Ubuntu has more or less gone from being the defacto default Linux desktop to the black sheep, and rightfully so.

  • Paradiesstaub says:

    Will there be the possibility to turn the always-online-search the other way around and only search online when typing a special keyword?

    An option in the privacy settings link this would be nice:
    search online sources when search is prefixed with ‘[editable character/keyword] [ ] <– check box

    This way the search remains local – the way it should be – and queries only the net when desired.

  • […] Más información – Unity: Cambio entre ventanas a través de las listas rápidas Fuente – The Open Sourcerer […]

  • Patrick says:

    Pure garbage, Canonical. You sold out our privacy for profit. It would have been better to ask users for a nominal price to purchase Ubuntu OS. This trend of usurped privacy, data collections and push notifications from software developers is spirialing out of control and, quite frankly, I’m tired of it. I choose Ubuntu years ago because they were “not” likely their competitors in this. Now, you’ve disappointed myself and many others. Poor choice, Canonical.

Leave a Reply

XHTML: You can use these tags:
<a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <s> <strike> <strong>