Snappy Packaging – Don’t ignore the hype!

This week I was lucky enough to be invited along as a guest to a Canonical sprint in Heidelberg, Germany, to topic of which is the Snappy packaging system(http://snapcraft.io), pioneered by the folks here at Canonical.

I like to think I keep up with current events pretty well in the Linux world and it hadn’t escaped my attention over the previous couple of months that there had been a lot of negative press surrounding the Snappy packaging format quoting a lot of  Redhat based distros. As with Juju, I turned up to an event not fully understanding the project, not fully understanding its purpose, or for that matter,  why I’d been invited!

The thing that struck me on the first day was the diverse group of people here at the sprint. Of course the majority were Canonical employees, but there were people here from a number of different Linux distributions and a number of big Linux based open source projects and of course people like me. Canonical had really excelled in making sure people from across the Linux world had been invited to attend and it really showed.

Day 1 was like a who’s who of people who I’ve known virtually for many years in the Linux community but never seen in the flesh, so once I’d picked my jaw back up off of the floor and got to work, it was very interesting sitting in a number of discussions where very clever people who understand how the Linux systems all inter-operate , figure out how to make snappy work across the whole Linux ecosystem.

So I guess at this point it’s worth mentioning what Snaps are.  Originally developed by Canonical for their phones and mobile devices,  Snaps allow programs to be run in a confined  environment wherever the snap daemon runs. So you can run the same code on Ubuntu Classic or Core,  Arch, Redhat etc without any modifications. How can this be assured?  Snaps allow developers to bundle all the dependencies they require in a single package, they can also delegate certain dependencies back to the underlying system. To ensure data security and enhanced platform security, Snaps are also sandboxed and their file system access tightly controlled. For application developers, this is great because it allows them to build once, run everywhere. For end users it also allows them to roll back should they mess up an upgrade. Not only that but they will also be able to restore all the old configuration data at the same time!

Back to the sprint, day 2 I wanted to try some stuff, it’s all well and good sitting in planning meetings but I still didn’t understand how it all worked. So I started “snapping” a couple of Java applications to see how hard it was… it wasn’t, which is great. But I was also aware that to get people using Snaps there needs to be a simple way for them to be able to build and integrate Snaps into their build chain. They don’t want to have to learn lots of new stuff because open source developers generally have enough on their plate. I’m a Java developer, so I wanted a way for me to be able to leverage my existing Java builds but package up the finished artifacts into something that I can deploy instantly. To fit that requirement I started work on my first Maven plugin: https://github.com/buggtb/snappy-maven-plugin . This is very much work in progress but it means I can now build snaps for all my Java apps, hopefully more build tools for different build systems will appear like this to help onboard developers with minimum disruption to what they already do.

Wednesday night was a show and tell night and there was lots of cool stuff on display, I ended up being the last talk of the night and presented “Don’t forget the end users” (https://docs.google.com/presentation/d/1UdKSsuXpYSy25V9HuxnqgzQRmlC1DEtLJUgEY-5PDoc/edit?usp=sharing), I took a bit of a risk and trolled Mark slightly whilst he was in the room, I hope he see’s the funny side otherwise I’m not being invited back to another Canonical event! But I think the message was important, when you’re going and building all this cool new stuff, don’t forget, to make them a success you need the buy in and support of a diverse community of developers. If you just inflict new build tools and standards upon them, they will push back, you need to engage and work with them to make sure they understand why the new platform is going to be of benefit to them!

One interesting take away we had from chatting on Wednesday night, and I think Juju suffers from the same affliction, is the naming.

  • Snaps
  • Snappy
  • Snapcraft
  • Snapstore

  • Juju
  • Juju Charms
  • Charming
  • Charmers

and so on. Canonical need to do more  to make it easier for both developers and end users to find and ingest the information they need.

Snaps could easily be the future of packaging on the Linux platform, but there is competition from Flatpak and others. Engaging the communities and proving its worth is key and they are off to a good start. Similarly ease of usage and access for both developers and end users is very important.

Finally, I’d like to thank Mark, Claire, Dustin for inviting me to the event, I’d also like to thank everyone I engaged with over the last week in understanding and helping me realise how useful snaps are going to be.

Dear Jeff Bezos…..

Hi Jeff,

I emailed you on September 3rd regarding your companies use of Saiku Analytics, but it would appear you’ve been busy launching rockets! I like that, I’m a rocket fan myself, not on the same scale as you but when I can afford it I like to fly model rockets in the fields behind my house, here’s a video for you:

Anyway, back to my point. I spoke to a few of your employees over the last few months, one of them was very helpful, he said they’d been running a Saiku Analytics proof of concept internally, but selected Tableau because they had no in house Mondrian skills. That response, whilst sad, isn’t the first time we’ve heard it and was very nice to hear the feedback from someone in your company.

I also made contact with someone else in another department within Amazon who use Saiku, and raised the prospect of having a chat about sponsoring Saiku or paying for licenses etc. Now, I know they got my mail, and I also know they use Saiku because here was the response I got:

Should I talk to this guy?

Besides us just buying EE Edition other possible scenarios are us contributing some development effort to the project or getting the features we want prioritized if we pay for EE edition.

This is comparison of Community and Enterprise editions: http://www.meteorite.bi/saiku-product-comparison

Rita

Now, I assumed that had been sent to me in error because, lets face it, I’m the vendor so it would be weird asking if I should talk to myself. So, I replied to Rita, and suggested they sent that mail to the correct person and that we should certainly have a chat. But then the line went dead, and I’ve followed up on that email 7 or 8 times without response. So I can only assume either Rita was a little embarrassed by the mistake, or their boss decided it wasn’t worth the effort. I suspect it was probably #2.

Now Jeff, Amazon earns a lot of money, and we are an open source software company, and unsurprisingly we don’t compete with Amazon in terms of revenue, so to help we are running a small kickstarter to fund Saiku and allow us to build Saiku Reporting, we have 5 days left and still need another £8000. To keep building open source software, vendors like us need the support of companies who use the software, because we have to be able to pay our bills and feed our families, and I won’t lie, that’s a struggle because the bean counters see the “free” sign, and take that literally.

In case you were afraid you couldn’t afford it, or might annoy your share holders with their lack of return, I’ve created a quick report in the tool we’re trying to build to demonstrate how much time it would take you to generate enough revenue to pay for the kickstarter in its entirety:

Untitled

As you can see its less than 8 seconds.

You can see the report proper here.

We have 5 days Jeff, just spend a minute considering what companies like yours do, you like supporting open source projects when they are in your customer facing tools and part of the “common good” or when you get some nice PR, but when its for internal reporting and use, the Amazon’s, Samsung’s, HP’s of this world, all like to take and not give back.

Happy Christmas Jeff, I hope you have a good one.

Tom

Dear Samsung (once more)……

Dear Samsung,

I thought I’d write you another letter. A few months back you’ll recall I wrote you a letter asking about your use of Saiku CE within your organisation. Very nicely, a couple of your developers in Vancouver got on a call with me where we discussed the use of Saiku within Samsung and I got the chance to demonstrate Saiku EE. It was a very enjoyable call and enlightening as well. I had assumed that Saiku was used occasionally as part of a Pentaho BI server or similar, on the call I learnt that Saiku Server 2.5 is actually quite a big part of the report distribution for the team, and judging by the analytics we receive from google, is certainly used by the Kies group and others for report distribution.

Anyway all that aside, after my demo we sadly ran out of time to talk about the bigger picture and I can’t help but feel from the follow up emails the people in charge of the Saiku Server project largely missed the point of our call, which is possibly my fault, who knows, which wasn’t EE upgrade or license sales, but just contributing to the project as a whole. It would be great if Samsung bought an EE license, hell, our most expensive license only costs $35,000 and would be a nice contribution to the platform which you have used for a year or two now, you don’t have to use it, it would just be a nice way to acknowledge its use. But your developers cited a number of issues preventing the upgrade to 3.x and that is fine, I understand and accept that. What I don’t really understand is why when we followed up asking politely if they wanted to sponsor development, fund the project or commit some resource, we were sent a reply suddenly diminishing the role Saiku played and told that we’d hear from you if anything required building but otherwise you were fine. I find it a shame a company that reported $305bn in revenue in 2014, can’t find the time or funds to support an open source project they have used for a good length of time. As ever, I like to bring up Mozilla’s recent commitment to open source with their fund, I think its a great platform and demonstrates the importance open source software plays in businesses in the 21st century.

Of course Samsung, you have an open source group, which I think is great, but I can’t help but feel that the projects you support on there are projects that are included in your end user apps, and not projects that make your life easier and more productive within the business. Don’t get me wrong, I love Samsung brands, I’ve had most of the Galaxy phones, a Galaxy tab and the monitor I’m writing this on is a Samsung flat screen, so I understand why supporting open source software is important to you, but I think you also need to see the bigger picture and realise that it’s not just the end user tools you should be supporting.

So here it is Samsung, you use Saiku, and I know from the call we had, there was some interest on your end regarding Saiku Reporting. We are running a kickstarter to allow us to release Saiku Reporting under the apache license. Or, send some developers our way and start contributing to the platform with resource! We only require £15,000 but we are short with 6 days to go.

Just in case you were worried you might not be able to afford it, I have created a report, in a similar format to what we plan to supply with Saiku Reporting, detailing the revenue you guys generate vs what we are asking:

samsung

You can even have a play around with it live, because you aren’t the only ones, Amazon, HP and others all pull the same trick.

So if you have some petty cash stuck down the back of one of your sofa’s, feel free to help support a project you make use of daily within your organisation and help keep Saiku open source.

Thanks

Tom

Programmatically administering users in Apache Karaf

So this is one of those bookmark type blog posts I’m just putting out there because I couldn’t find an example anywhere else and it took some figuring out.

I wanted to administer the users in Karaf programatically instead (or alongside) people logging into the console and typing stuff in, so I wanted to know how to get access to the Backin Engine that powers the user management within Karaf. This is how I did it(along with a bit of help from jbonofre on the #karaf IRC channel.

In my utility class I add:

private BackingEngine getEngine(){
  for (AppConfigurationEntry entry : realm.getEntries()) {
      String moduleClass = (String) entry.getOptions().get(ProxyLoginModule.PROPERTY_MODULE);
      if (moduleClass != null) {
        return backingEngineService.getEngineFactories().get(1).build(entry.getOptions());
      }
    }
  return null;
}

With that done I can then add a couple of setters and get the services via blueprint like:

<!-- JAAS Realm for user management -->
<reference id="realm" interface="org.apache.karaf.jaas.config.JaasRealm"/>

<!-- Backing Engine Service -->
<bean id="engineService" class="org.apache.karaf.jaas.modules.BackingEngineService">
    <property name="engineFactories" ref="engineFactories"/>
</bean>

<!-- Backing Engine Factories -->
<reference-list id="engineFactories" interface="org.apache.karaf.jaas.modules.BackingEngineFactory" availability="optional"/>

With that done I can now edit users, clearly the get(1) needs adjusting to be dynamic but the premise works.

Hope it helps anyone in the same situation!

A message to all the Pentaho Consultants out there who use Saiku :)

I sent this email to a bunch of Pentaho Consultants today after some discussions we had yesterday with some of them. I will no doubt have missed a bunch of people, so here’s a public version for everyone to flame….

Hi guys

After some feedback from a few of you I figured I’d email you all directly to try and explain the situation and where we currently stand with Saiku and especially the CE version. I’ve also included everyone’s email address in the TO, I hope you don’t mind, we’re all friends(I think), and it gives people the opportunity to REPLY ALL if they would like to have a public discussion about the below. Similarly, if you want to take this email off line, just reply to me and I’ll be happy to discuss it with you.

As you are all aware the CE version is Apache licensed and very permissive, which means you and your clients are all free to do pretty much whatever you like with it, and that is the way we like it. Those of you who know me well will also know that I don’t like the CE & EE model and would rather everything just be open source and free to use and we made money of support and sponsorship instead of license sales but that sadly isn’t the case. Before we launched EE we tried offering support and no one was interested because Saiku “just worked”, we aim to make life simple for people and that is probably the biggest downfall with the application because then the customers you work with aren’t interested in investing because “its working, why pay for support?”. I realise that I have no god given right to make money from an Apache Licensed project, but similarly, I’m also not duty bound to support the open source core if I choose not to, but this isn’t something I’d want.

So that leads us to where we are today, Saiku standalone generates a decent amount of leads and we are finally seeing a return on it, but Saiku CE & EE for the BI server generate next to nothing in terms of useful leads or revenue, yet they take up a lot of time and energy to maintain, especially now we have to support Pentaho 5 & Pentaho 6 in different plugins.

If you want to see Saiku CE remain in the BI server, we’re going to need the help of the Pentaho consultants to make that happen, because currently we are putting lots of effort into software that we then just give away for nothing, and have no chance of recouping any revenue on to pay myself or Breno. Thats not to say Saiku would go away from the platform, but if things don’t change there is a large chance it would go to an EE only plugin because at least then if people used it, it would generate revenue for us.

Which leads me onto my next point, and one that some people didn’t seem to realise. We are cheap! We are cheap on purpose. Most Pentaho BI server installatoins will be small, departmental installations. Saiku is priced per user per year to keep the cost down whilst generating revenue to continue developing both versions. Harris said to me yesterday “but my users don’t need a cube designer or dashboards”, well thats fine, but it doesn’t mean they can’t use EE or just use CE and pay for licenses anyway. Purchasing EE licenses does also mean the users would then get product support, quicker bug fix releases etc, so it does have its benifits.

If they really don’t want to pay, just suggest they make a donation to the project, we happily accept contributions via Credit Card, Paypal, Bank transfer, or whatever, or ask them to offer some support to the community, just because your client chooses to use open source software, shouldn’t exclude them from contributing. Having spoken to someone who consults for the Belgian Government a few weeks back and they said they couldn’t sell licenses because “they have an open source only policy”, well thats nice for them, but it doesn’t mean they shouldn’t support the open source projects they use. Look at Mozilla, https://blog.mozilla.org/blog/2015/10/23/mozilla-launches-open-source-support-program/ they have realised this and are now giving back.

As resellers you are also entitled to a 20% rebate on any completed deal, we’re not trying to take the money and run, we want to incentivise you guys to work with us to get Saiku into as many organisations as possible.

If every one of those 225,000 users gave 50 cents we clearly wouldn’t need EE!

To make it clear here are the list prices per user per year in USD, EUR and GBP

1-5   Users $300    €240    £180
6-10  Users $260    €205    £160
11-50 Users $230    €180    £140
51-99 Users $210    €165    £125
100 + Users    $180    €140    £110

Unlimited    $35,000    € 28,000.00    £22,000

Also, I’m not saying people have to give financially, if there were 50 regular committers, I could let the project tick over without spending hours every day maintaining the codebase because other people on the community would be there to help with the chores. If you make changes, push stuff back up stream, if you want to be a release manager, coder, ui guy, whatever let me know and we’ll find a role for you in the community, the code is easy to build and package we certainly make life as easy as possible for everyone.

I know most of you install Saiku then spend 5 minutes hiding the splash screen and removing the CE banners, please remember in doing so you are removing our link with the people who control the budgets and who are likely to have a say in sponsoring our project. So if you do remove those items, please offset it by having a chat with your client and discussing the prospect of contributing money or resource to the project, because as I said yesterday, the other option is JPivot or Analzyer, and that will make, you, me and your end users, very sad indeed I suspect.

Thanks for your time.

Tom

Dear Samsung…..

Hello Samsung.

A few weeks ago I wrote a letter to Jeff Bezos because his company use our product and don’t see the need to contribute back to the product ecosystem. Sadly I didn’t get a response for him, I assume he’s too busy trying to figure out which cloud tool to launch next, hopefully you are more responsive.

Let me introduce myself. My name is Tom, I have 2 small kids, and a Mrs who stays at home to look after them because we think that is better than farming them off to childminders every day, but that does mean we rely on one income. I live in the UK just outside a city called Norwich and I try and make a living developing primarily open source software.

The problem us small vendors have is that we rely very much on a steady income stream from companies who use our software, or we sell our souls to investors and we have to do as they say, or of course we could all give up and work for larger companies, but that’s not good for the world, so we keep trying to earn a living.

What irks me though is when large companies, Amazon, Samsung, Teradata, the US Government, the Brazilian Government, Cisco to name but a few use open source software without contributing back to the software you use.

I’ll give you an example, I like Sublime Text, it does the job for me and my general editing needs, and every so often it pops up a dialogue telling you to purchase a license if you are to continue using the software, it doesn’t demand that you do, but there is an expectation. After a few months of using Sublime Text I went ahead and spent the money on a license because I thought it was worth it. Saiku is Apache Licensed, which means its very business friendly and of course companies can use it however they see fit, but if businesses don’t contribute back to the code base, documentation, support or to the bank balance then the projects cease to be.

Its not a secret you folks use Saiku, hell, you can find it easily enough on Linkedin, not to mention the 3 people who are in our CRM.

Untitled

Untitled2

We’ve tried to reach out to these people via email, but sadly they can’t find enough time in their day to respond to our polite enquiry. So, if you can find someone with the time to have a look at what you guys do with Saiku internally and then want to get back to us to discuss what the best way to give back to the product would be, I would be very grateful, without companies giving back to projects like ours, we either have to close them up, wind them up or give them up.

Thanks

Tom