Elasticsearch is best known as the company that provides commercial support for the open source Elasticsearch search engine. But as the company evolved, it acquired Logstash, which scrubs and parses log data and converts it to JSON, and Kibana, which provides a data visualization layer.
The scope of the expanded offering, dubbed ELK, was one reason Elasticsearch decided to change its name to Elastic -- announced today at ElastiCon, the company's first big user conference. Prior to the announcement, CEO Steven Schuurman told me this rebranding is a logical next step for the company because the ELK stack goes beyond search and has already been adopted for a more diverse set of use cases.
I asked Shay Banon, Elastic's CTO, about what he sees as the most common Elasticsearch use cases besides the obvious.
Plain search and logging are the two most popular use cases. Once people start to use us, they realize they can do so much more. A prime example is that a company will use us to log application errors, but they find that if they stream metrics they can find a correlation between an error happening and high memory usage, for example. They also find there is a lot of business information in these log files -- for example, promotion codes. When using Kibana they can show business users what the effects are of promotion codes.
Banon also referenced a U.K. bank that started with logs and search, but ended up indexing Twitter to help find ATM outages, as well as a credit card company that uses Elasticsearch for fraud detection.
Elasticsearch in the cloud
The second big announcement from the company is that it has acquired Found, which provides Elasticsearch as a cloud service. According to Schuurman, “At this point we have massive corporations who run 10, 20, 30, or many more instances of ELK servicing different use cases and they’re coming to us asking how to manage and provision different deployments. On the other hand, people ask us when we’re going to provide something as a service.” Rather than build their own management tool and their own cloud service, they discovered someone else had thought through these problems and had a solid offering.
Next, I spoke to Alex Brasetvik, one of the co-founders of Found, who noted, "There are a lot of companies out there creating Elasticsearch products and services," including Amazon and Microsoft, which are adding Elasticsearch to their cloud offerings. “I think given the situation that joining Elasticsearch is the best possible exit strategy.”
I asked Schuurman what the Found acquisition would mean for Elastic’s existing partner offering. “It’s going to open completely new avenues of cooperation, which is wonderful," he said. "On the other hand, with partners that are search-as-a-service providers, we’re going to have to look at what that means. I can try to BS it, but you know what that means -- there is a reason you asked the question.”
Schuurman was quick to note the importance of open source to his company’s strategy:
We are a company that’s completely based on the success of a community and open source software. It is the very DNA of our company. We have to always work very carefully and closely with other folks that have a business that relies on our software. You can’t partner with someone you compete with, but that doesn’t mean you won’t help these guys out. It is still a member of the ecosystem, somebody that is part of the open source Elasticsearch, Logstash, and Kibana and we’ll still help them out.
Banon noted, “Without open source a single person from his bedroom would not have been successful. I was writing Elasticsearch on my own for a few years and open source is a wonderful platform to get your ideas out there and have users give feedback.” Logstash and Kibana evolved because of this collaboration as well.
In an environment with established players like Autonomy, open source “is a way to reach those alpha users and say here’s a new thing, try it out. It never ends, you’re always crossing the chasm. Open source is a great platform to keep your product fresh,” Banon added. Schuurman also mentioned that as an entrepreneur open source allows you to move fast and gain benefits from word of mouth rather than having to spend lots of money on hardcore sales and marketing.
Cracking enterprise search
Elasticsearch was a bit of an unlikely success story. Search was an established market and there was already an open source commoditizer: Apache Solr, primarily commercially marketed by LucidWorks. How did Elasticsearch succeed? Why did it even get funded?
As Banon volunteered, “Our elevator pitch ended up being the usage of Elasticsearch. When we were talking to investors, the only thing we asked them to do is pick up the phone and call their portfolio companies.”
Banon noted that his approach to the earlier Compass project, the predecessor to Elasticsearch, wasn’t to go the traditional route of enterprise search and create a bunch of connectors to Word, SharePoint, and so on. He set out to map business objects to Lucene at a much higher scale; this resonated, and people were using it to search stock trades, not only documents:
It was a moment in time when it made a lot of sense. Data volumes were growing and other systems were being built to just handle the vast amounts of storage like Hadoop and NoSQL solutions or what have you. There was a place for something like Elasticsearch -- after I have all of that, how do I make sense of that data? I managed to store it, but how do I search across it?
He also noted the central importance of an API-first architecture to Elasticsearch’s success.
I asked Schuurman about the next big steps for the company:
For at least the next 12 months we are going to be taking our existing products to the next level and expanding into a centralized management structure ... We have some other products already in the works. We will be releasing some more commercial products as well. The market is also saying: "We’re sticking so much data into Elasticsearch now, wouldn't it be cool if we stuck some machine-learning-style algorithms on some of the data so we can be a little bit more predictive? What you guys have done is to democratize data to enable interaction with hundreds of terabytes if not petabytes of data in real time ... with millisecond responses, which is awesome. We would love to have the same ease of use when going beyond what is there to be a little more predictive."
A team at the company is already looking at machine learning solutions and is testing the waters to see where it can add value. Banon noted the company is also considering an expansion of the Found management solution to other cloud providers.
This should be an interesting year for Elasticsearch, with the acquisition of Found as a turning point -- and with a rebranding that encourages people to think of the company as more than a search technology provider.