AI in production: make data as easy as using your phone
This is one of the articles about "A.I. in production." It contains a story about a real company using A.I. in of their products.
This is NOT a sponsored content. I don't endorse the interviewed company and its products in any way. They don't endorse me either.
What if writing a data pipeline was as easy as making a REST API call from a smartphone? What if we did not need to worry about differences between data formats when using data from multiple sources?
What if we could allow any developer in the organization to build a data pipeline and focus on the business value they are creating? What if we did not need to worry whether they make calls to external APIs, databases, files, etc. and instead, all the data they need is available?
What if the data engineering team could work as fast as front-end developers? What if we could test data-related ideas in a matter of hours, not days or weeks?
Last week, I talked with Gautam Bakshi - the CEO and product architect at 15 Rock. It is a startup that wants to “Make data as easy as using your phone.”
They think that the most significant problems which slow down organizations are: siloed data sources created by legacy applications, building custom integrations, the fact that people spend most of their time searching for the data they need.
Standardized data platform
According to Gautam, those problems can be solved by creating a standardized data platform. They build a solution capable of downloading data from various data sources and transforming it into a standard format. In addition to that, it allows users to write data processing code as applications running on 15 Rock data platform.
Their solution is based on a collection of connectors that download data from popular databases. It is worth noting that the connectors monitor the databases for changes and download the data continuously instead of making a big batch or large scheduled synchronization. They are trying to bring enterprises away from batch and closer to real-time.
Gautam admitted that creating such connectors was the largest technical challenge. However, it was worth the effort because its “stream-like” implementation allows them to give their clients real-time and event-based access to their data/analytics instead of batch views.
Right now, 15 Rock focuses on cooperation with financial institutions. This means that they must first and foremost support the databases popular in such institutions and leverage the power of a secured cloud for scale and performance.
Because the platform handles data security and does not allow access to the “outside world,” they built an application marketplace (currently invite-only). In the 15 Rock application store, the developers can offer their implementations of AI algorithms, analytics, or reporting pipelines.
It is yet another step on the road to achieving making data as easy as using the phone. Right now, the platform natively supports applications written in Python, but in the future, it is going to be extended to support also Java, Julia, or even Haskell.
What is an outage?
During the conversation, Gautam pointed out an interesting monitoring problem. What outage looks like in serverless applications as 15Rock’s platform is leveraging serverless for ease and scale?
A server being down does not constitute an outage anymore. We may even lose multiple servers and still be able to handle requests. Because of that, the monitoring solution created by 15 Rock replaces a simple health check with monitoring the number of repeating errors and detecting anomalies in service-related metrics.
Fast-growing businesses are AI-driven
Gautam Bakshi wants to help companies leverage AI systems because he thinks that all successful companies are based on AI algorithms. Nowadays, all awesome and fast-growing businesses are AI-driven.
Think about companies that struggle to get by. They probably postpone all AI-related projects. Their developers created brilliant proofs-of-concept during internal hackathons, but those things never get deployed in production because of “product roadmap” and urgent but not important goals.
It is not a surprise that when I asked Gautam Bakshi about a book recommendation, he suggested reading “Programming Collective Intelligence.” It is a book that teaches AI algorithms in the context of business. It shows how to implement those algorithms in Python and apply them to solve real problems in an organization.
You may also like
- Row number in Apache Spark window — row_number, rank, and dense_rank
- What is the difference between data lake, data warehouse, and data mart
- Apache Spark: should we use RDD, Dataset, or DataFrame?
- How to write to a Parquet file in Scala without using Apache Spark
- What are the 4 V's of big data, and which one is the most important?