Why You Might Bring Your Own S3 Bucket (BYOB) to the Observability Party
A company’s future increasingly lies in gleaning insights from its own telemetry data and events. This information is invaluable for observability and can guide a company in the right direction. Thus, companies often strive to maintain ownership and control over the data generated by their infrastructure.
Companies hire third-party monitoring and observability tools for plenty of good reasons. It’s less maintenance and can be much cheaper than internal options. Yet, there can be tradeoffs. You might need to sample data or truncate retention time to keep costs down. The vendor may charge to access your data for other analysis or to load older data from cold storage. Your company’s internal compliance policies may outright prevent it.
A complete SaaS approach is best for most companies. But for some, storing and owning all of their own data while giving third-party SaaS tools read and write permissions is a must. In the past, this option was simply not possible given prohibitive costs and non-cloud native architectures — but that has changed.
The Move to the Cloud Took Customer Data With It
Originally, companies captured, stored and analyzed their data onsite using on-prem tools. Cloud computing removed the costs and maintenance overhead, but early cloud vendors also designed their systems to store their customers’ event data in the vendor’s space to keep their architectures simple, efficient, and reliable.
This approach has two potential downsides for some customers. First, the business that creates the data ends up not holding it. They can only decide on retention length if they negotiate fees for longer archive periods in the vendor’s space. They can’t export event data to other tools to analyze as they wish. Vendors charge businesses for data access, whether retention fees, rehydration fees, or straight-up access fees to their own events.
Second, the early vendors’ now-outdated storage and handling costs, plus the limits of their architectures to scale with modern infrastructures’ event volume, have been baked into their pricing. As the volume of telemetry data and events expands non-linearly, it becomes shockingly expensive to ingest, store, and query 100% of events via old tools. This has made sampling a default approach to cost control.
Sampling has obvious drawbacks that customers have learned to accept: It ignores individual users, hoping a small percentage will speak for an entire cohort. It deletes data forever that you may only realize later that you needed to keep. It forces businesses to decide which events will matter for security analysis and for how long when cybercriminals spend their lives coming up with new ways to break in via seemingly innocuous behavior over months or even years. They’ve learned to use business sampling as an attack vector.
Deleting Data Is Dinosaur Behavior
Best practices need to evolve from keeping only a sample of generated events for a short time to capturing 100%, keeping 100% indefinitely, and analyzing 100% on demand without performing data gymnastics to contain costs based on outdated tools and pricing.
Legacy logging tools may take years to evolve for survival in the modern cloud, holding customers and their data hostage. But if you need to own and control your event history, ask yourself if there’s a way today to do so. After all, you created that data in the first place.
Why You Should Own and Control Your Telemetry Data and Events
The Most Important Data in Your Company
Event data has become the lifeblood of the digital enterprise, reflecting a complete record of human and machine interactions. You should be in control of this strategic asset, hot and unfiltered, for several reasons:
- Team productivity. Having a single source of truth — the whole truth — means that developers, operations, product, customer support, security, and even sales can make faster, more accurate, and harmonious decisions.
- Forecasting. You can’t figure out what you’ll need next year based on 30 days of data.
- Training AI. Enterprise AI engines are already real, and you need to fill them with high-quality fuel. The more detail and history you have to train them with, the more intelligent they become.
- The customer. Customers will find your company more effective to engage with if each touchpoint — human or not — has access to the same complete source of truth.
- Career impact. Technical leaders who increase productivity, team cohesion, forecasting accuracy, product quality, and customer satisfaction will rise to the top.
Streamline and Optimize Data Flow
Having all your event data in one place, under your control, lets you use it for varied analyses through multiple toolchains. You don’t need to transform it, filter it, pay fees to a vendor to keep your data for longer or pay again to access collected data for further use.
If you’ve had to stop in the middle of an incident to ask yourself questions like,” Can I run a high-cardinality query over the last four months of data right now, without having to consider where the data resides and how much it will cost me,” you’re aware of how a single complete repository could make your life more productive and lower the company’s total costs.
How Axiom Can Help
We designed Axiom from the ground up — rejecting off-the-shelf components for ingest, storage, and querying to craft our own optimized methods at each step — so that it’s affordable for businesses to capture and keep 100% of their telemetry data and events. However, some prospective customers told us they want to go one step further: They want Axiom to use one of their S3 buckets, not ours, for storage. They want to pay us once to ingest and store their data, then have the freedom to keep it as long as they want and export it to other tools without paying us again.
We said OK!
We’ve made it simple for customers to specify their own S3 bucket as the location of Axiom’s data store for their events. Using the tools we provide, they can access events in their original raw formats. We only charge once to ingest their data, after which they can do what they will with it without wondering what their next bill will be.
Our approach, which we call BYOB — Bring Your Own Bucket — leverages the efficiencies of the cloud without taking businesses’ data from them or forcing them to delete most of it to keep their costs down. It only takes a few minutes with us to set up Axiom in a BYOB configuration upon request. It’s certainly less complicated than the AWS console.
The size and value of businesses’ event data — all of it — will keep multiplying. It’s already the majority of data volume for most businesses, emerging as the most important data they create. To plot a successful path to your company’s future, you can’t afford to throw it away or cede it to someone else to hold and control. We realized this as former customers, so our mission from the start has been to make the best of all options possible and affordable.
