If you’re working for a company that handles a ton of data, chances are your company is constantly moving data from applications, APIs and databases and sending it to a data warehouse or a data lake. There are a few well-known platforms to extract, load and transform data, such as Fivetran and Stitchdata.
Airbyte is building an open-source data pipeline platform. An open-source alternative to Fivetran or Stitchdata is interesting because building and maintaining connectors require a ton of resources. If you’re just starting out with your data pipeline and you don’t have a big team, it creates a barrier to entry.
Sure, existing data pipeline platforms already provide dozens of integrations with popular sources and destinations. Popular sources include Salesforce, Stripe, Marketo, SendGrid, etc. Popular destinations include Redshift, Snowflake and BigQuery.
But there are a ton of small services that aren’t supported on traditional data pipeline platforms. If you can’t import all your data, you may only have a partial picture of your business.
A lot of companies end up building custom connectors for their applications. With this open-source approach, Airbyte thinks it can foster a community of users to build and maintain thousands of open-source connectors. When a company builds a custom connector, another company can use it, improve it and give back to the community.
“The open-source approach is the right approach,” co-founder and CEO Michel Tricot told me. “If somebody using a connector realizes that it’s broken, they can fix it for the entire community.”
The team behind Airbyte originally started with a product focused on marketing data. They attended Y Combinator and later pivoted because of the coronavirus outbreak. “In a COVID world, there’s no marketing budget,” co-founder John Lafleur told me.
Airbyte connectors run in Docker containers, which means they all operate independently from each other. You can schedule updates, refresh your connectors manually and monitor them individually.
Right now, there are 46 connectors — Airbyte is currently working on certifying them to make sure they’re production-ready. 250 companies are currently using Airbyte.
Leave a Reply
Leave a Reply
This site uses Akismet to reduce spam. Learn how your comment data is processed.
