4 Best Open Source Data Catalog Tools in 2022

Fundamentally, any data-driven organization needs data catalog tools. 데이터 카탈로그 help create a single environment from which all of an organization’s data and context about that data can be accessed, ensuring that organizations can reduce their insight time and quickly make high-quality data-driven business decisions. If you are looking for the best open source data catalog tools, then you’ve come to the right place. In this post, we’ve compiled a list of the best open source data catalog tools in 2022 to make your life easier.

최고의 오픈소스 데이터 카탈로그 도구

Best Open Source Data Catalog Tools – 1. Apache Atlas

As an open source 메타데이터 관리 tool and governance platform, Apache Atlas is incubated by Hortonworks under the umbrella of the Data Governance Initiative.

It joined the Apache Foundation Incubator in 2015 and grew to a top program in 2017. Apache Atlas is widely recognized as one of the building blocks of modern data platforms because of its early vision of using metadata to solve the challenges of data cataloging, sorting, discovery, governance, and collaboration.

Main capabilities of Apache Atlas:

1. Metadata classification: Apache Atlas enables you to automatically classify PII, sensitive data, and other sensitive data. Data assets can be associated with multiple classifications. These policies are also propagated through lineage to ensure that derived data inherits the same classification and security controls.
2. Metadata types and instances: According to the Apache documentation, a “type” is a definition of how a specific type of metadata object is stored and accessed in Atlas, which enables 데이터 관리자 to define technical and business metadata.
3. Search and lineage: The intuitive UI in Apache Atlas allows predefined and temporary exploration of data types by type, category, attribute value, or free text. In addition, it retains a history of how the data source or explicit data was constructed and how it evolved over time.
4. Security and data masking: Apache Atlas is primarily a data governance tool that allows fine-grained security of metadata access to set up access control over entity instances, as well as add/update/remove classifications, and so on.

Best Open Source Data Catalog Tools – 2. Amundsen Lyft

As an open source data catalog platform originally built by Lyft’s engineering team, Amundsen opened source in October 2019, a year after launching for internal use.

Amundsen has a cohesive community of contributors and users, and is widely adopted by other organizations built on top of this open source data catalog tool to advance their data democratization, governance, and metadata service initiatives.

Main Capabilities of Amundsen:

1. Easy to find trusted data: Amundsen helps find data from a variety of sources with simple text searches, and the search results even display online metadata.
2. Automated and curated metadata: When a data asset is clicked, the user is shown its detailed description and behavior, which are manually curated and automatically generated, respectively.
3. Ability to share context with colleagues: Descriptions of data assets can be updated, reducing the need for colleagues to go back and forth looking for more context for a particular data asset.
4. Learn and understand from data usage: Users can see which data assets are frequently used, owned, or bookmarked, and can even see the most common queries related to tables by looking at the dashboards built on a given table.

Best Open Source Data Catalog Tools – 3. LinkedIn DataHub

As an open source metadata management platform developed by LinkedIn’s engineering team, DataHub is really LinkedIn’s second attempt to address the challenges of data cataloging, discovery, observability and lineage.

Prior to DataHub, they built an open source data directory tool called WhereHows back in 2016. DataHub was announced in 2019 and opened source in 2020. As a result, LinkedIn maintains two different versions of DataHub — one for internal use and one open source for others that can be built.

Main Capabilities of DataHub:

1. Automated metadata ingestion: In LinkedIn DataHub, metadata is ingestion from different sources via API or Kafka stream push.
2. Easy data discovery: At the highest level for end users, the DataHub front end supports three types of interaction: search, browse, and view/edit metadata.
3. Understand data through context: Each data entity on DataHub comes with a profile page that displays all metadata associated with that data entity, providing the user with the information needed to develop that data context.

Best Open Source Data Catalog Tools – 4. Netflix Metacat

As a joint metadata management service built by Netflix and opened source in June 2018, Metacat aims to simplify the sorting, discovery, processing, and management of data.

Metacat primarily forms a single source of access to all of Netflix’s data assets. While Metacat is an open source data catalog, there appears to be a lack of significant public knowledge for others to effectively use its schema and extend it.

Main Capabilities of Metacat:

1. Data abstraction and interoperability: Metacat forms a common abstraction layer, and datasets can be accessed across Netflix’s multiple query engines.
2. Business and USER-DEFINED metadata stores: Metacat helps document business and user-defined metadata about data assets, ensures that data users are provided with more information about data assets, and standard rules for how to deal with them.
3. Data discovery: Metacat provides schema metadata and business/user defined metadata for data via ElasticSearch and this helps query through text search.
4. Data change audit and notification: Captures any metadata changes or updates, enabling push notifications for such events that might require the user’s attention.

What are the benefits of open source data catalog tools?

High-quality data catalogs not only allow you to properly categorize all your data, they also allow you to properly track data flows between different data types and even show you flaws in data flows that you can improve.

Another nice feature is that sensitive data can also be managed, and the tool can identify where sensitive data is displayed the most, thereby reducing the risk of leakage. Some high-end data catalog tools even offer machine learning capabilities that can understand how you manage your data and help you process large amounts of data. But why use the open source data directory tool?

Open source data catalog tools are still high-quality software, inexpensive, sometimes even free, but they scale well, offer a lot of customization options, and can work without any limitations, ideal for high data volumes.

Beyond that, as a business or organization, you don’t have to worry about relying on one developer for updates, as you can hire developers to further develop open source software, or easily customize it to suit your needs.

결론

저희 기사를 읽어주셔서 감사합니다. 이 기사가 여러분이 원하는 것을 찾는 데 도움이 되기를 바랍니다. best open source data catalog tools in 2022. If you want to find more information about open source data catalog tools, we would like to advise you to visit Gudu SQLFlow 자세한 내용은.

그 중 하나로서 최고의 데이터 계보 도구 오늘날 시장에 출시된 Gudu SQLFlow는 SQL 스크립트 파일을 분석하고 얻을 수 있을 뿐만 아니라 데이터 계보, 시각적 표시를 수행할 뿐만 아니라, 사용자가 CSV 형식으로 데이터 계보를 제공하고 시각적 표시를 수행할 수 있도록 합니다. (2022년 7월 16일 Ryan 게시)