How Do Companies Choose a Metadata Management Tool?
Under the influence of all kinds of digitization, it is crucial to integrate and utilize the various metadata in the enterprise environment. For enterprises, choosing a metadata management tool that suits them will maximize the role of metadata to assist enterprises in accomplishing their strategic goals in terms of data. Different roles in an enterprise may have different expectations for metadata tools, but these expectations can basically be mapped to the top ten capabilities of metadata management tools. Of course, these capabilities are supported by key technologies.
This article is divided into three parts. In the first part, we will first list the expectations of different roles for metadata management according to different roles in the enterprise. Then, the second part summarizes the ten capabilities that a metadata management tool needs to have in an ideal situation, and associates these capabilities with different roles, so that different enterprises can choose appropriate metadata management tools according to their own conditions. Finally, in the third part, several key technologies to realize metadata management at the present stage will be given.
Table of contents:
1. Different roles have different expectations for metadata management；
2. How to choose the right metadata tool?
3. What are the key technologies of metadata management?
1. Different roles have different expectations for metadata management.
In order to answer this question from the perspective of different roles, we first list several roles related to metadata management projects in the enterprise. These roles can be divided into executives, data developers, data analysts, data managers, operation and maintenance personnel, and other business users.
Business executives: Under the situation that data is becoming more and more important, executives are more concerned about the entire data picture of the enterprise and the use of data in the whole enterprise (or it can be said to pay more attention to the data assets and application level), but no one can directly tell the leading enterprise what the data is like and what the specific usage and circulation are. Effective metadata management can well answer these questions of enterprise executives.
Data developers: For data development, the most common problem is a lot of duplication of work: there are already identical interfaces or scripts, but because they are written by others, they are not uniformly identified and managed, so they cannot be found at all. , even if it is found, it may not be reused at all due to the lack of relevant explanations, which reduces the efficiency of data development and also causes a lot of redundancy. Metadata management can make it easier for data developers to find information that they want to reuse, and explanations can be achieved through business metadata management.
Data analysts: Data analysts usually need to achieve the company’s goals related to strategic decision-making, business or assessment through higher-level statistical analysis of data. For them, intricate data relationships, uneven data quality, and lack of business metadata are major problems. Metadata management lowers the threshold for obtaining such information, and also provides support for the traceability of data quality problems.
Data managers: Data managers are usually responsible for the full lifecycle management of data from design, testing to deployment and delivery. For them, it is usually necessary to manage various versions of data information and manage the life cycle of enterprise data. How to control the coordination and consistency of data in various states and timely determine what cycle the data needs to do is an urgent problem to be solved at present. This can be achieved by managing enterprise metadata.
Operation and maintenance personnel: For operation and maintenance personnel, it is necessary to ensure the stability of the system at all times, especially when the enterprise model changes, they must constantly judge the impact of the change. Obviously, the method of manual judgment is both accurate and real-time. It is difficult to guarantee, and it requires high business capabilities of operation and maintenance personnel, which greatly increases system risks. Through metadata management, when the system is changed, the impact of the change can be automatically analyzed based on the relationships between objects such as systems and tables that have been obtained, so as to reduce maintenance costs and improve user experience in an automated way.
Other business personnel: Because business personnel are familiar with business rules and business processes, they usually do not need to have a deep understanding of technical details. The technical threshold often makes it difficult for business personnel to obtain and understand data. Because they don’t know the data storage situation, it is difficult to communicate with the business needs technically, and often the data they finally get is not what they want, and it is difficult to match the rapid development of the business.
2. How to choose the right metadata tool?
It can be seen from the above that different users have different expectations for metadata management tools. Enterprises carrying out metadata management projects may need to solve the problems of one or several types of users. We first list the basic capabilities of some metadata management tools, and then match these capabilities with the expectations above. By referring to the corresponding relationship between the two, you can directionally choose a metadata management tool that suits you.
Through the practice of a large number of metadata projects, I have summarized ten capabilities that a metadata management tool needs to have in an ideal situation, as follows:
- Metadata Collection Capability: The ability to automatically parse and capture various metadata in real-time from complex enterprise environments. In order to cope with various data environments, this link usually requires the use of various technologies and syntaxes to support automated collection of big data platforms, relational databases, third-party tools, stored procedures, scripts, text files, and table files.
- Metadata Storage Capability: The ability to store the collected metadata in a unified manner. In order to support the storage of various metadata and the relationship between metadata, the metadata storage needs a flexible and scalable architecture support. In addition, it is also very important to be able to update the storage in real time.
- Metadata Lookup Capability: Provide a unified port to search for metadata, and a complete metadata management tool should be able to support the search for metadata according to various classification methods of the enterprise (some classification methods are contained in the metadata itself, which needs to be data after analysis). For example, you may search for information according to different dimensions such as systems, tables, indicators, and interfaces, or even create a completely different category according to your own search habits.
- Data lineage Analysis Capabilities: Analyze the source and flow of data, reveal the upstream and downstream relationships of the data, and analyze, describe and visualize the details in the metadata management tool to facilitate users to track key information. Perfect data lineage analysis needs to be available in both horizontal (current) and vertical (historical) directions to facilitate the analysis of different objects in the same period and the changes of the same object in different periods.
- Role-Based Access Control and Hierarchy: The control of permissions such as adding, deleting, and modifying metadata is something that needs special attention in the metadata management tool. The tool should support the control of access permissions. For example, data administrators have all permissions. Developers may pay more attention to the metadata of the development environment and test environment, while enterprise managers may only focus on the metadata of the production environment. Users at the general manager level can access metadata in various environments of the enterprise. data, while department heads may only focus on metadata relevant to their department.
- Business Metadata Management Capabilities: Collecting business metadata in the enterprise environment, completing the mapping between business metadata and technical metadata, and assigning business attributes to metadata is also a key to exerting the business value of metadata management tools.
- Metadata Change Control Capability: When the metadata needs to be changed, it provides the change auditing capability, clarifying the metadata version, saving the historical state of the metadata, and automatically restoring to the previous version in case of any problems. When a metadata item changes, it may also be necessary to analyze and evaluate the impact of the change.
- Metadata Comparative Analysis Capability: Compare and analyze the metadata in different environments, analyze the similarities and differences, and produce corresponding analysis reports according to the analysis results if necessary.
- Data Lifecycle Management Capabilities: Ideally, a metadata management tool should retain metadata in all states from creation, storage, to obsolete deletion/backup, to manage the flow of data throughout its lifecycle. As a rule, newer data and those that are likely to be accessed more frequently should be stored in easily accessible locations, while less important data can be backed up on cheaper, slightly slower media superior.
- Ability to Integrate with Other Systems: In order to make the metadata management system play the business value, another very important point is the ability of the metadata management tool to integrate with other systems.
3. What are the key technologies of metadata management?
Generally speaking, metadata management projects need to use many technologies, here are four types: highly flexible and extensible architecture, role access control and layering, business metadata and technical metadata correspondence, and integration with other systems.
- Highly Flexible and Scalable Architecture: The data in the enterprise data environment is messy, in various forms and with different standards. To achieve effective collection or automated collection and storage of all metadata, it must be supported by a highly flexible and scalable architecture, which means that the metadata architecture must be able to “communicate” with various models of the enterprise.
- Role Access Control and Hierarchy: As mentioned above, enterprise metadata management involves many different people. An excellent metadata management tool should do a good job of role access control. The specific implementation methods can be summed up in two ways: 1. On the platform, a role layering mechanism/role group is established, and different roles in the enterprise are classified into different role groups according to specific needs, and different functions are displayed for different role groups. 2. The mapping between roles and functions is established within the tool, and flexible configuration is supported according to the situation of the enterprise (after all, the roles and corresponding functions of each enterprise are different), and different functions are displayed for different roles according to the mapping.
- Correspondence between Business Metadata and Technical Metadata: This is the key to the transformation of enterprises from data management to knowledge management. For the correspondence between business metadata and technical metadata, you can refer to the articles we wrote before. In addition, I think that in addition to the construction of domain ontology, weaving model may also become a technology corresponding to the relationship between the two in the future. In this way, the relationship between different models of the enterprise is stored and managed by weaving model. For example, by establishing the relationship between the star model/snowflake model and the business process model, enterprise organizational structure, etc., the business information and manager information corresponding to the data can be automatically obtained.
- Integration with Other Systems: With the introduction of various data policies, metadata has become an indispensable and important part of various processes of enterprises, and people’s voice for metadata-driven is getting higher and higher. To achieve metadata-driven, in addition to metadata classification and model classification standardization, whether the metadata management tool is integrated with other systems of the enterprise (such as CRM, ERP, SCM, OA and other systems, as well as data standard systems and data quality systems related to data management) to provide metadata services for other systems, is the main key point. One method is to provide flexibly configurable interfaces to different roles, different users, and different systems in the enterprise, so as to achieve efficient collaboration across the entire enterprise; the other method is to directly integrate the metadata management tool into the enterprise portal. The entry to keep the metadata repository in other information systems of the enterprise.
Different enterprises have different requirements for metadata management, and enterprises should choose the most suitable metadata management tool for their own situation.
Thank you for reading our article and if you’ve enjoyed it, we would be very happy. If you want to learn more information about metadata management, we would like to advise you to visit Gudu SQLFlow for more information. As one of the best data lineage tools available on the market today, Gudu SQLFlow can not only analyze SQL script files, obtain data lineage, and perform visual display, but also allow users to provide data lineage in CSV format and perform visual display. (Published by Ryan on Jun 26, 2022)
If you enjoy reading this, then, please explore our other articles below:
Leave A Comment