Enterprise Metadata Management Platform

Under the influence of all kinds of digitization, it is crucial to integrate and utilize the various metadata in the enterprise environment. For enterprises, choosing a metadata management tool that suits them will maximize the role of metadata to assist enterprises in accomplishing their strategic goals in terms of data.

Different roles in an enterprise may have different expectations for metadata tools, but these expectations can basically be mapped to the top ten capabilities of metadata management tools. Of course, these capabilities are supported by key technologies. “Metadata management will be the core infrastructure of future enterprise informatization”.

Indeed, in the big data environment, if the enterprise does not manage a variety of complex information through metadata management, it is difficult to effectively utilize the information. However, many enterprises gradually find that the value directly brought by metadata management to enterprise business innovation is very limited.

Enterprise Metadata Management

What is the current state of metadata management?

The main reasons why metadata management cannot bring direct value to business innovation lie in the following four aspects:

1. Narrow management scope: The current narrow scope of metadata management is mainly reflected in two aspects: first, only metadata related to data warehouses are managed, and metadata management is not extended to the entire enterprise level; second, even in the field of data warehouses, only part of the technical metadata is managed. The lack of technical metadata prevents the metadata system from showing the real situation of the enterprise system. The lack of business metadata results in only a bunch of table structures that lack specific business meanings.
2. Difficulty in integrating business: Due to the lack of integration with business, after many companies complete the metadata management system, they find that only the internal personnel of the data management department are using it, and people in other departments cannot use it at all. All systems should serve the business. A system that is used by only one department is lifeless. It is very important that people in all departments use the metadata management system.
3. Lack of application scenarios: At present, most of the application scenarios of metadata are limited to the internal functions of the metadata system, such as lineage analysis, version management, etc. These functions are used by enterprises in informatization, but in fact, the role of metadata is not only reflected in these limited functions.
4. Imperfect technology: Imperfect technology has led to the emergence of the above series of problems. The scalability is not strong, so that many metadata management tools cannot manage the business data and structure of the enterprise; the collection ability is poor, resulting in only manual auxiliary records, high labor costs, and inability to establish a complete information link; the real-time performance is not high, resulting in the metadata management of many enterprises still in the T+1 (or even not T+1) stage, unable to understand the status of data assets in real time, and unable to keep up with the speed of enterprise data growth.

There is an increasing demand for enterprise metadata management in the market today:

1. Digitalization is accelerating, and the amount of data in the enterprise has grown exponentially.

With the advent of the digital age, many complex and changeable information can be transformed into measurable data and introduced into the computer for unified processing. Relevant studies have shown that the data collected and used by enterprises will increase exponentially every year. Effectively managing these massive amounts of data requires enterprise metadata management.

2. The emergence of big data-related technologies allows enterprises to see new opportunities.

Big data-related technologies allow the value of enterprise data to be fully exploited, but big data often means the collection, dissemination and sharing of many data sources, such as mobile personal data, social network data, public data, Internet of Things data, etc. The process needs the support of enterprise metadata management.

3. Increased business demand for data governance.

Nowadays, enterprises are paying attention to how to use big data, but the premise of big data application is to have high-quality data. Nowadays, many enterprises have various internal data forms and different standards. Big data applications often start with data governance. As an important means of enterprise data governance, data management will inevitably attract the attention of domestic and foreign enterprises.

How to fully unlock the business value of metadata management?

Metadata also defines attributes according to actual business scenarios. Metadata will have common attributes, such as name and type. Different types of metadata will also have their own specific attributes. Since metadata is also data, it must be stored in the database. Metadata repository refers to the physical database table that stores metadata. Usually, an open source relational database (MySQL) is used to realize metadata storage. The following describes how to manage metadata from several aspects:

1. Determine the metadata scope.

First, determine the scope of metadata sources. In actual work, not all data needs to be managed by metadata. Usually, we will choose business data for metadata management. Non-business data will not be included in the management scope, mainly because Metadata management is to provide business and developers with quick grasp of business data.

After the rules are determined, it is necessary to sort out which business systems, databases, database users, and which tables need metadata management based on the actual situation of the company. Of course, it can also support unstructured metadata extraction, such as: word, pdf, etc.

2. Access metadata

Where the metadata is accessed from, it is generally accessed from the source system. If the company already has a data warehouse or the real-time requirements are not high, in order to save the development workload, the existing metadata will be accessed from the data warehouse, and the unconnected will be accessed from the source system. However, this solution is also risky. If the data of the data warehouse is inconsistent with the source system, it will lead to metadata errors. Most of the metadata extraction is now done in the way of configuration automation.

3. Establish metadata standards.

In the process of sorting, there may be some databases or some data definitions that are not standardized, resulting in the inability of metadata management. Next, it is necessary to establish a metadata management specification to reverse the front-end source data for rectification, mainly to ensure the integrity and consistency of the metadata.

According to the requirements of different types of companies, metadata will be open to different groups. Therefore, the permission management process of metadata needs to be defined in the specifications, including the permission layer of metadata, the permission application process of metadata, the release process of metadata, and the approval process of metadata.

4. Maintenance of metadata.

Metadata maintenance is mainly to maintain and manage the metadata that has been released. If the metadata that has been released online needs to be adjusted or optimized, it must go through the metadata release process again, and direct modification of the metadata is not allowed. For security, all metadata operations must be recorded in the metadata operation log.

You can create catalogs for metadata, hang different metadata in the corresponding catalogs, and design the corresponding catalogs according to business processes, business subject domains, and development processes, mainly based on company requirements.

5. Metadata search, analysis and reporting.

There is a separate page to support fuzzy or precise fast search of metadata, and find corresponding metadata by entering key information. Metadata can also be regarded as a type of data assets, so we need to produce a metadata asset report, from which we can quickly understand the metadata access popularity, data value, data cost, data distribution and other related information.

What are the applications of metadata?

Enterprises have diverse businesses and complex products, forming a large amount of data in various systems and applications. With metadata, we can understand what data the enterprise has, what the data represents, where the data comes from, how it flows in the system, etc., perform metadata management, and build metadata applications, such as business terms, data standards, data dictionaries, data asset catalogs, data lineage analysis, data maps, etc. We  will mainly talk about data lineage relationship and data map in the following.

1. Data lineage analysis

Data lineage is an important application of metadata, which can describe the relationship between data and data. For example, this table is extracted from a certain system. What is the relationship between this field and that field, including cluster lineage relationship, system lineage relationship, table-level lineage relationship, and field lineage relationship. It points to the upstream source of the data and traces the source upstream.

The upward and downward table-level and field-level traceability data can clearly show the logical context of data processing, quickly locate the impact range of abnormal data fields, accurately delineate the minimum range of data backtracking, and reduce the cost of understanding data and solving data problems. Lineage analysis can meet the special regulatory and compliance requirements of the data presented in many industries, including healthcare, finance, banking, and manufacturing, among others.

In addition, impact analysis is also part of the lineage relationship application, which is used to analyze the downstream flow of data. When the system is upgraded, it can notify the downstream system of dynamic data structure changes and deletions in time. Through data-dependent impact analysis, it is possible to quickly locate which downstream systems, which tables and which fields will be affected by metadata modification, thereby reducing the risks brought by system upgrades.

2. Data map

In the entire data system, the data map assumes the role of a manager. It displays data information in a graphical way and indicates various information parameters necessary for data calculation. Not only data developers can use it, but also for products. And the operation is also very friendly. It contains the following parts:

• Quick search location: Search for relevant data through the search engine, supporting accurate query, fuzzy query, table name query, field query, remarks query and other methods;
• Standardized graphic presentation: Use a graphical way to organize page logic. For example, the data quality level is marked in the form of a Wifi icon, which is convenient and visible to the naked eye, and is convenient for the key information required by the user;
• Accumulate historical data information: In many scenarios, historical data does not need to be recalculated, and direct pulling can greatly avoid repeated development. For example, for the statistics of new users, historical user information can be pulled and associated with the daily user login log to produce daily new users;
• Direct association analysis tools: Because the data information is directly stored on the platform, the report plug-in can be called to quickly see the intuitive report information, without the need for secondary processing and development, which greatly improves the efficiency of development.

With these, business personnel can understand what the data of the enterprise is, and better understand the data through the business connotation of the data; technical personnel can grasp the overall situation of the data, establish database tables according to the data standards, achieve bids, and achieve data standardization. Data lineage and data map will make the data context particularly clear, and you will no longer feel that the data is a mess.

Conclusion

Thank you for reading our article and we hope it can help you to have a better understanding of enterprise metadata management. If you want to learn more about enterprise metadata management, we would like to advise you to visit Gudu SQLFlow for more information.

As one of the best data lineage tools available on the market today, Gudu SQLFlow can not only analyze SQL script files, obtain data lineage, and perform visual display, but also allow users to provide data lineage in CSV format and perform visual display. (Published by Ryan on Jul 1, 2022)