At present, many enterprises have realized that it is difficult to improve business innovation capabilities through big data because business personnel cannot understand the data stored in the system. This article will discuss the solution to this problem – business metadata management.
Table of contents:
A “semantic barrier” appears between computers and humans;
Business metadata – the business context behind the data;
Ontology-based business metadata management practice;
Summary and Outlook.
Business Metadata Management
1. A “semantic barrier” appears between computers and humans.
About 70 years ago, on Valentine’s Day, ENIAC was born at the University of Pennsylvania. Since then, mankind has started to explore the “intelligence” of computers. Various technologies such as “language recognition”, “image recognition” and “natural language processing” are increasingly Mature, we can input almost any information we want into the computer system, but in turn, can we correctly understand the output data of the computer?
As data has grown, we have found it difficult to figure out the specific meaning behind the data – a semantic barrier has been created between us and computer systems. The existence of semantic barriers brings a series of problems to enterprises:
Inconsistent business understanding: Inconsistent employees’ understanding of the business is very common in enterprises. Inconsistent understanding of business terms makes employees unable to communicate, causing misunderstandings and reducing communication efficiency; in meeting decision-making, leaders’ inconsistent understanding of business may lead to wrong decisions; in department statistics, inconsistent understanding of business definitions will lead to statistical methods Inconsistency results in inaccurate statistics, and even affects the statistical results of multiple indicators and KPIs of the enterprise.
Finding information is difficult: In the era of big data, the amount of enterprise data is growing explosively, and finding information for employees is more and more like “finding a needle in a haystack”. According to statistics, enterprise employees spend 15% to 35% of their time searching for the data they need in a large amount of information every day, and only less than 50% of the search results meet their needs. In most cases, the search results are unsatisfactory; Because the information that has been stored cannot be found, companies often do unnecessary duplication of work.
Great loss of personnel: According to statistics, the average annual turnover rate of employees in enterprises is about 12%. Because there is no set of business management methods, enterprises often spend a lot of time and cost on training new employees after employees who know “computer” very well, causing serious knowledge loss and money consumption.
The emergence of the above series of problems, in the final analysis, is because enterprise employees “can’t read” the data in the computer. Enterprises need to break the semantic barrier between computers and people, and turn computer output into a business language that can be understood by business personnel to avoid these problems. Business metadata is the key to solving the problem.
2. Business metadata – the business context behind the data.
To clarify what business metadata is, we must start with the classification of metadata. Currently, a classification method recognized by the industry is to divide metadata into two types: technical metadata and business metadata.
The technical metadata includes: field name, field length, database table structure, etc.
The business metadata includes: business name, business definition, business description, etc.
Business people pay more attention to content related to “customer”, “settlement date”, “sale amount”, etc., which are difficult to reflect from technical metadata.
Business metadata uses business name, definition, description and other information to represent various attributes and concepts in the enterprise environment. To a certain extent, the business context behind all data can be regarded as business metadata. Compared with technical metadata, business metadata enables users to better understand and use data in the enterprise environment. For example, users can clearly understand the meaning of each indicator and the calculation method of indicators by viewing the business metadata.
Business metadata widely exists in the enterprise environment. The main sources of business metadata are:
ERP: Enterprise ERP system stores a large amount of business metadata, such as financial calculation formulas, process logic, business rules, etc.
Report: The header of the report is also a kind of business metadata, especially those columns with summary properties such as totals and averages, and some calculation formulas in the report.
Table: Similar to reports, EXCEL headers and formulas are also important business metadata. Unlike reports, most tables will have a single column for “Description”, and some tables will also have a column for Code and Code Description, which are useful business metadata.
File: There are business metadata everywhere in the file, such as title, author, modification time, etc. It is relatively difficult to obtain business metadata in the file content, involving technologies such as machine learning.
BI tools: The operation often used in BI is the “drilling” operation. Drilling up and down usually defines various classification structures of the enterprise, such as product level and organizational structure level, which are very important business metadata.
Data warehouse: Business metadata also exists in data warehouses. For example, building a data warehouse often requires a lot of research on how to integrate multiple data sources, and there is a lot of business metadata in the files related to the data warehouse construction process.
At present, most enterprises only pay attention to technical metadata, ignoring business metadata management. The technical metadata lacks business meaning and is difficult to be understood by people other than technicians. For example, it may use “rec_temp_fld_a” to represent a field, and “236IN_TAB” to represent a table in the database. Difficult to bring benefits to the business. Business metadata can represent the business meaning behind the data. Enterprises need to pay attention to the business metadata managment while managing technical metadata.
Compared with technical metadata, the source of business metadata is more complex and scattered in all aspects of the enterprise environment. In order to realize the management of business metadata, enterprises need effective methods and means.
3. Ontology-based business metadata management practice.
The definition of ontology that is relatively recognized by the industry is: a clear formal specification of a shared conceptual model. Among them, the conceptual model is a model obtained by abstracting some phenomena in the objective world, which is the abstraction and simplification of the objective world; sharing means that the knowledge described in the ontology is not exclusive to individuals but recognized in the field; explicitly means that , the types of concepts used and the constraints on concept usage are clearly defined; formal means that the ontology is machine-readable and human-understandable.
To sum up, ontology can analyze the objects in the field and find out the relationship between these objects, so as to describe the business in this field clearly and formally.
To manage business metadata through ontology requires attention to three key points: the construction of the ontology, the storage of the ontology, and the use of the ontology.
1. Construction – Automatically construct ontology using metadata management.
The traditional way of building an ontology is to manually sort out the ontology of the business domain according to the suggestions of business experts. There are a series of problems in this manual sorting method:
Efficiency problem: In the big data environment, the data is complex, the sources are diverse, and the business fields continue to increase. The speed of manual sorting can no longer meet the needs of enterprises.
Tool problem: Business experts lack tools with automation capabilities, resulting in the construction of complex ontologies that consume a lot of time and resources.
Third-party data: Enterprise experts do not understand the related business of third-party data, so it is difficult to complete the construction of related ontology.
In the big data environment, enterprises need a new way of constructing ontology. Enterprises can automatically extract metadata from enterprise application systems and various documents through metadata management tools, and after the ontology is initially formed, it is then handed over to business experts for secondary review, and finally the construction of the enterprise ontology is completed.
2. Storage – based on MOF to implement OWL specification storage ontology.
The storage of the ontology needs to be based on certain standards, and the storage method needs to be flexible and extensible. The OWL specification is a specification recommended by W3C and is currently a widely recognized ontology storage and exchange specification. Since our metadata is based on MOF, the OWL meta model can be established in the meta model, so the storage and management of ontology in the metadata database can be realized.
Since both technical metadata and ontology have been stored in the metadata database, the ontology was originally extracted from the technical metadata. In this way, it is easy to obtain the relationship between ontology and technical metadata, so that business personnel can clearly understand the business meaning behind the data.
3. Use – obtain business context through business metadata services.
Finally, the service of business metadata needs to be provided to all business personnel, embedded in the working environment of business personnel, so that business personnel can quickly understand data from a business perspective, thereby helping business personnel to better use data.
4. Summary and Outlook
Finally, I will summarize in one sentence. Business metadata is the key to future metadata management. In the era of big data, enterprises need to strengthen business metadata management. Enterprises can use automated means to manage business metadata based on ontology, and integrate business metadata. Data is provided to business personnel in the form of services to help business personnel make better use of data.
Thank you for reading our article and we hope it can help you to have a better understanding of business metadata management. If you want to learn more about metadata management, we would like to advise you to visit Gudu SQLFlow for more information.
As one of the best data lineage tools available on the market today, Gudu SQLFlow can not only analyze SQL script files, obtain data lineage, and perform visual display, but also allow users to provide data lineage in CSV format and perform visual display. (Published by Ryan on June 30, 2022)