{"id":6761,"date":"2026-07-12T02:07:41","date_gmt":"2026-07-12T10:07:41","guid":{"rendered":"https:\/\/www.gudusoft.com\/spark-sql-data-lineage\/"},"modified":"2026-07-12T02:07:41","modified_gmt":"2026-07-12T10:07:41","slug":"spark-sql-data-lineage","status":"publish","type":"page","link":"https:\/\/www.gudusoft.com\/jp\/spark-sql-data-lineage\/","title":{"rendered":"Spark SQL\u30c7\u30fc\u30bf\u30ea\u30cd\u30fc\u30b8\uff1aSpark\u30d1\u30a4\u30d7\u30e9\u30a4\u30f3\u306e\u5217\u30ea\u30cd\u30fc\u30b8"},"content":{"rendered":"<p><strong>Spark SQL\u30c7\u30fc\u30bf\u30ea\u30cd\u30fc\u30b8<\/strong> \u3053\u308c\u306f\u3001\u30c7\u30fc\u30bf\u304c Spark SQL \u30b3\u30fc\u30c9\u5185\u3092\u3069\u306e\u3088\u3046\u306b\u79fb\u52d5\u3059\u308b\u304b\u3092\u793a\u3059\u5217\u30ec\u30d9\u30eb\u306e\u30de\u30c3\u30d7\u3067\u3059\u3002\u3069\u306e\u30bd\u30fc\u30b9\u5217\u304c\u3001\u306b\u3088\u3063\u3066\u66f8\u304d\u8fbc\u307e\u308c\u308b\u3059\u3079\u3066\u306e\u30c6\u30fc\u30d6\u30eb\u306b\u30c7\u30fc\u30bf\u3092\u4f9b\u7d66\u3057\u3066\u3044\u308b\u304b\u3092\u793a\u3057\u307e\u3059\u3002 <code>\u30c6\u30fc\u30d6\u30eb\u3092\u4f5c\u6210\u3057\u307e\u3059...AS SELECT<\/code> \u307e\u305f <code>\u633f\u5165\u4e0a\u66f8\u304d<\/code>\u305d\u3057\u3066\u3001\u305d\u306e\u904e\u7a0b\u3067\u4e00\u6642\u7684\u306a\u30d3\u30e5\u30fc\u3001\u7d50\u5408\u3001\u95a2\u6570\u3001\u30d5\u30a3\u30eb\u30bf\u30fc\u304c\u5b9f\u884c\u3055\u308c\u307e\u3059\u3002 <strong>Gudu SQLFlow<\/strong> \u5c02\u7528\u306eSpark\u65b9\u8a00\u30d1\u30fc\u30b5\u30fc\u3092\u4f7f\u7528\u3057\u3066Spark SQL\u3092\u9759\u7684\u306b\u89e3\u6790\u3059\u308b\u3053\u3068\u3067\u3001\u305d\u306e\u30de\u30c3\u30d7\u3092\u4f5c\u6210\u3057\u307e\u3059\u3002\u30af\u30e9\u30b9\u30bf\u30fc\u306e\u8a08\u6e2c\u3001\u30ea\u30b9\u30ca\u30fc\u306e\u30a2\u30bf\u30c3\u30c1\u3001\u30b8\u30e7\u30d6\u306e\u5b9f\u884c\u306f\u4e0d\u8981\u3067\u3059\u3002SQL\u3092\u8cbc\u308a\u4ed8\u3051\u308b\u3060\u3051\u3067\u3001\u7cfb\u7d71\u56f3\u3092\u53d6\u5f97\u3067\u304d\u307e\u3059\u3002<\/p>\n\n\n\n<div class=\"wp-block-group alignfull has-background is-layout-constrained\" style=\"background-color:#eef7fb;padding-top:24px;padding-bottom:24px\"><div class=\"wp-block-group__inner-container\">\n\n<p><strong>\u4eca\u3059\u3050\u304a\u8a66\u3057\u304f\u3060\u3055\u3044\uff1a<\/strong> Spark SQL \u30b9\u30af\u30ea\u30d7\u30c8\u3092\u8cbc\u308a\u4ed8\u3051\u308b <a href=\"https:\/\/sqlflow.gudusoft.com\/?utm_source=gudusoft&amp;utm_medium=website&amp;utm_campaign=spark-sql-data-lineage\" target=\"_blank\" rel=\"noreferrer noopener\">\u7121\u6599\u30aa\u30f3\u30e9\u30a4\u30f3SQL\u7cfb\u7d71\u53ef\u8996\u5316\u30c4\u30fc\u30eb<\/a>Spark\u65b9\u8a00\u3092\u9078\u629e\u3057\u3001\u4efb\u610f\u306e\u5217\u3092\u4e0a\u6d41\u307e\u305f\u306f\u4e0b\u6d41\u306b\u30c8\u30ec\u30fc\u30b9\u3057\u307e\u3059\u3002\u30af\u30e9\u30a6\u30c9\u7248\u306b\u306f\u7121\u6599\u67a0\u304c\u3042\u308a\u307e\u3059\u3002<\/p>\n\n<\/div><\/div>\n\n\n\n<h2 class=\"wp-block-heading\">\u30e9\u30f3\u30bf\u30a4\u30e0\u30ea\u30b9\u30ca\u30fc\u3068\u9759\u7684\u89e3\u6790\uff1aSpark\u306e\u5c65\u6b74\u3092\u53d6\u5f97\u3059\u308b2\u3064\u306e\u65b9\u6cd5<\/h2>\n\n\n\n<p>Spark\u306e\u4e16\u754c\u306b\u304a\u3051\u308b\u7cfb\u8b5c\u7ba1\u7406\u306b\u306f\u6839\u672c\u7684\u306b\u7570\u306a\u308b2\u3064\u306e\u30a2\u30d7\u30ed\u30fc\u30c1\u304c\u3042\u308a\u3001\u307b\u3068\u3093\u3069\u306e\u30c1\u30fc\u30e0\u306f\u6700\u7d42\u7684\u306b\u4e21\u65b9\u3092\u5fc5\u8981\u3068\u3057\u307e\u3059\u3002<\/p>\n\n\n\n<p><strong>\u30e9\u30f3\u30bf\u30a4\u30e0\u306e\u7cfb\u8b5c<\/strong> OpenLineage\u3068\u305d\u306eSpark\u7d71\u5408\u304c\u63a1\u7528\u3057\u3066\u3044\u308b\u30a2\u30d7\u30ed\u30fc\u30c1\u306f\u3001Spark\u30bb\u30c3\u30b7\u30e7\u30f3\u306b\u30ea\u30b9\u30ca\u30fc\u3092\u30a2\u30bf\u30c3\u30c1\u3057\u3001\u30b8\u30e7\u30d6\u306e\u5b9f\u884c\u3054\u3068\u306b\u30ea\u30cd\u30fc\u30b8\u30a4\u30d9\u30f3\u30c8\u3092\u767a\u884c\u3057\u307e\u3059\u3002\u3053\u308c\u306f\u975e\u5e38\u306b\u512a\u308c\u305f\u6a5f\u80fd\u3067\u3001\u9759\u7684\u89e3\u6790\u3067\u306f\u628a\u63e1\u3067\u304d\u306a\u3044\u5b9f\u884c\u30ec\u30d9\u30eb\u306e\u30e1\u30bf\u30c7\u30fc\u30bf\u3068\u3068\u3082\u306b\u3001\u5b9f\u969b\u306b\u4f55\u304c\u3001\u3044\u3064\u3001\u3069\u306e\u30c7\u30fc\u30bf\u30bb\u30c3\u30c8\u306b\u5bfe\u3057\u3066\u5b9f\u884c\u3055\u308c\u305f\u304b\u3092\u8a18\u9332\u3057\u307e\u3059\u3002\u300c\u6628\u6669\u306e\u30d1\u30a4\u30d7\u30e9\u30a4\u30f3\u306f\u5b9f\u969b\u306b\u4f55\u306b\u5f71\u97ff\u3092\u4e0e\u3048\u305f\u306e\u304b\uff1f\u300d\u3068\u3044\u3046\u7591\u554f\u306b\u3001\u30e9\u30f3\u30bf\u30a4\u30e0\u30ea\u30cd\u30fc\u30b8\u304c\u7b54\u3048\u3066\u304f\u308c\u307e\u3059\u3002<\/p>\n\n\n\n<p><strong>\u9759\u7684\u7cfb\u7d71<\/strong> SQLFlow\u306e\u30a2\u30d7\u30ed\u30fc\u30c1\u306f\u3001\u30b3\u30f3\u30d1\u30a4\u30e9\u304c\u884c\u3046\u3088\u3046\u306b\u3001SQL\u30c6\u30ad\u30b9\u30c8\u81ea\u4f53\u3092\u89e3\u6790\u3057\u3001\u30b3\u30fc\u30c9\u304b\u3089\u7cfb\u7d71\u60c5\u5831\u3092\u5c0e\u304d\u51fa\u3057\u307e\u3059\u3002\u3053\u308c\u306b\u3088\u308a\u3001\u30e9\u30f3\u30bf\u30a4\u30e0\u30ad\u30e3\u30d7\u30c1\u30e3\u3067\u306f\u69cb\u9020\u7684\u306b\u51e6\u7406\u3067\u304d\u306a\u3044\u3059\u3079\u3066\u306e\u3053\u3068\u304c\u7db2\u7f85\u3055\u308c\u307e\u3059\u3002<\/p>\n\n\n\n<ul><li><strong>\u307e\u3060\u5b9f\u884c\u3057\u3066\u3044\u306a\u3044SQL\u3002<\/strong> \u30d7\u30eb\u30ea\u30af\u30a8\u30b9\u30c8\u306b\u542b\u307e\u308c\u308b\u65b0\u3057\u3044\u30d1\u30a4\u30d7\u30e9\u30a4\u30f3\u3001\u5225\u306e\u30c1\u30fc\u30e0\u304b\u3089\u5f15\u304d\u7d99\u3044\u3060\u30b9\u30af\u30ea\u30d7\u30c8\u3001\u56db\u534a\u671f\u3054\u3068\u306b\u30b9\u30b1\u30b8\u30e5\u30fc\u30eb\u3055\u308c\u305f\u30b8\u30e7\u30d6\u306e\u30d0\u30c3\u30af\u30ed\u30b0\u306a\u3069\u3001\u3053\u308c\u3089\u306f\u5b9f\u884c\u3055\u308c\u308b\u307e\u3067\uff08\u5834\u5408\u306b\u3088\u3063\u3066\u306f\u672c\u756a\u30c7\u30fc\u30bf\u306b\u5bfe\u3057\u3066\uff09\u30e9\u30f3\u30bf\u30a4\u30e0\u30a4\u30d9\u30f3\u30c8\u3092\u767a\u884c\u3057\u307e\u305b\u3093\u3002<\/li>\n<li><strong>\u30b3\u30fc\u30c9\u30ec\u30d3\u30e5\u30fc\u3002<\/strong> \u30ec\u30d3\u30e5\u30fc\u62c5\u5f53\u8005\u306f\u3001\u5909\u66f4\u3092\u627f\u8a8d\u3059\u308b\u524d\u306b\u3001\u5909\u66f4\u306e\u5217\u30ec\u30d9\u30eb\u306e\u4f9d\u5b58\u95a2\u4fc2\u30b0\u30e9\u30d5\u5168\u4f53\u3092\u78ba\u8a8d\u3067\u304d\u308b\u305f\u3081\u3001\u30c7\u30d7\u30ed\u30a4\u5f8c\u306b\u305d\u306e\u5f71\u97ff\u3092\u767a\u898b\u3059\u308b\u5fc5\u8981\u304c\u306a\u304f\u306a\u308a\u307e\u3059\u3002<\/li>\n<li><strong>\u79fb\u884c\u76e3\u67fb\u3002<\/strong> \u30ef\u30fc\u30af\u30ed\u30fc\u30c9\u3092 Spark \u306b\u79fb\u884c\u3059\u308b\u5834\u5408 (\u901a\u5e38\u306f Hive \u304b\u3089)\u3001\u4f9d\u5b58\u95a2\u4fc2\u30b0\u30e9\u30d5\u304c\u5fc5\u8981\u306b\u306a\u308a\u307e\u3059\u3002 <em>\u5168\u4f53<\/em> \u6708\u672b\u306b\u306e\u307f\u5b9f\u884c\u3055\u308c\u308b\u30b8\u30e7\u30d6\u3092\u542b\u3080\u30b3\u30fc\u30c9\u30d9\u30fc\u30b9\u3002\u9759\u7684\u89e3\u6790\u3067\u306f\u3001\u3059\u3079\u3066\u3092\u4e00\u5ea6\u306b\u8aad\u307f\u8fbc\u3080\u3053\u3068\u304c\u3067\u304d\u307e\u3059\u3002<\/li><\/ul>\n\n\n\n<figure class=\"wp-block-table\"><table><thead><tr><th><\/th><th>\u30e9\u30f3\u30bf\u30a4\u30e0\u30ea\u30b9\u30ca\u30fc (OpenLineage)<\/th><th>\u9759\u7684\u89e3\u6790\uff08SQLFlow\uff09<\/th><\/tr><\/thead><tbody>\n<tr><td>\u8840\u7d71\u304c\u6349\u3048\u3089\u308c\u305f<\/td><td>\u30b8\u30e7\u30d6\u5b9f\u884c\u3054\u3068\u306b\u3001\u5b9f\u884c\u3055\u308c\u308b<\/td><td>SQL\u30c6\u30ad\u30b9\u30c8\u304b\u3089\u3001\u4f55\u304b\u304c\u5b9f\u884c\u3055\u308c\u308b\u524d\u306b<\/td><\/tr>\n<tr><td>\u5fc5\u8981<\/td><td>Spark\u30bb\u30c3\u30b7\u30e7\u30f3\u3067\u30ea\u30b9\u30ca\u30fc\u304c\u8a2d\u5b9a\u3055\u308c\u3066\u3044\u307e\u3059\u3002\u30b8\u30e7\u30d6\u306f\u5b9f\u884c\u3055\u308c\u308b\u5fc5\u8981\u304c\u3042\u308a\u307e\u3059\u3002<\/td><td>SQL\u30d5\u30a1\u30a4\u30eb\u3068\u3001\u30aa\u30d7\u30b7\u30e7\u30f3\u306e\u30b9\u30ad\u30fc\u30de\u30e1\u30bf\u30c7\u30fc\u30bf<\/td><\/tr>\n<tr><td>\u672a\u5b9f\u884c\u306e\u30b3\u30fc\u30c9\u3092\u30ab\u30d0\u30fc\u3057\u307e\u3059<\/td><td>\u3044\u3044\u3048\u3001\u30e9\u30f3\u30cb\u30f3\u30b0\u3082\u30a4\u30d9\u30f3\u30c8\u3082\u3042\u308a\u307e\u305b\u3093\u3002<\/td><td>\u306f\u3044\u3001PR\u3001\u30d0\u30c3\u30af\u30ed\u30b0\u3001\u3081\u3063\u305f\u306b\u5b9f\u884c\u3055\u308c\u306a\u3044\u30b8\u30e7\u30d6<\/td><\/tr>\n<tr><td>\u5b9f\u884c\u30e1\u30bf\u30c7\u30fc\u30bf\uff08\u30bf\u30a4\u30df\u30f3\u30b0\u3001\u30d0\u30fc\u30b8\u30e7\u30f3\uff09<\/td><td>\u306f\u3044<\/td><td>\u3044\u3044\u3048 \u2014 \u30b3\u30fc\u30c9\u30ec\u30d9\u30eb\u306e\u307f<\/td><\/tr>\n<tr><td>\u5178\u578b\u7684\u306a\u4f7f\u7528\u4f8b<\/td><td>\u7a3c\u50cd\u4e2d\u306e\u30d1\u30a4\u30d7\u30e9\u30a4\u30f3\u306e\u904b\u7528\u4e0a\u306e\u53ef\u89b3\u6e2c\u6027<\/td><td>\u5f71\u97ff\u5206\u6790\u3001\u30b3\u30fc\u30c9\u30ec\u30d3\u30e5\u30fc\u3001\u79fb\u884c\u76e3\u67fb\u3001\u30b3\u30f3\u30d7\u30e9\u30a4\u30a2\u30f3\u30b9\u6587\u66f8\u4f5c\u6210<\/td><\/tr>\n<\/tbody><\/table><\/figure>\n\n\n\n<p>\u3053\u308c\u3089\u306f\u4e92\u3044\u306b\u88dc\u5b8c\u3057\u5408\u3046\u3082\u306e\u3067\u3042\u308a\u3001\u7af6\u5408\u3059\u308b\u3082\u306e\u3067\u306f\u3042\u308a\u307e\u305b\u3093\u3002\u30e9\u30f3\u30bf\u30a4\u30e0\u30ea\u30cd\u30fc\u30b8\u306f\u767a\u751f\u3057\u305f\u4e8b\u8c61\u3092\u793a\u3057\u3001\u9759\u7684\u30ea\u30cd\u30fc\u30b8\u306f\u30b3\u30fc\u30c9\u306e\u52d5\u4f5c\u3092\u793a\u3057\u307e\u3059\u3002SQLFlow \u306e DataHub\u3001Microsoft Purview\u3001\u304a\u3088\u3073 OpenMetadata \u7528\u306e\u30a8\u30af\u30b9\u30dd\u30fc\u30c8\u30a2\u30c0\u30d7\u30bf\u3092\u4f7f\u7528\u3059\u308b\u3068\u3001\u9759\u7684\u306b\u751f\u6210\u3055\u308c\u305f\u30ea\u30cd\u30fc\u30b8\u3092\u30e9\u30f3\u30bf\u30a4\u30e0\u30a4\u30d9\u30f3\u30c8\u3068\u540c\u3058\u30ab\u30bf\u30ed\u30b0\u306b\u683c\u7d0d\u3067\u304d\u308b\u305f\u3081\u30012 \u3064\u306e\u30d3\u30e5\u30fc\u3092\u4e26\u3079\u3066\u8868\u793a\u3067\u304d\u307e\u3059\u3002<\/p>\n\n\n\n<h2 class=\"wp-block-heading\">\u4f8b\uff1aCTAS\u3078\u306e\u4e00\u6642\u30d3\u30e5\u30fc\u30c1\u30a7\u30fc\u30f3<\/h2>\n\n\n\n<p>Spark SQL \u3067\u5358\u7d14\u306a\u30ea\u30cd\u30fc\u30b8\u30c4\u30fc\u30eb\u3092\u7121\u52b9\u5316\u3059\u308b\u30d1\u30bf\u30fc\u30f3\u306f\u3001\u4e00\u6642\u30d3\u30e5\u30fc\u30c1\u30a7\u30fc\u30f3\u3067\u3059\u3002\u30ce\u30fc\u30c8\u30d6\u30c3\u30af\u3068\u30b8\u30e7\u30d6\u306f\u3001\u5b9a\u671f\u7684\u306b\u30ed\u30b8\u30c3\u30af\u3092\u30b9\u30c6\u30fc\u30b8\u30f3\u30b0\u3057\u307e\u3059\u3002 <code>\u4e00\u6642\u30d3\u30e5\u30fc\u306e\u4f5c\u6210\u307e\u305f\u306f\u7f6e\u63db<\/code> \u6700\u7d42\u6bb5\u968e\u307e\u3067\u306e\u30b9\u30c6\u30c3\u30d7 <code>\u30c6\u30fc\u30d6\u30eb\u3092\u4f5c\u6210\u3057\u307e\u3059...AS SELECT<\/code>\u4ee5\u4e0b\u306b\u3001\u7c21\u6f54\u306a\u304c\u3089\u3082\u73fe\u5b9f\u7684\u306a\u4f8b\u3092\u793a\u3057\u307e\u3059\u3002<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>CREATE OR REPLACE TEMP VIEW clean_events AS SELECT CAST(event_time AS TIMESTAMP) AS event_ts, user_id, lower(event_name) AS event_name, amount FROM raw.events WHERE event_date &gt;= date_sub(current_date(), 30); CREATE OR REPLACE TEMP VIEW user_daily AS SELECT user_id, to_date(event_ts) AS event_day, SUM(amount) AS daily_spend FROM clean_events WHERE event_name = &#039;purchase&#039; GROUP BY user_id, to_date(event_ts); CREATE TABLE analytics.user_spend_30d USING PARQUET AS SELECT u.user_id, u.segment, d.event_day, d.daily_spend FROM user_daily d JOIN dim.users u ON u.user_id = d.user_id;<\/code><\/pre>\n\n\n\n<p>\u300c\u3069\u3053\u3067 <code>\u30a2\u30ca\u30ea\u30c6\u30a3\u30af\u30b9.user_spend_30d.daily_spend<\/code> \u300c\u3069\u3053\u304b\u3089\u6765\u305f\u306e\u304b\uff1f\u300d\u3068\u3044\u3046\u7591\u554f\u306b\u5bfe\u3059\u308b\u6b63\u76f4\u306a\u7b54\u3048\u306f\u30012\u3064\u306e\u30d3\u30e5\u30fc\u30ec\u30a4\u30e4\u30fc\u3092\u89e3\u6c7a\u3059\u308b\u5fc5\u8981\u304c\u3042\u308b\u3002 <code>\u6bce\u65e5\u306e\u652f\u51fa<\/code> \u306f <code>SUM(clean_events.amount)<\/code>\u3001 \u3068 <code>clean_events.amount<\/code> \u306f <code>\u751f\u306e\u30a4\u30d9\u30f3\u30c8\u6570<\/code>SQLFlow \u306f\u307e\u3055\u306b\u305d\u306e\u89e3\u6c7a\u3092\u5b9f\u884c\u3057\u3001\u5b8c\u5168\u306a\u30c1\u30a7\u30fc\u30f3\u3092\u30ec\u30f3\u30c0\u30ea\u30f3\u30b0\u3057\u307e\u3059\u3002 <code>\u751f\u306e\u30a4\u30d9\u30f3\u30c8\u6570<\/code> \u3092\u901a\u3057\u3066 <code>\u30af\u30ea\u30fc\u30f3\u30a4\u30d9\u30f3\u30c8<\/code> \u3068 <code>\u30e6\u30fc\u30b6\u30fc\u30c7\u30a4\u30ea\u30fc<\/code> \u6700\u7d42\u30c6\u30fc\u30d6\u30eb\u306b\u9032\u51fa\u3057\u3001 <code>\u548c<\/code> \u96c6\u7d04\u306f\u3001\u305d\u308c\u304c\u8d77\u3053\u308b\u30a8\u30c3\u30b8\u306b\u4ed8\u968f\u3059\u308b\u3002<\/p>\n\n\n\n<p>\u307e\u305f\u3001\u307b\u3068\u3093\u3069\u306e\u30c4\u30fc\u30eb\u304c\u5b8c\u5168\u306b\u7121\u8996\u3057\u3066\u3044\u308b\u60c5\u5831\u3082\u6349\u3048\u3066\u3044\u307e\u3059\u3002 <strong>\u9593\u63a5\u7684\u306a\u7cfb\u7d71<\/strong>\u5217 <code>\u30a4\u30d9\u30f3\u30c8\u65e5\u4ed8<\/code>, <code>\u30a4\u30d9\u30f3\u30c8\u540d<\/code>\u3001\u305d\u3057\u3066\u7d50\u5408\u30ad\u30fc <code>\u30e6\u30fc\u30b6\u30fcID<\/code> \u6c7a\u3057\u3066\u767b\u5834\u3057\u306a\u3044 <code>\u6bce\u65e5\u306e\u652f\u51fa<\/code>\u3057\u304b\u3057\u30013\u3064\u3059\u3079\u3066\u304c\u305d\u306e\u4fa1\u5024\u3092\u5de6\u53f3\u3059\u308b\u300230\u65e5\u9593\u30d5\u30a3\u30eb\u30bf\u30fc\u3092\u5909\u66f4\u3059\u308b\u304b\u3001 <code>&#039;\u8cfc\u5165&#039;<\/code> \u8ff0\u8a9e\u3068\u51fa\u529b\u306e\u3059\u3079\u3066\u306e\u6570\u5024\u304c\u5909\u66f4\u3055\u308c\u307e\u3059\u3002SQLFlow \u306f\u3053\u308c\u3089\u3092\u500b\u5225\u306e\u5207\u308a\u66ff\u3048\u53ef\u80fd\u306a\u95a2\u4fc2\u30bf\u30a4\u30d7\u3068\u3057\u3066\u30e2\u30c7\u30eb\u5316\u3059\u308b\u305f\u3081\u3001\u7d14\u7c8b\u306a\u30c7\u30fc\u30bf\u30d5\u30ed\u30fc\u3092\u8868\u793a\u3059\u308b\u3053\u3068\u3082\u3001\u5f71\u97ff\u7bc4\u56f2\u5168\u4f53\u3092\u8868\u793a\u3059\u308b\u3053\u3068\u3082\u3067\u304d\u307e\u3059\u3002\u30b9\u30ad\u30fc\u30de\u5909\u66f4\u306e\u30ec\u30d3\u30e5\u30fc\u3067\u306f\u3001\u5f71\u97ff\u7bc4\u56f2\u3092\u8868\u793a\u3059\u308b\u30d3\u30e5\u30fc\u304c\u5b9f\u969b\u306b\u5fc5\u8981\u306a\u30d3\u30e5\u30fc\u3067\u3059\u3002<\/p>\n\n\n\n<h2 class=\"wp-block-heading\">Spark SQL \u306f SQLFlow \u3092\u89e3\u6c7a\u3057\u307e\u3059<\/h2>\n\n\n\n<p>SQLFlow\u306b\u306f\u3001Spark SQL\u65b9\u8a00\u30d1\u30fc\u30b5\u30fc\u304c\u540c\u68b1\u3055\u308c\u3066\u3044\u307e\u3059\u3002\u3053\u308c\u306f\u3001\u6c4e\u7528\u7684\u306aANSI\u6587\u6cd5\u306bSpark\u30ad\u30fc\u30ef\u30fc\u30c9\u3092\u8ffd\u52a0\u3057\u305f\u3082\u306e\u3067\u306f\u306a\u304f\u300139\u7a2e\u985e\u3042\u308b\u65b9\u8a00\u56fa\u6709\u306e\u30d1\u30fc\u30b5\u30fc\u306e1\u3064\u3067\u3059\u3002\u7279\u306bSpark SQL\u3067\u306f\u3001\u4ee5\u4e0b\u306e\u5c65\u6b74\u3092\u6301\u3064\u30b9\u30c6\u30fc\u30c8\u30e1\u30f3\u30c8\u3092\u51e6\u7406\u3057\u307e\u3059\u3002<\/p>\n\n\n\n<ul><li><strong><code>\u30c6\u30fc\u30d6\u30eb\u3092\u4f5c\u6210\u3057\u307e\u3059...AS SELECT<\/code> \uff08CTAS\uff09<\/strong> \u2014 Spark \u30d0\u30c3\u30c1 \u30d1\u30a4\u30d7\u30e9\u30a4\u30f3\u306e\u4e3b\u529b\u6a5f\u80fd\u3002\u3059\u3079\u3066\u306e\u51fa\u529b\u5217\u306f\u3001\u5b8c\u5168\u306a SELECT \u3092\u901a\u3058\u3066\u305d\u306e\u30bd\u30fc\u30b9\u307e\u3067\u8ffd\u8de1\u3055\u308c\u307e\u3059\u3002<\/li>\n<li><strong><code>\u633f\u5165\u5148<\/code> \u3068 <code>\u633f\u5165\u4e0a\u66f8\u304d<\/code><\/strong> \u2014 SELECT\u30ea\u30b9\u30c8\u3068\u5bfe\u8c61\u30c6\u30fc\u30d6\u30eb\u306e\u30b9\u30ad\u30fc\u30de\u9593\u306e\u4f4d\u7f6e\u7684\u306a\u5217\u30de\u30c3\u30d4\u30f3\u30b0\u3092\u542b\u307f\u307e\u3059\u3002<\/li>\n<li><strong><code>\u4e00\u6642\u30d3\u30e5\u30fc\u306e\u4f5c\u6210\u307e\u305f\u306f\u7f6e\u63db<\/code><\/strong> \u2015 \u4e00\u6642\u30d3\u30e5\u30fc\u306f\u89e3\u6c7a\u3055\u308c\u3001\u9023\u9396\u3055\u308c\u308b\u305f\u3081\u3001\u30ea\u30cd\u30fc\u30b8\u306f\u30b9\u30c6\u30fc\u30b8\u30f3\u30b0\u30ec\u30a4\u30e4\u30fc\u3067\u505c\u6b62\u3059\u308b\u306e\u3067\u306f\u306a\u304f\u3001\u30b9\u30c6\u30fc\u30b8\u30f3\u30b0\u30ec\u30a4\u30e4\u30fc\u3092\u901a\u904e\u3057\u307e\u3059\u3002<\/li>\n<li><strong>CTE\u3001\u30b5\u30d6\u30af\u30a8\u30ea\u3001 <code>\u9078\u629e *<\/code><\/strong> \u5217\u53c2\u7167\u306f\u5171\u901a\u30c6\u30fc\u30d6\u30eb\u5f0f\u3068\u30cd\u30b9\u30c8\u3055\u308c\u305f\u30b5\u30d6\u30af\u30a8\u30ea\u306b\u3088\u3063\u3066\u89e3\u6c7a\u3055\u308c\u3001\u30b9\u30ad\u30fc\u30de\u30e1\u30bf\u30c7\u30fc\u30bf\u304c\u5229\u7528\u53ef\u80fd\u306a\u5834\u5408\u306f\u30b9\u30bf\u30fc\u5f0f\u304c\u5b9f\u969b\u306e\u5217\u306b\u5c55\u958b\u3055\u308c\u307e\u3059\u3002<\/li>\n<li><strong>\u95a2\u6570\u3001\u578b\u5909\u63db\u3001\u304a\u3088\u3073\u96c6\u5408\u6f14\u7b97\u5b50<\/strong> \u2014 \u5404\u51fa\u529b\u5217\u306e\u7cfb\u7d71\u306f\u3001\u305d\u308c\u304c\u901a\u904e\u3057\u305f\u5909\u63db\u3092\u8a18\u9332\u3057\u307e\u3059\u3002 <code>\u6bce\u65e5\u306e\u652f\u51fa<\/code> \u300c\u91d1\u984d\u304b\u3089\u300d\u3060\u3051\u3067\u306f\u306a\u304f\u300c <code>\u984d<\/code> \u7d4c\u7531 <code>\u548c<\/code>&#8220;.<\/li><\/ul>\n\n\n\n<p>\u5185\u90e8\u7684\u306b\u306f\u3001\u3053\u308c\u306f\u6c4e\u7528SQL\u30d1\u30fc\u30b5\u30fc\u30a8\u30f3\u30b8\u30f3\u3067\u3059\u3002\u3053\u308c\u306f\u30012000\u5e74\u4ee3\u534a\u3070\u304b\u3089\u958b\u767a\u3055\u308c\u3001\u65b9\u8a00\u3054\u3068\u306b\u7d0413,600\u500b\u306e\u30c6\u30b9\u30c8\u30d5\u30a3\u30af\u30b9\u30c1\u30e3\u3067\u691c\u8a3c\u3055\u308c\u305f\u5546\u7528SQL\u30b3\u30f3\u30d1\u30a4\u30e9\u306e\u30d5\u30ed\u30f3\u30c8\u30a8\u30f3\u30c9\u3067\u3059\u3002\u7cfb\u7d71\u54c1\u8cea\u306f\u3001\u4f55\u3088\u308a\u3082\u307e\u305a\u69cb\u6587\u89e3\u6790\u306e\u554f\u984c\u3067\u3042\u308a\u3001\u3053\u306e\u30b3\u30fc\u30d1\u30b9\u3053\u305d\u304c\u3001Spark\u306e\u7279\u6b8a\u306a\u69cb\u6587\u304c\u610f\u56f3\u305b\u305a\u8aa4\u3063\u305f\u30a8\u30c3\u30b8\u3092\u751f\u6210\u3059\u308b\u306e\u3092\u9632\u3044\u3067\u3044\u308b\u306e\u3067\u3059\u3002<\/p>\n\n\n\n<h2 class=\"wp-block-heading\">Spark SQL\u30c7\u30fc\u30bf\u30ea\u30cd\u30fc\u30b8\u304c\u30b9\u30bf\u30c3\u30af\u5185\u3067\u3069\u306e\u3088\u3046\u306b\u4f4d\u7f6e\u3065\u3051\u3089\u308c\u308b\u304b<\/h2>\n\n\n\n<p>Spark\u306f\u5358\u72ec\u3067\u52d5\u4f5c\u3059\u308b\u3053\u3068\u306f\u307b\u3068\u3093\u3069\u3042\u308a\u307e\u305b\u3093\u3002\u305d\u306e\u30c6\u30fc\u30d6\u30eb\u306f\u901a\u5e38Hive\u30e1\u30bf\u30b9\u30c8\u30a2\u306b\u683c\u7d0d\u3055\u308c\u3001Spark\u30ef\u30fc\u30af\u30ed\u30fc\u30c9\u306e\u5927\u90e8\u5206\u306fDatabricks\u4e0a\u3067\u5b9f\u884c\u3055\u308c\u308b\u3088\u3046\u306b\u306a\u3063\u3066\u3044\u307e\u3059\u3002SQLFlow\u306f\u3053\u308c\u3089\u3092\u3001\u305d\u308c\u305e\u308c\u72ec\u7acb\u3057\u3066\u89e3\u6790\u3055\u308c\u308b\u7b2c\u4e00\u7d1a\u306e\u65b9\u8a00\u3068\u3057\u3066\u6271\u3044\u307e\u3059\u3002<\/p>\n\n\n\n<ul><li>Hive\u30b8\u30e7\u30d6\u3092Spark\u306b\u79fb\u884c\u3057\u307e\u3059\u304b\uff1f\u30ec\u30ac\u30b7\u30fcHQL\u3092\u5206\u6790\u3057\u307e\u3059 <a href=\"https:\/\/www.gudusoft.com\/jp\/%e3%83%8f%e3%82%a4%e3%83%96%e3%83%87%e3%83%bc%e3%82%bf%e7%b3%bb%e7%b5%b1\/\">Hive\u30c7\u30fc\u30bf\u30ea\u30cd\u30fc\u30b8\u30d1\u30fc\u30b5\u30fc<\/a> \u305d\u3057\u3066\u3001Spark\u30d1\u30fc\u30b5\u30fc\u3092\u4f7f\u7528\u3057\u3066\u66f8\u304d\u63db\u3048\u3089\u308c\u305f\u30b8\u30e7\u30d6\u3092\u4f5c\u6210\u3057\u30012\u3064\u306e\u4f9d\u5b58\u95a2\u4fc2\u30b0\u30e9\u30d5\u3092\u6bd4\u8f03\u3057\u3066\u3001\u5909\u63db\u6642\u306b\u4f55\u3082\u5931\u308f\u308c\u3066\u3044\u306a\u3044\u3053\u3068\u3092\u78ba\u8a8d\u3057\u307e\u3059\u3002<\/li>\n<li>Databricks\u4e0a\u3067\u5b9f\u884c\u4e2d\u3067\u3059\u304b\uff1f <a href=\"https:\/\/www.gudusoft.com\/jp\/%e3%83%87%e3%83%bc%e3%82%bf%e3%83%96%e3%83%aa%e3%83%83%e3%82%af%e3%82%b9%e3%81%ae%e3%83%87%e3%83%bc%e3%82%bf%e3%83%aa%e3%83%8d%e3%83%bc%e3%82%b8\/\">Databricks \u30c7\u30fc\u30bf\u30ea\u30cd\u30fc\u30b8 \u30d1\u30fc\u30b5\u30fc<\/a> \u672c\u66f8\u306f\u3001\u30aa\u30fc\u30d7\u30f3\u30bd\u30fc\u30b9\u306eSpark SQL\u3068\u306f\u4e56\u96e2\u3057\u3066\u304a\u308a\u3001\u5358\u4e00\u306e\u5171\u901a\u6587\u6cd5\u3067\u306f\u9069\u5207\u306b\u51e6\u7406\u3067\u304d\u306a\u3044Databricks SQL\u65b9\u8a00\u3092\u5bfe\u8c61\u3068\u3057\u3066\u3044\u307e\u3059\u3002<\/li>\n<li>\u6df7\u5408\u74b0\u5883\uff1f SQLFlow \u306f\u30b5\u30dd\u30fc\u30c8\u3055\u308c\u3066\u3044\u308b 39 \u306e\u65b9\u8a00\u3059\u3079\u3066\u3092 1 \u3064\u306e\u7cfb\u7d71\u30ea\u30dd\u30b8\u30c8\u30ea\u306b\u5206\u6790\u3059\u308b\u305f\u3081\u3001Snowflake \u30bf\u30b9\u30af\u306b\u3088\u3063\u3066\u751f\u6210\u3055\u308c\u305f\u30c6\u30fc\u30d6\u30eb\u3092\u8aad\u307f\u53d6\u308b Spark \u30b8\u30e7\u30d6\u306f\u30011 \u3064\u306e\u9023\u7d9a\u3057\u305f\u30b0\u30e9\u30d5\u3068\u3057\u3066\u8868\u793a\u3055\u308c\u307e\u3059\u3002 <a href=\"https:\/\/www.gudusoft.com\/jp\/sql-data-lineage-tool\/\">SQL\u30c7\u30fc\u30bf\u30ea\u30cd\u30fc\u30b8\u30c4\u30fc\u30eb\u306e\u6982\u8981<\/a> \u5168\u4f53\u50cf\u3092\u628a\u63e1\u3059\u308b\u306b\u306f\u3002<\/li><\/ul>\n\n\n\n<p>\u5165\u529b\u65b9\u6cd5\u306f\u67d4\u8edf\u3067\u3001SQL\u306e\u8cbc\u308a\u4ed8\u3051\u3001\u30d5\u30a1\u30a4\u30eb\u306e\u30a2\u30c3\u30d7\u30ed\u30fc\u30c9\u3001JDBC\u7d4c\u7531\u3067\u306e\u30e1\u30bf\u30c7\u30fc\u30bf\u306e\u53d6\u5f97\u3001\u307e\u305f\u306fdbt-on-Spark\u30d7\u30ed\u30b8\u30a7\u30af\u30c8\u7528\u306edbt\u30de\u30cb\u30d5\u30a7\u30b9\u30c8\u306e\u30a4\u30f3\u30dd\u30fc\u30c8\u304c\u53ef\u80fd\u3067\u3059\u3002\u51fa\u529b\u306f\u3001\u30a4\u30f3\u30bf\u30e9\u30af\u30c6\u30a3\u30d6\u306a\u30c9\u30ea\u30eb\u30c0\u30a6\u30f3\u53ef\u80fd\u306a\u56f3\u3001JSON\u3001CSV\u3001PNG\u5f62\u5f0f\u306e\u30a8\u30af\u30b9\u30dd\u30fc\u30c8\u3001\u304a\u3088\u3073\u81ea\u52d5\u5316\u306e\u305f\u3081\u306eREST API\u3067\u3059\u3002\u4f8b\u3048\u3070\u3001\u30d7\u30eb\u30ea\u30af\u30a8\u30b9\u30c8\u3067\u5909\u66f4\u3055\u308c\u305fSQL\u3092CI\u30b9\u30c6\u30c3\u30d7\u3068\u3057\u3066\u5206\u6790\u3059\u308b\u3068\u3044\u3063\u305f\u3053\u3068\u304c\u53ef\u80fd\u3067\u3059\u3002<\/p>\n\n\n\n<h2 class=\"wp-block-heading\">\u5c55\u958b\u3068\u62e1\u5f35<\/h2>\n\n\n\n<p>SQLFlow Cloud\u306b\u306f\u30a4\u30f3\u30bf\u30e9\u30af\u30c6\u30a3\u30d6\u306b\u4f7f\u7528\u3067\u304d\u308b\u7121\u6599\u30d7\u30e9\u30f3\u304c\u3042\u308a\u3001\u30d7\u30ec\u30df\u30a2\u30e0\u30d7\u30e9\u30f3\u306f\u6708\u984d$49.99\u3067\u3059\u3002SQL\u304c\u30cd\u30c3\u30c8\u30ef\u30fc\u30af\u5916\u306b\u51fa\u3089\u308c\u306a\u3044\u30c1\u30fc\u30e0\u306f\u3001 <a href=\"https:\/\/www.gudusoft.com\/jp\/sqlflow-%e3%82%aa%e3%83%b3%e3%83%97%e3%83%ac%e3%83%9f%e3%82%b9-%e3%83%90%e3%83%bc%e3%82%b8%e3%83%a7%e3%83%b3\/\">SQLFlow \u30aa\u30f3\u30d7\u30ec\u30df\u30b9<\/a> \u2014 Docker\u307e\u305f\u306fKubernetes\u306b\u5bfe\u5fdc\u3001\u30a8\u30a2\u30ae\u30e3\u30c3\u30d7\u74b0\u5883\u306b\u3082\u5bfe\u5fdc\u3001\u9078\u629e\u3057\u305f\u30c7\u30fc\u30bf\u30d9\u30fc\u30b9\u30bf\u30a4\u30d7\u3054\u3068\u306b\u6708\u984d$500\u307e\u305f\u306f1\u56de\u9650\u308a\u306e$4,800\u3002\u3044\u305a\u308c\u306e\u5834\u5408\u3082\u3001SQLFlow\u306fSQL\u30b3\u30fc\u30c9\u306e\u9759\u7684\u89e3\u6790\u306e\u307f\u3092\u5b9f\u884c\u3057\u3001\u30c6\u30fc\u30d6\u30eb\u306e\u884c\u3092\u8aad\u307f\u53d6\u308b\u3053\u3068\u306f\u3042\u308a\u307e\u305b\u3093\u3002\u30a8\u30f3\u30bf\u30fc\u30d7\u30e9\u30a4\u30ba\u5c55\u958b\u3067\u306f\u3001\u5897\u5206\u30b9\u30ad\u30e3\u30f3\u3068\u6c38\u7d9a\u7684\u306a\u30ea\u30cd\u30fc\u30b8\u30ea\u30dd\u30b8\u30c8\u30ea\u3092\u4f7f\u7528\u3057\u3066\u3001100\u4ee5\u4e0a\u306e\u30c7\u30fc\u30bf\u30d9\u30fc\u30b9\u3068100\u4e07\u4ee5\u4e0a\u306e\u5217\u304b\u3089\u306a\u308b\u74b0\u5883\u3092\u30d0\u30c3\u30c1\u30b9\u30ad\u30e3\u30f3\u3057\u307e\u3059\u3002<\/p>\n\n\n\n<h2 class=\"wp-block-heading\">\u3088\u304f\u3042\u308b\u8cea\u554f<\/h2>\n\n\n\n<h3 class=\"wp-block-heading\">\u7cfb\u7d71\u60c5\u5831\u3092\u53d6\u5f97\u3059\u308b\u306b\u306f\u3001\u7a3c\u50cd\u4e2d\u306eSpark\u30af\u30e9\u30b9\u30bf\u30fc\u304c\u5fc5\u8981\u3067\u3059\u304b\uff1f<\/h3>\n\n\n<p>\u3044\u3044\u3048\u3002SQLFlow\u306fSpark SQL\u30c6\u30ad\u30b9\u30c8\u3092\u9759\u7684\u306b\u89e3\u6790\u3057\u307e\u3059\u3002\u30d7\u30eb\u30ea\u30af\u30a8\u30b9\u30c8\u3001\u30d9\u30f3\u30c0\u30fc\u304b\u3089\u306e\u7d0d\u54c1\u7269\u3001\u79fb\u884c\u30d0\u30c3\u30af\u30ed\u30b0\u306a\u3069\u3001\u3053\u308c\u307e\u3067\u4e00\u5ea6\u3082\u5b9f\u884c\u3055\u308c\u305f\u3053\u3068\u306e\u306a\u3044\u30b9\u30af\u30ea\u30d7\u30c8\u3092\u89e3\u6790\u3057\u3066\u3082\u3001\u672c\u756a\u30b3\u30fc\u30c9\u304b\u3089\u5f97\u3089\u308c\u308b\u306e\u3068\u540c\u3058\u5217\u30ec\u30d9\u30eb\u306e\u30c7\u30fc\u30bf\u7cfb\u5217\u3092\u53d6\u5f97\u3067\u304d\u307e\u3059\u3002<\/p>\n\n\n\n<h3 class=\"wp-block-heading\">\u3053\u308c\u306fOpenLineage\u306eSpark\u7d71\u5408\u3068\u3069\u3046\u9055\u3046\u306e\u3067\u3059\u304b\uff1f<\/h3>\n\n\n<p>OpenLineage\u306f\u3001Spark\u30bb\u30c3\u30b7\u30e7\u30f3\u4e0a\u306e\u30ea\u30b9\u30ca\u30fc\u3092\u4ecb\u3057\u3066\u5b9f\u884c\u6642\u306b\u30ea\u30cd\u30fc\u30b8\u3092\u30ad\u30e3\u30d7\u30c1\u30e3\u3057\u307e\u3059\u3002\u3053\u308c\u306f\u3001\u5b9f\u884c\u4e2d\u306e\u30b8\u30e7\u30d6\u304c\u5b9f\u969b\u306b\u4f55\u3092\u3057\u305f\u304b\u3092\u76e3\u8996\u3059\u308b\u306e\u306b\u6700\u9069\u3067\u3059\u3002SQLFlow\u306fSQL\u30b3\u30fc\u30c9\u81ea\u4f53\u304b\u3089\u30ea\u30cd\u30fc\u30b8\u3092\u751f\u6210\u3059\u308b\u305f\u3081\u3001\u5b9f\u884c\u3055\u308c\u3066\u3044\u306a\u3044\u30b3\u30fc\u30c9\u3082\u5bfe\u8c61\u3068\u3057\u3001\u30de\u30fc\u30b8\u524d\u306e\u30ec\u30d3\u30e5\u30fc\u3084\u79fb\u884c\u76e3\u67fb\u3092\u30b5\u30dd\u30fc\u30c8\u3057\u3001\u30af\u30e9\u30b9\u30bf\u3078\u306e\u30a2\u30af\u30bb\u30b9\u3082\u4e0d\u8981\u3067\u3059\u3002\u3053\u306e2\u3064\u306f\u76f8\u4e92\u88dc\u5b8c\u7684\u306a\u95a2\u4fc2\u306b\u3042\u308a\u3001SQLFlow\u306f\u5b9f\u884c\u6642\u306e\u30ea\u30cd\u30fc\u30b8\u3068\u4e26\u884c\u3057\u3066\u3001DataHub\u3001Purview\u3001\u307e\u305f\u306fOpenMetadata\u306b\u30a8\u30af\u30b9\u30dd\u30fc\u30c8\u3067\u304d\u307e\u3059\u3002<\/p>\n\n\n\n<h3 class=\"wp-block-heading\">SQLFlow\u306f\u4e00\u6642\u30d3\u30e5\u30fc\u3092\u901a\u3057\u3066\u30c7\u30fc\u30bf\u7cfb\u5217\u3092\u8ffd\u8de1\u3067\u304d\u307e\u3059\u304b\uff1f<\/h3>\n\n\n<p>\u306f\u3044\u3002 <code>\u4e00\u6642\u30d3\u30e5\u30fc\u306e\u4f5c\u6210\u307e\u305f\u306f\u7f6e\u63db<\/code> \u30b9\u30c6\u30fc\u30c8\u30e1\u30f3\u30c8\u306f\u30a8\u30f3\u30c9\u30c4\u30fc\u30a8\u30f3\u30c9\u3067\u89e3\u6c7a\u3055\u308c\u308b\u305f\u3081\u3001\u6700\u7d42\u7684\u306aCTAS\u306e\u5217\u306f\u3001\u5404\u30b9\u30c6\u30c3\u30d7\u3067\u95a2\u6570\u3068\u96c6\u8a08\u304c\u9069\u7528\u3055\u308c\u305f\u72b6\u614b\u3067\u3001\u3059\u3079\u3066\u306e\u30b9\u30c6\u30fc\u30b8\u30f3\u30b0\u30d3\u30e5\u30fc\u3092\u305f\u3069\u3063\u3066\u7269\u7406\u7684\u306a\u30bd\u30fc\u30b9\u30c6\u30fc\u30d6\u30eb\u307e\u3067\u30c8\u30ec\u30fc\u30b9\u3055\u308c\u307e\u3059\u3002<\/p>\n\n\n\n<h3 class=\"wp-block-heading\">Databricks SQL\u306f\u3001\u3053\u3053\u3067\u3044\u3046Spark SQL\u3068\u540c\u3058\u3082\u306e\u3067\u3059\u304b\uff1f<\/h3>\n\n\n<p>\u3044\u3044\u3048\u3002SQLFlow\u306f\u3001\u305d\u308c\u305e\u308c\u7570\u306a\u308b\u65b9\u8a00\u30d1\u30fc\u30b5\u30fc\u3092\u4f7f\u7528\u3057\u3066\u89e3\u6790\u3057\u307e\u3059\u3002Databricks SQL\u306b\u306f\u72ec\u81ea\u306e\u69cb\u6587\u62e1\u5f35\u304c\u3042\u308b\u305f\u3081\u3001\u72ec\u81ea\u306e\u6587\u6cd5\u3092\u6301\u3061\u307e\u3059\u3002Databricks\u4e0a\u3067\u5b9f\u884c\u3059\u308b\u5834\u5408\u306fDatabricks\u65b9\u8a00\u3092\u3001\u30aa\u30fc\u30d7\u30f3\u30bd\u30fc\u30b9\u306eSpark\u3001EMR\u3001\u307e\u305f\u306f\u81ea\u5df1\u7ba1\u7406\u578b\u30af\u30e9\u30b9\u30bf\u30fc\u306e\u5834\u5408\u306fSpark SQL\u65b9\u8a00\u3092\u4f7f\u7528\u3057\u3066\u304f\u3060\u3055\u3044\u3002<\/p>\n\n\n\n<h3 class=\"wp-block-heading\">SQLFlow\u306f\u79c1\u306e\u30c7\u30fc\u30bf\u3092\u8aad\u307f\u53d6\u308a\u307e\u3059\u304b\uff1f<\/h3>\n\n\n<p>\u3044\u3044\u3048\u3002SQL\u30b3\u30fc\u30c9\u3068\u3001\u5fc5\u8981\u306b\u5fdc\u3058\u3066\u30b9\u30ad\u30fc\u30de\u30e1\u30bf\u30c7\u30fc\u30bf\uff08\u30c6\u30fc\u30d6\u30eb\u3068\u5217\u306e\u5b9a\u7fa9\uff09\u3092\u5206\u6790\u3057\u307e\u3059\u3002\u884c\u30c7\u30fc\u30bf\u306b\u306f\u4e00\u5207\u30a2\u30af\u30bb\u30b9\u3057\u307e\u305b\u3093\u3002\u30aa\u30f3\u30d7\u30ec\u30df\u30b9\u7248\u3067\u306f\u3001SQL\u30c6\u30ad\u30b9\u30c8\u3082\u30cd\u30c3\u30c8\u30ef\u30fc\u30af\u5185\u306b\u7559\u307e\u308a\u307e\u3059\u3002<\/p>\n\n\n\n<h3 class=\"wp-block-heading\">Spark SQL\u306e\u7cfb\u7d71\u8ffd\u8de1\u3068SQLFlow\u306e\u8cbb\u7528\u306f\u3044\u304f\u3089\u3067\u3059\u304b\uff1f<\/h3>\n\n\n<p>SQLFlow Cloud\u306f\u7121\u6599\u304b\u3089\u30b9\u30bf\u30fc\u30c8\u3002\u30d7\u30ec\u30df\u30a2\u30e0\u30d7\u30e9\u30f3\u306f\u6708\u984d$49.99\u3002\u30aa\u30f3\u30d7\u30ec\u30df\u30b9\u7248\u306f\u6708\u984d$500\u3001\u307e\u305f\u306f\u9078\u629e\u3057\u305f\u30c7\u30fc\u30bf\u30d9\u30fc\u30b9\u30bf\u30a4\u30d7\u3054\u3068\u306b1\u56de\u9650\u308a\u306e$4,800\u3067\u30012\u53f0\u306e\u30b5\u30fc\u30d0\u30fc\u306b\u30a4\u30f3\u30b9\u30c8\u30fc\u30eb\u53ef\u80fd\u3002\u8a73\u7d30\u306f\u4ee5\u4e0b\u3092\u53c2\u7167\u3002 <a href=\"https:\/\/www.gudusoft.com\/jp\/%e4%be%a1%e6%a0%bc%e8%a8%ad%e5%ae%9a\/\">\u4fa1\u683c\u8a2d\u5b9a<\/a> \u8a73\u7d30\u306f<\/p>\n\n\n\n<div class=\"wp-block-group alignfull has-background is-layout-constrained\" style=\"background-color:#60d5f6;padding-top:32px;padding-bottom:32px\"><div class=\"wp-block-group__inner-container\">\n\n<h2 class=\"wp-block-heading\">Spark SQL \u3092\u4eca\u3059\u3050\u30c8\u30ec\u30fc\u30b9\u3057\u307e\u3057\u3087\u3046<\/h2>\n\n\n<p>\u7121\u6599\u306e\u30d3\u30b8\u30e5\u30a2\u30e9\u30a4\u30b6\u30fc\u306bSpark SQL\u30b9\u30af\u30ea\u30d7\u30c8\u3092\u8cbc\u308a\u4ed8\u3051\u3066\u3001\u30bd\u30fc\u30b9\u304b\u3089\u30bf\u30fc\u30b2\u30c3\u30c8\u307e\u3067\u306e\u4efb\u610f\u306e\u5217\u3092\u8ffd\u8de1\u3059\u308b\u304b\u3001\u307e\u305f\u306f\u304a\u5ba2\u69d8\u306e\u30b7\u30b9\u30c6\u30e0\u5168\u4f53\u306e\u30b9\u30ad\u30e3\u30f3\u306b\u3064\u3044\u3066\u3054\u76f8\u8ac7\u304f\u3060\u3055\u3044\u3002<\/p>\n\n\n<div class=\"wp-block-buttons is-layout-flex\">\n<div class=\"wp-block-button\"><a class=\"wp-block-button__link wp-element-button\" href=\"https:\/\/sqlflow.gudusoft.com\/?utm_source=gudusoft&amp;utm_medium=website&amp;utm_campaign=spark-sql-data-lineage\" target=\"_blank\" rel=\"noreferrer noopener\">SQLFlow\u3092\u7121\u6599\u3067\u304a\u8a66\u3057\u304f\u3060\u3055\u3044<\/a><\/div>\n<\/div>\n\n\n<div class=\"wp-block-buttons is-layout-flex\">\n<div class=\"wp-block-button\"><a class=\"wp-block-button__link wp-element-button\" href=\"https:\/\/www.gudusoft.com\/jp\/%e3%82%b3%e3%83%b3%e3%82%bf%e3%82%af%e3%83%88\/\">\u30a8\u30f3\u30bf\u30fc\u30d7\u30e9\u30a4\u30ba\u5411\u3051\u30c7\u30e2\u3092\u30ea\u30af\u30a8\u30b9\u30c8\u3059\u308b<\/a><\/div>\n<\/div>\n\n<\/div><\/div>\n\n\n\n<script type=\"application\/ld+json\">{\n    \"@context\": \"https:\\\/\\\/schema.org\",\n    \"@graph\": [\n        {\n            \"@type\": \"SoftwareApplication\",\n            \"name\": \"Gudu SQLFlow\",\n            \"applicationCategory\": \"DeveloperApplication\",\n            \"applicationSubCategory\": \"SQL Data Lineage Tool\",\n            \"operatingSystem\": \"Web, Linux, Windows, macOS\",\n            \"url\": \"https:\\\/\\\/www.gudusoft.com\\\/spark-sql-data-lineage\\\/\",\n            \"description\": \"Automated Spark SQL data lineage from static parsing: column-level lineage through CTAS, INSERT INTO\\\/OVERWRITE, and temp view chains, with no cluster or job run required.\",\n            \"featureList\": \"Spark SQL dialect parser, column-level lineage, temp view resolution, CTAS and INSERT OVERWRITE analysis, indirect\\\/impact lineage, dbt support, REST API, DataHub\\\/Purview\\\/OpenMetadata export\",\n            \"softwareVersion\": \"8.2.3\",\n            \"offers\": [\n                {\n                    \"@type\": \"Offer\",\n                    \"name\": \"SQLFlow Cloud Free\",\n                    \"price\": \"0\",\n                    \"priceCurrency\": \"USD\"\n                },\n                {\n                    \"@type\": \"Offer\",\n                    \"name\": \"SQLFlow Cloud Premium\",\n                    \"price\": \"49.99\",\n                    \"priceCurrency\": \"USD\",\n                    \"priceSpecification\": {\n                        \"@type\": \"UnitPriceSpecification\",\n                        \"price\": \"49.99\",\n                        \"priceCurrency\": \"USD\",\n                        \"billingIncrement\": 1,\n                        \"unitText\": \"MONTH\"\n                    }\n                },\n                {\n                    \"@type\": \"Offer\",\n                    \"name\": \"SQLFlow On-Premise\",\n                    \"price\": \"4800\",\n                    \"priceCurrency\": \"USD\"\n                }\n            ],\n            \"publisher\": {\n                \"@type\": \"Organization\",\n                \"name\": \"Gudu Software\",\n                \"url\": \"https:\\\/\\\/www.gudusoft.com\\\/\"\n            }\n        },\n        {\n            \"@type\": \"FAQPage\",\n            \"mainEntity\": [\n                {\n                    \"@type\": \"Question\",\n                    \"name\": \"Do I need a running Spark cluster to get lineage?\",\n                    \"acceptedAnswer\": {\n                        \"@type\": \"Answer\",\n                        \"text\": \"No. SQLFlow parses the Spark SQL text statically. You can analyze a script that has never executed anywhere and get the same column-level lineage you would get from production code.\"\n                    }\n                },\n                {\n                    \"@type\": \"Question\",\n                    \"name\": \"How is this different from OpenLineage's Spark integration?\",\n                    \"acceptedAnswer\": {\n                        \"@type\": \"Answer\",\n                        \"text\": \"OpenLineage captures lineage at run time via a listener on the Spark session, which is excellent for observing what live jobs actually did. SQLFlow derives lineage from the SQL code itself, so it also covers code that hasn't run, supports pre-merge review and migration audits, and needs no cluster access. The two are complementary.\"\n                    }\n                },\n                {\n                    \"@type\": \"Question\",\n                    \"name\": \"Can SQLFlow trace lineage through temp views?\",\n                    \"acceptedAnswer\": {\n                        \"@type\": \"Answer\",\n                        \"text\": \"Yes. Chains of CREATE OR REPLACE TEMP VIEW statements are resolved end to end, so a column in a final CTAS traces back through every staging view to the physical source table, with the functions and aggregations applied at each step.\"\n                    }\n                },\n                {\n                    \"@type\": \"Question\",\n                    \"name\": \"Is Databricks SQL the same as Spark SQL here?\",\n                    \"acceptedAnswer\": {\n                        \"@type\": \"Answer\",\n                        \"text\": \"No. SQLFlow parses them with separate dialect parsers. Databricks SQL has its own syntax extensions, so it gets its own grammar. Use the Databricks dialect on Databricks and the Spark SQL dialect for open-source Spark, EMR, or self-managed clusters.\"\n                    }\n                },\n                {\n                    \"@type\": \"Question\",\n                    \"name\": \"Does SQLFlow read my data?\",\n                    \"acceptedAnswer\": {\n                        \"@type\": \"Answer\",\n                        \"text\": \"No. It analyzes SQL code and, optionally, schema metadata such as table and column definitions. Row data is never accessed. With the On-Premise edition, even the SQL text stays inside your network.\"\n                    }\n                },\n                {\n                    \"@type\": \"Question\",\n                    \"name\": \"What does Spark SQL lineage with SQLFlow cost?\",\n                    \"acceptedAnswer\": {\n                        \"@type\": \"Answer\",\n                        \"text\": \"SQLFlow Cloud starts free; premium is $49.99\\\/month. SQLFlow On-Premise is $500\\\/month or $4,800 one-time per selected database type, installable on two servers.\"\n                    }\n                }\n            ]\n        }\n    ]\n}<\/script>","protected":false},"excerpt":{"rendered":"<p>Spark SQL data lineage is the column-level map of how data moves through your Spark SQL code: which source columns feed every table written by a CREATE TABLE &#8230; AS SELECT or INSERT OVERWRITE, and through which temp views, joins, functions, and filters along the way. Gudu SQLFlow builds that map by statically parsing your Spark SQL with a dedicated Spark dialect parser \u2014 no cluster to instrument, no listener to attach, and no job run required. Paste the SQL, get the lineage diagram. Try it now: paste any Spark SQL script into the free online SQL lineage visualizer, select\u2026<\/p>","protected":false},"author":1,"featured_media":0,"parent":0,"menu_order":0,"comment_status":"closed","ping_status":"closed","template":"","meta":[],"blocksy_meta":{"styles_descriptor":{"styles":{"desktop":"","tablet":"","mobile":""},"google_fonts":[],"version":5}},"_links":{"self":[{"href":"https:\/\/www.gudusoft.com\/jp\/wp-json\/wp\/v2\/pages\/6761"}],"collection":[{"href":"https:\/\/www.gudusoft.com\/jp\/wp-json\/wp\/v2\/pages"}],"about":[{"href":"https:\/\/www.gudusoft.com\/jp\/wp-json\/wp\/v2\/types\/page"}],"author":[{"embeddable":true,"href":"https:\/\/www.gudusoft.com\/jp\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/www.gudusoft.com\/jp\/wp-json\/wp\/v2\/comments?post=6761"}],"version-history":[{"count":0,"href":"https:\/\/www.gudusoft.com\/jp\/wp-json\/wp\/v2\/pages\/6761\/revisions"}],"wp:attachment":[{"href":"https:\/\/www.gudusoft.com\/jp\/wp-json\/wp\/v2\/media?parent=6761"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}