{"id":24427,"date":"2019-12-09T08:00:20","date_gmt":"2019-12-08T23:00:20","guid":{"rendered":"https:\/\/www.techscore.com\/blog\/?p=24427"},"modified":"2019-12-09T10:01:32","modified_gmt":"2019-12-09T01:01:32","slug":"clojure-aws-textract","status":"publish","type":"post","link":"https:\/\/www.techscore.com\/blog\/2019\/12\/09\/clojure-aws-textract\/","title":{"rendered":"Clojure + AWS Textract \u3067 OCR \u3057\u3066\u307f\u308b"},"content":{"rendered":"

\u3053\u306e\u8a18\u4e8b\u306f\ud83d\ude3aTECHSCORE Advent Calendar 2019\ud83d\ude3a<\/a>\u306e9\u65e5\u76ee\u306e\u8a18\u4e8b\u3067\u3059\u3002<\/p>\n

\ud83d\udc08 Clojure + AWS Textract \u3067 OCR ! \ud83d\udc08<\/h1>\n

\u7d19\u66f8\u7c4d\u3067\u82f1\u8a9e\u306e\u6587\u66f8\u3092\u8aad\u3080\u6642\u306b\u9762\u5012\u306b\u601d\u3046\u306e\u306f\u3001\u8f9e\u66f8\u3092\u3072\u304f\u3053\u3068\u3067\u3059\u3002
\n\u7d19\u66f8\u7c4d\u3067\u306f\u3001\u305d\u306e\u5ea6\u306b\u30b9\u30de\u30db\u306a\u3069\u3092\u53d6\u308a\u81ea\u5206\u3067\u5165\u529b\u305b\u306d\u3070\u306a\u3089\u305a\u3001\u6975\u3081\u3066\u9762\u5012\u3067\u3059\u3002<\/p>\n

\u5e78\u3044\u30c9\u30ad\u30e5\u30e1\u30f3\u30c8\u30b9\u30ad\u30e3\u30ca\u306f\u6240\u6301\u3057\u3066\u3044\u308b\u306e\u3067\u3001\u96fb\u5b50\u5316\u3057\u3066\u307f\u308b\u3053\u3068\u306b\u3057\u307e\u3057\u305f\u3002
\n\u30b9\u30ad\u30e3\u30ca\u3067\u753b\u50cf\u3068\u3057\u3066\u53d6\u308a\u3053\u3093\u3060\u3082\u306e\u3092\u8aad\u307f\u53d6\u308a\u3001\u30d7\u30ec\u30a4\u30f3\u30c6\u30ad\u30b9\u30c8\u306b\u3057\u3066\u3001\u666e\u6bb5\u4f7f\u3063\u3066\u3044\u308bPC\u4e0a\u306e\u30a2\u30d7\u30ea\u3067\u30b5\u30af\u3063\u3068\u8abf\u3079\u3089\u308c\u308b\u3088\u3046\u306b\u3059\u308b\u306e\u3067\u3059\u3002<\/p>\n

\u30d7\u30ec\u30a4\u30f3\u30c6\u30ad\u30b9\u30c8\u5316\u3057\u305f\u3082\u306e\u3092\u4e00\u7b87\u6240\u306b\u307e\u3068\u3081\u3066\u304a\u3051\u3070\u3001\u691c\u7d22\u3082\u5bb9\u6613\u306b\u306a\u308b\u306e\u3067\u4fbf\u5229\u305d\u3046\u3067\u3059\u3002<\/p>\n

\ud83d\udc08 AWS Textract \ud83d\udc08<\/h1>\n

\u306a\u3093\u3067\u3082\u3042\u308b\u611f\u306e\u3042\u308b AWS \u3067\u3059\u304c\u3001Textract<\/a> \u3068\u3044\u3046 OCR \u7684\u306a\u30b5\u30fc\u30d3\u30b9\u3082\u3042\u308a\u307e\u3059\u3002
\nTextract<\/a> \u306b\u306f\u5358\u7d14\u306a\u6587\u5b57\u306e\u8aad\u307f\u53d6\u308a\u3060\u3051\u3067\u306a\u304f\u3001\u30d5\u30a9\u30fc\u30e0\u3084\u8868\u306e\u8aad\u307f\u53d6\u308a\u306e\u6a5f\u80fd\u307e\u3067\u3042\u308a\u307e\u3059\u304c\u3001\u4eca\u56de\u306f\u5358\u7d14\u306a\u6587\u5b57\u306e\u8aad\u307f\u53d6\u308a\u306e\u307f\u3092\u884c\u3044\u307e\u3059\u3002<\/p>\n

AWS CLI \u3092\u4f7f\u3048\u3070\u30b7\u30a7\u30eb\u30b9\u30af\u30ea\u30d7\u30c8\u3067\u3082\u5341\u5206\u53ef\u80fd\u3067\u3059\u304c\u3001\u4eca\u56de\u306f\u6bb5\u843d\u3054\u3068\u306b\u307e\u3068\u3081\u308b\u51e6\u7406\u3092\u5165\u308c\u305f\u304b\u3063\u305f\u306e\u3067\u3001\u7c21\u5358\u306b Clojure<\/strong> \u306e\u30b3\u30fc\u30c9\u3092\u66f8\u3044\u3066\u3044\u307e\u3059\u3002
\n\u3061\u306a\u307f\u306b Java \u3068\u3044\u3046\u8a00\u8a9e\u3067\u3082\u66f8\u3051\u308b\u3088\u3046\u3067\u3059\u304c\u3001\u4e00\u822c\u7684<\/strong>\u306a\u8a00\u8a9e\u306e\u307b\u3046\u304c\u826f\u3044\u3068\u601d\u3044 Clojure<\/strong> \u306b\u3057\u307e\u3057\u305f\u3002<\/p>\n

API \u306f\u7c21\u5358\u3067\u3001S3 \u3068\u3044\u3046\u30b9\u30c8\u30ec\u30fc\u30b8\u4e0a\u306e\u5834\u6240\u3092\u4e0e\u3048\u308b\u3060\u3051\u3067\u3059\u3002
\n\u305d\u3046\u3059\u308b\u3068 JSON \u304c\u8fd4\u3063\u3066\u304f\u308b\u306e\u3067\u3001\u9069\u5f53\u306b\u89e3\u91c8\u3059\u308b\u3060\u3051\u3067\u3059\u3002<\/p>\n

JSON \u306f\u4ee5\u4e0b\u306e\u3088\u3046\u306a\u5f62\u5f0f\u306b\u306a\u3063\u3066\u3044\u307e\u3059\u3002
\n\u30da\u30fc\u30b8\u3068\u884c\u3068\u5358\u8a9e\u3068\u3044\u3063\u305f\u30d6\u30ed\u30c3\u30af\u4e26\u3093\u3067\u3044\u3066\u3001\u305d\u308c\u305e\u308c\u306e Relationships<\/code> \u30d7\u30ed\u30d1\u30c6\u30a3\u3067\u305d\u306e\u95a2\u9023\u304c\u628a\u63e1\u3067\u304d\u308b\u3088\u3046\u306b\u306a\u3063\u3066\u3044\u307e\u3059\u3002
\n\u3053\u306e\u5185\u306e\u884c\u30d6\u30ed\u30c3\u30af\u3092\u62bd\u51fa\u3057\u3066\u3001\u30d7\u30ec\u30a4\u30f3\u30c6\u30ad\u30b9\u30c8\u3092\u751f\u6210\u3057\u307e\u3059\u3002
\n\u3064\u3044\u3067\u306b\u3001\u5168\u3066\u306e\u884c\u304c\u9023\u7d9a\u3057\u3066\u3044\u308b\u3068\u8aad\u307f\u306b\u304f\u3044\u306e\u3067\u3001\u6bb5\u843d\u6bce\u306b\u307e\u3068\u3081\u3066\u9593\u306b\u6539\u884c\u3092\u5165\u308c\u307e\u3059\u3002<\/p>\n

<\/p>\n

\n(ns textract.core\n  (:gen-class)\n  (:require [clojure.data.json :as json]\n            [clojure.string :as string]\n            [clojure.java.io :as io]\n            [cognitect.aws.client.api :as aws]\n            [clj-time.core :as tm]\n            [clj-time.format :as tf]))\n\n\n(def ^:dynamic *bucket* \"YOUR-BUCKET-NAME\")\n\n(def key-format (tf\/formatter \"yyyyMMdd-HHmmss\" (tm\/default-time-zone)))\n\n(def uniq-num (atom 1))\n\n\n(defn base-line\n  \"\u30d6\u30ed\u30c3\u30af\u306e\u57fa\u6e96\u4f4d\u7f6e\u3092\u8fd4\u3059\"\n  [block]\n  (let [{top :Top height :Height} (-> block :Geometry :BoundingBox)]\n    (+ top\n       (\/ height 2))))\n\n(defn extract-line-blocks\n  \"JSON \u304b\u3089\u884c\u3092\u8868\u3059\u30aa\u30d6\u30b8\u30a7\u30af\u30c8\u306e\u307f\u3092\u62bd\u51fa\u3059\u308b\"\n  [content]\n  (->>\n    (:Blocks content)\n    (filter (comp #(= \"LINE\" %) :BlockType))\n    (sort-by base-line)))\n\n(defn get-min-dist\n  \"\u30d6\u30ed\u30c3\u30af\u9593\u306e\u8ddd\u96e2(y\u8ef8)\u3067\u6700\u5c0f\u306e\u3082\u306e\u3092\u8fd4\u3059\"\n  [blocks]\n  (let [base-lines (map base-line blocks)]\n    (if (= 1 (count base-lines))\n      (first base-lines)\n      (apply min (map - (rest base-lines) base-lines)))))\n\n(defn make-near?\n  \"\u5145\u5206\u306b\u63a5\u8fd1\u3057\u3066\u3044\u308b\u304b\u3092\u5224\u65ad\u3059\u308b\u95a2\u6570\u3092\u8fd4\u3059\"\n  [blocks]\n  (let [dist (* 1.5 (get-min-dist blocks))]\n    (fn [a b]\n      (if (and a b)\n        (< (Math\/abs (- (base-line a) (base-line b)))\n           dist)\n        true))))\n\n(defn paragraphize\n  \"\u884c\u306e\u30d6\u30ed\u30c3\u30af\u3092\u30d1\u30e9\u30b0\u30e9\u30d5\u5358\u4f4d\u306b\u307e\u3068\u3081\u308b\"\n  [blocks]\n  (if (empty? blocks)\n    []\n    (let [near? (make-near? blocks)]\n      (loop [result []\n             last-part []\n             coll blocks]\n        (let [[cur follow & tail] coll\n              new-part #(conj result\n                              (conj last-part cur))]\n          (if follow\n            (if (near? cur follow)\n              (recur result\n                     (conj last-part cur)\n                     (cons follow tail))\n              (recur\n                (new-part)\n                []\n                (cons follow tail)))\n            (new-part)))))))\n\n(defn textize\n  \"\u30d1\u30e9\u30b0\u30e9\u30d5\u3092\u30c6\u30ad\u30b9\u30c8\u306b\u3059\u308b\"\n  [paragraphs]\n  (->>\n    paragraphs\n    (map #(map :Text %))\n    (map #(string\/join \"\\n\" %))\n    (string\/join \"\\n\\n\")))\n\n(defn textract\n  \"S3 \u4e0a\u306b\u3042\u308b\u6587\u66f8\u753b\u50cf\u3092 AWS API \u3067 OCR \u3059\u308b\"\n  [client s3-key]\n  (aws\/invoke\n    client\n    {:op :DetectDocumentText\n     :request {:Document\n               {:S3Object {:Bucket *bucket*\n                           :Name s3-key}}}}))\n\n(defn s3-put\n  \"S3 \u306b\u30d5\u30a1\u30a4\u30eb\u3092\u30a2\u30c3\u30d7\u30ed\u30fc\u30c9\u3059\u308b\"\n  [client file-path s3-key]\n  (aws\/invoke\n    client\n    {:op :PutObject\n     :request {:Bucket *bucket*\n               :Key s3-key\n               :Body (java.io.FileInputStream. file-path)}}))\n\n(defn s3-rm\n  \"S3 \u304b\u3089\u30d5\u30a1\u30a4\u30eb\u3092\u524a\u9664\u3059\u308b\"\n  [client s3-key]\n  (aws\/invoke\n    client\n    {:op :DeleteObject\n     :request {:Bucket *bucket*\n               :Key s3-key}}))\n\n(defn -main\n  ([]\n   (println \"Usage: textract   \")\n   (println \"                 \"))\n  ([input-png output-json output-text]\n   (let [s3-key (str (tf\/unparse key-format (tm\/now))\n                     \"-\"\n                     (swap! uniq-num inc)\n                     \".png\")\n         s3-client (aws\/client {:api :s3 :region \"ap-southeast-1\"})\n         tx-client (aws\/client {:api :textract :region \"ap-southeast-1\"})]\n     (s3-put s3-client input-png s3-key)\n     (let [result (textract tx-client s3-key)\n           text (-> result extract-line-blocks paragraphize textize)]\n       (doall(map io\/make-parents [output-text output-json]))\n       (spit output-json (json\/write-str result))\n       (spit output-text text)\n       (s3-rm s3-client s3-key)\n       (println text))))\n  ([input-json output-text]\n   (let [text (-> input-json slurp (json\/read-str :key-fn keyword) extract-line-blocks paragraphize textize)]\n     (io\/make-parents output-text)\n     (spit output-text text)\n     (println text))))\n\n; dependencies [[org.clojure\/clojure \"1.10.1\"]\n;               [org.clojure\/data.json \"0.2.7\"]\n;               [com.cognitect.aws\/api       \"0.8.391\"]\n;               [com.cognitect.aws\/endpoints \"1.1.11.670\"]\n;               [com.cognitect.aws\/s3        \"770.2.568.0\"]\n;               [com.cognitect.aws\/textract  \"747.2.533.0\"]\n;               [clj-time \"0.15.2\"]]\n<\/pre>\n
<\/code><\/p>\n
\ud83d\udc08 \u307e\u3068\u3081 \ud83d\udc08<\/h1>\n
\u5b9f\u969b\u306b\u6570\u30da\u30fc\u30b8\u8aad\u3093\u3067\u307f\u307e\u3057\u305f\u304c\u3001\u8aa4\u8aad\u53d6\u3082\u7279\u306b\u898b\u3064\u304b\u308a\u307e\u305b\u3093\u3057\u3001\u5358\u8a9e\u306b\u30ab\u30fc\u30bd\u30eb\u3092\u5408\u308f\u305b\u308b\u3060\u3051\u3067\u8f9e\u66f8\u3092\u3072\u3051\u308b\u306e\u306f\u4fbf\u5229\u3067\u3059\u306d\u3002<\/p>\n
\u307e\u305f\u3001\u6539\u3081\u3066\u611f\u3058\u305f\u3053\u3068\u3067\u3059\u304c Clojure (\u3068\u3044\u3046\u304b Lisp)\u306f\u3001REPL (\u5bfe\u8a71\u74b0\u5883) \u304c\u4fbf\u5229\u3067\u3059\u3002
\n\u95a2\u6570\u5358\u4f4d\u3067\u30b3\u30fc\u30c9\u3092\u66f8\u3044\u3066\u3044\u304f\u306e\u3067\u3001\u30b3\u30fc\u30c9\u3092\u7de8\u96c6\u3057\u306a\u304c\u3089\u8a66\u884c\u932f\u8aa4\u3059\u308b\u3053\u3068\u304c\u3001\u6c17\u8efd\u306b\u3067\u304d\u307e\u3059\u3002
\nREPL \u306e\u5bfe\u8a71\u74b0\u306f\u95a2\u6570\u578b\u8a00\u8a9e\u3068\u7279\u306b\u76f8\u6027\u304c\u826f\u3044\u3067\u3059\u306d\u3002<\/p>\n
\ud83d\udc08 \u62ec\u5f27\u3044\u3044\u306d\u3053 \ud83d\udc08<\/h1>\n
\u753b\u50cf\u3067\u3059\u3002<\/p>\n
\"\"<\/a><\/p>\n
\u2191\u306e\u62ec\u5f27\u3044\u3044\u732b\u306e\u753b\u50cf\u3092 OCR \u3057\u305f\u3089\u8089\u7403\u306b\u30e1\u30c3\u30bb\u30fc\u30b8\u304c\u767a\u898b\u3055\u308c\u307e\u3057\u305f\u3002\u300cad 4\u300d\u306f\u305f\u3057\u3066\u3069\u3046\u3044\u3046\u610f\u5473\u304c\u2026\uff1f<\/p>\n
\"\"<\/a><\/p>\n","protected":false},"excerpt":{"rendered":"
\u3053\u306e\u8a18\u4e8b\u306f\ud83d\ude3aTECHSCORE Advent Calendar 2019\ud83d\ude3a\u306e9\u65e5\u76ee\u306e\u8a18\u4e8b\u3067\u3059\u3002<\/p>\n
\ud83d\udc08 Clojure + AWS Textract \u3067 OCR ! \ud83d\udc08<\/p>\n
\u7d19\u66f8\u7c4d\u3067\u82f1\u8a9e\u306e\u6587\u66f8\u3092\u8aad\u3080\u6642\u306b\u9762\u5012\u306b\u601d\u3046\u306e\u306f\u3001\u8f9e\u66f8\u3092\u3072\u304f\u3053\u3068\u3067\u3059\u3002
\u7d9a\u304d\u3092\u8aad\u3080...<\/a><\/p>\n","protected":false},"author":43,"featured_media":0,"comment_status":"closed","ping_status":"open","sticky":false,"template":"","format":"standard","meta":[],"categories":[332,18],"tags":[141,119,340],"_links":{"self":[{"href":"https:\/\/www.techscore.com\/blog\/wp-json\/wp\/v2\/posts\/24427"}],"collection":[{"href":"https:\/\/www.techscore.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.techscore.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.techscore.com\/blog\/wp-json\/wp\/v2\/users\/43"}],"replies":[{"embeddable":true,"href":"https:\/\/www.techscore.com\/blog\/wp-json\/wp\/v2\/comments?post=24427"}],"version-history":[{"count":16,"href":"https:\/\/www.techscore.com\/blog\/wp-json\/wp\/v2\/posts\/24427\/revisions"}],"predecessor-version":[{"id":24465,"href":"https:\/\/www.techscore.com\/blog\/wp-json\/wp\/v2\/posts\/24427\/revisions\/24465"}],"wp:attachment":[{"href":"https:\/\/www.techscore.com\/blog\/wp-json\/wp\/v2\/media?parent=24427"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.techscore.com\/blog\/wp-json\/wp\/v2\/categories?post=24427"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.techscore.com\/blog\/wp-json\/wp\/v2\/tags?post=24427"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}