{"id":5217,"date":"2014-12-15T14:12:49","date_gmt":"2014-12-15T05:12:49","guid":{"rendered":"http:\/\/www.techscore.com\/blog\/?p=5217"},"modified":"2018-11-14T16:33:50","modified_gmt":"2018-11-14T07:33:50","slug":"fluentd_parser_and_formatter","status":"publish","type":"post","link":"https:\/\/www.techscore.com\/blog\/2014\/12\/15\/fluentd_parser_and_formatter\/","title":{"rendered":"fluentd \u306e parser \u3068 formatter \u3092\u30ab\u30b9\u30bf\u30de\u30a4\u30ba\u3057\u3066\u3044\u3044\u611f\u3058\u306b\u30ed\u30b0\u3092\u96c6\u7d04\u3059\u308b"},"content":{"rendered":"<p>\u306f\u3058\u3081\u307e\u3057\u3066\u3001\u30a4\u30f3\u30d5\u30e9\u30a8\u30f3\u30b8\u30cb\u30a2\u306e\u672b\u5ee3\u3067\u3059\u3002<\/p>\n<p>\u3053\u306e\u8a18\u4e8b\u306f <a title=\"TECHSCORE Advent Calendar 2014\" href=\"http:\/\/www.techscore.com\/blog\/2014\/11\/27\/techscore-advent-calendar-2014\/\">TECHSCORE Advent Calendar 2014<\/a>\u300115\u65e5\u76ee\u306e\u8a18\u4e8b\u3067\u3059\u3002<\/p>\n<p><a title=\"fluentd\" href=\"http:\/\/www.fluentd.org\/\" target=\"_blank\">fluentd<\/a> \u306f\u4eca\u3084\u5e83\u304f\u4f7f\u308f\u308c\u3066\u3044\u308b\u30ed\u30b0\u30b3\u30ec\u30af\u30bf\u30fc\u3067\u3042\u308a\u3001\u3082\u306f\u3084\u8aac\u660e\u306e\u5fc5\u8981\u3082\u306a\u3044\u3067\u3057\u3087\u3046\u3002<br \/>\n\u5f0a\u793e\u3067\u3082\u30ed\u30b0\u96c6\u7d04\u30fb\u30ed\u30b0\u89e3\u6790\u306e\u524d\u51e6\u7406\u30fb\u30a4\u30d9\u30f3\u30c8\u691c\u77e5\u306a\u3069\u306b\u5229\u7528\u3057\u3066\u3044\u307e\u3059\u3002<\/p>\n<p>\u3053\u306e\u8a18\u4e8b\u3067\u306f fluentd (td-agent 2.1.2, fluentd-0.10.57) \u306b\u3088\u308b\u751f\u30ed\u30b0\u96c6\u7d04\u3092\u984c\u6750\u306b\u3001Parser plugin \u3084 Formatter plugin \u306b\u3088\u308b\u30ab\u30b9\u30bf\u30de\u30a4\u30ba\u306b\u3064\u3044\u3066\u8aac\u660e\u3057\u307e\u3059\u3002<\/p>\n<p>\u203b 12\u670814\u65e5\u306b v0.12 \u304c\u30ea\u30ea\u30fc\u30b9\u3055\u308c\u3066 Parser plugin \u3068 Formatter plugin \u306e\u65b0 API \u304c\u5229\u7528\u3067\u304d\u308b\u3088\u3046\u306b\u306a\u3063\u305f\u306e\u3067\u3001\u6700\u5f8c\u306b\u65b0 API \u306b\u3064\u3044\u3066\u3082\u89e6\u308c\u307e\u3059\u3002<\/p>\n<h2>\u751f\u30ed\u30b0\u306e\u8ee2\u9001<\/h2>\n<p>\u304b\u3064\u3066\u3001\u7d20\u306e fluentd \u3067\u306f\u30ed\u30b0\u3092\u751f\u306e\u307e\u307e\u8ee2\u9001\u3059\u308b\u3053\u3068\u306f\u3067\u304d\u307e\u305b\u3093\u3067\u3057\u305f\u3002<br \/>\n\u5f53\u6642\u306f\u3001<a title=\"fluent-agent-lite\" href=\"https:\/\/github.com\/tagomoris\/fluent-agent-lite\" target=\"_blank\">fluent-agent-lite<\/a>\u00a0\u3068\u00a0<a title=\"file-alternative \u30d7\u30e9\u30b0\u30a4\u30f3\" href=\"https:\/\/github.com\/tagomoris\/fluent-plugin-file-alternative\" target=\"_blank\">file-alternative plugin<\/a>\u00a0\u3092\u7d44\u307f\u5408\u308f\u305b\u305f\u308a\u3001\u751f\u30ed\u30b0\u3092\u305d\u306e\u307e\u307e\u6271\u3046\u30d7\u30e9\u30b0\u30a4\u30f3\u3092\u81ea\u4f5c\u3057\u305f\u308a\u3057\u3066\u3044\u307e\u3057\u305f\u3002<\/p>\n<p>v0.10.39 \u3067 in_tail \u306a\u3069\u306e format \u30d1\u30e9\u30e1\u30fc\u30bf\u306b none \u304c\u8ffd\u52a0\u3055\u308c\u307e\u3057\u305f\u3002\u3053\u308c\u306b\u3088\u308a\u3001\u30ed\u30b0\u306e\u884c\u3092\u305d\u306e\u307e\u307e \"message\" key \u306e\u5024\u3068\u3057\u3066\u53d6\u308a\u8fbc\u3080\u3053\u3068\u304c\u3067\u304d\u308b\u3088\u3046\u306b\u306a\u308a\u307e\u3057\u305f\u3002\u307e\u305f\u3001v0.10.50 \u3067 out_file \u306a\u3069\u306e\u51fa\u529b formatter \u306b SingleValue \u304c\u4f7f\u3048\u308b\u3088\u3046\u306b\u306a\u308a\u3001\u7279\u5b9a\u306e key \u306b\u5bfe\u5fdc\u3059\u308b\u5024\u306e\u307f\u3092\u51fa\u529b\u3067\u304d\u308b\u3088\u3046\u306b\u306a\u308a\u307e\u3057\u305f\u3002\u3053\u308c\u3089\u3092\u7d44\u307f\u5408\u308f\u305b\u308b\u3068\u3001\u7d20\u306e fluentd \u3067\u751f\u30ed\u30b0\u3092\u8ee2\u9001\u3059\u308b\u3053\u3068\u304c\u3067\u304d\u307e\u3059\u3002<\/p>\n<p>\u305f\u3068\u3048\u3070\u3001 \u6b21\u306e\u3088\u3046\u306b in_tail \u3068 out_file \u3092\u8a2d\u5b9a\u3059\u308b\u3068 Apache \u306e access_log \u3092\u751f\u30ed\u30b0\u306e\u307e\u307e\u8ee2\u9001\u3059\u308b\u3053\u3068\u304c\u3067\u304d\u307e\u3059\u3002<\/p>\n<pre class=\"\"><source>\r\n  type tail\r\n  format none\r\n  path \/var\/log\/httpd\/access_log\r\n  pos_file \/var\/log\/td-agent\/access_log\r\n  tag apache.access\r\n<\/source>\r\n<\/pre>\n<pre class=\"\"><match apache.access>\r\n  type file\r\n  format SingleValue\r\n  path \/var\/log\/access_log\r\n<\/match><\/pre>\n<h2>\u8907\u6570\u30db\u30b9\u30c8\u306e\u751f\u30ed\u30b0\u306e\u96c6\u7d04<\/h2>\n<p>\u8907\u6570\u306e\u30db\u30b9\u30c8\u304b\u3089\u306a\u308b\u30b5\u30fc\u30d3\u30b9\u3067\u306f\u3001\u8907\u6570\u30db\u30b9\u30c8\u306e\u30ed\u30b0\u30921\u3064\u306e\u30d5\u30a1\u30a4\u30eb\u306b\u96c6\u7d04\u3057\u305f\u304f\u306a\u308b\u3053\u3068\u304c\u3042\u308a\u307e\u3059\u3002\u3053\u306e\u3068\u304d\u3001\u30ed\u30b0\u306e\u5404\u884c\u306e\u5148\u982d\u306b\u30db\u30b9\u30c8\u540d\u3092\u5165\u308c\u305f\u304f\u306a\u308b\u306e\u3067\u3059\u304c\u3001\u7d20\u306e fluentd \u3067\u306f\u3067\u304d\u307e\u305b\u3093\u3002\u305d\u3053\u3067\u3001v0.10.46 \u3067\u5c0e\u5165\u3055\u308c\u305f Parser plugin \u3068 v0.10.49 \u5c0e\u5165\u3055\u308c\u305f Formatter \u00a0plugin \u3092\u5229\u7528\u3057\u307e\u3059\u3002\u30ab\u30b9\u30bf\u30e0\u306e Parser plugin \u3067\u751f\u30ed\u30b0\u3068\u30db\u30b9\u30c8\u540d\u3092\u30ec\u30b3\u30fc\u30c9\u306b\u633f\u5165\u3057\u3001\u30ab\u30b9\u30bf\u30e0\u306e Formatter plugin \u3067\u751f\u30ed\u30b0\u306e\u5148\u982d\u306b\u30db\u30b9\u30c8\u540d\u3092\u4ed8\u4e0e\u3057\u3066\u51fa\u529b\u3057\u307e\u3059\u3002<\/p>\n<h3>Parser Plugin<\/h3>\n<p>Parser plugin \u306f TextParser \u30af\u30e9\u30b9\u306e\u5185\u90e8\u30af\u30e9\u30b9\u3068\u3057\u3066\u5b9f\u88c5\u3057\u307e\u3059\u3002<\/p>\n<p>\u4e0b\u8a18\u306e NoneWithHostnameParser \u306f\u3001\u751f\u30ed\u30b0\u3068\u30db\u30b9\u30c8\u540d\uff08\u30c7\u30d5\u30a9\u30eb\u30c8\u306f hostname -s \u3067\u5f97\u3089\u308c\u308b\u5024\uff09\u3092\u30ec\u30b3\u30fc\u30c9\u306b\u683c\u7d0d\u3059\u308b Parser \u3067\u3059\u3002NoneParser \u3092\u304a\u624b\u672c\u306b\u3057\u3066\u3001\u30db\u30b9\u30c8\u540d\u3092\u30ec\u30b3\u30fc\u30c9\u306b\u633f\u5165\u3059\u308b\u30b3\u30fc\u30c9\u3092\u8ffd\u52a0\u3057\u3066\u3044\u307e\u3059\u3002register_template \u3067 \u3053\u306e Parser \u3092 none_with_hostname \u3068\u3057\u3066\u767b\u9332\u3057\u3066\u3044\u307e\u3059\u3002<\/p>\n<p>\/etc\/td-agent\/plugin\/parser_none_with_hostname.rb<\/p>\n<pre class=\"lang:ruby decode:true\">module Fluent\r\n  class TextParser\r\n    class NoneWithHostnameParser\r\n      include Configurable\r\n\r\n      config_param :message_key, :string, :default => 'message'\r\n      config_param :hostname, :string, :default => `hostname -s`.chomp\r\n\r\n      attr_accessor :estimate_current_event\r\n\r\n      def initialize\r\n        super\r\n        @estimate_current_event = true\r\n      end\r\n\r\n      def configure(conf)\r\n        super\r\n      end\r\n\r\n      def call(text)\r\n        record = {}\r\n        record[@message_key] = text\r\n        record['hostname'] = @hostname\r\n        time = @estimate_current_event ? Engine.now : nil\r\n        if block_given?\r\n          yield time, record\r\n        else\r\n          return time, record\r\n        end\r\n      end\r\n    end\r\n\r\n    register_template(\"none_with_hostname\", Proc.new { NoneWithHostnameParser.new })\r\n  end\r\nend<\/pre>\n<p>in_tail \u306e\u8a2d\u5b9a\u3067 format \u306b none_with_hostname \u3092\u6307\u5b9a\u3059\u308b\u3068\u3001\u751f\u30ed\u30b0\u3068\u30db\u30b9\u30c8\u540d\u304c\u30ec\u30b3\u30fc\u30c9\u306b\u683c\u7d0d\u3055\u308c\u307e\u3059\u3002<\/p>\n<pre class=\"\"><source>\r\n  type tail\r\n  format none_with_hostname\r\n  path \/var\/log\/httpd\/access_log\r\n  pos_file \/var\/log\/td-agent\/access_log\r\n  tag apache.access\r\n<\/source><\/pre>\n<h3>Formatter Plugin<\/h3>\n<p>Formatter plugin \u306f TextFormatter \u30e2\u30b8\u30e5\u30fc\u30eb\u306e\u5185\u90e8\u30af\u30e9\u30b9\u3068\u3057\u3066\u5b9f\u88c5\u3057\u307e\u3059\u3002<\/p>\n<p>\u4e0b\u8a18\u306e HostnameAndMessageFormatter \u306f\u3001SingleValueFormatter \u3068\u307b\u307c\u540c\u3058\u3067\u3059\u304c\u3001\u51fa\u529b\u884c\u306e\u5148\u982d\u306b\u30db\u30b9\u30c8\u540d\u3092\u633f\u5165\u3059\u308b Formatter \u3067\u3059\u3002register_template \u3067 \u3053\u306e Formatter \u3092 hostname_and_message \u3068\u3057\u3066\u767b\u9332\u3057\u3066\u3044\u307e\u3059\u3002<\/p>\n<p>\/etc\/td-agent\/plugin\/formatter_hostname_and_message.rb<\/p>\n<pre class=\"lang:ruby decode:true \">module Fluent\r\n  module TextFormatter\r\n    class HostnameAndMessageFormatter\r\n      include Configurable\r\n\r\n      config_param :message_key, :string, :default =>; 'message'\r\n      config_param :add_newline, :bool, :default =>; true\r\n\r\n      def configure(conf)\r\n        super\r\n      end\r\n\r\n      def format(tag, time, record)\r\n        text = \"#{record['hostname']} #{record[@message_key].to_s}\"\r\n        text << \"\\n\" if @add_newline\r\n      end\r\n    end\r\n\r\n    register_template(\"hostname_and_message\", Proc.new { HostnameAndMessageFormatter.new })\r\n  end\r\nend<\/pre>\n<p>out_file \u306e\u8a2d\u5b9a\u3067 format \u306b hostname_and_message \u3092\u6307\u5b9a\u3059\u308b\u3068\u30ed\u30b0\u306e\u5404\u884c\u306e\u5148\u982d\u306b\u30db\u30b9\u30c8\u540d\u304c\u3064\u304f\u3088\u3046\u306b\u306a\u308a\u307e\u3059\u3002<\/p>\n<pre class=\"\"><match apache.access>\r\n  type file\r\n  format hostname_and_message\r\n  path \/var\/log\/access_log\r\n<\/match><\/pre>\n<p>\u3053\u308c\u3089\u306e plugin \u3092\u7d44\u307f\u5408\u308f\u305b\u308b\u3053\u3068\u306b\u3088\u308a\u3001\u8907\u6570\u30db\u30b9\u30c8\u306e\u751f\u30ed\u30b0\u3092\u3044\u3044\u611f\u3058\u306b\u96c6\u7d04\u3057\u3066\u3044\u307e\u3059\u3002<\/p>\n<h2>\u304a\u307e\u3051\uff1afluentd v0.12<\/h2>\n<p>\u3053\u3053\u307e\u3067\u66f8\u3044\u305f\u3068\u3053\u308d\u3067\u300112\u670814\u65e5\u306b v0.12 \u304c\u30ea\u30ea\u30fc\u30b9\u3055\u308c\u305f\u3053\u3068\u306b\u6c17\u3065\u304d\u307e\u3057\u305f\u3002\u3053\u306e\u30ea\u30ea\u30fc\u30b9\u3067 Parser \u304a\u3088\u3073 Formatter plugin \u306b\u65b0\u3057\u3044 API\u00a0\u304c\u8ffd\u52a0\u3055\u308c\u3001plugin \u3092\u3088\u308a\u7c21\u6f54\u306b\u8a18\u8ff0\u3067\u304d\u308b\u3088\u3046\u306b\u306a\u308a\u307e\u3057\u305f\u3002v0.10.58 \u3067\u3082\u540c\u69d8\u306e\u8a18\u8ff0\u304c\u3067\u304d\u308b\u3088\u3046\u306b\u306a\u308b\u3088\u3046\u3067\u3059\u3002<\/p>\n<p>\u65b0API \u306e\u5909\u66f4\u70b9\u306f\u6b21\u306e\u901a\u308a\u3067\u3059\u3002<\/p>\n<ul>\n<li>\u30d9\u30fc\u30b9\u3068\u306a\u308b Parser \u304a\u3088\u3073 Formatter \u30af\u30e9\u30b9\u304c\u5b9a\u7fa9\u3055\u308c\u305f\u306e\u3067\u3001\u3053\u308c\u3089\u3092\u7d99\u627f\u3059\u308b\u5f62\u3067 plugin \u3092\u5b9f\u88c5\u3059\u308b\u3002<\/li>\n<li>plugin \u306e\u767b\u9332\u306f\u3001\u4ed6\u306e plugin \u3068\u540c\u69d8\u306b \u00a0Plugin.register_parser \u304a\u3088\u3073 Plugin.register_formatter \u3067\u884c\u3048\u308b\u3088\u3046\u306b\u306a\u3063\u305f\u3002<\/li>\n<li>Parser \u306e\u5b9f\u969b\u306e\u51e6\u7406\u3092\u5b9f\u88c5\u3059\u308b\u30e1\u30bd\u30c3\u30c9\u304c call \u304b\u3089 parse \u306b\u5909\u66f4\u3055\u308c\u305f\u3002<\/li>\n<\/ul>\n<p>\u65b0 API \u3092\u5229\u7528\u3059\u308b\u3068\u4e0a\u8a18\u306e plugin \u306f\u305d\u308c\u305e\u308c\u6b21\u306e\u3088\u3046\u306b\u8a18\u8ff0\u3067\u304d\u307e\u3059\u3002<\/p>\n<p>\/etc\/td-agent\/plugin\/parser_none_with_hostname.rb<\/p>\n<pre class=\"lang:ruby decode:true\">module Fluent\r\n  class TextParser\r\n    class NoneWithHostnameParser < Parser\r\n      Plugin.register_parser('none_with_hostname', self)\r\n\r\n      config_param :message_key, :string, :default => 'message'\r\n      config_param :hostname, :string, :default => `hostname -s`.chomp\r\n \r\n      def configure(conf)\r\n        super\r\n      end\r\n\r\n      def parse(text)\r\n        record = {}\r\n        record[@message_key] = text\r\n        record['hostname'] = @hostname\r\n        time = @estimate_current_event ? Engine.now : nil\r\n        if block_given?\r\n          yield time, record\r\n        else\r\n          return time, record\r\n        end\r\n      end\r\n    end\r\n  end\r\nend<\/pre>\n<p>\/etc\/td-agent\/plugin\/formatter_hostname_and_message.rb<\/p>\n<pre class=\"lang:ruby decode:true\">module Fluent\r\n  module TextFormatter\r\n    class HostnameAndMessageFormatter > Formatter\r\n      Plugin.register_formatter('hostname_and_message', self)\r\n\r\n      config_param :message_key, :string, :default => 'message'\r\n      config_param :add_newline, :bool, :default => true\r\n \r\n      def configure(conf)\r\n        super\r\n      end\r\n\r\n      def format(tag, time, record)\r\n        text = \"#{record['hostname']} #{record[@message_key].to_s}\"\r\n        text << \"\\n\" if @add_newline\r\n      end\r\n    end\r\n  end\r\nend<\/pre>\n<p>\u4ee5\u524d\u3068\u6bd4\u3079\u308b\u3068\u5206\u304b\u308a\u3084\u3059\u304f\u306a\u308a\u307e\u3057\u305f\u306d\u3002<\/p>\n<h2>\u53c2\u8003<\/h2>\n<ul>\n<li>\u516c\u5f0f\u30c9\u30ad\u30e5\u30e1\u30f3\u30c8\u00a0<a title=\"Writing plugins\" href=\"http:\/\/docs.fluentd.org\/articles\/plugin-development\" target=\"_blank\">Writing plugins<\/a><\/li>\n<li><a title=\"DevNullParser\u3067\u59cb\u3081\u308bFluentd \u30d1\u30fc\u30b5\u30d7\u30e9\u30b0\u30a4\u30f3\u5165\u9580\" href=\"http:\/\/qiita.com\/kiyoto\/items\/9903718147adde4d6c32\" target=\"_blank\">DevNullParser\u3067\u59cb\u3081\u308bFluentd \u30d1\u30fc\u30b5\u30d7\u30e9\u30b0\u30a4\u30f3\u5165\u9580<\/a><\/li>\n<li><a title=\"Fluentd Formatter \u30d7\u30e9\u30b0\u30a4\u30f3\u5165\u9580\" href=\"http:\/\/qiita.com\/sonots\/items\/d3c69e754814b8923e64\" target=\"_blank\">Fluentd Formatter \u30d7\u30e9\u30b0\u30a4\u30f3\u5165\u9580<\/a><\/li>\n<\/ul>\n","protected":false},"excerpt":{"rendered":"<p>\u306f\u3058\u3081\u307e\u3057\u3066\u3001\u30a4\u30f3\u30d5\u30e9\u30a8\u30f3\u30b8\u30cb\u30a2\u306e\u672b\u5ee3\u3067\u3059\u3002<\/p>\n<p>\u3053\u306e\u8a18\u4e8b\u306f TECHSCORE Advent Calendar 2014\u300115\u65e5\u76ee\u306e\u8a18\u4e8b\u3067\u3059\u3002<\/p>\n<p>fluentd \u306f\u4eca\u3084\u5e83\u304f\u4f7f\u308f\u308c\u3066\u3044\u308b\u30ed\u30b0\u30b3\u30ec\u30af\u30bf\u30fc\u3067\u3042\u308a\u3001\u3082\u306f\u3084\u8aac\u660e\u306e\u5fc5\u8981\u3082\u306a\u3044\u3067\u3057\u3087\u3046\u3002<br \/><a href=\"https:\/\/www.techscore.com\/blog\/2014\/12\/15\/fluentd_parser_and_formatter\/\">\u7d9a\u304d\u3092\u8aad\u3080...<\/a><\/p>\n","protected":false},"author":22,"featured_media":0,"comment_status":"closed","ping_status":"open","sticky":false,"template":"","format":"standard","meta":[],"categories":[18],"tags":[141,145],"_links":{"self":[{"href":"https:\/\/www.techscore.com\/blog\/wp-json\/wp\/v2\/posts\/5217"}],"collection":[{"href":"https:\/\/www.techscore.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.techscore.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.techscore.com\/blog\/wp-json\/wp\/v2\/users\/22"}],"replies":[{"embeddable":true,"href":"https:\/\/www.techscore.com\/blog\/wp-json\/wp\/v2\/comments?post=5217"}],"version-history":[{"count":6,"href":"https:\/\/www.techscore.com\/blog\/wp-json\/wp\/v2\/posts\/5217\/revisions"}],"predecessor-version":[{"id":5239,"href":"https:\/\/www.techscore.com\/blog\/wp-json\/wp\/v2\/posts\/5217\/revisions\/5239"}],"wp:attachment":[{"href":"https:\/\/www.techscore.com\/blog\/wp-json\/wp\/v2\/media?parent=5217"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.techscore.com\/blog\/wp-json\/wp\/v2\/categories?post=5217"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.techscore.com\/blog\/wp-json\/wp\/v2\/tags?post=5217"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}