updated proc.rb to get stats on all processes

Author: eperry

Gemfile

@@ -1,2 +1,3 @@
 source 'https://rubygems.org'
+gem "logstash-input-proc", :path => "/home/eperry/ws-logstash/workspace/logstash-input-proc"
 gemspec

README.md

@@ -2,62 +2,43 @@
 
 This is a plugin for [Logstash](https://github.com/elasticsearch/logstash).
 
-It is fully free and fully open source. The license is Apache 2.0, meaning you are pretty much free to use it however you want in whatever way.
+This plugin is to read the /proc virtual file system , decode the files in it.
 
 ## Documentation
 
-Logstash provides infrastructure to automatically generate documentation for this plugin. We use the asciidoc format to write documentation so any comments in the source code will be first converted into asciidoc and then into html. All plugin documentation are placed under one [central location](http://www.elasticsearch.org/guide/en/logstash/current/).
-
-- For formatting code or config example, you can use the asciidoc `[source,ruby]` directive
-- For more asciidoc formatting tips, see the excellent reference here https://github.com/elasticsearch/docs#asciidoc-guide
-
-## Need Help?
-
-Need help? Try #logstash on freenode IRC or the https://discuss.elastic.co/c/logstash discussion forum.
-
-## Developing
 
 ### 1. Plugin Developement and Testing
 
 #### Code
 - To get started, you'll need JRuby with the Bundler gem installed.
-
-- Create a new plugin or clone and existing from the GitHub [logstash-plugins](https://github.com/logstash-plugins) organization. We also provide [example plugins](https://github.com/logstash-plugins?query=example).
-
-- Install dependencies
-```sh
+ ```sh
 bundle install
 ```
 
-#### Test
+- Then clone this repo
+- You will need to either clone the logstash repo or download the binary
 
-- Update your dependencies
 
-```sh
-bundle install
-```
 
-- Run tests
-
-```sh
-bundle exec rspec
-```
-
-### 2. Running your unpublished Plugin in Logstash
+### 2. Running the unpublished Plugin in Logstash
 
 #### 2.1 Run in a local Logstash clone
 
 - Edit Logstash `Gemfile` and add the local plugin path, for example:
 ```ruby
-gem "logstash-filter-awesome", :path => "/your/local/logstash-filter-awesome"
+gem "logstash-output-proc", :path => "/your/local/logstash-output-proc"
 ```
 - Install plugin
 ```sh
 bin/plugin install --no-verify
 ```
+- install Ruby Debug
+```sh
+bin/plugin install logstash-codec-rubydebug
+```
 - Run Logstash with your plugin
 ```sh
-bin/logstash -e 'filter {awesome {}}'
+bin/logstash -e 'input {proc {interval=>60}}  output { stdout{ codec=>"rubydebug"}}'
 ```
 At this point any modifications to the plugin code will be applied to this local Logstash setup. After modifying the plugin, simply rerun Logstash.
 
@@ -67,20 +48,24 @@ You can use the same **2.1** method to run your plugin in an installed Logstash
 
 - Build your plugin gem
 ```sh
-gem build logstash-filter-awesome.gemspec
+gem build logstash-output-proc.gemspec
 ```
 - Install the plugin from the Logstash home
 ```sh
-bin/plugin install /your/local/plugin/logstash-filter-awesome.gem
+bin/plugin install /your/local/plugin/logstash-output-proc.gem
 ```
 - Start Logstash and proceed to test the plugin
-
-## Contributing
-
-All contributions are welcome: ideas, patches, documentation, bug reports, complaints, and even something you drew up on a napkin.
-
-Programming is not a required skill. Whatever you've seen about open source and maintainers or community members  saying "send patches or die" - you will not see that here.
-
-It is more important to the community that you are able to contribute.
-
-For more information about contributing, see the [CONTRIBUTING](https://github.com/elasticsearch/logstash/blob/master/CONTRIBUTING.md) file.
+- 
+# Example Config
+```ruby
+input {
+    proc {
+        interval=>60
+    }
+}  
+output { 
+    stdout{ 
+        codec=>"rubydebug"
+    }
+}
+```

lib/logstash/inputs/proc.rb

@@ -32,7 +32,7 @@ def readVmStats(queue)
   file = Pathname.new("/proc/vmstat")
   lines = file.readlines
   lines.each { |line|
-      @logger.info? && @logger.info("LINE: "+line)
+      #@logger.info? && @logger.info("LINE: "+line)
       m = /(\w+)\s+([\.\d]+)/.match(line)
       if (m && m.length >= 3 )
       event = LogStash::Event.new("raw"=>line, m[1] => m[2], "file" => file.to_s,"host" => @host, "type" => "vmstats" )
@@ -72,18 +72,158 @@ def readMemInfo(queue)
   }
 end
 
-#discoverSearch(Pathname.new("/proc"))
+def readPidStats(queue)
+  @logger.info? && @logger.info("in ReadPidStats ")
+  process = Hash.new
+  #Loosely based on the GEM ProcTable  which was  based on the Perl Module ProcTable
+     Dir.foreach("/proc"){ |file|
+        next if file =~ /\D/ # Skip non-numeric directories
+        
+        # Get /proc/<pid>/cmdline information. Strip out embedded nulls.
+        begin
+          data = IO.read("/proc/#{file}/cmdline").tr("\000", ' ').strip
+          process["cmdline"] = data 
+        rescue Exception => ex
+          next # Process terminated, on to the next process
+        end
+
+        # Get /proc/<pid>/cwd information
+        process["cwd"] = File.readlink("/proc/#{file}/cwd") rescue nil
+
+        # Get /proc/<pid>/environ information. Environment information
+        # is represented as a Hash, with the environment variable as the
+        # key and its value as the hash value.
+        process["environ"] = Hash.new
+
+        begin
+          IO.read("/proc/#{file}/environ").split("\0").each{ |str|
+            key, value = str.split('=')
+            process["environ"][key] = value
+          }
+        rescue Errno::EACCES, Errno::ESRCH, Errno::ENOENT
+          # Ignore and move on.
+        end
+
+        # Get /proc/<pid>/exe information
+        process["exe"] = File.readlink("/proc/#{file}/exe") rescue nil
+
+        # Get /proc/<pid>/fd information. File descriptor information
+        # is represented as a Hash, with the fd as the key, and its
+        # symlink as the value.
+        process["fd"] = Hash.new
+
+        begin
+          Dir["/proc/#{file}/fd/*"].foreach { |fd|
+            process["fd"][File.basename(fd)] = File.readlink(fd) rescue nil
+          }
+        rescue
+        #  # Ignore and move on
+        end
+
+        # Get /proc/<pid>/root information
+        process["root"] = File.readlink("/proc/#{file}/root") rescue nil
+
+        # Get /proc/<pid>/stat information
+        stat = IO.read("/proc/#{file}/stat") rescue next
+
+        # Get number of LWP, one directory for each in /proc/<pid>/task/
+        # Every process has at least one thread, so if we fail to read the task directory, set nlwp to 1.
+        process["nlwp"] = Dir.glob("/proc/#{file}/task/*").length rescue process["nlwp"] = 1
+
+        # Deal with spaces in comm name. Courtesy of Ara Howard.
+        re = %r/\([^\)]+\)/
+        comm = stat[re]
+        comm.tr!(' ', '-')
+        stat[re] = comm
+
+        stat = stat.split
+
+        process["pid"]         = stat[0].to_i
+        process["comm"]        = stat[1].tr('()','') # Remove parens
+        process["state"]       = stat[2]
+        process["ppid"]        = stat[3].to_i
+        process["pgrp"]        = stat[4].to_i
+        process["session"]     = stat[5].to_i
+        process["tty_nr"]      = stat[6].to_i
+        process["tpgid"]       = stat[7].to_i
+        process["flags"]       = stat[8].to_i
+        process["minflt"]      = stat[9].to_i
+        process["cminflt"]     = stat[10].to_i
+        process["majflt"]      = stat[11].to_i
+        process["cmajflt"]     = stat[12].to_i
+        process["utime"]       = stat[13].to_i
+        process["stime"]       = stat[14].to_i
+        process["cutime"]      = stat[15].to_i
+        process["cstime"]      = stat[16].to_i
+        process["priority"]    = stat[17].to_i
+        process["nice"]        = stat[18].to_i
+        # Skip 19
+        process["itrealvalue"] = stat[20].to_i
+        process["starttime"]   = stat[21].to_i
+        process["vsize"]       = stat[22].to_i
+        process["rss"]         = stat[23].to_i
+        process["rlim"]        = stat[24].to_i
+        process["startcode"]   = stat[25].to_i
+        process["endcode"]     = stat[26].to_i
+        process["startstack"]  = stat[27].to_i
+        process["kstkesp"]     = stat[28].to_i
+        process["kstkeip"]     = stat[29].to_i
+        process["signal"]      = stat[30].to_i
+        process["blocked"]     = stat[31].to_i
+        process["sigignore"]   = stat[32].to_i
+        process["sigcatch"]    = stat[33].to_i
+        process["wchan"]       = stat[34].to_i
+        process["nswap"]       = stat[35].to_i
+        process["cnswap"]      = stat[36].to_i
+        process["exit_signal"] = stat[37].to_i
+        process["processor"]   = stat[38].to_i
+        process["rt_priority"] = stat[39].to_i
+        process["policy"]      = stat[40].to_i
+
+        # Get /proc/<pid>/status information (name, uid, euid, gid, egid)
+        begin
+          IO.foreach("/proc/#{file}/status") do |line|
+            case line
+              when /Name:\s*?(\w+)/
+                process["name"] = $1
+              when /Uid:\s*?(\d+)\s*?(\d+)/
+                process["uid"]  = $1.to_i
+                process["euid"] = $2.to_i
+              when /Gid:\s*?(\d+)\s*?(\d+)/
+                process["gid"]  = $1.to_i
+                process["egid"] = $2.to_i
+            end
+          end
+        rescue Errno::ESRCH, Errno::ENOENT
+          next
+        end
+
+        # If cmdline is empty use comm instead
+        process["cmdline"] = process["comm"] if process["cmdline.empty?"]
+
+        @logger.info? && @logger.info("output  ")
+        event = LogStash::Event.new( "file" => "/proc" ,"host" => @host, "type" => "pidstats" , "process" => process);
+        decorate(event)
+        queue << event
+
+      }
+
+
+end
 
   def run(queue)
     loop do
       begin
       start = Time.now
       @logger.info? && @logger.info("Reading VmStats ")
-      readVmStats(queue)
+      #readVmStats(queue)
       @logger.info? && @logger.info("Reading LoadAverage ")
-      readLoadAverage(queue)
+      #readLoadAverage(queue)
       @logger.info? && @logger.info("Reading MemInfo ")
-      readMemInfo(queue)
+      #readMemInfo(queue)
+      @logger.info? && @logger.info("Getting list of PID ")
+      readPidStats(queue);
+  
       duration = Time.now - start
       @logger.info? && @logger.info("Parsing completed", 
                                      :duration => duration,