Pulling performance data from XtremIO and pushing it to InfluxDB

If you followed my previous blog posts you have seen how cool Grafana is and what you can start to do when you have your data in a time series database like InfluxDB. I’ve written a script in Ruby that will pull data from XtremIO via REST and will feed it into InfluxDB. The script will just pull data from the cluster object but it would be fairly easily modified to pull from other objects. XtremIO returns the data in a JSON format that is easily modified and sent on to InfluxDB. You will need to configure your specific settings in the settings.local.yml file.

settlings.local.yml

xtremio_user: "user"
xtremio_pass: "password"

influx_url: "http://<hostname>:8086/db/<dbname>/series"
influx_user: "user"
influx_pass: "password"

hosts: [xio01, xio02]
metrics: [ compression-factor-text, dedup-ratio-text, rd-bw, wr-bw, rd-latency, wr-latency, avg-latency, rd-iops, wr-iops]

xtrempull.rb

require 'rest-client'
require 'json'
require 'pry-byebug'
require 'yaml'
require 'ostruct'

#pull in the settings file
Settings = OpenStruct.new YAML::load_file(File.join(__dir__, 'settings.local.yml'))

#method for "get" operations against a REST api
def get_url(url, username, password)
  result = RestClient::Request.execute(user: username,
                                       password: password,
                                       url: url,
                                       method: :get,
                                       headers: {:accept => :json},
                                       verify_ssl: false,
                                       timeout: 30,
                                       open_timeout: 30)
  return result
end

#method for "put" operations against a REST api
def puts_influx(in_url, in_user, in_pass, myjson)
  result = RestClient::Request.execute(user: in_user,
                                       password: in_pass,
                                       url: in_url,
                                       method: :post,
                                       headers: {:accept => :json},
                                       verify_ssl: false,
                                       timeout: 30,
                                       payload: myjson,
                                       open_timeout: 30)
  return result
end

#user/pass for XtremIO
username = Settings.xtremio_user
password = Settings.xtremio_pass

#URL,user, and pass for InfluxDB
in_url = Settings.influx_url
in_user = Settings.influx_user
in_pass = Settings.influx_pass

#for each host - determine the cluster name
Settings.hosts.each do |xbrick|

  #### build the URL for the API
  baseurl = "https://#{xbrick}"
  clusterapi = '/api/json/types/clusters'
  xiourl = baseurl + clusterapi

  #### pull list of clusters from each xio brick
  getclusters = get_url(xiourl, username, password)
  clusternames = JSON.parse(getclusters)

  myxio = clusternames["clusters"].select { |name| !name["name"].nil? }.map{ |x| x['name']}
#### pull XIO data from each clusters
  myxio.each do |name|
    clusterurl = xiourl + "?name=#{name}"
    getperf = get_url(clusterurl, username, password)
    perfdata = JSON.parse(getperf)

    #### pull the metrics from the JSON
    data = []
    Settings.metrics.each do |metric|
      var  = perfdata['content'][metric]
      hash = {}
      hash[:name] = xbrick + "_" + metric
      hash[:columns] = ['value']
      hash[:points] = [[var]]
      data << hash
    end

    #### convert back to JSON and send to influx
    payload = data.to_json
    puts_influx(in_url, in_user, in_pass, payload)

  end
end

Advertisements

One thought on “Pulling performance data from XtremIO and pushing it to InfluxDB

  1. Great script,
    I’ve extended the script to push metrics directly into Graphite and pull CPU along with per-volume stats.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s