Recently I had a request internally for how to access the Export endpoint from Splunk from a node.js application. The Export endpoint is useful for exporting large amounts of data efficiently out of Splunk as it will stream the results directly rather than requiring you to continually poll for more results. It turns out we don’t support the Export endpoint currently in our JS SDK, but it is very easy do access it yourself using Mikael’s super simple request module.

A picture (or a snippet in this case) tells a thousand words. Below you can see how to export Splunk’s internal index. Once you start it up it will instantly start streaming. Make sure you have enough disk space, or stop it before you run out

var request = require('request');
request.get(
    {
        strictSSL: false,
        uri: 'https://localhost:8089/servicesNS/admin/search/search/jobs/
              export?output_mode=json',
        qs: {
            search: 'search index=_internal'
        }
    }
)
.auth('admin', 'changeme', false)
.pipe(process.stdout);

Here is what is going on above:

• Loads the request module.
• Calls get to issue a GET request passing in the following params:
  • strictSSL – set to false tells request to not validate the server cert returned by Splunk, which by default is not a valid cert.
  • uri – set to the Splunk host along with the path for the export endpoint. I’ve also specified in the query string to force a JSON response.
  • qs – set to supply the search param. Passing it in this way allows me to not have to URI encode the search string as request will do it.\
• Calls auth to use HTTP Basic Auth passing the Splunk username and password.
• Pipe’s the results to stdout

Any questions?

In case you haven’t heard, an emerging and fast-growing use case for Splunk is using Splunk for anti-fraud, theft, and abuse (which I will just call “fraud”). Many Splunk customers across a wide range of industries Splunk their machine data and log files for a wide range of anti-fraud use cases, including fraud investigations, detection, and analytics/reporting. They also put the event data from other point anti-fraud tools into Splunk and use Splunk to: (1) break down the siloed nature of these point tools to present a more unified view on fraud, and (2) correlate fraud events with other data sources. Splunk’s flexibility enables it to be an anti-fraud solution and/or enhance existing fraud tools.

A few weeks ago, Splunk conducted a compelling webinar on how to use Splunk for anti-fraud use cases. The webinar recording can be viewed here .  In it we had a former 2-time CISO and Splunk customer explain how a large online retailer used Splunk to better detect and defeat online fraud. Also, a Splunk partner detailed how they have deployed Splunk to detect employee theft at thousands of stores and branches by Splunking point-of-sale machine data. Lastly, the webinar also covered a sampling of the wide range of possible fraud patterns that Splunk can detect.

We have also produced new material on Splunk for anti-fraud use cases I encourage you to view. This includes a new Splunk anti-fraud web page here with the 101 on this use case and sample patterns of fraud. On this web page also are a case study and solutions guide. Please feel free to read these and, if you have additional questions, contact Splunk sales here. Happy Splunking!

Joe Goldberg
Product Marketing
Fraud, Compliance, Security

Today we’re releasing a new suite of tools for .NET developers so you can supercharge your .NET development with Splunkl!!

CC image Supercharger by Eaday on Flickr

This release is a continuation of our commitment to provide developers a rich platform for developing Splunk solutions.

• C# SDK 2.0 –  A new, modern, C# SDK for building cross-platform solutions that consume Splunk’s API and/or which extend Splunk.
• Logging libraries – These libraries allow you to easily wire logging in your existing .NET applications to send log data to Splunk via TCP or UDP. It provides .NET Trace Listeners as well as sinks for the Semantic Logging Application Block (SLAB).
• Visual Studio Extension – This extension makes it really easy to get started with the C# SDK and to build Modular Inputs which extend Splunk.

C# SDK

The new SDK is designed from scratch to take advantage of the latest advances in the .NET Platform by including rich support for async/await, and the dynamic keyword. Introducing support for async/await means your custom apps can be more scalable and responsive. The SDK includes several nuget packages, Splunk.Client and Splunk.ModularInputs.

Splunk.Client

The Client package is used for connecting to Splunk instances and accessing the Splunk API. Below you can see I am using the client’s Service class to connect to a Splunk instance.

var service = new Service(Scheme.Https, “localhost”, 8089);

await service.LogOnAsync(“admin”, “changeme”);

Once I have logged on, I can then issue queries, for example below is a real time query searching for errors from within the last hour and going forward.

var results = await service.ExportSearchResultsAsync(

  “search error”,

  new SearchExportArgs {EarliestTime = “rt-1h”, LatestTime = “rt”});

foreach (dynamic result in results)

{

  Console.WriteLine(result._raw);

}

await index.RemoveAsync(); 

Cross Platform and Mobile / Device development

Something really exciting about the new Splunk.Client package is it is a Portable Class Library (PCL) allowing it to be used for cross-platform development. You can use it for .NET 4.5,  Windows Store, and  Windows Phone 8 development. Additionally thanks to the awesome power of Xamarin, you can use it for Linux, IOS and even Android development!

Below is a screenshot showing an IOS search app being developed in Xamarin Studio. This app lets users issue ad-hoc searches against a Splunk instance which are sent using the new Splunk Client. You can see on the left that the Splunk.Client PCL package has been referenced. The app is also using Rx to provide a responsive user interface. As events stream in the UI instantly updates without blocking.

Below you can see the Application running and doing a search for shopping cart related errors.

We also have created a Windows Store app using the SDK which we’re releasing on Github. Below you can see a screenshot:

Similarly in the code you can see the calls to the Splunk Client.

Yes, you can now build Splunk Mobile / Device apps with the Splunk C# SDK 2.0!

Splunk.ModularInputs

Using the Splunk.ModularInputs package you can create Modular Inputs that extend Splunk to pull events from custom sources. You could use this to talk to devices, interact with the operating system, or connect with external APIs like Twitter or Facebook.

As an example, in our samples for the SDK, you’ll find a sample input which monitors environment variables and triggers an event whenever they change. You can find the full sample code for this sample input here. To create a Modular Input, you create a .NET 4.5 console app and then include the Splunk.ModularInputs package. Next you change your Program class to derive from the ModularInput class and implement methods for returning the scheme, performing validation of the input parameters, and actually streaming the events.

The streaming logic is done in the new StreamEventsAsync method. Below is the implementation of that method for the environment variable monitor mentioned earlier. As you can see, similar to the Splunk Client, Modular Inputs in the new SDK are completely async.

public override async Task StreamEventsAsync(InputDefinition inputDefinition, EventWriter eventWriter)

{

  int interval = 1000;

  try

  {

    // if user didn’t give value for polling_interval, type conversion to

    SingleValueParameter will throw

    interval = int.Parse(((SingleValueParameter)inputDefinition

      .Parameters["polling_interval"]).ToString());

  }

  catch (Exception)

  {

  }

  const string Seperator = @”://”;

  int index = inputDefinition.Name.IndexOf(Seperator) + Seperator.Length;

  string varName = inputDefinition.Name.Substring(index);

  string lastVarValue = null;

  while (true)

  {

    await Task.Delay(interval);

    string varValue = Environment.GetEnvironmentVariable(varName,

      EnvironmentVariableTarget.Machine);

    // Event data can’t be null for real events.

    varValue = varValue ?? “(not exist)”;

    // Splunk does not record lines with only white spaces.

    varValue = string.IsNullOrWhiteSpace(varValue) ? “(white space)” : varValue;

    if (varValue != lastVarValue)

    {

      await eventWriter.QueueEventForWriting(new Event

      {

        Stanza = varName,

        Data = string.Format(

          “{0}, interval={1}, inputDefinition.Name={2} , varName={3}”,

          varValue, interval, inputDefinition.Name, varName)

      });

      lastVarValue = varValue;

    }

  }

}

This method first retrieves a polling interval and environment variable from the input configuration. These were configured when the input was added in Splunk. Next the code loops continually monitoring the specified variable at the interval. If the variable changes then the QueueEventForWriting async method is invoke to write back an event to Splunk.

This is just one example of what you can do with Modular Inputs, the possibilities are only limited by your imagination!

Logging Libraries

A common desire we hear from developers is to make it easy to just send log data from their applications to Splunk directly without having to first write to a file. This is very useful both for development and production environments. Enter our new logging libraries! They provide everything you need to wire up your existing apps to log directly over UDP or TCP to a Splunk instance or forwarder.

The libraries include standard .NET Trace Listeners which popular OSS logging frameworks like log4net, NLog and Enterprise Library support.

Below you can see I am configuring .NET Tracing to use the new Splunk UdpTraceListener.

var listener = new TcpTraceListener(“localhost”, 9000);

Trace.Listeners.Add(listener);

Next I am modifying the log4net configuration to configure it to write to .NET Tracing.

<appender name=“TraceAppender“ type=“log4net.Appender.TraceAppender“>

  …

</appender> 

Now when I start my application, I see my log entries instantly pouring in!

Additionally the libraries also include support for the Semantic Logging Application Block (SLAB), which leverages the highly performant ETW infrastructure that ships in Windows.  Here is a snippet of wiring up SLAB to send log data to Splunk over UDP.

var _listener = newObservableEventListener();

var sink = new SplunkSink(“127.0.0.1″, 9000);

_listener.EnableEvents(TestEventSource.Log, EventLevel.LogAlways, Keywords.All);

_subscription = _listener.Subscribe(sink);

The new logging libraries are not invasive making them easy to wire up in any of your existing .NET 4.5 apps.

Visual Studio Extension

In the first part of this post you’ve learned about new .NET libraries we’ve introduced for enhancing your Splunk development. We’re also introducing a new Visual Studio extension which will further streamline your development in Visual Studio 2013 for using them.

The VS extension includes the following:

• A template for creating a new .NET Project using the Splunk Client, and optionally using the new logging libraries to send log data over TCP or UDP.
• Snippets for performing common tasks using the Splunk Client, which you can use in any .NET application that references the SDK.
• A template for creating a custom C# Modular Input.

To install the extension, go download it from the Visual Studio Gallery here.

As I have previously installed it, I can see the available templates from the “New Project” dialog and select the SDK project.

Once I hit “OK” a wizard is displayed for logging configuration. I’’ve selected to use the SLAB event sink.

Next it displays a list of transports to choose from. I’ve selected UDP.

Finally, it generates for me a starter project with a sample implementation. The project pulls in the required NuGet packages as well. You can see below that it has configured SLAB, generated a sample log entry, and also configured the Splunk Client.

I am all ready to start developing with Splunk!

Where to go next?

Head over to dev.splunk.com and checkout our docs on the SDK, Logging Libraries, and the Visual Studio Extension. You’ll find overview docs, snippets and walkthroughs as well as links for downloading the new components.

Also check out our source on github here, here and here. All our new deliverables release under the standard Apache 2 OSS license. We’d welcome your pull requests!

It’s a great time to develop with Splunk!

If you have ever used Splunk, you can probably come up with a number of reasons why you should use a Splunk forwarder whenever possible to send data to Splunk. To quickly illustrate some of the benefits, a Splunk forwarder maintains an internal index of where it left off when sending data. If for some reason the Splunk Indexer has to be taken offline, the forwarder can resume its task after the indexer is brought back up. Additionally, a forwarder can automatically load balance traffic between multiple Splunk indexers. There’s already a Splunk blog here devoted to getting data into Splunk that highlights a forwarder’s benefits that I encourage you to review.

But what if using a Splunk Forwarder is not an option due to some political or other reason. Well, let’s consider such a use case and its challenges.

Use Case

Company XYZ has already invested time and resources deploying Apache Flume throughout its infrastructure. Apache Flume is currently in use by many teams and they all claim it’s great at collecting and moving log data. Thus, your boss now insists on leveraging Flume to send the SMTP log to Splunk for analysis and visualization. But, that’s not without challenge.

Challenges

A Splunk sink does not ship by default with Apache Flume. The development teams tells you that any custom component of an Apache Flume agent can be developed. However, coding skills are limited within your team and the development team cannot take on more requests for now. Sounds familiar?  Sigh.

Flume Data Flow

So, you’re on your own again having to design a solution that involves Flume. Before we embark on providing a solution, a short description of a Flume agent and its data flow is warranted.

An Apache Flume agent manages data flow from a point of origin to a destination. It’s a JVM process with a source component for consuming events with a specific format, a channel that holds the event, and a sink that forwards it to the next hop or destination. To learn more on Flume, you can visit this page, but this should be enough for this post.

Now, it’s time to explore our solution.

Solution

Do not panic as a solution that does not require development exists. Splunk is a universal machine data platform and does not impose hard requirements like other point solutions. We will therefore leverage the default Flume Thrift sink to send our SMTP log to Splunk after consuming it. The diagram below illustrates the process.

Configuration

Our configuration will involve three steps:

1) Configuring the Splunk input

A TCP input will be configured for accepting data from the Flume agent. Instructions for doing so can be found in the “Getting Data In” Splunk guide.

2) Creating a Flume agent configuration file

The sample configuration file provided below continuously reads the SMTP log file and sends new events via TCP to the Splunk indexer. In our example, the Flume agent is called Splunk.

##################################################

# Flume Configuration Example

# Sources, channels and sinks are defined per agent.

# In this example the Flume agent is called “splunk”,

# the sink “indexer”, the source is a tail of the SMTP log

# Defining sources, channels and sinks

# We will be reading a log file as source

# Our channel si-1 will hold data for the Splunk Indexer

splunk.sources = reader

splunk.channels = si-1

splunk.sinks = indexer

# For each one of the sources defined above, the type is defined

# The mail log will be read using the tail command

splunk.sources.reader.type = exec

splunk.sources.reader.command = tail -f /var/log/maillog

# Error is simply discarded, unless logStdErr=true

splunk.sources.reader.logStdErr = true

splunk.sources.reader.restart = true

# Memory channel si-1 will be used.

splunk.sources.reader.channels = si-1

# Each sink’s type must be defined with the following:

# Type: the default thrift sink is used

# Hostname or IP address of the sink, our Splunk indexer

# The IP address is 10.0.0.153

# The TCP port the Splunk indexer is configured to listen on

# in this case, port 1997

splunk.sinks.indexer.type = thrift

splunk.sinks.indexer.hostname = 10.0.0.153

splunk.sinks.indexer.port = 1997

# Specify the channel(s) the sink will use

splunk.sinks.indexer.channel = si-1

# Each channel’s type is defined

splunk.channels.si-1.type = memory

# Specify the capacity of the memory channel

# The maximum number of events stored in the channel

splunk.channels.si-1.capacity = 100

##################################################

3) Configuring line breaking in Splunk

Because some of the SMTP log events will come across merged, we need to configure line breaking in Splunk to properly handle the events. This is easily handled using the following regex “LINE_BREAKER = (\\x\w{2})+”, in our use case. We also used Splunk to set the appropriate host at input creation time as Splunk offers many options for setting parameters.  Below is a screenshot of  those SMTP events after being ingested and indexed in Splunk.

Conclusion

If you have a choice between using a Splunk forwarder and another method to get data into Splunk, always pick the forwarder for its ease of use, flexibility and various benefits. However, Splunk lets you implement other ingestion methods as well, thereby allowing you to conform to corporate policies and requirements without the need for an army of developers and consultants, and associated costs. So next time, you’re faced with a mandate to use Flume or another ingestion method instead of a forwarder, do not despair.

Happy Splunking!

There was a nice article published on GovDataDownload today about the potential for big data to impact the business of higher education.  The material does a nice job of explaining big data in simple concepts, then cites an excellent example of how it can help the bottom line of a university directly.  Perhaps more importantly, the article closes with a mention of big data being used to help with learning analytics  by “helping identify predictors and patterns for student success”, which is near and dear to my heart as a former educator.…

Splunk Answers has just been migrated to a new platform!  Read more about the process and goals.

What to expect

You won’t see much in the way of UI changes, but the site underneath will be more stable and more flexible.  You should experience faster loading times, more responsive controls, and very importantly, an improved search experience. We will now also have access to new and improved spam blocking features, a much-needed improvement.

The goal of the initial migration is to maintain feature parity with the existing Splunk Answers site. This will help us make sure we don’t break anything you’ve come to rely on. Over time, we will be able to launch new features and improved functionality.

Update: We’re experiencing some performance issues, but will be continuing to troubleshoot through the weekend.

Report issues!

If you have feedback, suggestions, or want to report a problem with the new site, please email rachel@splunk.com.

While disk spindles get cheaper, disk space on servers hosting mainstream services like Email or Messaging Service, still remains a big budget item. As organizations continue to grow and more people join hands (employees, contractors, service providers, developers, et al), it is important for organizations to monitor and make optimal usage of the critical disk space.

In the Infrastructure and IT Operations space, Microsoft Exchange continues to retain top-spot in the Gartner’s Magic Quadrant for Unified Communications report. Splunk App for Microsoft Exchange provides valuable insight regarding various aspect of Microsoft Exchange deployment landscape.

Splunk App for MS Exchange provides granular insight regarding the complete lifecycle of an email right from the time an email arrives within an org (will cover this in my next blog) till it is delivered to mailbox.

The Splunk App for Microsoft Exchange also provides current status of the organizations’s Outbound Mail Reputation. This status is generally reported by external agencies as part of their due-diligence and/or audit reports.

This blog aims at highlighting a particular dashboard panel which has been helping Splunk customers gain important details from operational data and assists in taking timely business decisions regarding their next investments in IT Operations.

As it can be seen in the above screen shots, this panel provides granular details for the default folders created for each mailbox on the Exchange server. These folders can be further drilled down to individual mailboxes to find out users occupying valuable disk space for unimportant data which can freed up for good reasons.

In the above example, we can see a lot disk space being consumed by Deleted Items, Junk Items and RSS Feeds folders. The Deleted Items’ and Junk Items folders can emptied by the users themselves, while on the other hand, an Exchange Administrator can shorten the life span of RSS Feeds by reviewing the policy configuration related to it.

We have seen customers running internal campaigns announcing the change in policy and educating users to periodically clean their mailboxes on the Exchange server. This helps org and enterprises reclaim disk space to an extent where they can now defer their capital expenditure, investments to buy new drives  or expand disk capacities on mailbox database servers.

Hope this helps you take a closer look at your Messaging Service utilization. Until next time, Happy Splunking !!

I’m somewhat of a Heroku fan boy. I’ve been using it for some time because it is just so simple to deploy applications. However, I’ve never really looked too deeply into the logs produced by my apps via the command line.

Que Spunk.

In this post we’ll look at how you can start Splunking data from apps deployed in Heroku, and some recipes to visualise it using the SPL.

Prepare Our Indexer

We will send logs from Heroku to our Splunk Indexer via syslog. To configure the Indexer to accept these logs we’ll need to enable receiving:

Settings > Forwarding and Receiving > Recieve Data > Add New

Now all we need to do is select the port via which we want to receive this data. We’ll use port:9997 for this example but you can select an open port of your choice.

Get Heroku Logging

Heroku provides comprehensive app, system, and API logging by default. But lets turn things up a bit and include some debug and runtime logs to help us with app development, after all this is Splunk. Assuming you have the Heroku Toolbelt installed, you can run the command below to enable logging:

heroku config:add LOG_LEVEL=DEBUG --app <YOUR APP NAME>

And…

heroku labs:enable log-runtime-metrics --app <YOUR APP NAME>

Restart your app:

heroku restart --app <YOUR APP NAME>

Then add a syslog drain to send these logs to your indexer:

drains:add syslog://<YOUR INDEXER'S IP>:9997 --app <YOUR APP NAME>

Using AWS? When you use EC2 security groups, the hostname used when you add the drain must resolve to the 10/8 private IP address of your instance (which must be in the us-east Amazon region). If you use the EC2 public IP address, or a name that resolves to the public IP address, then logplex will not be able to connect to your drain.

Heroku meet Splunk, Splunk Heroku

Assuming everything is working correctly we should start to see data being indexed using a simple search:

sourcetype="syslog"

SPL Recipes

@chuckg has written a great set of recipes to start searching and visualising your data. You can find them all on Github here. From application performance metrics to Heroku errors, it’s a really comprehensive list.

Not using Heroku? Not a problem. Most PAAS providers offer logging functionality and are therefore Splunkable. A quick read of their docs should answer this question. I’d be interested to hear from people who have successfully Splunked data from other providers too.

Virtualization is difficult to manage given the complex moving parts from storage to networking to hardware. When you have a dynamic VMware environment with Distributed Resource Scheduler (DRS) and High Availability (HA) enabled, Virtual Machine’s (VM) in the environment can transition through multiple hosts and clusters and can potentially become unregistered VM’s. This can lead a VMWare Administrator to loose visibility for these VMs. In addition each VM in a datacenter could cost from a couple hundred dollars into the thousands (http://roitco.vmware.com) based on your environment and infrastructure costs.

In this blog post I will cover three types of VM’s that can exist in your VMware Infrastructure and requires additional attention. The definition of these VM’s vary, but I’m sure you will be able to recognize them regardless of the name I give them.

Zombie VM : Virtual Machine that uses less than certain amount of CPU for a period of time. (Example: VM using less than 5% CPU for over a thirty-day period.) Since Zombie VM’s are the VMs running very low CPU usage, it could be repurposed to run other applications when needed.

Chatty VM (Opposite of Zombie) : Virtual Machine that uses more than certain amount of CPU for a period of time. (Example : VM using more than 80% CPU over a week). Chatty VM’s are the ones probably moving from ESXi to ESXi host using vMotion based on utilization.

Orphan VM : There are multiple definitions for this type of VM. Here are a just some examples of what an Orphaned VM can look like:

• Virtual Machine that was unregistered from vCenter Server but still running within the environment unmanaged.
• Virtual Machine that exists in the vCenter database but is no longer present on the ESXi host.
• Virtual Machine that exists on a different ESXi host than expected by the vCenter Server.

In many occasions, actively running Orphan VMs is a security concern since they are not visible to vCenter Server and thus the VMware administrator is unaware of them as well. The VM’s will not be patched and can go undetected from compliance and operational audits.

Orphan VM’s happen because of some of the following reasons:

• After a vMotion or VMware DRS migration event.
• After a VMware HA host failure occurs, or after the ESX host comes out of maintenance mode.
• A virtual machine is deleted outside of vCenter Server.
• vCenter Server is restarted while a migration is in progress
• Too many virtual machines are scheduled to be relocated at the same time.
• Attempting to delete virtual machines when an ESX/ESXi host local disk (particularly the root partition) has become full.
• Rebooting the host within 1 hour of moving or powering on virtual machines.
• A .vmx file contains special characters or incomplete line item entries.

In order to gather information from a complex environment like VMware, we will need to collect performance, log and configuration data from vCenter Server and ESXi hosts.

Splunk App for VMware provides deep operational visibility into granular performance metrics, logs, tasks and events and topology from hosts, virtual machines and virtual centers.

Splunk App for VMware provides:

• Proactive monitoring of your virtual infrastructure.
• A visual interactive topology map of your virtual environment, highlighting problems and statistical comparisons based on predefined customizable thresholds.
• Views that provide insight into how you environment performs with details on performance, availability, security, and capacity and change tracking.
• Capacity Planning and Capacity Forecasting dashboards.
• Correlation of VMware virtualization data with NetApp NFS datastores.
• Views that show the operational health of your environment, identifying underperforming or distressed hosts, virtual machines, and datastores.
• A security view that provides visibility into potential security breaches and non-compliant usage patterns.
• The collection of granular performance metrics and log data all in one place, directly collected from VMware vCenter Servers and ESXi and vCenter logs (collected via syslog).
• The ability to explore very large data volumes, at speed, with access to fast queries on performance data.
• Track changes with visibility into VMware vCenter Server tasks and events in the context of your virtual environment.

Going back to basics of core Splunk, we can create our own searches, reports, alerts and dashboards on top of any Splunk app. With these additional dashboards we can identify, validate and repurpose these VMs that was mentioned above.

Lets go ahead and identify Zombie, Chatty and Orphan VMs by custom search command.

(sourcetype=”vmware:perf:cpu” source=”VMPerf:VirtualMachine”) OR (sourcetype=”vmware:inv:vm” changeSet.name=*) | eval detect = if(p_average_cpu_usage_percent < 5.00, “zombie”, if(p_average_cpu_usage_percent > 80.00, “chatty”, “normal”)) | stats first(detect) as “CPU Status” by moid

We can put together a very cool dashboard to show all the Zombie, Orphan and Chatty VMs.

Since the zombie and/or orphan VM’s could be repurposed for other usage, we can calculate the total cost for removing or repurposing the troubled VM’s.

This could help you show your management how much you saved the business with real savings!

(sourcetype=”vmware:perf:cpu” source=”VMPerf:VirtualMachine”) OR (sourcetype=”vmware:inv:vm” changeSet.name=*) | stats first(detect) as “CPU Status” first(changeSet.name) as “VM Name” first(p_average_cpu_usage_percent) as “Avg CPU Usage” by moid | stats count(moid) as “moid”, count(“VM Name”) as “vms” | eval cost = (moid – vms)*$price$ | table cost

Splunk can help your organization repurpose zombie and orphan VM’s to fully utilize your virtualization effort and to keep it secure. Splunk can also help identify chatty VM’s and move them to properly sized ESXi hosts.

Happy Splunking.

This blog post was jointly written by Tolga Tohumcu and Kam Amir…

From one side of the globe to the other, I continue to be impressed with the innovation and success of Splunk customers. In my last post, we discussed the success of UK retailer John Lewis and today we’re headed over to New Zealand to check out the innovation of online marketplace and classified ad site Trade Me.

Like many Splunk customers, Trade Me initially downloaded Splunk to address IT operations challenges (covered in detail in our Trade Me case study). As the company found success with Splunk, the Trade Me team realized that correlating clickstream data with structured data could uncover new business insights.

By mashing up clickstream and mobile data with structured data from relational databases, Trade Me began to gain insights into user preferences, bidding actions and listings popularity. This has also helped the company understand customer interactions across the web, mobile phones and tablets to determine which listings are popular and which site features attract the most attention from mobile users in real time.

Using Splunk for multi-channel analytics is having a real impact on business. The insights are being used to drive marketing campaigns and get a deeper understanding of listings on the platform and customers across channels. For a tangible example, I highly recommend checking out the Trade Me video below, which details how the company used Splunk Enterprise to track the success of a recent advertising campaign in real time.

If you want to hear from more Splunk customers, it’s not too late to register for. conf2014. The event will take place October 6-9, 2014, in Las Vegas. We hope to see you there!

Welcome to part three of my three-part blog on the ascending role of big data for healthcare analytics in the federal government. In this final part of the series we look at DHMSM, a very large project to find efficiency and insight in near real time. Part one and part two can be found here and here.

DHMSM and the problem to be addressed

Department of Defense Healthcare Management System Modernization (DHMSM) Program is administering an RFP for a potential $11B effort which calls for the modernization the Department of Defense healthcare system by uniting multiple legacy healthcare systems and data stores, developed over decades. I’ve reviewed most of the RFP consisting over 20 attachments which also calls for the implementation of an EHR (electronic health record) and HIE (health information exchange). This RFP was originally part of a joint DoD and Veterans Administration RFP that was “considered to be too costly.”

DoD has recognized the rising costs of healthcare as a percentage of the total DoD budget. Between 2006 and 2015, DoD health case related costs have moved from 8% of the overall budget to 12% by 2015. This is due in large part to:

• “Medical forces are not sufficiently net-centric and interoperable to enable a fully integrated future health system and accelerate its ability to know, decide and act in real time.
• Current medical information systems do not fully facilitate data sharing between government agencies, DoD and industry partners.
• Current medical training strategies and platforms cannot adequately prepare medical forces to operate in a future operational environment in which joint forces deliberately create dynamic situations that change at great speed and intensity.
• Current joint medical logistics processes are not sufficiently integrated and networked to respond efficiently and effectively to medical material requirements of future medical forces.
• Current medical systems, equipment, and forces lack capabilities to operate in all types of environments, including multinational operations; security, transition, and reconstruction operations; operations with NGOs and IGOs; medical capacity building; and public health services.
  The growing population of persons to be treated will increasingly challenge the ability of the MHS to provide the expected level of care.”

The EHR System is expected to unify and increase accessibility of integrated, evidenced-based healthcare delivery and decision-making. The EHR System will support the, “availability of longitudinal medical records for 9.6 million DoD beneficiaries and approximately 153,000+ MHS personnel globally. MHS personnel include activated members of the National Guard and Reserve, estimated at 50,000 additional personnel.”

The system will need to support:

• Access to historical information with role-based access control,
• Synchronization/integration with legacy systems,
• Migration required to support transitions for legacy information portals,
• Data, Identity Management and Access Management Architecture and Design Inputs,
• Health Data Sharing Recommendations, while;

Establishing “appropriate administrative, technical, and physical safeguards to protect all Government data, to ensure the confidentiality, integrity, and availability of Government data.”

The system is created to support, “…first responders, advanced trauma/emergency, Ancillary support services (e.g., Laboratory, Pharmacy, Radiology), in-theater hospitalization including robust care for resuscitation, surgery, and postoperative care, Critical Care services from Critical Care Air Transport Teams (CCATT).

In other words, the DoD wants to gather a high volume (I’m guessing north of 100 terabytes) of EHR data from a wide variety of legacy systems that likely produce a wide variety of structured and unstructured data from across the DoD health systems at high velocity. All while looking for better cost efficiency and better patient outcomes across a wide variety of use cases.

The DoD has finally recognized that it must, “…fully exploit information technology, infrastructure, training and research to support the four basic categories of military activity.”

Data types

While no specific data types are mentioned in the DHMSM RFP, we have some hints about the types of data that must be collected for the project to succeed.

• Access to historical information – Building baselines and trending for insight requires this kind of access.
• Synchronization with legacy systems – Tons of mainframes and legacy applications likely no longer supported putting out a variety of different structured and unstructured data.
• Data, Identity Management and Access Management Architecture and Design Inputs – Data access and identity management data are critical for HIPAA compliance and mitigating data breach risks.
• Health Data Sharing Recommendations – It is desirable to have data flow out of the big data system in ways that allow it to be used in other systems or as part of a mash-up in a web portal.

Security

The emphasis on security can be seen in several parts of DHMSM. Specifically, the contractor shall, “…provide cybersecurity that conforms to the DoD cybersecurity and the DoD Risk Management Framework requirements as outlined in the Defense Information Systems Agency (DISA) Information Assurance Support Environment (IASE).

The system shall also:

• “Ensure the confidentiality, integrity, and availability of Government data in compliance with all applicable laws and regulations, including data breach reporting and response requirements, in accordance with Defense Federal Regulations Subpart 224.1 (Protection of Individual Privacy), which incorporates by reference current versions DoDD 5400.11, “DoD Privacy Program,” and DoD 5400.11-R, “DoD Privacy Program.”
• Comply with federal laws relating to freedom of information and records management.
• Analyze any breach of PII/PHI for which it is responsible under the terms of this Contract under both the Privacy Act and Health Insurance Portability and Accountability Act (HIPAA), if applicable, to determine the appropriate course of action under each requirement, if any.”

Figure 1 — RMF for IS and PIT systems (page 28) – DoD Risk Management Framework

It should be noted that this approach closely mirrors the new CDM framework published by DHS.

“The system shall maintain situational awareness/continuous monitoring of any vulnerability, flaw or weakness in the network defense that could be exploited to gain unauthorized access to, damage or otherwise affect the network.” This is in accordance with the approved DoD Information Assurance Certification and Accreditation Process (DIACAP) baseline in accordance with the DoDI 8510.01. Also, [the system operator shall] mitigate all newly discovered vulnerabilities within the specified timelines based on Category level of the findings in accordance with DoD (DoDI 8510.01 section E2.56), DHA policies, and/or PEO DHMS.

Availability

Ideally, to support the use cases listed, the system must be able to be supported through full data redundancy, disaster recovery, and emergency failover and have 100% up-time. First responders (a use case listed in the RFP) can’t count on a system with only ‘five-nines’ of up time. The performance standards summary (appendix A) in the RFP calls for the EHR system reliability to be “The EHR System shall meet > 98% Systems Operational (network) Availability. Operational availability assesses the total time the system is capable of being used to perform clinical functions during a given interval.” Established baselines and benchmarks for application performance, error rates, and system downtime can all be tracked for accountability and alerts routed to the right person for

Confidentiality

In addition to all applicable HIPAA compliance standards, the DoD has it’s own Department of Defense Privacy Program. The section of collecting personal information discusses how SSNs are used, how private data shall be accessed (For example, an individual requiring access, “…may be required to provide reasonable proof his or her identity.”).

Splunk has the ability to collect all authentication records and correlate them with context data about the user to understand their rights to view the data based on who they are, where they are, what system they are authenticating from, the authentication channel or method (i.e. VPN), and when the data is being accessed. The user can assign a risk score to these factors and use the information to determine whether or not the user should be allowed access.

Splunk’s big data solution provides role based access controls and auditing capabilities needed to restrict, monitor and alert on user access to specific patient data. This meets DoD privacy objectives. Alerts and access dashboards can be created that call out any combination of abnormal access based on a lack of relationship between the patient and caregiver, time or date of access, or access by a caregiver that may be inappropriate based on a current or past family relationship.

Use Cases and Splunk’s Role

Of all the documents that are a part of the RFP, the most interesting documents in the RFP package are Attachments 8 and 9 both titled Healthcare Service Delivery Concept of Operations – CONOPS and Attachment 2 Government Requirements Traceability Matrix. These document outline some of the use cases that the DoD has thought of and seen to include in the RFP package. In the table below we’ll some of the capabilities in the tracibility matrix and outline Splunk’s possible role.

The table above represents a small sample of use cases for data driven healthcare decisions.

Splunk provides two very compelling solutions that can be applied to healthcare data in support of health information exchanges and electronic health record systems:

Hunk (Splunk Analytics on Hadoop) – provides the benefits of Splunk’s statistical analysis capabilities across extremely large datasets in Hadoop. Use cases for this data would be related to historical data analysis and the monitoring of ongoing trends. This cost effective solution supports non real-time use cases.

Splunk Enterprise – provides support for extremely large data sets where answers to questions in specific use cases are needed in near real-time. Emergency workers, first responders, and military theaters of operations should be using Splunk Enterprise for real-time insight.

Big data has a huge role to play in bringing together desperate healthcare data sources and types without the cost of upfront normalization, monitoring for proper usage of data, finding efficiencies that reduce cost and finding data relationships that can lead to better quality of care.

//

//

//

//

//

Splunk is really good at knowing what has been read and what is new when dealing with machine data that is on disk (using the fishbucket).  However, there is a lot of machine data that does not exist on disk.  Some examples of this type of data are 3^rd party APIs or in-memory data.  In order to get to this data, we often use scripted or modular inputs.

When dealing with scripted or modular inputs, it is important to only get new events or information since the last time the input ran.  You don’t want to keep indexing the same events over and over because this will cause index bloat and performance issues.  So, you need a way of knowing what you have already read and where to pick up.  There are a couple of ways of doing this and where your input is running often dictates which method you should use.

Running Inputs from a Splunk Indexer or Search Head

When querying a 3rd party API (especially SaaS APIs), it may make sense to run the input on a search head or indexer. There are less moving parts involved and there are some built-in Splunk script methods you can use.

Step 1 – Create a default .conf file

The first thing we need to do is create a .conf file to store our parameters. This file can optionally have any defaults defined. I called my file my_app.conf, but you can call it whatever you want.

$SPLUNK_HOME/etc/apps/<your_app>/default/my_app.conf

Here are the contents of my_app.conf:

[API_data]
last_record_read =

Note: after the script runs, updates will be made in the following location (notice it is in local instead of default):
$SPLUNK_HOME/etc/apps/<your_app>/local/my_app.conf

Step 2 – Create a script to query data

The script should be in $SPLUNK_HOME/etc/apps//bin/scripts

Here is an example in Python using Splunk Entities:
import splunk.entity as en
import splunk, sys, re, time, logging, os
# Constants
APP_NAME = "my_app"
CONF_FILE = "my_app"
STANZA_NAME = "API_data"
# Set up script logging
def getExceptionLogger():
logger = logging.getLogger(APP_NAME)
SPLUNK_HOME = os.environ['SPLUNK_HOME']
LOGGING_DEFAULT_CONFIG_FILE = os.path.join(SPLUNK_HOME, 'etc', 'log.cfg')
LOGGING_LOCAL_CONFIG_FILE = os.path.join(SPLUNK_HOME, 'etc', 'log-local.cfg')
LOGGING_STANZA_NAME = 'python'
LOGGING_FILE_NAME = APP_NAME + '.log'
BASE_LOG_PATH = os.path.join('var', 'log', 'splunk')
LOGGING_FORMAT = "%(asctime)s %(levelname)-s\t%(module)s:%(lineno)d - %(message)s"
splunk_log_handler = logging.handlers.RotatingFileHandler(os.path.join(SPLUNK_HOME, BASE_LOG_PATH, LOGGING_FILE_NAME), mode='a')
splunk_log_handler.setFormatter(logging.Formatter(LOGGING_FORMAT))
logger.addHandler(splunk_log_handler)
splunk.setupSplunkLogger(logger, LOGGING_DEFAULT_CONFIG_FILE, LOGGING_LOCAL_CONFIG_FILE, LOGGING_STANZA_NAME)
return logger
def updateStanza(key,value):
conf_stanza[key] = value
en.setEntity(conf_stanza,sessionKey=sessionKey)
def run_script():
try:
# Get the last record read from the conf file
last_record_read = conf_stanza["last_record_read"]
# Perform your connection to the API here.
# You can use the variable last_record_read to pass to the API call.
# Write to stdout (i.e. print) the records returned from the API however you want them to show up in the index.
# Once you have written your records to stdout, update the conf file using a time stamp or value from the API call.
# In this case, we are just making up a value, but this would normally be returned from the API.
last_record_read_from_API = 400
updateStanza(key="last_record_read", value=last_record_read_from_API)
except IOError, err:
logger.error('ERROR - %s' % str(err))
pass
conf_stanza = None
if __name__ == '__main__':
logger = getExceptionLogger()
logger.info("Script started")
# Get the sessionKey from splunkd
# Note: inputs.conf shoud specify passAuth = splunk-system-user
sk = sys.stdin.readline().strip()
sessionKey = re.sub(r'sessionKey=', "", sk)
try:
# Get the stanza key/value pairs
conf_stanza = en.getEntity('configs/conf-' + CONF_FILE, STANZA_NAME, namespace=APP_NAME, owner='nobody', sessionKey=sessionKey)
run_script()
except IOError, err:
logger.error('ERROR - %s' % str(err))
pass

Step 3 – Add the script to inputs.conf

Once you have your script written, the script needs to be added to inputs.conf. Here is an excerpt:

[script://./bin/scripts/test.py]
disabled = 0
interval = 300
passAuth = splunk-system-user

The passAuth part is very important in order to obtain a session key in the script.

Running Inputs on Universal Forwarders

When you have a lot of universal forwarders running inputs and forwarding data to indexers, it is impractical to keep up with each and every forwarder’s position from a search head or indexer. Therefore, it is better to have the forwarder itself keep a position file to mark where it is in the running lifecycle. Unfortunately, Python does not ship with Universal Forwarders, so the Entity method used above will not work. You are free to implement this type of position placeholder using any scripting language the OS understands. For an example using PowerShell, refer to the blog post about Measuring Windows Group Policy Logon Performance.

Specifically, here is the code that utilized a position file:

https://github.com/JasonConger/SplunkWinLogon/blob/master/SplunkWinLogonExample/appserver/addons/TA-WinLogon/bin/powershell/GPO-Ops-Log.ps1

Modular inputs allow you to teach Splunk Enterprise new ways to pull in events from internal systems, third party APIs or even devices. Modular Inputs extend Splunk Enterprise and are deployed on the Splunk Enterprise instance or on a forwarder.  In version 1.4.0 of the Splunk SDK for JavaScript we added support for creating modular inputs in Node.js!

In this post, I’ll show you how to create a modular input with Node.js that pulls commit data from GitHub into Splunk.

Why Node.js

Node.js is designed for I/O intensive workloads. It offers great support for streaming data into and out of a Node application in an asynchronous manner. It also has great support for JSON out of the box. Finally, Node.js has a huge ecosystem of packages available via npm that are at your disposal. An input pulls data from a source and then streams those results directly into a Splunk instance. This makes modular inputs a great fit for Node.js.

Getting started

You can get the Splunk SDK for JavaScript from npm (npm install splunk-sdk), the Splunk Developer Portal or by grabbing the source from our GitHub repo. You can find out more about the SDK here. The SDK includes two sample modular inputs, random numbers, and GitHub commits. For the remainder of this post we’ll look at the GitHub example.

This input indexes all commits on the master branch of a GitHub repository using GitHub’s API. This example illustrates how to pull in data from an external source, as well as showing how to create checkpoints when you are periodically polling in order to prevent duplicate events from getting created.

Prerequisites

• You have installed a Splunk Enterprise instance, version 5.0 or later
• You have Node.js installed (v0.8 or later, check this by running node -v from the command line).
• You have downloaded the Splunk SDK for JavaScript.

Installing the example

1. Set the $SPLUNK_HOME environment variable to the root directory of your Splunk Enterprise instance.
2. Copy the GitHub example from

   /splunk-sdk-javascript/examples/modularinputs/github_commits

   to

   $SPLUNK_HOME/etc/apps

3. Open a command prompt or terminal window and go to the following directory:

   $SPLUNK_HOME/etc/apps/github_commits/bin/app

4. Then type npm install, this will install the Node modules which are required, which includes the splunk-sdk itself and the github module.
5. Restart Splunk Enterprise by typing the following into the command line:

   $SPLUNK_HOME/bin/splunk restart

Configuring the GitHub commits modular input example

Modular Inputs integrate with Splunk Enterprise, allowing Splunk Administrators to create new instances and provide necessary configuration right in the UI similar to other inputs in Splunk. To see this in action, follow these steps:

1. From Splunk Home, click the Settings menu. Under Data, click Data inputs, and find “GitHub commits”, the input you just added. Click Add new on that row. 
2. Click Add new and fill in:
   • name (whatever name you want to give this input)
   • owner (the owner of the GitHub repository, this is a GitHub username or org name)
   • repository (the name of the GitHub repository)
   • (optional) token if using a private repository and/or to avoid GitHub’s API limits

   To get a GitHub API token visit the GitHub settings page and make sure the repo and public_repo scopes are selected. 

3. Save your input, and navigate back to Splunk Home.
4. Do a search for sourcetype=github_commits and you should see some events indexed; if your repository has a large number of commits indexing them may take a few moments.

Analyzing GitHub commit data

Now that your GitHub repository’s commit data has been indexed by Splunk Enterprise, you can leverage the power of Splunk’s Search Processing Language to do interesting things with your data. Below are some example searches you can run:

• Want to know who the top contributors are for this repository? Run this search:

  sourcetype="github_commits" source="github_commits://[your input name]" | stats count by author | sort count DESC
  

• Want to see a graph of the repository’s commits over time? Run this search:

  sourcetype="github_commits" source="github_commits://[your input name]" | timechart count(sha) as "Number of commits"

  Then click the Vizualization tab, and select line from the drop down for visualization types (pie may be already selected).

Write your own modular input with the Splunk SDK for JavaScript

Adding a modular input to Splunk Enterprise is a two-step process: First, write a modular input script, and then package the script with several accompanying files and install it as a Splunk app.

Writing a modular input

A modular input will:

1. Return an introspection scheme. The introspection scheme defines the behavior and endpoints of the script.  When Splunk Enterprise starts, it runs the input to determine the modular input’s behavior and configuration.
2. Validate the script’s configuration (optional). Whenever a user creates or edits an input, Splunk Enterprise can call the input to validate the configuration.
3. Stream events into Splunk. The input streams event data that can be indexed by Splunk Enterprise. Splunk Enterprise invokes the input and waits for it to stream events.

To create a modular input in Node.js, first require the splunk-sdk Node module. In our examples, we’ve also assigned the classes we’ll be using to variables, for convenience. At the very least, we recommend defining a ModularInputs variable as shown here:

var splunkjs        = require("splunk-sdk");
var ModularInputs   = splunkjs.ModularInputs;

The preceding three steps are accomplished as follows using the Splunk SDK for JavaScript:

1. Return the introspection scheme: Define the getScheme method on the exports object.
2. Validate the script’s configuration (optional): Define the validateInput method on the exports object. This is required if you set the scheme returned by getScheme to use external validation (that is, set Scheme.useExternalValidation to true).
3. Stream events into Splunk: Define the streamEvents method on the exports object.

In addition, you must run the script by calling the ModularInputs.execute method, passing in the exports object you just configured along with the module object which contains the state of this script:

ModularInputs.execute(exports, module);

To see the full GitHub commits input source code, see here.

Woah. Let’s take a deeper dive into the code so we can understand what’s really going on.

The getScheme method

When Splunk Enterprise starts, it looks for all the modular inputs defined by its configuration, and tries to run them with the argument –scheme. The scheme allows your input to tell Splunk arguments that need to be provided for the input, these arguments are then used for populating the UI when a user creates an instance of an input. Splunk expects each modular input to print a description of itself in XML to stdout. The SDK’s modular input framework takes care of all the details of formatting the XML and printing it. You only need to implement a getScheme method to return a new Scheme object, this makes your job much easier!

As mentioned earlier, we will be adding all methods to the exports object.

Let’s begin by defining getScheme, creating a new Scheme object, and setting its description:

exports.getScheme = function() {
        var scheme = new Scheme("GitHub Commits"); 
        scheme.description = "Streams events of commits in the specified GitHub repository (must be public, unless setting a token).";

For this scheme, the modular input will show up as “GitHub Commits” in Splunk.

Next, specify whether you want to use external validation or not by setting the useExternalValidation property (the default is true). If you set external validation to true without implementing the validateInput method on the exports object, the script will accept anything as valid. We want to make sure the GitHub repository exists, so we’ll define validateInput once we finish with getScheme.

       scheme.useExternalValidation = true;

If you set useSingleInstance to true (the default is false), Splunk will launch a single process executing the script which will handle all instances of the modular input. You are then responsible for implementing the proper handling for all instances within the script. Setting useSingleInstance to false will allow us to set an optional interval parameter in seconds or as a cron schedule(available under more settings when creating an input).

      scheme.useSingleInstance = false;

The GitHub commits example has 3 required arguments (name, owner, repository), and one optional argument (token). Let’s recap what these are for:

• name: The name of this modular input definition (ex: Splunk SDK for JavaScript)
• owner: The GitHub organization or user that owns the repository (ex: splunk)
• repository: The GitHub repository (ex: splunk-sdk-javascript), don’t forget to set the token argument if the repository is private
• token: A GitHub access token with at least the repo and public_repo scopes enabled. To get an access token, see the steps outlined earlier in this post.

Now let’s see how these arguments are defined within the Scheme. We need to set the args property of the Scheme object we just created to an array of Argument objects:

      scheme.args = [
            new Argument({
                name: "owner",
                dataType: Argument.dataTypeString,
                description: "GitHub user or organization that created the repository.",
                requiredOnCreate: true,
                requiredOnEdit: false
            }),
            new Argument({
                name: "repository",
                dataType: Argument.dataTypeString,
                description: "Name of a public GitHub repository, owned by the specified owner.",
                requiredOnCreate: true,
                requiredOnEdit: false
            }),
            new Argument({
                name: "token",
                dataType: Argument.dataTypeString,
                description: "(Optional) A GitHub API access token. Required for private repositories (the token must have the 'repo' and 'public_repo' scopes enabled). Recommended to avoid GitHub's API limit, especially if setting an interval.",
                requiredOnCreate: false,
                requiredOnEdit: false
            })
        ];

Each Argument constructor, takes a parameter of a JavaScript object with the required property name and the optional properties:

• dataType: What kind of data is this argument? (Argument.dataTypeBoolean, Argument.dataTypeNumber, or Argument.dataTypeString)
• description: A description for the user entering this argument (string)
• requiredOnCreate: Is this a required argument? (boolean)
• requiredOnEdit: Does a new value need to be specified when editing this input? (boolean)

After adding arguments to the scheme, return the scheme and we close the function:

        return scheme;
    };

The validateInput method

The validateInput method is where the configuration of an input is validated, and is only needed if you’ve set your modular input to use external validation. If validateInput does not call the done callback with an error argument, the input is assumed to be valid. Otherwise it throws an error when it tells Splunk that the configuration is not valid.

When you use external validation, after splunkd calls the modular input with the –scheme argument to get the scheme, it calls it again with the –validate-arguments argument for each instance of the modular inputs in its configuration files, feeding XML on stdin to the modular input to validate all enabled inputs. Splunk calls the modular input the same way again whenever the modular input’s configuration is changed.

In our GitHub Commits example, we’re using external validation since we want to make sure the repository is valid. Our validateInput method contains logic used the GitHub API to check that there is at least one commit on the master branch of the specified repository:

    exports.validateInput = function(definition, done) { 
        var owner = definition.parameters.owner;
        var repository = definition.parameters.repository;
        var token = definition.parameters.token;

        var GitHub = new GitHubAPI({version: "3.0.0"});

        try {
            if (token && token.length > 0) {
                GitHub.authenticate({
                    type: "oauth",
                    token: token
                });
            }

            GitHub.repos.getCommits({
                headers: {"User-Agent": SDK_UA_STRING},
                user: owner,
                repo: repository,
                per_page: 1,
                page: 1
            }, function (err, res) {
                if (err) {
                    done(err);
                }
                else {
                    if (res.message) {
                        done(new Error(res.message));
                    }
                    else if (res.length === 1 && res[0].hasOwnProperty("sha")) {
                        done();
                    }
                    else {
                        done(new Error("Expected only the latest commit, instead found " + res.length + " commits."));
                    }
                }
            });
        }
        catch (e) {
            done(e);
        }
    };

The streamEvents method

Here’s the best and most important part, streaming events!

The streamEvents method is where the event streaming happens. Events are streamed into stdout using an InputDefinition object as input that determines what events are streamed. In the case of the GitHub commits example, for each input, the arguments are retrieved before connecting to the GitHub API. Then, we go through each commit in the repository on the master branch.

Creating Events and Checkpointing

For each commit, we’ll check to see if we’ve already indexed it by looking in a checkpoint file. This is a file that Splunk allows us to create in order to track which data has been already processed so that we can prevent duplicates. If we have indexed the commit, we simply move on – we don’t want to have duplicate commit data in Splunk. If we haven’t indexed the commit we’ll create an Event object, set its properties, write the event using the EventWriter, then append the unique SHA for the commit to the checkpoint file. We will create a new checkpoint file for each input (in this case, each repository).

The getDisplayDate function, is used to transform the date we get back from the GitHub API into something more readable format.

exports.streamEvents = function(name, singleInput, eventWriter, done) {
        // Get the checkpoint directory out of the modular input's metadata.
        var checkpointDir = this._inputDefinition.metadata["checkpoint_dir"];

        var owner = singleInput.owner;
        var repository = singleInput.repository;
        var token      = singleInput.token;

        var alreadyIndexed = 0;

        var GitHub = new GitHubAPI({version: "3.0.0"});

        if (token && token.length > 0) {
            GitHub.authenticate({
                type: "oauth",
                token: token
            });
        }

        var page = 1;
        var working = true;

        Async.whilst(
            function() {
                return working;
            },
            function(callback) {
                try {
                    GitHub.repos.getCommits({
                        headers: {"User-Agent": SDK_UA_STRING},
                        user: owner,
                        repo: repository,
                        per_page: 100,
                        page: page
                    }, function (err, res) {
                        if (err) {
                            callback(err);
                            return;
                        }

                        if (res.meta.link.indexOf("rel=\"next\"") < 0) {
                            working = false;
                        }
                        
                        var checkpointFilePath  = path.join(checkpointDir, owner + " " + repository + ".txt");
                        var checkpointFileNewContents = "";
                        var errorFound = false;

                        var checkpointFileContents = "";
                        try {
                            checkpointFileContents = utils.readFile("", checkpointFilePath);
                        }
                        catch (e) {
                            fs.appendFileSync(checkpointFilePath, "");
                        }

                        for (var i = 0; i < res.length && !errorFound; i++) {
                            var json = {
                                sha: res[i].sha,
                                api_url: res[i].url,
                                url: "https://github.com/" + owner + "/" + repository + "/commit/" + res[i].sha
                            };

                            if (checkpointFileContents.indexOf(res[i].sha + "\n") < 0) {
                                var commit = res[i].commit;

                                json.message = commit.message.replace(/(\n|\r)+/g, " ");
                                json.author = commit.author.name;
                                json.rawdate = commit.author.date;
                                json.displaydate = getDisplayDate(commit.author.date.replace("T|Z", " ").trim());

                                try {
                                    var event = new Event({
                                        stanza: repository,
                                        sourcetype: "github_commits",
                                        data: JSON.stringify(json),
                                        time: Date.parse(json.rawdate)
                                    });
                                    eventWriter.writeEvent(event);

                                    checkpointFileNewContents += res[i].sha + "\n";
                                    Logger.info(name, "Indexed a GitHub commit with sha: " + res[i].sha);
                                }
                                catch (e) {
                                    errorFound = true;
                                    working = false;
                                    Logger.error(name, e.message, eventWriter._err);
                                    fs.appendFileSync(checkpointFilePath, checkpointFileNewContents);

                                    done(e);
                                    return;
                                }
                            }
                            else {
                                alreadyIndexed++;
                            }
                        }

                        fs.appendFileSync(checkpointFilePath, checkpointFileNewContents);

                        if (alreadyIndexed > 0) {
                            Logger.info(name, "Skipped " + alreadyIndexed.toString() + " already indexed GitHub commits from " + owner + "/" + repository);
                        }

                        page++;
                        alreadyIndexed = 0;
                        callback();
                    });
                }
                catch (e) {
                    callback(e);
                }
            },
            function(err) {
                done(err);
            }
        );
    };

Logging (optional)

Logging is an optional feature we’ve included with modular inputs the Splunk SDK for JavaScript.

It’s best practice for your modular input script to log diagnostic data to splunkd.log ($SPLUNK_HOME/var/log/splunk/splunkd.log). Use a Logger method to write log messages, which include a standard splunkd.log severity level (such as “DEBUG”, “WARN”, “ERROR” and so on) and a descriptive message. For instance, the following code is from the GitHub Commits streamEvents example, and logs a message if any GitHub commits have already been indexed:

if (alreadyIndexed > 0) {
    Logger.info(name, "Skipped " + alreadyIndexed.toString() + " already indexed GitHub commits from " + owner + "/" + repository);
}

Here we call the Logger.info method to log a message with the info severity, we’re also passing in the name argument, which the user set when creating the input.

That’s all the code you have to write to get started with modular inputs using the Splunk SDK for JavaScript!

Add the modular input to Splunk Enterprise

With your modular input completed, you’re ready to integrate it into Splunk Enterprise. First, package the input, and then install the modular input as a Splunk app.

Package the input

Files

Create the following files with the content indicated. Wherever you see modinput_name — whether in the file name or its contents — replace it with the name of your modular input JavaScript file. For example, if your script’s file name is github_commits.js, give the file indicated as modinput_name.cmd the name github_commits.cmd.

If you haven’t already, now is a good time to set your $SPLUNK_HOME environment variable.

We need to make sure all the names match up here, or Splunk will have problems recognizing your modular input.

modinput_name.cmd

@"%SPLUNK_HOME%"\bin\splunk cmd node "%~dp0\app\modinput_name.js" %*

modinput_name.sh

#!/bin/bash

current_dir=$(dirname "$0")
"$SPLUNK_HOME/bin/splunk" cmd node "$current_dir/app/modinput_name.js" $@

package.json

When creating this file, replace the values given with the corresponding values for your modular input. All values (except the splunk-sdk dependency, which should stay at “>=1.4.0″) can be changed.

{
    "name": "modinput_name",
    "version": "0.0.1",
    "description": "My great modular input",
    "main": "modinput_name.js",
    "dependencies": {
        "splunk-sdk": ">=1.4.0"
    },
    "author": "Me"
}

app.conf

When creating this file, replace the values given with the corresponding values for your modular input:

• The is_configured value determines whether the modular input is preconfigured on install, or whether the user should configure it.
• The is_visible value determines whether the modular input is visible to the user in Splunk Web.

inputs.conf.spec

[install]
is_configured = 0

[ui]
is_visible = 0
label = My modular input

[launcher]
author=Me
description=My great modular input
version = 1.0

When creating this file, in addition to replacing modinput_name with the name of your modular input’s JavaScript file, do the following:

• After the asterisk (*), type a description for your modular input.
• Add any arguments to your modular input as shown. You must list every argument that you define in the getScheme method of your script.

The file should look something like this:

[github_commits://<name>]
*Generates events of GitHub commits from a specified repository.

owner = <value>
repository = <value>
token = <value>

File structure

Next, create a directory that corresponds to the name of your modular input script—for instance, “modinput_name” — in a location such as your Documents directory. (It can be anywhere; you’ll copy the directory over to your Splunk Enterprise directory at the end of this process.)

1. Within this directory, create the following directory structure:

   modinput_name/
       bin/
           app/
       default/
       README/

2. Copy your modular input script (modinput_name.js) and the files you created in the previous section so that your directory structure looks like this:

   modinput_name/
       bin/
           modinput_name.cmd
           modinput_name.sh
           app/
               package.json
               modinput_name.js
       default/
           app.conf
       README/
           inputs.conf.spec

Install the modular input

Before using your modular input as a data input for your Splunk Enterprise instance, you must first install it.

1. Set the SPLUNK_HOME environment variable to the root directory of your Splunk Enterprise instance.
2. Copy the directory you created in Package the script to the following directory:

   $SPLUNK_HOME/etc/apps/

3. Open a command prompt or terminal window and go to the following directory, where modinput_name is the name of your modular input script:

   $SPLUNK_HOME/etc/apps/modinput_name/bin/app

4. Type the following, and then press Enter or Return: npm install
5. Restart Splunk Enterprise: From Splunk Home, click the Settings menu. Under System, click Server Controls. Click Restart Splunk; alternatively you can just run

   $SPLUNK_HOME/bin/splunk restart

   from command prompt or terminal.

Your modular input should now appear long the native Splunk input by going to Splunk Home, click the Settings menu. Under Data, click Data inputs, and find the names of the modular inputs you just created.

In Summary

In this post you’ve seen how to create a modular input using the Splunk SDK for JavaScript.

Now you can use your Node.js skills to extend Splunk and pull data from any source, even Github!

NOTE: Rather than read this post, you can come see this use case presented live at our upcoming Worldwide User Conference in Las Vegas. My colleague Andrew Gerber from Wipro will be reviewing this and a few other recent use cases we have worked on together.

For the past 14 years (yes, I am old) I have worked out of a home office. This means that during my workdays, I can freely receive YouTube cat video links from my high-school ex-girlfriends, grab some carrot sticks and hummus from the kitchen, and watch as many of them as I care to using my trusty copy of Internet Explorer 6. The reason I can do this? No pesky corporate web proxy monitoring my outgoing browsing activity, thank you very much.

But for those in the corporate world, there’s all sorts of infrastructure deployed specifically so that you can’t get your fix of YouTube, and various other non-essential sites. Web traffic is generally funneled through a “restrictive corporate Internet proxy.” There’s good reason for this of course – companies like to limit personal use of corporate resources, and want to ensure that employees aren’t accessing content that could be considered offensive and/or potentially harmful. And, modern proxies can do things like cache content, limit bandwidth, and scan for viruses in addition to the standard content filtering. (By the way – the logs from those proxies are chock-full of great information, and if you aren’t feeding them into Splunk, you should. We have code and documentation to help you onboard logs from Websense, Bluecoat, Palo Alto, and many other vendors).

These days, if you want your fix of cat videos at work, you’d probably just fire up your iPhone 6 Plus with LTE connection and prop it up on your desk using one-half of yesterday’s tuna sandwich as a stand. But another method to usurp the corporate proxy, and one that is pretty commonplace, is “off net browsing.” Corporations often provide a “guest wireless network” for use by visitors, contractors, vendors, and so forth. Often, this guest network is less restrictive than the corporate network, simply because of the needs of the computing population that uses it. For example, in a hospital, it is common to offer a guest network for use by patients and families. More services and sites may be available via this unsecured network than on the hospital’s regular “corporate” network. It’s the same thing for many retail locations.

The bottom line is, even if you have one of these guest networks readily available in the same physical location as your corporate employees, they shouldn’t be jumping onto the guest network to get their fill of cat videos, or anything else. It’s potentially quite dangerous to the corporate-owned laptop or desktop being used to do this: i.e. Jimmy drops off the corporate network, associates his laptop with the guest network, browses to “http://www.SeeKittyPerformTheLambada.biz” and is immediately entertained, then infected with a piece of malware. Not knowing his laptop is infected, he then re-associates it with the corporate network, and now you have an infected client sitting on your LAN.

How can we monitor this behavior? Enter Splunk (four paragraphs to get to the heart of the matter? Really? Sorry. )

Recently I spent time at a local customer that wanted to monitor this behavior across their environment. Users that leverage the guest network excessively should be reprimanded, especially if they haven’t taken the appropriate security skills assessment. In order to create a dashboard that reports on this behavior, we brought the following data into Splunk:

• DHCP log events from Fortinet firewalls servicing the “Guest” network
• Log events categorizing Guest browsing activity from Fortinet firewalls
• Log events categorizing Corporate browsing activity from Palo Alto devices
• DNS log events from Microsoft DNS servers categorizing DNS lookups across the environment

Splunk allows us to correlate all of these sources together over time. We can, for example, find a MAC address of a corporate resource showing up on our guest network. Using lookups, or using authentication data found in the logs already, we can determine which employee has associated with the guest network. Next, we can connect that MAC address to the IP address given on the guest network, and determine what sites they have visited on the guest network. We can correlate these pieces of information with the same types of data found in our corporate web proxies, to get a feel of the sites that an employee has tried to visit (hence the need to move to the guest network). And finally, we can correlate the employee name, or asset MAC address, with identity and asset databases to determine whether or not the employee has attended security skills training.

Here’s a walkthrough of a dashboard called, appropriately, “Off Net Jumpers.” We start with a dashboard panel of off-net jumping activity. The corporate assets seen moving from corporate IP address space to guest IP addresses are plotted over time.

How do we determine that an asset has moved from “corporate” to “guest?” We look at the firewall logs, which log DHCP requests and happen to have the IP address of one of our corporate networks, as well as the MAC address, showing up on our guest network. Note that in the search, we are eliminating hosts that are known to be mobile, and we are further limiting the hostnames to only ones that meet the pattern for our corporate standard.

Next, a panel detailing these jumpers, all with corporate-assigned IP addresses, hostnames, and MAC addresses.

This is a drilldown panel. Clicking on a row will give us the details of that “jumper” in the panels below (using the new panel tags available in SimpleXML in Splunk 6.1+). The appearing panels look like this:

We can see the jumper at IP address 10.248.30.80 tried to go to a lot of YouTube links, under the category of “streaming media” on our corporate site (and they were blocked). These logs came from Palo Alto devices. On the left, we see the DNS requests that the system has been making (blurred to protect the innocent). And at the top, the name (corporate AD login or owner of asset) of the jumper – in our case this was in the DHCP request, but this just as easily could come from a lookup against the MAC address and a corporate database (CMDB) or from authentication logs.

The searches above do one more thing – they retrieve the Guest IP address of the “jumper” based on MAC address seen in the DHCP requests from the same logs. In this case, it was 192.168.200.62. We can then use this guest IP address to search for which sites have been accessed by this IP address during the same timeframe of the jump. And we can see that YouTube was in fact passed:

And finally, based on the jumping activity found in the first few panels, we now scope to just that one asset, to determine how many times over the past 7 days this device has been seen moving from our corporate network to our guest network. This search could be turned into an alert, or a e-mailable report to a superior, or or into a lookup against a database containing training records, to ensure that this individual has (or has not) attended security training. This allows us to link Splunk capabilities to prove our adherence with one of the 20 Critical Security Controls (#9 – Security Skills Assessment).

If you want to hear more about how this was accomplished, my colleague Andrew Gerber will be detailing this use case, and several others, at our upcoming .conf2014 in Fabulous Las Vegas. What? You haven’t signed up yet? What are you waiting for? If you’re coming…I will see you there.

Please bring me some carrots and hummus from the buffet.

UPDATE: I changed the script a little bit to include platform information in the output by using the uname command and bash version information in the output with –version. This should work on Linux and OSX.

UPDATE 2: The first script below is specific to find the original shellshock: CVE-2014-6271. The second, less severe shellshock vulnerability, CVE-2014-7169, requires a different test. See the new script at the end of the post to cover this.

I knew that eventually, a vulnerability would find its way into the wild with its own theme song.

Today, David Millis, one of our fine Client Architects, sent out this link concerning a new vulnerability (CVE-2014-6271) in bash, found installed on, oh, just about every modern UNIX variant. Read about it here and here and about 4,000 other places by simply typing “shellshock” into Google News. And basically – stop reading Splunk blogs and go patch bash. Now.

I’ll let others tell you how you could use Splunk to search through your various logs for evidence that evildoers are trying to exploit this in your environment. But how can we use the trusty Splunk forwarders installed on your servers to ensure that, after your sysadmins patch bash everywhere, you don’t have systems that are still vulnerable? How about a scripted input?

Create a shell script that looks like this, somewhere on your system you want to test for vulnerable bash. A good place to put it could be right alongside the Splunk_TA_nix scripts that already gather information from your UNIX systems. I created it there, and called it shellshock.sh, and made it executable. You can always use Splunk’s deployment server to get this script out to your systems.

#!/bin/sh
HOSTNAME=$(/bin/hostname)
RUNNING=$(/bin/date)
BASHVERSION=$(/bin/bash --version)
if [ -f /bin/uname ]; then UNAME=$(/bin/uname -srvmpio); else UNAME=$(/usr/bin/uname -v); fi
THECHECK=$(env='() { :;}; echo status=VULNERABLE' bash -c "ls -al /bin/bash" 2>&1 /dev/null)
if [[ $THECHECK == *VULNERABLE* ]] ; then echo "$RUNNING hostname=$HOSTNAME platform=$UNAME cve=2014-6271 status=VULNERABLE version=$BASHVERSION"; fi
if [[ $THECHECK != *VULNERABLE* ]] ; then echo "$RUNNING hostname=$HOSTNAME platform=$UNAME cve=2014-6271 status=NOTVULNERABLE version=$BASHVERSION"; fi

Yeah, it looks funky above, but copy and paste it, all will be well. Now, create an entry in inputs.conf like this (this runs the check every hour…):

[script://./bin/shellshock.sh]
sourcetype = shellshock
source = shellshock
interval = 3600
index = os
disabled = 0

And restart your forwarder.

You can now search for any hosts with sourcetype=shellshock, and of course use the field “status” to determine if the host is vulnerable or not, and with the “platform” field and “version” field you can filter on various OS versions and levels (you may need to adjust the field extractions depending on what you end up with in your versions and platforms…). The middle screenshot below shows that I’ve patched a Linux box and an OSX box in my lab, but one remains unpatched (because showing all OK is never fun for examples!)

I have tested this on recent Linux distributions (CentOS) and OSX, and it appears functional on both. It’s pretty simple – my guess is it will work on a lot of other *nix flavors. However, your mileage may vary. Please let me know how this works out for you, and improve as you see fit!

2014-7169 UPDATE: Here’s a modification of the above script to check for CVE-2014-7169. It’s a little more complex as there’s a need to capture standard error to assess the vulnerability, so a temp file is used. Tested on CentOS and OSX, but again, your mileage may vary. If you have both of these running as scripted inputs, you can simply use Splunk to filter on the “cve” field to display results from one or both vulnerabilities.

#!/bin/sh
HOSTNAME=$(/bin/hostname)
RUNNING=$(/bin/date)
BASHVERSION=$(/bin/bash --version)
TEMPFILE=/tmp/7169check.txt
if [ -f /bin/uname ]; then UNAME=$(/bin/uname -srvmpio); else UNAME=$(/usr/bin/uname -v); fi

env X='() { (a)=>\' sh -c "echo date > /dev/null" 2>> $TEMPFILE; if [ -f echo ]; then cat echo; fi
THECHECK=$(cat $TEMPFILE)
rm $TEMPFILE
if [ -f echo ]; then rm echo; fi

if [[ $THECHECK == *unexpected* ]] ; then echo "$RUNNING hostname=$HOSTNAME platform=$UNAME cve=2014-7169 status=VULNERABLE version=$BASHVERSION"; fi
if [[ $THECHECK != *unexpected* ]] ; then echo "$RUNNING hostname=$HOSTNAME platform=$UNAME cve=2014-7169 status=NOTVULNERABLE version=$BASHVERSION"; fi

Search output from BOTH scripts above, showing that the freshly-compiled bash on my test OSX host is not vulnerable to 2014-6271, but is still for vulnerability 2014-7169:

A few years ago, I created a publicly available traffic app for monitoring traffic incidents in major US cities configured by user. Since then, the provider of the feed has cut down on the number of cities they monitor and no longer provide incident counts per intersection. Nevertheless, they still provide a Jam Factor. A Jam Factor is a subjective number provided for a roadway that indicates how busy (or jammed) the roadway is.

For my reference implementation, I used this Jam factor field to visually allow you to to see your city’s (assuming the provider covers it) current Jam Factor for major highways. This updated traffic app that you can download has new dashboards that you can use to get a current read of traffic conditions. For illustration purposes, I used a Jam factor of 0 to 1 as minor, 2-4 as moderate, and anything greater as severe. The searches use the rangemap command to create the range field and you can change them to match what you think should be minor, moderate, and severe. Here’s the initial dashboard covering Jam Factor using a red/yellow/green chart and an xyseries search.

Jam Factor

Next, I wanted to show a timechart and a Splunk map for where these Jams are happening in this included dashboard.

Traffic Map

Finally, I retained Ron Naken’s Donut Chart to give you an idea of the distribution of Jam Factors within each city and overall for the cities you are monitoring.

Jam Factor Distribution

Of course, the familiar 3rd party Frogger game is referenced in the app for entertainment purposes. Enjoy the updated app.

Are you a Splunk Answers user? Are you attending Splunk’s 5th Worldwide User Conference next week in Las Vegas? Do you want a way to show your appreciation for other .conf attendees, presenters, vendors, your Splunk University instructors (besides buying them a drink*)?

Introducing SplunKarma, the mobile karma dispenser!

Starting on Saturday, October 4th (the first day of Splunk University), you can visit http://answers.splunk.com/karma from your mobile device and log in with your Splunk Answers credentials. You’ll be given a cache of karma points to use to reward the members of the Splunk Community around you at .conf. All you need to do is find out what their Splunk Answers userID is.

If you’re speaking at .conf, tell your session audience your Splunk Answers username so they can show their appreciation!

Karma: the perfect gift for every occasion

*Why not both?

In less then a week, .conf2014 kicks off at the MGM Grand in Las Vegas. As in past years, there won’t only be tons of great keynotes, sessions and training for the entire Splunk community, but also plenty of things tailored just for developers.

• Once again, Splunk University starts off the week with hands-on training, including an intense Splunk App Developer Bootcamp
• This year we’re introducing the Splunk Dev Lounge, a dedicated space for hacking on Splunk throughout the conference. All throughout the week, you’ll find members of the engineering and evangelist teams ready to answer any question or guide you in the right direction. We’ll also have chalk talk sessions (heavy on code, light on slides) led by Splunkers as well as partners like Auth0 (maybe you read about them in USA Today) and community developers like Rich and Erica who won the Splunk App Contest last year with their Splunk for your Car App. Whether you’re new to Splunk and want to dig in at your own pace or a Splunk Ninja looking to pick up new skills building Splunk Apps, working with the SDKs or extending Splunk with modular inputs and custom search commands, the Splunk Dev Lounge is the place to be at .conf2014.
• We have another full docket of sessions, covering everything from Splunk for .NET developers (led by community luminary and Splunker Glenn Block) to Splunking the Java Virtual Machine (led by Splunk Worldwide Evangelist Damien Dallimore) to building Splunk Apps to DevOps topics.
• We’ll also have a Splunk for Developers booth at the Splunk Apps Showcase, with Splunkers on hand to show you the latest and greatest in building Splunk Apps, integrating Splunk with the REST API and SDKs and using Splunk to monitor App Dev and DevOps processes. Stop by to see what’s new and find out how you can do more as a developer with Splunk.

Throw in the great networking and socializing, and .conf2014 looks to be the biggest and best .conf yet. See you in Vegas!

It has been a busy few weeks for Splunk EMEA with eight SplunkLive events in Cape Town, Johannesburg, Frankfurt, Vienna, Oslo, Copenhagen, Stockholm and Amsterdam. There have been close to a thousand people hearing some great customer stories about how organisations use Splunk to get operational intelligence across a huge range of different industries. I was lucky enough to be at six of them and thought it would be worth sharing some of the stories from across the region.

It has been a bit of a flying visit to each country with the usual plane-airport-taxi-hotel-presentation-taxi-airport-plane but the range of use cases for Splunk, the many different industries and the different kinds of value the organisations are getting from their data made for a really busy but rewarding few weeks.

South Africa

We started the customer presentations with Rudi Pretorius who has been a Chief Information Security Officer for a number of the leading FS organisations in South Africa. He explained how to use Splunk to “go beyond a SIEM” and the African perspective on having to secure and operate mission critical financial services systems. It was interesting to see Hadoop as one of the data sources they managed and monitored with Splunk. It is always fascinating to see what customers say about the Splunk products and the benefits they get from it – everyone has a different story. To quote Rudi, “the biggest benefit of Splunk is the ease of customization to get the intelligence you really need”. He ended his presentation with a question and challenge to the audience: “Talk to the business – ask them what they want from their data”.

Next up was Paul Gilowey, a Foundation Technology Specialist from Santam, one of the biggest life insurer providers in South Africa. Paul gave one of the best presentations I’ve seen on logging strategy, how to get it right, how to get developer buy in and how to find the right champion to turn logging into something with business value. Based on his experience, Paul’s written the 8 steps to a successful Splunk implementation – it is well worth the read. It was interesting to hear about the use of Splunk in life insurance together with the popular insurance package Guidewire.  You can see Paul’s presentation on Slideshare here – http://www.slideshare.net/paulcgt/sustainable-logging-splunklive-2014

Copenhagen

Next up was a tour of the Nordics. We started our first ever SplunkLive Copenhagen with a great turn out and some brilliant customer stories. We were lucky enough to have Tobias Gustavsson, Scrum Master from the DevOps team of Sweden’s largest fashion retailer, talking about how they use Splunk for online retail. He spoke about who uses Splunk:

• Development – incident investigation & product testing
• Business analysts – visibility into transaction volumes and customer intelligence
• Customer support – targeting customers who’ve had issues with apology/voucher
• Operations – web performance monitoring & incident investigation

The business team at the retailer use insights from Splunk to analyse shopping cart abandonment and make decisions on how to build and improve the site, manage stock levels etc.

Finally, Tobias explained how the operational intelligence they achieved with Splunk gave them an ROI within five minutes:

After Tobias, we had Peter Almqvist, Head of IT Operations and Per Hamrin, Systems Developer from Avanza Bank present. Avanza’s customers have been voted “most satisfied savers in Sweden” for the last four years and are also number one in stock market transactions in Sweden. Their online, multi-channel banking platform is core to their business and they recently moved from a traditional three-tier architecture to an in-memory computing model to be able to deal with their growth, increased customer demand and the need to deliver leading customer experience. They use Splunk for troubleshooting, business analytics around new product/feature adoption and availability/performance monitoring. Since the adoption of Splunk, Avanza have benefitted from improved availability (core to customer satisfaction), the ability to solve problems twice as fast, reduced development time thanks to DevOps insight and an enhanced customer support function.

Last, but by no means least, we had Maersk, one of the world’s largest shipping and logistic companies, as well as oil and gas. We were privileged to have Carsten Neubert, Head of Maersk’s Information Security Monitoring Center. Maersk was faced with the challenge of an increasing amount of security data, new kinds of unstructured security information and the need to cover all their individual business units. They take a three-pronged approach to security monitoring – external monitoring, security device monitoring and monitoring of threat feeds. They use Splunk to identify notable security events but also to assess the impact of these incidents and what the response should be. Through the use of Splunk they can now correlate across many types of security device. To quote Carsten, “before Splunk, any correlation across our infrastructure had to be done manually. It took a long time. That time has now been reduced to nothing.” The Maersk CISOs have been empowered through the use of Splunk dashboards and have better insight into the business’ security posture through access to their own data via a dedicated console.

Oslo

On next to Oslo, where last year I confessed my love of the band A-ha (when I was growing up – I’m obviously much cooler now…). First on stage was Ruben Olsen from a major Norwegian financial institution. They have over 2 million private customers and 200,000 corporate customers. Their online banking services have over a million users. When they went live in 2011, from their extensive application portfolio, they were generating 1,500,000,000 log events per day (that’s a lot of zeros) from over 2000 log files. They had a lot of data in a lot of different places. They use Splunk to:

• Enhance software quality
• Search and monitor  their machine data in real time
• Investigate incidents
• Monitor Application Servers
• Improve QA before deployment to production

There was an interesting footnote to the presentation about how to best work with an outsourcer and make sure you keep the operation of Splunk under your control but also work collaboratively with the service provider to make sure they use Splunk for their operations. You can find Ruben’s presentation on Slideshare here.

Carrying on the theme of insurance and Financial Services, next up was Rune Hellem, Senior IT Operations Consultant from KLP, which is Norway’s largest life insurance company. KLP started using Splunk like a lot of customers – with troubleshooting. They then expanded this to more proactive monitoring of their IT estate. Finally their use has grown and they are now looking at how Splunk could be used for auditing and compliance. KLP are taking data from Websphere, Filenet, IBM Process Server, WebLogic, JIRA, .Net servers and their content management system. Both development and IT ops use Splunk and the power of the Splunk alerting capabilities meant they could save money and reduce complexity by retiring other monitoring systems.

Some of the benefits that KLP have got from using Splunk include reducing incident troubleshooting by at least an hour, reducing the impact of any incidents on users and the bottom line, alerts now notify CSC if there’s an issue and the visibility they have now means they aren’t blamed for issues that aren’t their responsibility. One of the top tips they gave the audience was particularly interesting – “always refer to Splunk when sharing information”. We see this a lot with Splunk – it does provide a great “system of record” of what is actually happening (or has happened) and it is very easy to share and use the Splunk search language to collaborate over any particular issues.

Stockholm

Last in the mini Nordic tour was Stockholm, one of my favourite cities. We had Avanza Bank and Tobias speak for us again and they were joined by Magnus Norling from a leading online gaming company. The big challenge Magnus’ organisation faced was a lack of visibility. They started using Splunk for PCI compliance but quickly grew their adoption of Splunk to deliver analytical dashboards and proactive monitoring as part of their 24/7 NOC. They manage around 1,200,000,000 events every day in Splunk. Magnus described their situation with an entertaining quote from “The Art of War”:

“Superior commanders succeed in situations where ordinary people fail because they obtain more timely information and use it more quickly.”

Magnus then went on to explain how operational intelligence is key to driving real-time insight from machine and business data and how this is critical in making faster and smarter decisions across an organisation.

Amsterdam

This week started with the final stage of the tour to Amsterdam. The first presenter was Bas Zimmermann, a Technical Project Leader with Stedin, who was talking about how they use Splunk for renewable energy, smart metering and Internet of Things data from sensors. It was one of the first presentations where I’d seen a customer really tackled the challenge of combining IoT data, business data and security information to give a complete view of operational intelligence around sensor data. Through the use of Splunk and the machine data, Stedin’s IT team were able to better collaborate with the business thanks to the insights, dashboards and data correlation they could get from the different silos of information.

After Bas was Gert Kremer, a Mission Critical Engineer with Schuberg Philis, a leading managed service and cloud provider in The Netherlands. They are using Splunk as part of their multi-tenant managed service offering. They faced a number of challenges supporting their growing customer base including a lack of flexibility and adaptability in home grown tools, a growing number of sources of information they needed to analyse issues, increasing data volumes, a challenge around ensuring security and the need to centralise monitoring. To add to these challenges, they needed to ensure this service across two data centres that are active/active – they are using the multi-sire clustering features of Splunk 6.1 to good effect. The cloud solution deployment also needed to be automated through the use of configuration management tools such as Chef and customers needed self-service access through a portal.

Schuberg Philis have a built a highly effective architecture to deliver their managed cloud offerings. Splunk plays a key part in assuring the quality of service and making sure that some of the region’s biggest customers get a high quality managed cloud service.

Many thanks to all the awesome customer speakers. Apologies for the length of the post if you’ve got this far – there are a lot of customer stories to tell!

It is a few days back in the UK and then off to Las Vegas for .conf2014 – keep an eye out next week for some very, very exciting news and even more customer stories. If you’re going, then see you there.

As always – thanks for reading….

In the past, IT professionals have had to maintain specialized and expensive tools to monitor their mainframes. Mainframe operational insights are important but even more so in the context of the rest of the infrastructure and application performance and operational data. Now getting these insights is only a few steps away – I am very excited to announce Syncsort’s Ironstream, the latest addition to our Splunk apps.

Ironstream enables our and Syncsort’s joint customers to collect, visualize and report on mainframe log data in Splunk Enterprise and Splunk Cloud. This rich source of mainframe machine data includes z/OS log files such as syslog, various SMF records, WebSphere Log4j and more.

There are many benefits of analyzing mainframe data alongside all other technology data sources in Splunk software. With all your data in one place, you can dramatically reduce incident resolution times, proactively monitor both mainframe and distributed systems, improve applications delivery and performance with critical insights from underlying infrastructure, and enhance security.

You can also start getting end-to-end operational visibility into critical mainframe events, troubleshoot failed transactions, trace transactions across your entire enterprise infrastructure (which now includes mainframes), understand usage patterns for capacity planning, audit mainframe user logins for change management or identify security incidents… to name just a few.

For examples of what is possible with Ironstream, check out this video or read Syncsort’s blog. Also, don’t miss our Fifth Annual Worldwide Splunk Users’ Conference  in Las Vegas, Oct. 6-9, where you can engage in discussion with Splunk and Syncsort experts and see a live Ironstream demo. In the meantime, just download Ironstream here and start Splunking!