Splunk with PowerShell? Yes, Please

Do you manage Windows servers? If the answer is yes, then the likelihood is that you utilize PowerShell in your daily operations. As many know, PowerShell is an extraordinarily powerful shell command language that Microsoft invented to manage their most complex server applications. Exchange, SharePoint, Lync, SQL Server and Active Directory can all be managed through PowerShell; and that’s just the start. The Splunk App for Exchange and the Splunk App for Active Directory both use this facility to get inventory and usage information from the depths of the systems.

But it isn’t easy. Scripted inputs are, well, expensive. Firstly, you have to wrap the PowerShell executable inside a CMD batch file. When it executes, you are running a CMD prompt plus a full PowerShell environment. You are incurring this start up cost whenever the scripted input fires, and the start up cost can be significant. In addition, you have to cook the output into events yourself and deal with a lot of the scaffolding. Just take a look at the Splunk App for Exchange for an example of this. A lot of the code inside the PowerShell scripts is dealing with output formatting.

When Splunk 5.0 was released, it included a feature that will be a major advantage for this sort of thing – Modular Inputs. A modular input is a long-running data input that starts when the Splunk server starts and remains continually running. We can use this feature to remove the double process and the start-up cost. Instead of executing a batch file that runs PowerShell, we can instantiate a PowerShell run-time at server initiation and then run the scripts within that run-time. In addition, the modular input can handle all the scaffolding for us. PowerShell is an object-passing command language. Instead of doing all the output processing within the script, we can do it within the modular input and just pass objects back to the modular input.

Thus was born the Splunk Add-on for PowerShell. You can download it today for free from Splunkbase. It only works on Windows servers and we recommend you deploy it to universal forwarders. You need to install the .NET Framework 4.5 and PowerShell 3.0 first.

What can you do with it? Well, you can do a lot with it. It can replace WMI, for example. One of the things that we often use WMI for is to retrieve services. Try this input stanza to get the same effect:

[powershell://Running_Services]
script = Get-Service | Select Name,DisplayName,Status
schedule = 0 */5 * ? * *

You will note that we have our PowerShell script embedded in the inputs.conf. You can run this one-liner from the PowerShell prompt on your Windows host. We also have a schedule. The schedule takes a cron schedule (just like scripted inputs) and not an interval (unlike scripted inputs). This allows you to schedule your inventory scripts for the early morning.

Something more complex? We’ve been playing around with SQL Server and one of the things we want to do is to get an inventory of the SQL Databases. This is more complex. We place the code in a script:

[Reflection.Assembly]::LoadWithPartialName(‘Microsoft.SqlServer.Smo’)

function Get-SQLInstances {
	(Get-ItemProperty ‘HKLM:\SOFTWARE\Microsoft\Microsoft SQL Server’).InstalledInstances |
	Select @{n=’Cluster’;e={$env:ComputerName}}, `
		@{n=’Name’;e={$_}}, `
		@{n=’State’;e={‘Active’}}, `
		@{n=’VirtualServerName’;e={$env:ComputerName}}, `
		@{n=’ServerInstance’;e={{“{0}\{1}” –f $env:ComputerName,$_)}}, `
		@{n=’Node’;e={$env:ComputerName}}
}

function Get-SQLDatabases {
	[CmdletBinding()]
	Param([Parameter(Mandatory=$True,ValueFromPipeline=$True,ValueFromPipelineByPropertyName=$True)]
		$Instance
	)
	PROCESS {
		Foreach ($i in $Instance) {
			$s = New-Object(‘Microsoft.SqlServer.Management.Smo.Server’) $i.VirtualServerName
			$s.Databases | %{ Write-Output $_ }
		}
	}
}

Get-SQLInstances | Get-SQLDatabases | Select ID,Name,Owner,UserName,Version

Obviously, this is not a straight-forward script, nor is it complete – it does not deal with clusters and it only outputs a handful of the many parameters that are available. You can place this in the bin directory of a Splunk app, and then add the following inputs.conf stanza:

[powershell://My_DBInventory]
script = . “$SplunkHome\etc\apps\MyApp\bin\dbinventory.ps1”
schedule = 0 30 3 ? * *

Now the script will execute, outputting the list of databases per host to the index at 3:30am every morning.

The Splunk Add-on for PowerShell isn’t good for everything. One of the things it isn’t good for is long-running scripts. This is primarily because the output is not sent to the Splunk server until the end of the script when the modular input receives the generated objects. Instead of a long-running script (which is best suited for a scripted input still), we use a short-lived script and keep state. For example, let’s say we want to output the windows server boot time every 24 hours or when it reboots, whichever is first, we can use this script:

$State = Import-LocalStorage “Boot.xml” –Module “MyApp” -DefaultValue (New-Object PSObject –Property @{PE=[DateTime]::MinValue})

$wmi = gwmi –Class Win32_OperatingSystem –Property LastBootUpTime
$lastboot = $wmi.ConvertToDateTime($wmi.LastBootUpTime)
if ($State.PE.AddHours(24) –ge [DateTime]::Now –and $State.PE –ge $lastboot)
{
	return
}

$State.PE = [DateTime]::Now
$State | Export-LocalStorage “Boot.xml” –Module “MyApp”

$wmi

In this case, we only output an object at the prescribed time. We use a set of new cmdlets to import and export the state. The state is simply an object and we can put whatever information we want in that object, as long as it can be serialized. Now our inputs.conf stanza looks like this:

[powershell://LastBoot]
script = . “$SplunkHome\etc\apps\MyApp\bin\lastboot.ps1”
schedule = 0 * * ? * *

This executes the script once a minute, allowing us a 1-minute resolution on the boot time.

With the Splunk Add-on for PowerShell, you get the full power of the Windows .NET framework and PowerShell v3.0 for generating events. These can be as complex or as simple as you want them to be. Look for future Microsoft apps to take advantage of this new powerful data input.