Cyber Defense

PowerShell Script To Parse nmap XML Output

-

The nmap port scanner can produce XML output of the results of its scanning and OS fingerprinting work. But how can these XML files be conveniently handled from the command line when you want to query and extract specific data or to convert that data into other formats like CSV or HTML files?

PowerShell has great XML handling capabilities, so let's look at a script which takes nmap XML as input and converts that data into PowerShell objects where each object represents a host on the network and the properties of those objects contain the data from the scan. Once we have our scan results as an array of PowerShell objects, it becomes easy to pipe those objects into other cmdlets and scripts. (This script is from the Securing Windows and PowerShell Automation [SEC505] course at SANS.)

Prerequisites

To follow along with the examples below, get the samplescan.xml log file and the parse-nmap.ps1 script from the SEC505 zip file on the Downloads page of this blog. The scripts zip file contains these two files (in the Day1-PowerShell folder) plus many other folders and files. Every script in the zip file is in the public domain, so feel free to do whatever you wish with them. This blog has a number of articles about these scripts.

You also need PowerShell running of course and then set your working directory to the folder where you placed the log file and script. Don't forget that you must give the full path to the script at the command line, and that the current working directory is ".".

Finally, by default PowerShell does not permit scripts to run, so right-click on the PowerShell shortcut in the Start Menu, select Run As Administrator, then execute "set-executionpolicy unrestricted", which means you can now run any script from any source, so don't run scripts you download from blogs!

Examples

To process just one XML log file:

parse-nmap.ps1 samplescan.xml

And then the output will look like the following, but remember, this output is not really text, the output is an array of .NET objects with properties!

FQDN           : srv-4mm1vr3.sans.org
HostName : srv-4mm1vr3
Status : up
IPv4 : 10.54.23.2
IPv6 : <no-ipv6>
MAC : <no-mac>
Ports : open:TCP:135:msrpc
open:TCP:139:netbios-ssn
Services : TCP:135:msrpc:Microsoft Windows RPC <100%-confidence>
OS : Microsoft Windows 7 <95%-accuracy>
Script : <no-script>
FQDN           : wks-88d70.sans.org
HostName : wks-88d70
Status : up
IPv4 : 10.54.23.3
IPv6 : <no-ipv6>
MAC : <no-mac>
Ports : open:TCP:135:msrpc
open:TCP:139:netbios-ssn
Services: : open:TCP:135:msrpc:Microsoft Windows RPC <100%-confidence>
OS : Microsoft Windows Server 2008-R2 <93%-accuracy>
Script : <no-script>
FQDN           : srv-9ryc3.sans.org
HostName : srv-9ryc3
Status : up
IPv4 : 10.54.23.4
IPv6 : <no-ipv6>
MAC : <no-mac>
Ports : open:TCP:135:msrpc
open:TCP:139:netbios-ssn
Services : open:TCP:135:msrpc:Microsoft Windows RPC <100%-confidence>
OS : Microsoft Windows XP SP2 <100%-accuracy>
Script : <no-script>

To process a bunch of XML log files and consolidate their output:

parse-nmap.ps1 *.xml 

# Or, if you prefer piping instead:
dir *.xml | parse-nmap.ps1

To extract the IP addresses and hostnames of the machines fingerprinted as running Windows XP:

parse-nmap.ps1 samplescan.xml | where {$_.OS -like "*Windows XP*"} |
format-table IPv4,HostName,OS

To find the hosts listening on TCP/23 for telnet:

parse-nmap.ps1 samplescan.xml | where {$_.Ports -like "*open:tcp:23*"}

The "-like" operator is for simple wildcard matching, while the "-match" operator is for regular expression matching. You have the full .NET Framework regular expression engine available to you. Also, note that the Ports property is a space-character-delimited list of scanned ports in the format of state:protocol:portnumber:servicename (see the nmap documentation) for each port in the list.

To find the hosts listening on either TCP/80 or TCP/443 or both:

parse-nmap.ps1 samplescan.xml |
where {$_.Ports -match "open:tcp:80|open:tcp:443"}

Export to CSV or HTML (-OutputDelimiter)

The script uses a parameter named "-OutputDelimiter" to separate the strings in the Ports, Services, OS and Script properties. The default is a newline, but you will want to change the delimiter to a space character or some other symbol when exporting to a comma-delimited file or HTML report (whatever is best for how you intend to save, parse or print the data). The default is a newline to make the output easier to read within the shell.

To export the data to a CSV file for grepping, import into a spreadsheet, etc.:

parse-nmap.ps1 samplescan.xml -outputdelimiter " " |
where {$_.Ports -match "open:tcp:80"} |
export-csv weblisteners.csv

Then you could later re-import from the CSV file into a new array, keeping the same properties as before for the sake of further processing:

$data = import-csv weblisteners.csv
$data | where {($_.IPv4 -like "10.57.*") -and ($_.Ports -match "open:tcp:22")}

To export the processed data to an HTML file for viewing in a browser:

parse-nmap.ps1 samplescan.xml -outputdelimiter " " |
select-object IPv4,HostName,OS |
ConvertTo-Html | out-file \serversharereport.html

View RunStats Information (-RunStatsOnly)

The script has a -RunStatsOnly switch which shows general information about the scan instead of information about each scanned host; for example, the runstats output includes the scan's start time, end time, command-line arguments when the scan was run, etc. Keep in mind that different versions of nmap have different XML output formats, so not every field can be shown from every scan file.

Caveats

The script is slower than a C++ binary, of course; for example, on an Intel Core i7 at 3.6GHz, it can process host records in the XML file at a rate of about 900 records/second, which isn't blazing, but hopefully good enough. I'm sure there are other inefficiencies and shortcomings in the script too, so feel free to modify the script as you wish, it's in the public domain (or let me know how to improve it!).

Also, if you are unhappy with using PowerShell, there are great resources out there to parse nmap XML output in Perl or in Python.

To Learn More

Get the latest version of PowerShell (for free) from http://www.microsoft.com/powershell/

Get the nmap scanner (for free) from http://insecure.org (or http://nmap.org) and find the documentation.

Get PowerShell training (not free) as part of the SANS Institute's six-day Securing Windows Course. :-)

[Updated: 9.July.2009, and script edited to increase performance.]

[Updated: 18.July.2009, the -runstatsonly switch added.]

[Updated: 30.Jan.2010, fixed output issue with the -runstatsonly switch.]

[Updated: 23.May.2010, support for nmap script output added.]

[Updated: 1.Nov.2010, fixed bug for FQDN output.]

[Updated: 16.Nov.2010, removed a PowerShell 2.0 dependency]

[Updated: 14.Mar.2014, fixed an FQDN parsing bug.]

[Updated: 6.Jun.2015, improved performance, added the -Verbose switch.]

31 Comments

Posted June 24, 2009 at 6:21 PM | Permalink | Reply

MartinR.

Kick ass, it works great! I wish youd add the rest of the nmap data to the output ofyour script though, like the trace routes

Posted June 24, 2009 at 9:33 PM | Permalink | Reply

Jason Fossen

Hi Martin:
Thank you, I want to maximize performance of the script, but I'll keep your suggestion in mind: if most people would be happier with slower throughput in exchange for more detailed output, then I can change the script ''" and of course you're welcome to modify the script too for your own purposes!

Posted July 9, 2009 at 2:11 PM | Permalink | Reply

CodeRover

Jason, nice script, thank you for sharing!!
When using -D option in NMAP, is that source IP also captured anywhere?
I'd like to include it in the final report, since my scans are often performed from a secondary to a tertiary node(s).

Posted July 9, 2009 at 7:51 PM | Permalink | Reply

Jason Fossen

Hi CodeRover:
It can be extracted from the XML because the command line used to run nmap is in the XML output file. The script doesn't show this, though, since that info is not a part of the host entries. Here's one way to extract the command line:
[xml] $x = gc .scan.xml
$x.nmaprun.args
You could then capture that output and exact the -D option arguments for your report.

Posted July 15, 2009 at 9:43 AM | Permalink | Reply

TravisV

Amazing script! It does everything I needed it to with good results. Thanks for saving me a lot of time and energy.

Posted July 18, 2009 at 5:27 PM | Permalink | Reply

Jason Fossen

Hi Again CodeRover:
I modified the script to include the -runstatsonly switch in order to display command-line arguments and other general information about a scan. This update also puts the code into a function so that it can be dot-sourced, and after Win7 and PowerShell 2.0 have been out for a few months, I'll finish converting it to an "advanced function". Cheers

Posted July 30, 2009 at 5:44 AM | Permalink | Reply

Bobdog

Great script! Thank you very much.

Posted September 21, 2009 at 11:36 PM | Permalink | Reply

Jebediah Webb

Wow thanks for posting this script. I thought I was going to have to write something from scratch but this script did exactly what I needed.

Posted October 20, 2009 at 4:15 PM | Permalink | Reply

CodeRover

Very nice indeed! Now here is my challenge, can the script be modified to test/probe protocol and port connectivity between node2 and node3? In other words, I need something like psexec launching this functionality on node2 to test and log if it can connect to node3. That's my pain right now. I don't see myself using nmap, since it would need deployed across all my systems''

Posted October 28, 2009 at 10:59 PM | Permalink | Reply

wayne

I'm wondering if one can take an nmap xml file, parse it and remove host where All ports scanned were filtered, e.g. "All 1000 scanned ports on are filtered. Then save it back to valid nmap xml format with those hosts removed.
(Tried ''"open, but it still listed the filtered hosts for some reason).

Posted May 4, 2010 at 8:18 PM | Permalink | Reply

Tony

Wow, this script is great. I just ran across this. i usually use the -oX ''"webxml output but was looking for a way to dump the data into a spreadsheet for better data massaging and this script works like a champ! Now if only it would parse the service detection data from a -sV switch'' (can't really decide how I'd want it to display anyway since services and versions would likely get mangled)

Posted May 24, 2010 at 1:56 AM | Permalink | Reply

Jason Fossen

Hi Tony:
The script now supports parsing the service detection data as well as nmap scripts output. It also has a new parameter (-outputdelimiter) to help with exporting those strings to CSV or HTML files. Cheers.

Posted June 7, 2010 at 2:29 AM | Permalink | Reply

al bell

Thanks Jason for this script. Two questions:
1. It seems the output of scripts is not parsed:
Script :
OS: Windows 2000 (Windows 2000 LAN Manager)
Name: WORKGROUPHK190104240
System time: 2010-06-01 22:36:28 UTC+0
2. It would be great to hook this output into a different script to import the data into a SQL database''''

Posted June 9, 2010 at 10:20 PM | Permalink | Reply

Jason Fossen

Hi Al Bell:
Please confirm you've got the latest version of the script (or just download it again) and if it still doesn't work, please e-mail me the problematic XML output file (after sanitizing any IPs or other confidential data).
For SQL import, that can certainly be done, the output objects can be easily parsed. In the Downloads section of this blog you can get the SEC505 zip file, which includes a couple sample PowerShell scripts for ADO database access. You're welcome to modify these for use with SQL Server if you wish.

Posted August 10, 2010 at 8:41 PM | Permalink | Reply

Trenton Ivey

You Just saved me so much time! I have another script I wanted to try and run against all hosts with a few specific ports open I was about to (attempt to) program my own nmap parser in perl to fit my needs, but I think this will work perfectly. I am sure I will use this awesome tool for much more complicated projects in the future. Thanks!

Posted October 26, 2010 at 3:00 AM | Permalink | Reply

Kieran Jacobsen

Hi,
I was using your script to clean up some nmap scans for reporting purposes and found a particular weird and annoying bug. I had a number of hosts that if I performed the scan against the FQDN, there was no hostname or FQDN in the output of the parser script. I also discovered that if the ip address was used, then the hostname and FQDN fields would be properly populated.
The difference in the output of the two scans of the identical box come down to the hostnames element in the XML output. When the hostname was used:
When the ip address was used:
The following line of the script is what reads these entries:
$hostnode.hostnames | foreach-object { $entry.FQDN += $_.hostname.name + " " }
If this is changed to:
$hostnode.hostnames.hostname | ForEach-Object { $entry.FQDN += $_.name + " " }
then both situations will produce some output, in fact the hostname field will be correct, however there will be two entries in the FQDN field.

Posted November 1, 2010 at 7:19 PM | Permalink | Reply

Jason Fossen

Hi Kieran, thanks for the bug report!! I've updated the script. Note that duplicate FQDNs in the output, if any, are removed.

Posted November 22, 2010 at 1:23 PM | Permalink | Reply

DaricSmith

The work wonderfully. The only thing I can't figure out is how to run this as a scheduled task. It won't allow me to pass the arguments properly through a batch file. Any insight would be great.
Thank you.

Posted November 24, 2010 at 2:56 AM | Permalink | Reply

Jason Fossen

Hi Daric: Try scheduling powershell.exe with the path to the script and its arguments inside single quotes, and then surround the quotes with . For more info, see: http://technet.microsoft.com/en-us/library/ee176949.aspx

Posted November 30, 2010 at 8:21 PM | Permalink | Reply

DaricSmith

I found the fix for it. I had to replace anything with a " " around it with just '' ''. I ended up with what I have below. Now I am able to run an nmap scan along with parsing the data all from a scheduled task in windows.
''"''"''"''"''"''"''"''"''"''"''"''"''"''"-
C:WINDOWSsystem32WindowsPowerShellv1.0powershell.exe & 'c:systemscannerparse-nmap.ps1' c:temppingscan.xml | where {$_.os -like ''*Microsoft*'} | select-object IPv4 | export-csv c:tempparseresultswindowssystems.csv
''"''"''"''"''"''"''"''"''"''"''"''"''"''"-
Note: Task scheduler filled out the entire path to the executable. I initially had just "powershell.exe"
Thank you for your help.

Posted October 23, 2013 at 10:13 PM | Permalink | Reply

Giovanni Heward

Here's a quick way to run the script from the command line, without globally modifying the execution policy or unnecessarily elevating privileges.
powershell -ExecutionPolicy Bypass -NoLogo -NoProfile parse-nmap.ps1 output.xml ^| export-csv output.csv
The escaped pipe ( ^| ) is not a mistype.
Enjoy!

Posted March 5, 2015 at 9:12 AM | Permalink | Reply

prasa

Its working perfect for the samplescan.xml, but when I use my own nmap xml output, its not able to parse. I Used nmap -oX filename.xml 192.168.2.1-10
Error:
You cannot call a method on a null-valued expression.
category :InvalidOperation: (Trim:String) []. RuntimeException
please help.

Posted March 10, 2015 at 5:51 PM | Permalink | Reply

Jason Fossen

Hello Prasa:
I've sent you an e-mail, would you be able to send me the XML file?
Cheers, Jason

Posted September 16, 2015 at 2:59 PM | Permalink | Reply

Chris

Great Script Jason,
Quick question. How do I get the script to only parse hosts that are alive. I am working on another script that compares nmap results to what is contained in WSUS and the problem I am having is when I parse the nmap xml file it lists all the addresses and I just want to display the ones that are alive. Any help? Fairly new to PS.

Posted September 24, 2015 at 12:16 AM | Permalink | Reply

Jason Fossen

Hello Chris, try changing the nmap.exe arguments themselves to ping targets first, then, if necessary, filter the output of the parse-nmap script with Where-Object to only return live hosts.

Posted September 18, 2015 at 6:38 AM | Permalink | Reply

Henrique Souza

How can I make the port column to output only the port number?

Posted September 24, 2015 at 12:15 AM | Permalink | Reply

Jason Fossen

Hello Henrique, you'd have to edit the script or parse the output strings, which shouldn't be too bad (see the -OutputDelimeter parameter).

Posted February 10, 2016 at 10:01 PM | Permalink | Reply

Chris

Hey Jason,
Great script I have been using it for a while now to parse my nmap scans. One note. The parser does not properly check the status values for each host.
I believe I have the most current version of this script and in this section where it tries to pull the status of the host it will always return ""
foreach ($hostnode in $xmldoc.nmaprun.host)
{
# Init some variables, with $entry being the custom object for each .
$service = " " #service needs to be a single space.
$entry = ($entry = " " | select-object HostName, FullDomainName, Status, IPAddress, IPv6, MAC, Ports, Services, OS, Script)
''" Part with issues
What I ended up doing was modifying the above 2 lines relating to $entry.Status
Changed it to this:
if ($hostnode.status.state -ne $null -and $hostnode.status.state.Length -ne 0) {$entry.Status = $hostnode.status.state}
if ($hostnode.status.state.Length -lt 2) { $entry.Status ="" }
Which now produces the status of the IP.

Posted February 27, 2016 at 4:37 PM | Permalink | Reply

Jason Fossen

Hi Chris, thanks, I've updated the script!

Posted August 9, 2016 at 12:41 PM | Permalink | Reply

Daniel

Would be great if it could take hostname and FQDN from the script smb-os-discovery if it's there.

Posted February 14, 2017 at 1:46 AM | Permalink | Reply

Rod

Hi, Great script!
I need the vendor information associated with the Mac Address. In my nmap xml it looks like this:
I think your script is truncating this info? Do you know how I can use your script to get this info for each host?
Thanks,
Rod

Post a Comment






Captcha


* Indicates a required field.