Tuesday, November 12, 2019

Create vCenter Graph

Import vCenter infrastructure into a knowledge graph using Neo4j

Yes, I could have directly queried the Vmware WebAPI, but dealing with self-signed certificates and discovering all the API queries would have been a LOT of work.  RVTools conveniently already gathers ALL the data I'm looking for and exports it into a single Excel file, which makes this process quite a bit easier.

When complete this process will create the following database schema in your neo4j database:


Known Issues
    • Only tested against vCenter clusters (not standalone vsphere host output)
    • The script only builds Standard vSwitch and ports/portgroups.
      distributed virtual switches and ports ARE present in the .xls data export, but the .cypher will need modifications to properly map DV objects.

Installation: Steps (powershell)

Login using the account you intend to use (particularly if scheduling for automation) 
Now download the script files to run the veeam data collector from the github repositories

cd "$env:programfiles\blue net inc\graph-commit"
.\update-modules.ps1 -gitrepo pdrangeid/vmware-graph -gitfile refresh-vmware.cypher

If this is the first time using your neo4j database with my scripts, you will need to identify your Neo4j server location and provide credentials. This cmdlet will also verify you have the DotNET neo4j driver installed (The set-regcredentials cmdlet can install it automatically for you using the nuget package manager)
.\set-regcredentials.ps1 -credname myneo4jserver -n4j

    The prerequisites (Nuget, Neo4J dotNet driver) will be validated and prompted to be installed if missing.  Once complete it will validate connectivity to your neo4j database instance.  A successful result should look like this:

    First let's generate your output file from rvtools.
    The example below assumes we will use passthru authentication for the vCenter server.  Review the RVTools documentation for specifying credentials.
    The resulting excel document will be placed in the import subfolder within the neo4j installation path (adjust this for your environment)

    [string] $RVToolsPathexe = ${env:ProgramFiles(x86)}+"\Robware\RVTools\RVTools.exe"
    $Arguments = " -passthroughAuth -s fqdn.yourvcenterserver.com -c ExportAll2xlsx -d c:\neo4j-community-3.5.12\import 
    -f fqdn.yourvcenterserver.com.xlsx"
    $Process = Start-Process -FilePath $RVToolsPathExe -ArgumentList $Arguments -NoNewWindow -Wait

    If all went well you should have your vcenter environment exported into the excel document in c:\neo4j-community-3.5.12\import

    Now we want to run the import process to ingest the data into the graph.

    The $findstring variable is used to perform a find/replace the placeholder (in the .cypher script you downloaded earlier) for the path/file to your excel document.

    Replace the 'neo4jserver' with the name of the neo4j datasource credential you used with the set-regcredentials.ps1 earlier. 

    cd "$env:programfiles\blue net inc\graph-commit"
    $scriptpath = -join ($env:ProgramFiles,"\blue net inc\graph-commit\get-cypher-results.ps1")
    $csp=$(-join ($env:programfiles,"\blue net inc\graph-commit\refresh-vmware.cypher"))
    $result = . $scriptPath -Datasource 'myneo4jserver' -cypherscript $csp -logging 'myneo4jserver' -findrep $findstring

    A successful import will cycle through the transactions and give you log queries to validate:

    Use the Neo4j browser: http://your-neo4jserver:7474Login with your credentials
    Review the cypher logs (run the log queries that were output from the script execution above)
    Review the VMware data that was imported.Here are some sample cypher queries that will present an explorable graph:

    // SHOW vcenter, datacenter, cluster, folders and resource groups:
    MATCH (vc:Vcenterserver)
    MATCH (vc)--(vdc:Vspheredatacenter)
    MATCH (vc)--(vcc:Vcentercluster)
    WITH *,'/'+vdc.name as startpath
    OPTIONAL MATCH (vf:Vfolder) where vf.path starts with startpath
    OPTIONAL MATCH (vrp:Vresourcepool) where vrp.path starts with startpath
    WITH *
    MATCH (vm:Virtualmachine) where (vm)--(vf) or (vm)--(vrp) or (vm)--(vcc) or (vm)--(vdc)
    return vc,vdc,vcc,vf,vrp,vm

    DNS and NTP query:
    // SHOW vSphereHosts DNS,NTP, and vCenter relationships MATCH (vh:Vspherehost)
    OPTIONAL MATCH (vh)--(ds:Dnsserver)
    OPTIONAL MATCH (vh)--(ns:Ntpserver)
    OPTIONAL MATCH (vh)--(vc:Vcenterserver)
    return vh,ds,ns,vc

    vSphere Hosts and datastores:

    // SHOW vSpherehost datastores, types, and vcenter
    MATCH (vh:Vspherehost)
    OPTIONAL MATCH (vh)--(ds:Vdatastore)
    OPTIONAL MATCH (ds)--(dst:Vdatastoretype)
    OPTIONAL MATCH (vh)--(vc:Vcenterserver)
    return vh,ds,dst,vc

    vSwitch, Portgroups, and Loadbalancing policies:

    // SHOW vSwitch portgroups, and lbpolicies
    MATCH (vh:Vspherehost)
    OPTIONAL MATCH (vh)--(vs:Vswitch)
    OPTIONAL MATCH (vs)--(vlbp:Vlbpolicy)
    OPTIONAL MATCH (vpg:Vportgroup)
    OPTIONAL MATCH (vhpg:Vhostportgroup)--(vpg)
    RETURN vh,vs,vpg,vhpg,vlbp

    No comments:

    Post a Comment

    Have a comment? Would love to hear it!