IPSEC, SCEP, NDES and other stuff you probably wish you never heard of

I’m in an environment now where I have to proof-of-concept complicated/large ideas for environments to prove their feasibility.

The latest project:

  • High performance IPSEC for site to site encryption
  • BGP over the IPSEC tunnels
  • Use of certificates for authentication
  • Automatic enrollment/renewal of said certificates
  • Verification of certificates via OCSP

Well then, lets get started.

IPSEC and BGP aren’t much new and are pretty straight forward.  However, most places I’ve ever been have used Pre-Shared Keys for their IPSEC as using certificates means that you have some sort of CA infrastructure and can require a lot of overhead.  Getting static certificates to work is one thing, but getting them automatically created and renewed is another thing entirely.  Then there’s OCSP, again typically not used because of the infrastructure that is required to support it.

First, the topology:

We have the following:

  • a vRouter (vyOS, which deserves its own post for how awesome it is)
  • a Windows 2012 server running AD and a compliment of Certificate services (which we will cover)
  • two virtual appliances running IPSEC that are running eBGP between the vRouter (and each other)

Phase 0 - What is in (and out) of scope for this post

This post isn’t intended as an all inclusive guide to setting up the underlying environment, but rather configuring the pieces that exist on top of said environment.

A quick summary of the underlying environment:

  • 2 Dell R810 ESXi 6 servers managed by VCSA 6
  • Virtual router provided by vyOS
  • The ‘Internet’ as depicted in the diagram is an upstream router that has access to other lab networks (as well as the upstream Internet)
  • All of the point to point networks are vSwitches (actually a dvSwitch) with a corresponding backend VLAN on my lab switch

Phase 1 - Setting up the underlying network architecture (and making sure it works)

First lets make sure that we get the vRouter properly configured, here is what it should look like when finished.

interfaces {
    ethernet eth0 {
        address 192.168.254.60/24
        duplex auto
        hw-id 00:50:56:a4:63:ad
        smp_affinity auto
        speed auto
    }
    ethernet eth1 {
        address 10.106.0.1/24
        duplex auto
        hw-id 00:50:56:a4:fb:13
        smp_affinity auto
        speed auto
    }
    ethernet eth2 {
        address 10.106.1.1/24
    }
    ethernet eth3 {
        address 10.106.2.1/24
    }
    loopback lo {
    }
}
protocols {
    bgp 65600 {
        neighbor 10.106.1.11 {
            remote-as 65601
            soft-reconfiguration {
                inbound
            }
        }
        neighbor 10.106.2.11 {
            remote-as 65602
            soft-reconfiguration {
                inbound
            }
        }
        neighbor 192.168.254.1 {
            remote-as 65001
        }
        network 10.106.0.0/16 {
        }
    }
    static {
        route 10.106.0.0/16 {
            blackhole {
            }
        }
    }
}

Now that we have that portion, lets go ahead and configure the A10 devices (IPSEC-A and IPSEC-B)

IPSEC-A

multi-config enable
!
ip dns primary 10.106.0.2
!
hostname IPSEC-A
!
timezone America/Los_Angeles
!
ntp server 10.0.1.1
!
interface management
  ip address 10.106.0.11 255.255.255.0
  ip default-gateway 10.106.0.1
  enable
!
interface ethernet 1
  enable
  ip address 10.106.1.11 255.255.255.0
!
interface ethernet 2
  enable
  ip address 10.106.3.1 255.255.255.0
!
interface tunnel 1
  ip address 10.106.99.1 255.255.255.0
!
!
ip route 0.0.0.0 /0 10.106.1.1
!
!
vpn ike-gateway IKE_GATEWAY_IPSEC_B
  ike-version v2
  auth-method pre-share-key 12345
  encryption 3des hash sha1
  local-address ip 10.106.1.11
  remote-address ip 10.106.2.11
!
vpn ipsec IPSEC_IPSEC_B
  ike-gateway IKE_GATEWAY_IPSEC_B
  bind tunnel 1 10.106.99.2
  dh-group 1
  encryption 3des hash sha1
!
logging console information
!
bgp extended-asn-cap
!
router bgp 65601
  network 10.106.3.0 mask 255.255.255.0
  neighbor 10.106.99.2 remote-as 65602
  neighbor 10.106.99.2 soft-reconfiguration inbound
  neighbor 10.106.1.1 remote-as 65600
  neighbor 10.106.1.1 soft-reconfiguration inbound
!
end

IPSEC-B

!
multi-config enable
!
ip dns primary 10.106.0.2
!
hostname IPSEC-B
!
timezone America/Los_Angeles
!
ntp server 10.0.1.1
  prefer
!
interface management
  ip address 10.106.0.12 255.255.255.0
  ip default-gateway 10.106.0.1
!
interface ethernet 1
  enable
  ip address 10.106.2.11 255.255.255.0
!
interface ethernet 2
  enable
  ip address 10.106.4.1 255.255.255.0
!
interface tunnel 1
  ip address 10.106.99.2 255.255.255.0
!
!
ip route 0.0.0.0 /0 10.106.2.1
!
!
vpn ike-gateway IKE_GATEWAY_IPSEC_A
  ike-version v2
  auth-method pre-share-key 12345
   encryption 3des hash sha1
  local-address ip 10.106.2.11
  remote-address ip 10.106.1.11
!
vpn ipsec IPSEC_IPSEC_A
  ike-gateway IKE_GATEWAY_IPSEC_A
  bind tunnel 1 10.106.99.1
  dh-group 1
  encryption 3des hash sha1
!
vpn revocation IPSEC-REVOCATION
  ca IPSEC-CA
  ocsp pri IPSEC_OCSP
!
logging console information
!
bgp extended-asn-cap
!
router bgp 65602
  network 10.106.4.0 mask 255.255.255.0
  neighbor 10.106.99.1 remote-as 65601
  neighbor 10.106.99.1 soft-reconfiguration inbound
  neighbor 10.106.2.1 remote-as 65600
  neighbor 10.106.2.1 soft-reconfiguration inbound
!
end

Now that we have that configured, lets make sure that IPSEC is working, and that BGP is working.

From here we can see that both A10 devices have peered with the Lab6 vRouter and are both advertising two networks.

vyos@Lab6-vRouter:~$ sh ip bgp sum
BGP router identifier 192.168.254.60, local AS number 65600
IPv4 Unicast - max multipaths: ebgp 1 ibgp 1
RIB entries 24, using 2304 bytes of memory
Peers 3, using 13 KiB of memory

Neighbor        V    AS MsgRcvd MsgSent   TblVer  InQ OutQ Up/Down  State/PfxRcd
10.106.1.11     4 65601   12453   10668        0    0    0 2d18h25m        2
10.106.2.11     4 65602   12430   10653        0    0    0 2d18h25m        2
192.168.254.1   4 65001   12513   11365        0    0    0 01w0d21h       11

Total number of neighbors 3

And if we take a deeper look:

vyos@Lab6-vRouter:~$ sh ip bgp neighbors 10.106.1.11 received-routes
BGP table version is 0, local router ID is 192.168.254.60
Status codes: s suppressed, d damped, h history, * valid, > best, i - internal,
              r RIB-failure, S Stale, R Removed
Origin codes: i - IGP, e - EGP, ? - incomplete

   Network          Next Hop            Metric LocPrf Weight Path
*> 10.106.3.0/24    10.106.1.11                            0 65601 i
*> 10.106.4.0/24    10.106.1.11                            0 65601 65602 i

Total number of prefixes 2

vyos@Lab6-vRouter:~$ sh ip bgp neighbors 10.106.2.11 received-routes
BGP table version is 0, local router ID is 192.168.254.60
Status codes: s suppressed, d damped, h history, * valid, > best, i - internal,
              r RIB-failure, S Stale, R Removed
Origin codes: i - IGP, e - EGP, ? - incomplete

   Network          Next Hop            Metric LocPrf Weight Path
*> 10.106.3.0/24    10.106.2.11                            0 65602 65601 i
*> 10.106.4.0/24    10.106.2.11                            0 65602 i

Total number of prefixes 2


Based on this, we can determine the following:

  1. BGP is working between the Lab6 vRouter and both A10 Devices
  2. IPSEC is working between both A10 devices
    1. We know this because of the path of the learned networks from the A10 peer, which could only be learned over the IPSEC tunnel

But still, its probably worth confirming IPSEC on the A10 devices anyway:

On IPSEC-A

IPSEC-A#sh vpn
IKE Gateway total:   1
IPsec total:         1

IKE SA total:        1
IPsec SA total:      1

IPsec mode: software
IPsec passthrough traffic

CPU 0 processed 0 packets
IPSEC-A#sh vpn ike-sa

Name             Local/Peer IP  Enc/Hash  Lifetime  Status
---------------------------------------------------------------
IKE_GATEWAY_...  10.106.1.11    3des        28687s  Established
                 10.106.2.11    sha1
---------------------------------------------------------------
Total: 1
IPSEC-A#sh vpn ipsec-sa
Gateway:IKE_GATEWAY_IPSEC_B    Local IP:10.106.1.11    Remote IP:10.106.2.11
Name           Selectors  In/Out SPI  Mode/xform  Time/Bytes
------------------------------------------------------------
IPSEC_IPSEC_B  0.0.0.0/0  0x620955f6  esp-tunnel      14218s
               0.0.0.0/0  0x3e039678  3des-sha1    unlimited

And on IPSEC-B

IPSEC-B#sh vpn
IKE Gateway total:   1
IPsec total:         1

IKE SA total:        1
IPsec SA total:      1

IPsec mode: software
IPsec passthrough traffic

CPU 0 processed 0 packets
IPSEC-B#sh vpn ike-sa

Name             Local/Peer IP  Enc/Hash  Lifetime  Status
---------------------------------------------------------------
IKE_GATEWAY_...  10.106.2.11    3des        28639s  Established
                 10.106.1.11    sha1
---------------------------------------------------------------
Total: 1
IPSEC-B#sh vpn ipsec-sa
Gateway:IKE_GATEWAY_IPSEC_A    Local IP:10.106.2.11    Remote IP:10.106.1.11
Name           Selectors  In/Out SPI  Mode/xform  Time/Bytes
------------------------------------------------------------
IPSEC_IPSEC_A  0.0.0.0/0  0x3e039678  esp-tunnel      14171s
               0.0.0.0/0  0x620955f6  3des-sha1    unlimited

Furthermore, lets verify that we are learning routes via BGP over the IPSEC tunnel:

IPSEC-A

IPSEC-A#sh ip route
Codes: K - kernel, C - connected, S - static, R - RIP, B - BGP
       O - OSPF, IA - OSPF inter area
       N1 - OSPF NSSA external type 1, N2 - OSPF NSSA external type 2
       E1 - OSPF external type 1, E2 - OSPF external type 2
       i - IS-IS, L1 - IS-IS level-1, L2 - IS-IS level-2, ia - IS-IS inter area
       * - candidate default

Gateway of last resort is 10.106.1.1 to network 0.0.0.0

S*      0.0.0.0/0 [1/0] via 10.106.1.1, ethernet 1
B       10.106.0.0/16 [20/0] via 10.106.1.1, ethernet 1, 2d18h33m
C       10.106.1.0/24 is directly connected, ethernet 1
C       10.106.3.0/24 is directly connected, ethernet 2
B       10.106.4.0/24 [20/0] via 10.106.99.2, tunnel 1, 1d15h51m
C       10.106.99.0/24 is directly connected, tunnel 1

and on IPSEC-B

IPSEC-B#sh ip route
Codes: K - kernel, C - connected, S - static, R - RIP, B - BGP
       O - OSPF, IA - OSPF inter area
       N1 - OSPF NSSA external type 1, N2 - OSPF NSSA external type 2
       E1 - OSPF external type 1, E2 - OSPF external type 2
       i - IS-IS, L1 - IS-IS level-1, L2 - IS-IS level-2, ia - IS-IS inter area
       * - candidate default

Gateway of last resort is 10.106.2.1 to network 0.0.0.0

S*      0.0.0.0/0 [1/0] via 10.106.2.1, ethernet 1
B       10.106.0.0/16 [20/0] via 10.106.2.1, ethernet 1, 2d18h32m
C       10.106.2.0/24 is directly connected, ethernet 1
B       10.106.3.0/24 [20/0] via 10.106.99.1, tunnel 1, 1d15h50m
C       10.106.4.0/24 is directly connected, ethernet 2
C       10.106.99.0/24 is directly connected, tunnel 1

So, IPSEC is up, and BGP is working perfectly. Lets move on to Phase 2.

Phase 2 - Deploying Windows and Active Directory

I’m going to give you the cliff-notes version here, because you either already have AD installed and running, or you can go to some MSFT blog to figure it out.

So…  Install your Windows 2012 server, deploy Active Directory, and make sure that it is reachable by your lab environment (I attached mine to the ‘management’ network.

Phase 3 - Configuring Certificate Services for Windows 2012

This is where it gets a bit hairy.  Windows Server makes it really easy to deploy services by just choosing to add a feature.  Unfortunately sometimes it can be really confusing.

The good folks at Microsoft have published a pretty good guide for configuring a CA, and OCSP responder and NDES (SCEP) server.

You can find all of that hotness here:

https://technet.microsoft.com/en-us/library/cc772393(v=ws.10).aspx

Unfortunately NDES seems to have been a bit of an after-thought for the Windows Server environment, so once you have the role set up, the only way you can make meaningful configuration changes is via registry keys (and then restarting the IIS service).

However, once again, the Microsoft folks come through with more solid documentation and have a posting that shares everything you ever wanted to know about NDES (including appropriate registry keys!)

More hotness here:

http://social.technet.microsoft.com/wiki/contents/articles/9063.network-device-enrollment-service-ndes-in-active-directory-certificate-services-ad-cs.aspx

Here are some of the common problems that I found when setting this up:

  1. OCSP
    1. Did you remember to do the post deployment tasks available from Server Manager?
    2. Have you created your Revocation configuration? (available from the Online Responder MMC snap-in)
    3. Did you remember to create a copy of the Online Responder Template that allowed auto-enroll and put it into the list of available templates for your Certificate Authority? (this can be done using the Certificate Templates and Certificate Authority MMC snap-ins)
    4. Did you verify that it is actually working? (you can do this by creating a certificate and then running the following command against that cert ‘certutil -URL ' then checking the revocation information)
  2. NDES
    1. Some devices (like the A10’s) don’t yet use the old certificates for the renewal process and will require re-use of the old password
      1. You need to enable password re-use
      2. You probably want to make that password much longer, since it can be re-used
      3. You probably need to increase the password cache to the number of devices you can support so that you don’t have to re-use passwords
      4. You probably want a certificate that doesn’t last two years for your IPSEC devices (plus we’re auto-rotating so its not like we personally have to do it (do this by creating a copy of the IPSEC (offline request) template and making appropriate changes
      5. See the registry configuration values from the NDES link above to change the first 3 items and to update NDES to point to your new certificate template for the fourth

Phase 4 - Configuring the A10 devices to use certificates

Now we have a few things that need to be done:

  1. Install the trusted Root CA on the A10 Devices
  2. Configure SCEP on the A10 devices so that they can automatically get and renew certificates
  3. Configure OCSP on the A10 devices so that they can actually verify that the certificates haven’t been revoked
  4. Update the VPN config to use the new certs

Install the trusted root CA (you know from the CA you created earlier) on both of your A10 devices.

There are a couple of ways to do this (FTP, TFTP,SCP,SFTP or GUI).

Since all of the command line ways require you have another server from which to upload, I suggest using the GUI, where you can upload the file directly.

Log into the GUI on your A10 devices and then go to ADC->SSL Management->Import

Choose the name that the A10 will reference the CA by (I choose IPSEC-CA), choose the appropriate options (don’t forget to click the radio button for CA certificate) choose the file that is your CA Cert (that you downloaded from your CA) and click import.

Configuring SCEP on IPSEC-A
The password will be the password that you received from the URL on the http:///mscep_admin/ You will should get a new one for each device you enroll

IPSEC-A#conf
IPSEC-A(config)#pki scep-cert IPSEC-A-SCEP
IPSEC-A(config-scep cert:IPSEC-A-SCEP)#url http://10.106.0.2/certsrv/mscep/mscep.dll
IPSEC-A(config-scep cert:IPSEC-A-SCEP)dn CN=IPSEC-A,DC=IPSEC,DC=LOCAL
IPSEC-A(config-scep cert:IPSEC-A-SCEP)subject-alternate-name email IPSEC-A@IPSEC.LOCAL
IPSEC-A(config-scep cert:IPSEC-A-SCEP)password ASDFLKSDF8908ASDFSDFH083NKW
IPSEC-A(config-scep cert:IPSEC-A-SCEP)renew-every minute 5
IPSEC-A(config-scep cert:IPSEC-A-SCEP)enroll

Configure SCEP on IPSEC-B

IPSEC-B#conf
IPSEC-B(config)#pki scep-cert IPSEC-B-SCEP
IPSEC-B(config-scep cert:IPSEC-B-SCEP)#url http://10.106.0.2/certsrv/mscep/mscep.dll
IPSEC-B(config-scep cert:IPSEC-B-SCEP)dn CN=IPSEC-B,DC=IPSEC,DC=LOCAL
IPSEC-B(config-scep cert:IPSEC-B-SCEP)subject-alternate-name email IPSEC-B@IPSEC.LOCAL
IPSEC-B(config-scep cert:IPSEC-B-SCEP)password ASDFLKSDF8908ASDFSDFH083NKW
IPSEC-B(config-scep cert:IPSEC-B-SCEP)renew-every minute 5
IPSEC-B(config-scep cert:IPSEC-B-SCEP)enroll

Now let’s verify that we actually got our SCEP certs:

Verification on IPSEC-A:

IPSEC-A#show pki scep-cert status
Certificate name: IPSEC-A-SCEP  status: SUCCESS
    Renew every 5 minutes
    rotated files: 4

Verification on IPSEC-B:

IPSEC-B#show pki scep-cert status
Certificate name: IPSEC-B-SCEP  status: SUCCESS
    Renew every 5 minutes
    rotated files: 4

Woo-hoo- we got our SCEP certs!

Now let’s configure OCSP for checking certificate revocation:
On IPSEC-A

IPSEC-A#conf
IPSEC-A(config)#aam authentication server ocsp IPSEC_OCSP
IPSEC-A(config-ocsp auth server:IPSEC_OCSP)#url http://10.106.0.2/ocsp
IPSEC-A(config-ocsp auth server:IPSEC_OCSP)#responder-ca IPSEC-CA
IPSEC-A(config-ocsp auth server:IPSEC_OCSP)#exit
IPSEC-A(config)#vpn revocation IPSEC-REVOCATION
IPSEC-B(config-revocation:IPSEC-REVOCATION)#ca IPSEC-CA
IPSEC-B(config-revocation:IPSEC-REVOCATION)#ocsp pri IPSEC_OCSP

and on IPSEC-B

IPSEC-B#conf
IPSEC-B(config)#aam authentication server ocsp IPSEC_OCSP
IPSEC-B(config-ocsp auth server:IPSEC_OCSP)#url http://10.106.0.2/ocsp
IPSEC-B(config-ocsp auth server:IPSEC_OCSP)#responder-ca IPSEC-CA
IPSEC-B(config-ocsp auth server:IPSEC_OCSP)#exit
IPSEC-B(config)#vpn revocation IPSEC-REVOCATION
IPSEC-B(config-revocation:IPSEC-REVOCATION)#ca IPSEC-CA
IPSEC-B(config-revocation:IPSEC-REVOCATION)#ocsp pri IPSEC_OCSP

Finally, lets update our VPN to use all this new auto-certificate hotness!

On IPSEC-A

IPSEC-A(config-ike-gateway:IKE_GATEWAY_IPSE)#auth-method rsa-signature
IPSEC-A(config-ike-gateway:IKE_GATEWAY_IPSE)#key IPSEC-A-SCEP
IPSEC-A(config-ike-gateway:IKE_GATEWAY_IPSE)#local-cert IPSEC-A-SCEP
IPSEC-A(config-ike-gateway:IKE_GATEWAY_IPSE)#remote-ca-cert IPSEC-CA
IPSEC-A(config-ike-gateway:IKE_GATEWAY_IPSE)#local-id DC=IPSEC,DC=LOCAL,CN=IPSEC-A
IPSEC-A(config-ike-gateway:IKE_GATEWAY_IPSE)#remote-id DC=IPSEC,DC=LOCAL,CN=IPSEC-B

and on IPSEC-B

IPSEC-B(config-ike-gateway:IKE_GATEWAY_IPSE)#auth-method rsa-signature
IPSEC-B(config-ike-gateway:IKE_GATEWAY_IPSE)#key IPSEC-B-SCEP
IPSEC-B(config-ike-gateway:IKE_GATEWAY_IPSE)#local-cert IPSEC-B-SCEP
IPSEC-B(config-ike-gateway:IKE_GATEWAY_IPSE)#remote-ca-cert IPSEC-CA
IPSEC-B(config-ike-gateway:IKE_GATEWAY_IPSE)#local-id DC=IPSEC,DC=LOCAL,CN=IPSEC-B
IPSEC-B(config-ike-gateway:IKE_GATEWAY_IPSE)#remote-id DC=IPSEC,DC=LOCAL,CN=IPSEC-A

Now, once you’ve done this, you’ll want to verify that it is working with the same VPN commands that were shown earlier.
If you are having trouble getting the tunnels to come up you can troubleshoot by using the following commands:

debug vpn level 5
show vpn log follow

These will set the debugging to the highest level possible for VPN (which will show you keys so you can verify they are working correctly)
The log command will follow your vpn debug packet by packet and will provide helpful information as to whether or not you are failing auth (or something else like OCSP)

I hope this was somewhat helpful.

Time for…

Phase 6 - Go get a beer.

Or a Scotch.  Or both.  Yeah, probably both- better play it safe.

Tagged #A10, #CA, #Certificate Authority, #Certificates, #NDES, #OCSP, #SCEP, #vyOS, #Windows Server 2012

2021

Back to top ↑

2020

Self Service with Satellite

less than 1 minute read

Recently I had a customer who was running into some legacy infrastructure challenges with VM consistency, meeting (internal) customer expectations, and gener...

Winning with Soft Tokens

2 minute read

Well, I’d call that a sabbatical, but I think that would mean I was relaxing. Time to revive this thing (again).

Back to top ↑

2018

Back to top ↑

2017

Using Powershell and Posh-SSH to GSD.

5 minute read

I’m a guy who thinks you should use the right tool for the job. For instance, if you’re in a Windows environment, and you need to script something, installi...

iDrac, RACADM, sshpass, and BASH

8 minute read

If it were up to me, I suppose that the only thing that I’d really be responsible for would be core networking infrastructure (and consumption of craft brews...

Dynamic DNS and you

3 minute read

Okay, so I know the popular thing with network engineers is to remember the IP of EVERYTHING.  I’m pretty good at it too.  But having a lab at home and needi...

Back to top ↑