Juniper SRX: How to manage fxp0 across a VPN (Remote Management Best Practices)

This is one of the most common questions I see, both in my professional life as well as on popular Juniper technical forums.  Most of this confusion could be avoided if Juniper allowed for fxp0 to be placed in a non-default routing instance, however, for the time being, we're left with having to perform the following (moving all interfaces to a VR instead of just fxp0).

For those who are unaware, fxp0 represents a dedicated management interface to the routing-engine of the device. On the SRX, there is complete hardware separation between the routing-engine and the dataplane (which is responsible for the actual forwarding of transit traffic). This is accomplished on the high-end devices by a separate hardware module or blade (SRX 650 -> SRX 5800), and by dedicated CPU cores on the shared CPUs on the smaller branch devices.

In this example, our topology will contain the following:
1 Cluster of two SRX to be managed via fxp0 remotely.
1 Stand-Alone SRX acting as our VPN peer. Our SSH Proxy will reside behind this gateway.
1 Virtual-Chassis stack of EX Switches to attach the cluster's fxp0 and management reth to
1 SSH Proxy (172.16.88.100) for remote device management.

This example assumes the cluster has already been configured, the VPN (route-based) is up and running, etc. If you are unfamiliar with these topics, you can check out previous articles here on how to set them up (or support.juniper.net for the official documentation).

Step 1: Allocate a new VLAN for management purposes. This VLAN will contain both chassis' fxp0 interfaces, as well as a single reth interface (trunked or otherwise) from the dataplane. Assign the new reth interface to an appropriately named zone (careful not to use 'Management' as it's reserved):

Interface configurations are as follows:

set interfaces reth0 vlan-tagging
set interfaces reth0 redundant-ether-options redundancy-group 1
set interfaces reth0 redundant-ether-options minimum-links 1
set interfaces reth0 redundant-ether-options lacp passive
set interfaces reth0 unit 100 description "Outside interface"
set interfaces reth0 unit 100 vlan-id 100
set interfaces reth0 unit 100 family inet address 10.1.1.1/28
set interfaces reth1 redundant-ether-options redundancy-group 1
set interfaces reth1 unit 0 description "Attached to Management VLAN for fxp0 access"
set interfaces reth1 unit 0 family inet address 192.168.0.210/24
set interfaces st0 unit 0 family inet address 1.1.1.1/28
set groups node0 interfaces fxp0 unit 0 family inet address 192.168.0.211/24
set groups node1 interfaces fxp0 unit 0 family inet address 192.168.0.212/24
set security zones security-zone Outside interfaces reth0.100 host-inbound-traffic system-services ike
set security zones security-zone Mgmt interfaces reth1.0
set security zones security-zone VPN interfaces st0.0

Step 2: Divide the SRX into (at least) two virtual-routers. One with your management (fxp0) interfaces (in inet.0), and one with all of your revenue ports (reth's or otherwise):

set routing-instances Traffic instance-type virtual-router
set routing-instances Traffic interface reth0.100
set routing-instances Traffic interface reth1.0
set routing-instances Traffic interface st0.0
set routing-instances Traffic routing-options static route 172.16.88.0/24 next-hop 1.1.1.2

Step 3: Assign the correct routes to the default routing instance (that should only contain fxp0) so that it has a return path to our SSH Proxy regardless of what state it's in (RPD does not run on the standby node). This requires us to use several pieces of configuration.
The next-hop for all traffic for fxp0 should be our newly created "Mgmt" reth interface (reth1.0)

a) Create a backup router statement for the device to read at boot. This will only take effect during the RE boot sequence before RPD becomes available. Make sure to use host routes here (/32) with a next-hop address of your reth interface

set system backup-router 192.168.0.210
set system backup-router destination 172.16.88.100/32

b) Create a mirrored static route for the backup-router statement. This ensures that the secondary node remains reachable post-failover (backup-router is only read during boot).:

set routing-options static route 172.16.88.100/32 next-hop 192.168.0.210
set routing-options static route 172.16.88.100/32 retain
set routing-options static route 172.16.88.100/32 no-readvertise

c) Create the default route for all other RE (fxp0) traffic to use reth1.0 as well:

set routing-options static route 0.0.0.0/0 next-hop 192.168.0.210

d) Inform the device that all dataplane logs for traffic (Accept/Deny/IPS, from the  'security log' configuration) need to be routed to the new virtual-router (Traffic.inet.0) to reach the syslog/SIEM. This ensures they exit a revenue port and are not impacting the RE negatively:

set routing-options static route IP_of_SYSLOG_Server/32 next-table Traffic.inet.0

Step 4: Add appropriate security policies for the traffic from the Mgmt zone to the VPN zone (and vice-versa) to permit access to fxp0

Step 5: Validate that you have access to both devices and view it on the device (test it with both RG0 and RG1 failover scenarios):

 show security flow session source-prefix 172.16.88.100 node 0
node0:
--------------------------------------------------------------------------

Session ID: 382, Policy name: ANY_ANY_PERMIT/4, State: Active, Timeout: 1800, Valid
  In: 172.16.88.100/44411 --> 192.168.0.212/22;tcp, If: st0.0, Pkts: 459, Bytes: 27651
  Out: 192.168.0.212/22 --> 172.16.88.100/44411;tcp, If: reth1.0, Pkts: 480, Bytes: 161953
Total sessions: 1

Step 6 (Optional): Ensure you can still SSH to the master device if the VPN tunnel goes down by adding SSH accessibility on the outside reth interface. Ensure this is locked down appropriately with firewall filters!

set security zones security-zone Outside interfaces reth0.100 host-inbound-traffic system-services ssh

Step 7 (Optional): If your cluster requires access to things like public DNS, NTP, or application-services updates (IPS/AppFW, etc), you'll need to add NAT and security policies for fxp0 to access  the appropriate resources.

Juniper SRX - OSPF over GRE over IPSEC

Hi everyone,

As promised, and as a continuation of my JNCIE-SEC studies, a follow up to the basic Route Based VPN article. This is how you can join two separate OSPF domains together across an IPSEC/GRE tunnel. Keep in mind running GRE is not necessary on an SRX<->SRX IPSEC tunnel, however more limited platforms like ASA's require it.

As a recap, this is the topology we're working with:

We'll do this for SRX210_A only, mirror the config to SRX210_B with slight adjustments if you're following along:

Create the GRE Tunnel and add it to our DMZ Zone, making sure to use st0 as the source/destination:

set interfaces gr-0/0/0.0 tunnel source 172.16.30.1
set interfaces gr-0/0/0.0 tunnel destination 172.16.30.2
set interfaces gr-0/0/0.0 family inet address 172.16.30.5/30
set security zones security-zone DMZ interfaces gr-0/0/0.0 host-inbound-traffic system-services ping

Ensure VPN allows any system service for the time being (lock it down later):

set security zones security-zone VPN host-inbound-traffic system-services any-service

Test time - Make sure your GRE tunnel is up and running by trying to reach the remote side:

ping 172.16.30.6 rapid 
PING 172.16.30.6 (172.16.30.6): 56 data bytes
!!!!!
--- 172.16.30.6 ping statistics ---
5 packets transmitted, 5 packets received, 0% packet loss

round-trip min/avg/max/stddev = 4.811/5.521/6.624/0.703 ms

OSPF Area 0  configuration for GRE and DMZ interfaces:

set security zones security-zone DMZ host-inbound-traffic protocols ospf
set protocols ospf area 0 interface gr-0/0/0.0
set protocols ospf area 0 interface fe-0/0/5.0

Verify OSPF neighbor adjacency across gr-0/0/0.0 and that you're receiving the correct routes:

show ospf neighbor
Address          Interface              State     ID               Pri  Dead
172.16.30.6      gr-0/0/0.0             Full      10.200.200.1     128    39

show route protocol ospf 
inet.0: 16 destinations, 17 routes (16 active, 0 holddown, 0 hidden)
+ = Active Route, - = Last Active, * = Both
10.200.200.0/24    *[OSPF/10] 00:03:17, metric 2
                    > via gr-0/0/0.0
172.16.30.4/30      [OSPF/10] 00:03:28, metric 1
                    > via gr-0/0/0.0
224.0.0.5/32       *[OSPF/10] 00:03:38, metric 1
                      MultiRecv

You'll notice if you try and ping the remote hosts you'll get drop logs for DMZ -> DMZ:

May  2 09:47:27 192.168.0.212 RT_FLOW: RT_FLOW_SESSION_DENY: session denied 10.200.200.100/1->10.100.100.101/36102 icmp 1(8) global_drop_all(global) DMZ DMZ UNKNOWN UNKNOWN N/A(N/A) fe-0/0/5.0 No 

Finally, create a new policy to allow the new DMZ<->DMZ traffic we've just bridged:

set security policies from-zone DMZ to-zone DMZ policy DMZ_to_DMZ match source-address 10.100.100.0/24
set security policies from-zone DMZ to-zone DMZ policy DMZ_to_DMZ match source-address 10.200.200.0/24
set security policies from-zone DMZ to-zone DMZ policy DMZ_to_DMZ match destination-address 10.200.200.0/24
set security policies from-zone DMZ to-zone DMZ policy DMZ_to_DMZ match destination-address 10.100.100.0/24
set security policies from-zone DMZ to-zone DMZ policy DMZ_to_DMZ match application any
set security policies from-zone DMZ to-zone DMZ policy DMZ_to_DMZ then permit
set security policies from-zone DMZ to-zone DMZ policy DMZ_to_DMZ then log session-init

Verify our connectivity again by pinging from the remote hosts across the tunnel:

show security flow session destination-prefix 10.100.100/24 
Session ID: 38948, Policy name: DMZ_to_DMZ/15, Timeout: 2, Valid
  In: 10.200.200.100/17 --> 10.100.100.101/36614;icmp, If: gr-0/0/0.0, Pkts: 1, Bytes: 84
  Out: 10.100.100.101/36614 --> 10.200.200.100/17;icmp, If: fe-0/0/5.0, Pkts: 1, Bytes: 84

And we're done!

Full configuration will be uploaded to github shortly (once they're done their maintenance).
Full configuration on github

SPLAT/GAIA Static-Route migration scripts

Hi Everyone,

So I recently came across a situation where I needed to accomplish two things quite quickly:
1) Remove all active interfaces from a device and reconfigure them into load-sharing LACP bonds
2) Restore the previous routing configuration to the device post-interface removal.

Since #2 involved redoing over 1000 static routes, I of course didn't want to do this manually :)

I've created two sets of scripts; One for backing up the current configuration, and one for restoring the configuration post config-change.

(I'd suggest using wget to pull the raw files, however you can copy however you'd like):


Backups:
GAIA:
route_backup_gaia.sh
SPLAT:
route_backup_splat.sh

Restoring:
GAIA
route_rebuild_gaia.sh
SPLAT
route_rebuild_splat.sh


As for how to use them, I'll give you a basic scenario. Currently most routes for my test box are via eth2, however I want to move this link into a bond for better throughput and availability.


GAIA1# clish -c "show route"
Codes: C - Connected, S - Static, R - RIP, B - BGP,
       O - OSPF IntraArea (IA - InterArea, E - External, N - NSSA)
       A - Aggregate, K - Kernel Remnant, H - Hidden, P - Suppressed

S     0.0.0.0/0           via 192.168.0.1, eth0, cost 0, age 4 
C     10.100.100.0/24     is directly connected, eth2 
S     10.100.101.0/24     via 10.100.100.2, eth2, cost 0, age 5993 
S     10.100.102.0/24     via 10.100.100.2, eth2, cost 0, age 1090 
S     10.100.103.0/24     via 10.100.100.2, eth2, cost 0, age 1087 
S     10.100.104.0/24     via 10.100.100.2, eth2, cost 0, age 1084 
C     127.0.0.0/8         is directly connected, lo 
C     192.168.0.0/24      is directly connected, eth0 

Prior to making our changes, I run the backup script like so:
[Expert@GAIA1]# ./route_backup_gaia.sh
Backing up routes now...

DONE

You can find your routes in /home/admin/150812_195030_GAIA1_routes.txt

Looking through the route file you can see that it's parsed it into a useful format:
[Expert@GAIA1]# cat 150812_195030_GAIA1_routes.txt
0.0.0.0/0 192.168.0.1
10.100.101.0/24 10.100.100.2
10.100.102.0/24 10.100.100.2
10.100.103.0/24 10.100.100.2
10.100.104.0/24 10.100.100.2

We'll make our interface changes now (remove eth2 - migrate to bond0)

Post change we can see that we now have bond0 on 10.100.100/24, however all of our routes are now gone:

GAIA1> show route
Codes: C - Connected, S - Static, R - RIP, B - BGP,
       O - OSPF IntraArea (IA - InterArea, E - External, N - NSSA)
       A - Aggregate, K - Kernel Remnant, H - Hidden, P - Suppressed

S     0.0.0.0/0           via 192.168.0.1, eth0, cost 0, age 5490 
C     10.100.100.0/24     is directly connected, bond0 
C     127.0.0.0/8         is directly connected, lo 
C     192.168.0.0/24      is directly connected, eth0 
 
Now we want to restore our previous routes:
[Expert@GAIA1]# ./route_rebuild_gaia.sh
Hello, please enter the correct log file to analyze
150812_195030_GAIA1_routes.txt
150812_195030_GAIA1_routes.txt
Thank you - Rebuilding the routing table now
Finished rebuilding the routing table...

Please remember to verify if the routes were rebuilt correctly!!
Goodbye
[Expert@GAIA1]# clish -c "show route"
Codes: C - Connected, S - Static, R - RIP, B - BGP,
       O - OSPF IntraArea (IA - InterArea, E - External, N - NSSA)
       A - Aggregate, K - Kernel Remnant, H - Hidden, P - Suppressed

S     0.0.0.0/0           via 192.168.0.1, eth0, cost 0, age 5669 
C     10.100.100.0/24     is directly connected, bond0 
S     10.100.101.0/24     via 10.100.100.2, bond0, cost 0, age 12 
S     10.100.102.0/24     via 10.100.100.2, bond0, cost 0, age 12 
S     10.100.103.0/24     via 10.100.100.2, bond0, cost 0, age 12 
S     10.100.104.0/24     via 10.100.100.2, bond0, cost 0, age 12 
C     127.0.0.0/8         is directly connected, lo 
C     192.168.0.0/24      is directly connected, eth0

And there you have it - nice and simple :)

SPLAT works the same way, however the scripts themselves are different of course since we need to use CLISH now.

If you want to get this to work on IPSO, the GAIA script would only need very minor modifications to how it deals with write-locks. If you need some help, let me know :)