Tuesday, 11 December 2012

How to calculate the total amount of FireWall Logs per second

### Posting this here for the time being since the support site's SK is broken..

Edit: Not sure why CP runs this with three separate strings...just copy/paste this and you'll get your numbers (sleeps for 120 seconds):

SLEEP_TIME=120;SIZE_BEFORE=$(ls -l $FWDIR/log/fw.logptr | awk '{print $5}') ; sleep $SLEEP_TIME ; SIZE_AFTER=$(ls -l $FWDIR/log/fw.logptr | awk '{print $5}');expr \( $SIZE_AFTER - $SIZE_BEFORE \) \/ \( 4 \* $SLEEP_TIME \)

#######

Follow these steps to calculate/count the total amount of all FireWall Logs per second that arrive to this Security Management Server from all its managed Security Gateways:
  1. Connect to CLI on Security Management Server - over SSH, or console.

    Note:
    On Multi-Domain Management Server, go to the context of the relevant Domain Management Server: [Expert@HostName]# mdsenv [Domain_Name|Domain_IP]
  2. Go to the Log directory:

    [Expert@HostName]# cd $FWDIR/log
  3. Check by how much the size of the Pointer File grows during specific time
    (the time should be high enough to accumulate enough logs - e.g., 120 sec, 180 sec, etc):

    [Expert@HostName]# ls -l fw.logptr ; sleep SLEEP_TIME ; ls -l fw.logptr
  4. Calculate the log rate per this formula:

    RATE = ( SIZE_AFTER - SIZE_BEFORE ) / ( 4 * SLEEP_TIME )

    Use these three commands to automate the calculations:

    [Expert@HostName]# SLEEP_TIME=number_of_seconds

    [Expert@HostName]# SIZE_BEFORE=$(ls -l fw.logptr | awk '{print $5}') ; sleep $SLEEP_TIME ; SIZE_AFTER=$(ls -l fw.logptr | awk '{print $5}')

    [Expert@HostName]# expr \( $SIZE_AFTER - $SIZE_BEFORE \) \/ \( 4 \* $SLEEP_TIME \)


    Note: if the rate value has to be used in a shell script, then use this syntax:
    [Expert@HostName]# RATE=$(expr \( $SIZE_AFTER - $SIZE_BEFORE \) \/ \( 4 \* $SLEEP_TIME \))

Friday, 7 December 2012

VSX: Policy installation failing due to "Can't open..."

Hi everyone,

Had a new issue happen to me this morning while pushing to an R67 VSLS cluster. During the push, one MVS reported that all of it's configuration files plus those of it's VS were missing.

This output was taken from $CPDIR/log/cpd.elg, however the message within Dashboard was nearly identical:

[7 Dec 12:46:04] file_digest: Can't open /opt/CPsuite-V40/fw1/policy/local.dt

[7 Dec 12:46:04] file_digest: Can't open /opt/CPsuite-V40/fw1/policy/local.scv

[7 Dec 12:46:04] file_digest: Can't open /opt/CPsuite-V40/fw1/policy/local.lp

[7 Dec 12:46:04] file_digest: Can't open /opt/CPsuite-V40/fw1/policy/local.cfg

[7 Dec 12:46:04] file_digest: Can't open /opt/CPsuite-V40/fw1/CTX/CTX00002/policy/local.dt

[7 Dec 12:46:04] file_digest: Can't open /opt/CPsuite-V40/fw1/CTX/CTX00002/policy/local.scv

[7 Dec 12:46:04] file_digest: Can't open /opt/CPsuite-V40/fw1/CTX/CTX00002/policy/local.lp

[7 Dec 12:46:04] file_digest: Can't open /opt/CPsuite-V40/fw1/CTX/CTX00002/policy/local.cfg

etc, etc, for all of the VS (13).

Doing a quick 'ls' for any of those files returned no results...

I'm still not sure how the issue occured (and on only one of the three MVS in the cluster), but running this will correct the issue for you by creating the files all at once:

cat $CPDIR/log/cpd.elg | grep "file_digest: Can't open" | awk '{print "touch",$7}' | sh

Friday, 23 November 2012

SPLAT + GAIA : Inaccessible via Console/SSH/GUI

Hi Everyone,

Over the last few months I've seen a large amount of SPLAT appliances become completely inaccessible via "normal" methods (R71->R75). Upon further investigation it seems all of them are suffering from the same problem, however it's quite strange as all three methods use separate authentication schemes.

##It should be noted that *only* CP-branded appliances have been experiencing the issue, open servers seem to be safe from this.

Edit:: Upon further investigation, it appears that all SPLAT and GAIA devices can be affected by this bug.


Doing a debug of an SSH authentication attempt we can see the firewall immediately close the connection via a FIN, without ever presenting the 'reprompt' as if you typed the wrong password:

SSH attempt with the -vv flags:
admin@<firewall_IP>'s password:
debug2: we sent a password packet, wait for reply
Connection closed by
<host_IP>

TCPDUMP:

<firewall_IP>.22 > <host_IP>.33506: F 1393:1393(0) ack 1269 win 79 <nop,nop,timestamp 3558805529 352019454> (DF)

After trying multiple ways of "breaking in", we gave up and rebooted one of the devices and attempted to access it via maintenance mode, which was succesful.

Looking at /var/log/messages, we can see faillog is broken in some way:

cp_pam_tally[23237]: /var/log/faillog is either world writable or not a normal file


Looking at the file in detail, we saw that it is completely corrupted (filled with ascii/hex symbols).

Replacing the file with a fresh, empty copy (or simply removing it entirely) corrects the issue.

However, next time /usr/bin/faillog is called to do a rollover, the file becomes corrupted once again, and all access is lost...

To prevent this from happening, I've implemented a 'manual' rollover to prevent faillog from doing it itself via CRON (obviously from maintenance mode after the issue occured):

To create a cron entry:
crontab -e

The editor on SPLAT is still VI(m), so press 'i' to enter input mode, and type:

* * * * * /bin/bash /home/admin/faillog_rollover.sh

This will have crond run the faillog_rollover.sh script every minute, which you can grab here (chmod +x it):

faillog_rollover.sh

Make sure to adjust the path for the script in crontab if you don't place it in /home/admin/

CheckPoint R&D is also aware of the issue and are working on a corrected faillog binary (shadow-utils really), however for the time being this is definitely an easy fix. Until the fixed binary is included in a normal release however, I'd *highly* recommend having this cron job installed on any SPLAT-based Appliance, since fixing the issue once it's occured via maintenance mode isn't the easiest thing to schedule.

Follow Up: CP has released a fixed shadow-utils RPM that addresses the issue, however they have confirmed GAIA is currently susceptible, and that the rollover fix will be incorporated into the next release of GAIA.

Thanks for reading,









Wednesday, 15 August 2012

SPLAT/GAIA Static-Route migration scripts

Hi Everyone,

So I recently came across a situation where I needed to accomplish two things quite quickly:
1) Remove all active interfaces from a device and reconfigure them into load-sharing LACP bonds
2) Restore the previous routing configuration to the device post-interface removal.

Since #2 involved redoing over 1000 static routes, I of course didn't want to do this manually :)

I've created two sets of scripts; One for backing up the current configuration, and one for restoring the configuration post config-change.

(I'd suggest using wget to pull the raw files, however you can copy however you'd like):


Backups:
GAIA:
route_backup_gaia.sh
SPLAT:
route_backup_splat.sh

Restoring:
GAIA
route_rebuild_gaia.sh
SPLAT
route_rebuild_splat.sh


As for how to use them, I'll give you a basic scenario. Currently most routes for my test box are via eth2, however I want to move this link into a bond for better throughput and availability.


GAIA1# clish -c "show route"
Codes: C - Connected, S - Static, R - RIP, B - BGP,
       O - OSPF IntraArea (IA - InterArea, E - External, N - NSSA)
       A - Aggregate, K - Kernel Remnant, H - Hidden, P - Suppressed

S     0.0.0.0/0           via 192.168.0.1, eth0, cost 0, age 4 
C     10.100.100.0/24     is directly connected, eth2 
S     10.100.101.0/24     via 10.100.100.2, eth2, cost 0, age 5993 
S     10.100.102.0/24     via 10.100.100.2, eth2, cost 0, age 1090 
S     10.100.103.0/24     via 10.100.100.2, eth2, cost 0, age 1087 
S     10.100.104.0/24     via 10.100.100.2, eth2, cost 0, age 1084 
C     127.0.0.0/8         is directly connected, lo 
C     192.168.0.0/24      is directly connected, eth0 


Prior to making our changes, I run the backup script like so:
[Expert@GAIA1]# ./route_backup_gaia.sh
Backing up routes now...

DONE

You can find your routes in /home/admin/150812_195030_GAIA1_routes.txt


Looking through the route file you can see that it's parsed it into a useful format:
[Expert@GAIA1]# cat 150812_195030_GAIA1_routes.txt
0.0.0.0/0 192.168.0.1
10.100.101.0/24 10.100.100.2
10.100.102.0/24 10.100.100.2
10.100.103.0/24 10.100.100.2
10.100.104.0/24 10.100.100.2


We'll make our interface changes now (remove eth2 - migrate to bond0)

Post change we can see that we now have bond0 on 10.100.100/24, however all of our routes are now gone:

GAIA1> show route
Codes: C - Connected, S - Static, R - RIP, B - BGP,
       O - OSPF IntraArea (IA - InterArea, E - External, N - NSSA)
       A - Aggregate, K - Kernel Remnant, H - Hidden, P - Suppressed

S     0.0.0.0/0           via 192.168.0.1, eth0, cost 0, age 5490 
C     10.100.100.0/24     is directly connected, bond0 
C     127.0.0.0/8         is directly connected, lo 
C     192.168.0.0/24      is directly connected, eth0 
 

Now we want to restore our previous routes:
[Expert@GAIA1]# ./route_rebuild_gaia.sh
Hello, please enter the correct log file to analyze
150812_195030_GAIA1_routes.txt
150812_195030_GAIA1_routes.txt
Thank you - Rebuilding the routing table now
Finished rebuilding the routing table...

Please remember to verify if the routes were rebuilt correctly!!
Goodbye
[Expert@GAIA1]# clish -c "show route"
Codes: C - Connected, S - Static, R - RIP, B - BGP,
       O - OSPF IntraArea (IA - InterArea, E - External, N - NSSA)
       A - Aggregate, K - Kernel Remnant, H - Hidden, P - Suppressed

S     0.0.0.0/0           via 192.168.0.1, eth0, cost 0, age 5669 
C     10.100.100.0/24     is directly connected, bond0 
S     10.100.101.0/24     via 10.100.100.2, bond0, cost 0, age 12 
S     10.100.102.0/24     via 10.100.100.2, bond0, cost 0, age 12 
S     10.100.103.0/24     via 10.100.100.2, bond0, cost 0, age 12 
S     10.100.104.0/24     via 10.100.100.2, bond0, cost 0, age 12 
C     127.0.0.0/8         is directly connected, lo 
C     192.168.0.0/24      is directly connected, eth0


And there you have it - nice and simple :)

SPLAT works the same way, however the scripts themselves are different of course since we need to use CLISH now.

If you want to get this to work on IPSO, the GAIA script would only need very minor modifications to how it deals with write-locks. If you need some help, let me know :)