Friday, 23 November 2012

SPLAT + GAIA : Inaccessible via Console/SSH/GUI

Hi Everyone,

Over the last few months I've seen a large amount of SPLAT appliances become completely inaccessible via "normal" methods (R71->R75). Upon further investigation it seems all of them are suffering from the same problem, however it's quite strange as all three methods use separate authentication schemes.

##It should be noted that *only* CP-branded appliances have been experiencing the issue, open servers seem to be safe from this.

Edit:: Upon further investigation, it appears that all SPLAT and GAIA devices can be affected by this bug.

Doing a debug of an SSH authentication attempt we can see the firewall immediately close the connection via a FIN, without ever presenting the 'reprompt' as if you typed the wrong password:

SSH attempt with the -vv flags:
admin@<firewall_IP>'s password:
debug2: we sent a password packet, wait for reply
Connection closed by


<firewall_IP>.22 > <host_IP>.33506: F 1393:1393(0) ack 1269 win 79 <nop,nop,timestamp 3558805529 352019454> (DF)

After trying multiple ways of "breaking in", we gave up and rebooted one of the devices and attempted to access it via maintenance mode, which was succesful.

Looking at /var/log/messages, we can see faillog is broken in some way:

cp_pam_tally[23237]: /var/log/faillog is either world writable or not a normal file

Looking at the file in detail, we saw that it is completely corrupted (filled with ascii/hex symbols).

Replacing the file with a fresh, empty copy (or simply removing it entirely) corrects the issue.

However, next time /usr/bin/faillog is called to do a rollover, the file becomes corrupted once again, and all access is lost...

To prevent this from happening, I've implemented a 'manual' rollover to prevent faillog from doing it itself via CRON (obviously from maintenance mode after the issue occured):

To create a cron entry:
crontab -e

The editor on SPLAT is still VI(m), so press 'i' to enter input mode, and type:

* * * * * /bin/bash /home/admin/

This will have crond run the script every minute, which you can grab here (chmod +x it):

Make sure to adjust the path for the script in crontab if you don't place it in /home/admin/

CheckPoint R&D is also aware of the issue and are working on a corrected faillog binary (shadow-utils really), however for the time being this is definitely an easy fix. Until the fixed binary is included in a normal release however, I'd *highly* recommend having this cron job installed on any SPLAT-based Appliance, since fixing the issue once it's occured via maintenance mode isn't the easiest thing to schedule.

Follow Up: CP has released a fixed shadow-utils RPM that addresses the issue, however they have confirmed GAIA is currently susceptible, and that the rollover fix will be incorporated into the next release of GAIA.

Thanks for reading,