Saturday 20 April 2013

Juniper SRX - Determine exact cause of high CPU on PFE

Hi Everyone,

Instead of relying on the sanitized/basic information the SRX generally provides you via usual commands, it's possible to log into the PFE directly and determine which underlying system is causing an issue:

Log into your PFE (there are LOTS of hidden commands available here, but I'll focus on threads):

root@SRX210_A# run start shell pfe network fwdd


BSD platform (OCTEON processor, 416MB memory, 8192KB flash)

FLOWD_OCTEON(SRX210_A vty)# show threads
PID PR State     Name                   Stack Use  Time (Last/Max/Total) cpu
--- -- -------   ---------------------  ---------  ---------------------
  1 H  asleep    Maintenance            976/73824  0/7/14 ms  0%
  2 L  running   Idle                  1296/73824  0/8/10035 ms  0%
  3 H  asleep    Timer Services        1040/73824  0/8/270 ms  0%
  5 L  asleep    Ukern Syslog           856/73824  0/0/0 ms  0%
  6 L  asleep    Sheaf Background      1296/73824  0/0/0 ms  0%
  7 M  asleep    mac_db                 856/73824  0/0/0 ms  0%
  8 M  asleep    Docsis                1072/73824  0/8/184 ms  0%
  9 M  asleep    ATMX                  1136/73824  0/8/491 ms  0%
 10 M  asleep    XDSL                  1352/73824  0/8/30394 ms  0%
 11 M  asleep    DSX50ms               1272/73824  0/8/2491 ms  0%
 12 M  asleep    DSXonesec             1048/73824  0/8/104 ms  0%
 13 M  asleep    SFP                   1216/73824  0/8/295 ms  0%
 14 M  asleep    Ethernet              1184/73824  0/15/90202 ms  1%
 15 M  asleep    RSMON syslog thread   1024/73824  0/0/0 ms  0%
 16 L  asleep    Syslog                1264/73824  0/8/113 ms  0%
 17 M  asleep    Fwdd Notif Recv       1512/73824  0/8/2816 ms  0%
 18 M  asleep    Forwarding Thread     2456/73824  0/15/36745 ms  0%
 19 M  asleep    Periodic             12936/73824  0/15/21958 ms  0%
 20 M  asleep    USB Thread            1328/73824  0/8/776 ms  0%
 21 M  asleep    FPC_IPC-Thread        2368/73824  0/0/0 ms  0%
 22 H  asleep    TTP Receive           1320/73824  0/15/6219 ms  0%
 23 H  ready     TTP Transmit          1104/73824  0/8/1480 ms  0%
 24 M  asleep    UDP Input              904/73824  0/0/0 ms  0%
 25 H  asleep    TCP Timers            1264/73824  0/8/312 ms  0%
 26 H  asleep    TCP Receive            816/73824  0/0/0 ms  0%
 32 L  asleep    Console               3504/73824  7/7/7 ms  0%
.................etc

You can then look into a certain hook more closely if desired:

FLOWD_OCTEON(SRX210_A vty)# show threads 14
PID PR State     Name                   Stack Use  Time (Last/Max/Total) cpu
--- -- -------   ---------------------  ---------  ---------------------
 14 M  asleep    Ethernet              1184/73824  0/15/95814 ms  1%

Wakeups:
      Type  ID  Enabled  Pending   Context
 Semaphore  00      Yes       No  0x489ac008
     Timer  00      Yes       No  0x489ac158

Frame 00: sp = 0x48cbab38, pc = 0x08014d94
Frame 01: sp = 0x48cbabb0, pc = 0x08215c98
Frame 02: sp = 0x48cbabf0, pc = 0x0802b960
Frame 03: sp = 0x48cbac18, pc = 0x00012060


3 comments:

  1. I liked the content about exact cause of high CPU on PFE. But i am searching about configuration commands. Please update your blog with this info.
    juniper srx

    ReplyDelete
    Replies
    1. request pfe execute target tnp tnp-name node1.fpc0.pic0 command "show sched" cpu utilization

      Delete
    2. root@srx240h2# run request pfe execute target tnp tnp-name node1.fpc0.pic0 command "show sched"
      error: command is not valid on the srx240h2

      So you can use:
      request pfe execute target fwdd command "show threads cpu"

      Delete