F5 BIG-IP LTM Persistence Cookie Shows Wrong Timezone

As I'm working through the CBTnuggets course for f5 BIG-IP LTM I have set up the following lab as required in the course.

Chapter/Video 15 focuses on the concept of persistence.

Here's persistence from the f5 manual;

Using BIG-IP Local Traffic Manager, you can configure session persistence. When you configure session persistence, Local Traffic Manager tracks and stores session data, such as the specific pool member that serviced a client request. The primary reason for tracking and storing session data is to ensure that client requests are directed to the same pool member throughout the life of a session or during subsequent sessions.

Persistence is simply the BIG-IP tracking the client/user session in a way that the client is sent to the same server till that session expires or till the persistence timeout as set on the BIG-IP expires. For e.g. one type of persistence explained in the course is cookie persistence, which tracks the user by storing an HTTP cookie in the user's browser. 

This cookie is obviously added by the BIG-IP in the HTTP response. We as BIG-IP admins have the option to add a timer in mins for which the cookie remains active on the client and then the cookie expires.

A example would be a user hitting the BIG-IP public IP to fill a form. The first initially selection of the server from the back-end pool is done by the BIG-IP based on the load balancing method in place. Let's say server green is chosen, once the user hits green server, BIG-IP then adds an HTTP cookie to the HTTP response and from then onwards the remaining requests from the user will only be sent to the green server instead of being load balanced, this can go on till the cookie time as set on the BIG-IP expires or till the session expires—whichever is configured.

In my lab above the issue is that the cookie persistence profile was simply not taking effect and the client kept load balancing to all the three back-end servers in the pool.

f5 troubleshooting manuals strongly recommended setting the correct time on the BIG-IP. I did that rebooted.

[root@big-ip-test:Active:Standalone] config # date
Thu May 18 19:14:31 GST 2017
[root@big-ip-test:Active:Standalone] config #

Wireshark shows that the cookie was being added correctly to the HTTP response but expiry time was wrong. I am adding the cookie at 18:40 GST and the cookie expiry is set to 14:29.

Which means by the time I test this, the cookie has already expired!

This is strange. Also the cookie timezone in wireshark is GMT while I'm in Dubai and the BIG-IP is set to use Asia/Dubai timezone which is GST.



To be updated as I troubleshoot this.

UPDATE - 21 May 2017

DISCLAIMER: It turns out this could simply be a issue with my lab as I'm using my own windows host machine to test the load balancing. The issue was resolved when I created a different VM on virutal box and used GNS3 to connect it to the BIGIP external network and tested the persistence again.

But the troubleshooting is still worth it!

The first thing to get out of the way is the time issue. We can do this using perl.

Download and install strawberry perl for windows and simply find the scaler time.

root@(big-ip-test)(cfg-sync Standalone)(Active)(/Common)(tmos)# run /util bash
[root@big-ip-test:Standalone] # perl -le 'print scalar time()'
1495179814
root@big-ip-test:Standalone] # exit

[root@big-ip-test:Active:Standalone] config # perl -le 'print scalar time()'
1495183038

C:\Users\allwyn>perl -le "print scalar time"
1495183035

That looks good, so no issues here.

Also the timezone for the cookie is always recommended to be in the GMT timezone since a website has to serve an global audience, I need to do more reading on this though. For next time.

So the packet in the above wireshark pcap is the HTTP response packet which has the cookie inserted by the BIGIP after the initial load balancing. This cookie is then saved by the browser and presented to the BIGIP for the remainder of the session or the cookie timeout as configured on the BIGIP.

Now we know that the BIGIP is inserting the cookie but whether it's getting back needs to be verified, and the following iRule helps to log this.

when HTTP_REQUEST {
    if { [HTTP::header exists Cookie] } {
        log local0. "Client: [IP::client_addr] : Incoming Cookie header: [HTTP::header values Cookie]"
    } else {
        log local0. "Client: [IP::client_addr] : Incoming Cookie header: None"
    }
}
when LB_SELECTED {
    log local0. "client addr : [client_addr] : Selected pool member: [LB::server addr]"
}

Then to view the logs use this guide. You have to enter the tmsh utility and filter the log events with a pipe |.

[root@big-ip-test:Active:Standalone] config # tmsh
root@(big-ip-test)(cfg-sync Standalone)(Active)(/Common)(tmos)# show /sys log ltm | grep cookie

Test-iRule <lb_selected>: client addr : 192.168.1.5 : Selected pool member: 10.2.0.22
Fri May 19 15:28:00 GST 2017    info    big-ip-test     tmm[10349]      Rule /Common/Cookie-Test-iRule <http_request>: Client: 192.168.1.5 : Incoming Cookie header: None
Fri May 19 15:28:02 GST 2017    info    big-ip-test     tmm[10349]      Rule /Common/Cookie-Test-iRule <lb_selected>: client addr : 192.168.1.5 : Selected pool member: 10.2.0.33</lb_selected></http_request></lb_selected>

You can also view the same logs from in GUI at system - logs - LTM and filter the events.


We see here clearly that the incoming (from client to BIGIP) http_request does not have the cookie set and hence the BIGIP keeps load balancing across the servers.

At this stage, as I already mentioned I just created a new VM and used GNS3 to connect it to the VMware network and tested the persistence.

Here's the log which then shows the incoming cookie.

ltm 05-19 18:43:13 info big-ip-test tmm[10349]: Rule /Common/Cookie-Test-iRule <HTTP_REQUEST>: Client: 192.168.1.192 : Incoming Cookie header: BIGipServerour-http-pool=184549898.20480.0000
ltm 05-19 18:43:13 info big-ip-test tmm[10349]: Rule /Common/Cookie-Test-iRule <LB_SELECTED>: client addr : 192.168.1.192 : Selected pool member: 10.2.0.11

ltm 05-19 18:43:13 info big-ip-test tmm[10349]: Rule /Common/Cookie-Test-iRule <HTTP_REQUEST>: Client: 192.168.1.192 : Incoming Cookie header: BIGipServerour-http-pool=184549898.20480.0000

ltm 05-19 18:52:05 info big-ip-test tmm[10349]: Rule /Common/Cookie-Test-iRule <LB_SELECTED>: client addr : 192.168.1.192 : Selected pool member: 10.2.0.22

ltm 05-19 18:52:05 info big-ip-test tmm[10349]: Rule /Common/Cookie-Test-iRule <HTTP_REQUEST>: Client: 192.168.1.192 : Incoming Cookie header: BIGipServerour-http-pool=369099274.20480.0000

ltm 05-19 18:52:06 info big-ip-test tmm[10349]: Rule /Common/Cookie-Test-iRule <LB_SELECTED>: client addr : 192.168.1.192 : Selected pool member: 10.2.0.22

The BIGIP then halts the load balancing and sends all requests to just one back-end server.