HAproxy with Transparency

I recently came across another Transparency requirement for a customer. I dig around the web for a solution and more less found many. It all looked so easy and indeed after all it was, however during testing I found an issue I decided to share with you.

1. Requirements for HAproxy with Transparency.

In older version of kernel you would need to go via kernel and iptables patch procedure explained here: http://www.loadbalancer.org/uk/blog/configure-haproxy-with-tproxy-kernel-for-full-transparent-proxy

Since version 2.6.28+ you do not need to go through patching your kernel to get Transparency to work.

After HAproxy installation just check if you got TPROXY compiled in kernel:

[root@nfsec-lb ~]# lsmod
Module                  Size  Used by  
iptable_filter          2793  0  
nf_tproxy_core          1332  0 [permanent]  
xt_MARK                 1057  1  
xt_socket               5849  1  
nf_defrag_ipv4          1483  1 xt_socket  
iptable_mangle          3349  1  
ip_tables              17831  2 iptable_filter,iptable_mangle  

Also check if your installed HAproxy version got TPROXY support:

[root@nfsec-lb ~]# haproxy -vv
HA-Proxy version 1.5.4 2014/09/02  
Copyright 2000-2014 Willy Tarreau <w@1wt.eu>

Build options :  
  TARGET  = linux2628
  CPU     = generic
  CC      = gcc
  CFLAGS  = -O2 -g -fno-strict-aliasing

Default settings :  
  maxconn = 2000, bufsize = 16384, maxrewrite = 8192, maxpollevents = 200

Encrypted password support via crypt(3): yes  
Built with zlib version : 1.2.3  
Compression algorithms supported : identity, deflate, gzip  
Built with OpenSSL version : OpenSSL 1.0.1e-fips 11 Feb 2013  
Running on OpenSSL version : OpenSSL 1.0.1e-fips 11 Feb 2013  
OpenSSL library supports TLS extensions : yes  
OpenSSL library supports SNI : yes  
OpenSSL library supports prefer-server-ciphers : yes  
Built with PCRE version : 7.8 2008-09-05  
PCRE library supports JIT : no (USE_PCRE_JIT not set)  
Built with transparent proxy support using: IP_TRANSPARENT IPV6_TRANSPARENT IP_FREEBIND

Available polling systems :  
      epoll : pref=300,  test result OK
       poll : pref=200,  test result OK
     select : pref=150,  test result OK
Total: 3 (3 usable), will use epoll.  

2. Configuration

To get Transparency to work you need the following iptables rules. In fact iptables can be disabled at all, it just the mangle rules are required. The filter rules will be just default after iptables are disabled.

iptables -t mangle -N DIVERT  
iptables -t mangle -A PREROUTING -p tcp -m socket -j DIVERT  
iptables -t mangle -A DIVERT -j MARK --set-mark 111  
iptables -t mangle -A DIVERT -j ACCEPT  
ip rule add fwmark 111 lookup 100  
ip route add local dev lo table 100  

You also need kernel parameters set as follows in /etc/sysctl.conf

net.ipv4.ip_forward = 1  
net.ipv4.conf.all.forwarding = 1  
net.ipv4.conf.all.accept_redirects = 1  
net.ipv4.conf.all.send_redirects = 1  
net.ipv4.conf.eth0.send_redirects = 1  
net.ipv4.ip_nonlocal_bind = 1  
net.ipv4.conf.default.rp_filter = 2 #0 preferred  
net.ipv4.conf.default.accept_source_route = 1  

Again as I explained in one of my previous post about Transparency you may need Source Routing enabled so do it as follows depends on how many interfaces you got attached:

ip route add dev eth0 table 1  
ip route add default via dev eth0 table 1

ip route add dev eth1 table 2  
ip route add default via dev eth1 table 2

ip rule add iif eth0 table 1  
ip rule add iif eth1 table 2  

3. HAproxy configuration for Transparency

# Global settings
    log local2
    maxconn     4000

# common defaults that all the 'listen' and 'backend' sections will
# use if not designated in their block

        log     global
        mode    http            # Switch to HTTP mode (rather than basic TCP or HEALTH)
        option  httplog         # Include more HTTP information in the logs
        option  dontlognull     # Don't log mundane things like checks
        option  redispatch      # Allow sessions to move when a server goes down
#        option  http-server-close # Allow haproxy to close backend connections
#        option  forwardfor      # Add X-Forwarded-For headed logging original IP
        retries 3               # Number of failed check needed to consider a server down
        maxconn 4096
        timeout connect 60s      # Timeout for a TCP connection to backend server
        timeout check   60s      # Timeout for a HTTP health check to complete (normally less than server timeout)
        timeout client  120s     # Timeout for client inactivity (recommended equal to the server timeout)
        timeout server  120s     # Timeout for the backend server returning the page
log-format %ci:%cp\ [%t]\ %ft\ %H\ %b/%s\ %Tq/%Tw/%Tc/%Tr/%Tt\ %ST\ %B\ %CC\ %CS\ %tsc\ %ac/%fc/%bc/%sc/%rc\ %sq/%bq\ %hr\ %hs\ %{+Q}r

    mode http
    bind transparent
    default_backend TPROXY-BACKEND

    mode http
    balance source
    hash-type consistent
    source usesrc clientip
    timeout connect 1800s
    timeout server 5m
    server nfsec-web-01
    server nfsec-web-02
#    server backup backup source

NOTE: option http-server-close has to be disabled for the TPROXY backend. Otherwise requests from client on port 80 will not be able to be forwarded to backend servers ports.

This is probably an answer to this post:

4. Testing

Working scenario:

It will work from any other subnet apart from the subnet the LB and backend servers are on. Perhaps changing net.ipv4.conf.default.acceptsourceroute = 0 to 2 may fix it, however I doubt.

[root@nfsec-lb ~]# netstat -pan |grep "111:" tcp 0 0* LISTEN 31640/haproxy
tcp 0 0 ESTABLISHED 31640/haproxy
tcp 0 0 ESTABLISHED 31640/haproxy

If you are testing from the same subnet you will get below SYN_ACK to backend.
[root@nfsec-lb ~]# netstat -pan |grep "111:" tcp 0 0* LISTEN 31640/haproxy
tcp 0 0 ESTABLISHED 31640/haproxy
tcp 0 0 SYN_ACK 31640/haproxy

If you have global http-server-close enabled you will not get anything redirecting to the backend.

  1. Additional scenario.
    In situation when you have multiple VIP's each to be transparent pointing to the same backend servers which have multiple backend IP's, you won't be able to add multiple gateways on the backend servers pointing to each frontend VIP.

In this situation you just need to create a new frontend on LB and bind it to a new IP and set backend servers gateway to this IP.

What it does on LB it simply receives the returning traffic and via iptables mangle it then forwards it to the originating server via default route which is your ASA FW.

New GW is

Connection flows:
Public IP: -> (NAT) - LB VIP: sends to backend [Connection created: -]

Backend replies: -> New GW: LB sees the destination as and via iptables mangle replies to TCP ESTABLISHED OPEN connection on which routes to [Connection created. .3 recevices but creates connetion as -> ]



I hope you noticed that option http-server-close cannot be used in global configuration settings as well as uid/guid cannot be changed to haproxy as this will defeat the TPROXY requirements.

If you got a issue or want to discuss something - drop me a message or leave a comment.

Share this post:

by lo3k