I recently came across another Transparency requirement for a customer. I dig around the web for a solution and more less found many. It all looked so easy and indeed after all it was, however during testing I found an issue I decided to share with you.
1. Requirements for HAproxy with Transparency.
In older version of kernel you would need to go via kernel and iptables patch procedure explained here: http://www.loadbalancer.org/uk/blog/configure-haproxy-with-tproxy-kernel-for-full-transparent-proxy
Since version 2.6.28+ you do not need to go through patching your kernel to get Transparency to work.
After HAproxy installation just check if you got TPROXY compiled in kernel:
[root@nfsec-lb ~]# lsmod
Module Size Used by
iptable_filter 2793 0
nf_tproxy_core 1332 0 [permanent]
xt_MARK 1057 1
xt_socket 5849 1
nf_defrag_ipv4 1483 1 xt_socket
iptable_mangle 3349 1
ip_tables 17831 2 iptable_filter,iptable_mangle
Also check if your installed HAproxy version got TPROXY support:
[root@nfsec-lb ~]# haproxy -vv
HA-Proxy version 1.5.4 2014/09/02
Copyright 2000-2014 Willy Tarreau <w@1wt.eu>
Build options :
TARGET = linux2628
CPU = generic
CC = gcc
CFLAGS = -O2 -g -fno-strict-aliasing
OPTIONS = USE_LINUX_TPROXY=1 USE_ZLIB=1 USE_REGPARM=1 USE_OPENSSL=1 USE_PCRE=1
Default settings :
maxconn = 2000, bufsize = 16384, maxrewrite = 8192, maxpollevents = 200
Encrypted password support via crypt(3): yes
Built with zlib version : 1.2.3
Compression algorithms supported : identity, deflate, gzip
Built with OpenSSL version : OpenSSL 1.0.1e-fips 11 Feb 2013
Running on OpenSSL version : OpenSSL 1.0.1e-fips 11 Feb 2013
OpenSSL library supports TLS extensions : yes
OpenSSL library supports SNI : yes
OpenSSL library supports prefer-server-ciphers : yes
Built with PCRE version : 7.8 2008-09-05
PCRE library supports JIT : no (USE_PCRE_JIT not set)
Built with transparent proxy support using: IP_TRANSPARENT IPV6_TRANSPARENT IP_FREEBIND
Available polling systems :
epoll : pref=300, test result OK
poll : pref=200, test result OK
select : pref=150, test result OK
Total: 3 (3 usable), will use epoll.
2. Configuration
To get Transparency to work you need the following iptables rules. In fact iptables can be disabled at all, it just the mangle rules are required. The filter rules will be just default after iptables are disabled.
iptables -t mangle -N DIVERT
iptables -t mangle -A PREROUTING -p tcp -m socket -j DIVERT
iptables -t mangle -A DIVERT -j MARK --set-mark 111
iptables -t mangle -A DIVERT -j ACCEPT
ip rule add fwmark 111 lookup 100
ip route add local 0.0.0.0/0 dev lo table 100
You also need kernel parameters set as follows in /etc/sysctl.conf
net.ipv4.ip_forward = 1
net.ipv4.conf.all.forwarding = 1
net.ipv4.conf.all.accept_redirects = 1
net.ipv4.conf.all.send_redirects = 1
net.ipv4.conf.eth0.send_redirects = 1
net.ipv4.ip_nonlocal_bind = 1
net.ipv4.conf.default.rp_filter = 2 #0 preferred
net.ipv4.conf.default.accept_source_route = 1
Again as I explained in one of my previous post about Transparency you may need Source Routing enabled so do it as follows depends on how many interfaces you got attached:
ip route add 10.121.1.0/24 dev eth0 table 1
ip route add default via 10.121.1.254 dev eth0 table 1
ip route add 10.122.1.0/24 dev eth1 table 2
ip route add default via 10.122.1.254 dev eth1 table 2
ip rule add iif eth0 table 1
ip rule add iif eth1 table 2
3. HAproxy configuration for Transparency
#---------------------------------------------------------------------
# Global settings
#---------------------------------------------------------------------
global
log 127.0.0.1 local2
maxconn 4000
daemon
#---------------------------------------------------------------------
# common defaults that all the 'listen' and 'backend' sections will
# use if not designated in their block
#---------------------------------------------------------------------
defaults
log global
mode http # Switch to HTTP mode (rather than basic TCP or HEALTH)
option httplog # Include more HTTP information in the logs
option dontlognull # Don't log mundane things like checks
option redispatch # Allow sessions to move when a server goes down
# option http-server-close # Allow haproxy to close backend connections
# option forwardfor # Add X-Forwarded-For headed logging original IP
retries 3 # Number of failed check needed to consider a server down
maxconn 4096
timeout connect 60s # Timeout for a TCP connection to backend server
timeout check 60s # Timeout for a HTTP health check to complete (normally less than server timeout)
timeout client 120s # Timeout for client inactivity (recommended equal to the server timeout)
timeout server 120s # Timeout for the backend server returning the page
log-format %ci:%cp\ [%t]\ %ft\ %H\ %b/%s\ %Tq/%Tw/%Tc/%Tr/%Tt\ %ST\ %B\ %CC\ %CS\ %tsc\ %ac/%fc/%bc/%sc/%rc\ %sq/%bq\ %hr\ %hs\ %{+Q}r
#NFSEC TPROXY
frontend TPROXY-FRONTEND
mode http
bind 10.121.1.111:80 transparent
default_backend TPROXY-BACKEND
#NFROXY TPROXY BACKEND
backend TPROXY-BACKEND
mode http
balance source
hash-type consistent
source 0.0.0.0 usesrc clientip
timeout connect 1800s
timeout server 5m
server nfsec-web-01 10.121.1.49:80
server nfsec-web-02 10.121.1.50:80
# server backup 127.0.0.1:80 backup source 0.0.0.0
NOTE: option http-server-close has to be disabled for the TPROXY backend. Otherwise requests from client on port 80 will not be able to be forwarded to backend servers ports.
This is probably an answer to this post:
http://www.serverphorums.com/read.php?10,120994
http://www.serverphorums.com/read.php?10,129475
4. Testing
Working scenario:
It will work from any other subnet apart from the subnet the LB and backend servers are on. Perhaps changing net.ipv4.conf.default.acceptsourceroute = 0 to 2 may fix it, however I doubt.
[root@nfsec-lb ~]# netstat -pan |grep "111:"
tcp 0 0 10.121.1.111:80 0.0.0.0:* LISTEN 31640/haproxy
tcp 0 0 10.121.1.111:80 10.10.1.62:25467 ESTABLISHED 31640/haproxy
tcp 0 0 10.10.1.62:38343 10.121.1.50:80 ESTABLISHED 31640/haproxy
If you are testing from the same subnet you will get below SYN_ACK to backend.
[root@nfsec-lb ~]# netstat -pan |grep "111:"
tcp 0 0 10.121.1.111:80 0.0.0.0:* LISTEN 31640/haproxy
tcp 0 0 10.121.1.111:80 10.121.1.10:25467 ESTABLISHED 31640/haproxy
tcp 0 0 10.121.1.10:38343 10.121.1.50:80 SYN_ACK 31640/haproxy
If you have global http-server-close enabled you will not get anything redirecting to the backend.
- Additional scenario.
In situation when you have multiple VIP's each to be transparent pointing to the same backend servers which have multiple backend IP's, you won't be able to add multiple gateways on the backend servers pointing to each frontend VIP.
In this situation you just need to create a new frontend on LB and bind it to a new IP and set backend servers gateway to this IP.
What it does on LB it simply receives the returning traffic and via iptables mangle it then forwards it to the originating server via default route which is your ASA FW.
New GW is 192.168.1.3.
Connection flows:
Public IP: 33.33.11.77 -> 192.168.1.100 (NAT) - LB VIP: 192.168.1.100 sends to backend 192.168.1.201. [Connection created: 33.33.11.77 - 192.168.1.201]
Backend replies:
192.168.1.201 -> New GW: 192.168.1.3 LB sees the destination as 33.33.11.77 and via iptables mangle replies to TCP ESTABLISHED OPEN connection on 192.168.1.100 which routes to 33.33.11.77. [Connection created. .3 recevices but creates connetion as 192.168.1.100 -> 33.33.11.77 ]
:D
Conclusion
I hope you noticed that option http-server-close cannot be used in global configuration settings as well as uid/guid cannot be changed to haproxy as this will defeat the TPROXY requirements.
If you got a issue or want to discuss something - drop me a message or leave a comment.
