HAProxy is one of the most frequently used and efficient tools out there for load-balancing. It is highly configurable and can handle almost all of one’s needs to set up a HA, scalable infrastructure in both, HTTP and TCP. Its clientele is a testament to that as it is used and recommended by various heavy-hitters in the industry like Airbnb, Github, instagram, reddit, etc.
With this scale of usage, naturally, there’s a lot of help available in different communities if someone runs into a configuration problem. Especially when StackOverflow itself uses HAProxy extensively. That being said, in my experience, sometimes, it becomes a difficult and time-consuming process to troubleshoot HAProxy configuration issues when one is not well-versed in it. For this reason, we are going to discuss a few problems that we frequently encountered in our HAProxy setup and how we mitigated them.
Order of rules
HAProxy, like any other load-balancer, prioritises the rules that are written first in its configuration. That makes sense but sometimes, in path-based routing, one path could be a substring of the other path, for example,
acl is_kitchen_mapping_ui path_beg /kitchen use_backend dev_kitchen_mapping_ui if is_kitchen_mapping_ui acl is_kitchen_mapping_api path_beg /kitchen-api use_backend dev_kitchen_mapping_api if is_kitchen_mapping_api
Here, all the requests for /kitchen-api are most likely to go to /kitchen as /kitchen is written first and is a substring of /kitchen-api. Therefore, any request in the form of www.mapmykitchen.com/kitchen-api/?token=spoons is likely to go to www.mapmykitchen.com/kitchen/?token=spoons as it will be matched first.
This problem could be avoided by simply moving acl configuration for /kitchen-api above the one for /kitchen. All requests for /kitchen would still be routed to the correct path as they won’t be matched with /kitchen-api.
Redirect
Sometimes we have to move our endpoint to a new name. It could be due to any reason but it requires a change in the pathname for HTTP requests. In such cases, we have to redirect all requests directed at the old path to the new one. It could be configured in the HAProxy configuration like below,
acl old_kitchen_url path_beg /kitchen-king acl is_kitchen_mapping_ui path_beg /kitchen http-request redirect location /kitchen/ code 301 if old_kitchen_url use_backend dev_kitchen_mapping_ui if is_kitchen_mapping_ui
This would do the job but there’s an issue, it will not forward the whole request URL to the redirected endpoint as in www.mapmykitchen.com/kitchen-king/kitchen-api/?token=spoons will just become www.mapmykitchen.com/kitchen/. To redirect the whole request, we need to change the target string with regsub in captured request URL,
http-request redirect code 301 location //%[req.hdr(Host)]%[capture.req.uri,regsub("/kitchen-king/","/kitchen")] if old_kitchen_url
So the configuration becomes,
acl old_kitchen_url path_beg /kitchen-king
acl is_kitchen_mapping_ui path_beg /kitchen
http-request redirect code 301 location //%[req.hdr(Host)]%[capture.req.uri,regsub("/kitchen-king/","/kitchen")] if old_kitechen_url
use_backend dev_kitchen_mapping_ui if is_kitchen_mapping_ui
Conditions
We put a lot of conditions in our acl’s. Sometimes due to a miscalculation in logic, we might end up with an error or wrong path. Therefore, let’s get this concept straight once and for all. We work with three logical operators, AND, OR, and, NOT. The statements where we use these operators are if and unless.
Consider this example where we need to add a header called X-Datatype and route request on the basis of path and another header called X-Utensil-Category,
acl datatype_exists req.hdr(X-Datatype) -m found
This checks if X-Datatype header already exists.
http-request add-header X-Datatype “utensils” unless datatype_exists
This adds X-Datatype header if not already present.
acl is_spoons hdr(X-Utensil-Category) -i spoons
This does a case sensitive match on the value of header X-Utensil-Category.
acl is_kitchen_mapping_ui path_beg /kitchen use_backend dev_kitchen_mapping_ui if is_kitchen_mapping_ui is_spoons
Now, the request is routed to backend only when both is_kitchen_mapping_ui and is_spoons exist because the default operator is AND.
If we want to use backend when either of them exists, we can write,
use_backend dev_kitchen_mapping_ui if is_kitchen_mapping_ui or is_spoons
If we want to throw X-Datatype in the mix and as well and make it a required acl, we can write,
use_backend dev_kitchen_mapping_ui if is_kitchen_mapping_ui and datatype_exists or is_spoons and datatype_exists
Easily enough, we can put ‘!’ in front of an acl name to put a NOT condition,
use_backend dev_kitchen_mapping_ui if is_kitchen_mapping_ui !is_spoons
Here, the request will be routed to the backend when path /kitchen exists and header X-Utensil-Category does not exist.
Some useful points
- Sometimes we start getting 503 for the particular backend. We go to the servers and investigate. The application is working fine, listening to port/socket, ports are open, etc. In such cases make sure that if there’s a health check configured, it is working fine. If healthcheck fails, the request won’t go to the backend even if the backend is up.
- HAProxy provides a stats page where we can see the status of all backends. It gives detailed insight into live status of all the backends. It’s quite helpful if you have several backends configured. Read more about it https://www.haproxy.com/blog/exploring-the-haproxy-stats-page/
- HAProxy acl allows us to route requests based on a ton of conditions. The level of configuration it allows is quite interesting. Read more about it https://www.haproxy.com/blog/introduction-to-haproxy-acls/
In this blog, we mostly discussed issues and tips related to acl configuration. There’s a ton of things to talk about backend configuration as well. Backends also support regular expressions which can be used to redirect or edit requests, we can configure active/passive servers for our backends, configure healthchecks as mentioned above, configure consecutive healthchecks to mark a backend up or down, set or delete the header, load-balancing policy, and a lot of other things. We may cover those in another blog. Thanks for reading.
Image Source: NYTimes
