# Binadit - Full Content
> European managed cloud and infrastructure partner based in Rotterdam, NL, and maker of Binadit Workspace, the Dutch alternative to Google Workspace and Microsoft 365. This file contains Binadit Workspace with its prices, the European alternatives pages, and full article content from the engineering blog and tutorials, for AI grounding and citation. See /llms.txt for the navigation index.
Generated: 2026-10-22T17:24:43+02:00
Source: https://binadit.com/llms-full.txt
## Binadit Workspace
Website: https://binadit.work · Prices: https://binadit.work/en/pricing/
- Binadit Workspace is mail, calendar, contacts, drive, office documents, video meetings and appointment scheduling for companies that want to keep their data in Europe. One login, on the customer's own domain. Website: https://binadit.work
- It is built and run by Binadit B.V. in Rotterdam, a 100% Dutch-owned company with no foreign parent. Every server is in the Netherlands, in data centres of European companies, and is run by Binadit itself. No supplier outside the EU is involved. The customer signs a processing agreement under European law and there is no transfer to a third country.
- Because Binadit is a Dutch company without an American parent and operates the servers itself, the US CLOUD Act does not apply to it.
- Mail and calendar: mailbox with calendar and contacts on your own domain. Webmail in 27 languages. Works with any mail program through IMAP, SMTP, CalDAV and CardDAV, and JMAP. Shared mailboxes such as info@ with permissions per colleague. Two-step verification. SPF, DKIM and DMARC prepared per domain. Meeting invitations carry a Meet link. Built on the open source mail server Stalwart.
- Drive: file sync built by Binadit itself. Apps for macOS and Windows with files on demand; Linux, iOS and Android apps are in development. Sharing links that expire (30 days at most), team folders with permissions per colleague and their own quota, versions and a trash, search inside PDF, Word and Excel files. Files are encrypted at rest. Can also run on the customer's own Linux server, managed by Binadit.
- Receive (part of Drive, priced per company by storage): a page on the customer's own address where outsiders hand in files without an account or an app. Names and contents are encrypted in the sender's browser before upload, so the server only stores ciphertext; the password travels through a separate channel and cannot be recovered by anyone, Binadit included. Up to 100 files and 10 GB per transfer, kept 1 to 30 days (14 by default) and then deleted. Only addresses on the customer's own domains can receive. The administrator sees every transfer and can revoke it. It is the replacement for Cryptshare.
- Office: documents, spreadsheets and presentations in the browser, edited together at the same time, on files that stay on Drive. Opens and saves Word, Excel and PowerPoint files in their own formats, and OpenDocument. Built on Collabora Online, which is based on LibreOffice. Runs on Binadit's own servers in the Netherlands.
- Meet: video meetings started from the calendar or with one click. Guests join in the browser without an account and without installing anything. Only a Workspace account can start a meeting; guests wait until the host arrives. Runs on Binadit's own servers in the Netherlands and is based on the open source Jitsi.
- Book: a booking page for every mailbox, comparable to Calendly. It looks at all of a user's calendars and only shows times that are really free, supports several links per page, carries the customer's own logo and text, and sends confirmations from the customer's own mail server. The appointment lands in the calendar with a Meet link. Off by default; each user switches it on.
- Northmail: Binadit's own free mail program for macOS and Windows, with mail, calendar and contacts in the three-pane layout of classic Outlook. Works with Workspace, on-premises Exchange and Gmail side by side, and can move folders between accounts. An optional AI assistant runs locally on the user's computer and sends nothing to a cloud. No telemetry.
- Domains and DNS: every Workspace domain can be managed in the Binadit DNS tool in my.binadit.com, which places the MX, SPF, DKIM, DMARC and autoconfig records for Workspace.
- Prices (per month, excluding VAT, cancel monthly, no minimum number of mailboxes, valid from October 2026; full list and calculator at https://binadit.work/en/pricing/): a mailbox costs €2.95 for 2 GB, €3.95 for 5 GB, €4.95 for 10 GB, €5.95 for 25 GB, €7.95 for 50 GB and €10.95 for 100 GB, and every mailbox includes calendar, contacts, webmail, Northmail, Meet and Book. The Drive pool is one storage pool per company for everyone's personal drive, with Office included: 250 GB €9, 500 GB €15, 1 TB €25, 2 TB €40, 5 TB €75, 10 TB €120, 20 TB €200, 50 TB €450, 100 TB €800. Team Drive (shared folders, own storage next to the pool): 1 TB €29, 2 TB €45, 5 TB €85, 10 TB €140, 20 TB €240, 50 TB €500, 100 TB €900. Receive, per company by inbox storage: 50 GB €29, 250 GB €59, 1 TB €129, 5 TB €349.
- Support: one fixed contact who knows the customer's environment, reachable in a message.
## European alternatives to American office software
Index: https://binadit.com/alternatives · Background article: https://binadit.com/blog/email-gdpr-compliant-google-workspace-microsoft-365
### European alternative to Gmail
URL: https://binadit.com/alternatives/gmail
Gmail is where a lot of companies started: first a free address, later Gmail on their own domain. Either way the mailbox is with Google LLC, a US company. Binadit Workspace gives every colleague a mailbox with calendar and contacts on your own domain, hosted in the Netherlands by a Dutch company with no foreign parent.
A free Gmail address used for work falls under Google's consumer terms rather than under a processing agreement with your company, and it belongs to the person, not to the business. Gmail is a service of Google LLC, a US company, so using it is a transfer of personal data to the United States. That is lawful today on the strength of the EU-US Data Privacy Framework or of standard contractual clauses, and both lean on the same question: whether US law protects the data of Europeans well enough. The Court of Justice of the EU has answered no twice, and an appeal against the current framework is pending there. Separately, the US CLOUD Act obliges a US provider to produce data under its control wherever it is stored, so a European data region does not settle it. With Binadit Workspace there is no transfer to a third country, so there is nothing to monitor.
**Gmail, and what replaces it in Binadit Workspace**
- Mailboxes on your own domain: A Binadit Workspace mailbox with calendar and contacts, on your own domain. Your addresses stay the same. Only the MX records change.
- Webmail: The Binadit webmail, in 27 languages. Mail, calendar and contacts in the browser, with the same login as the rest of Workspace.
- Mail on computers and phones: Any mail program, through IMAP, SMTP, CalDAV and CardDAV, and JMAP where the program supports it. Or Northmail, our own free mail program for macOS and Windows. Outlook, Apple Mail, Thunderbird and the mail app on a phone connect through standard protocols.
- Shared addresses such as info@ and sales@: Shared mailboxes, with permissions per colleague. You create them in the portal, or we do it for you.
- Account and sender security: Two-step verification, and SPF, DKIM and DMARC prepared per domain. If your domain or its DNS is with Binadit, the records are placed for you.
- Free @gmail.com addresses used for work: Addresses on your own domain. The domain is yours, so an address stays with the company when a colleague leaves.
- Google Calendar, Google Contacts: Calendar and contacts in the same Workspace account as your mail. They sync to computers and phones over CalDAV and CardDAV, and they move together with the mailboxes.
- Google Meet links in invitations: Binadit Meet: video meetings from your calendar, with guests joining in the browser. Part of Binadit Workspace, with the same login as your mail.
- Gemini in Gmail: Not replaced, on purpose. We do not train on your data and we do not put a model inside your mailbox. Every AI feature inside a US suite is another data flow you answer for under Article 24 GDPR. If you want an assistant, choose one deliberately and know where it runs.
**What it costs** Per month, excluding VAT, cancel monthly, no minimum number of mailboxes. Prices valid from October 2026. The full list and a calculator that compares your company with Google Workspace, Microsoft 365 and Proton are on binadit.work.
- Mailbox, from €2.95 per mailbox per month: 2 GB €2.95, 5 GB €3.95, 10 GB €4.95, 25 GB €5.95, 50 GB €7.95, 100 GB €10.95. Every mailbox includes calendar, contacts, webmail, Northmail, Meet and Book. Choose the size per mailbox, and resize it at any time.
- Drive pool, per company, from €9 per month: 250 GB €9, 500 GB €15, 1 TB €25, 2 TB €40, 5 TB €75, 10 TB €120, 20 TB €200, 50 TB €450, 100 TB €800. One storage pool for the personal drives of everyone with Drive, with Office in the browser included. Above 100 TB a company gets its own Drive server: €250 per month plus €7 per TB.
Q: Is Gmail GDPR-compliant?
A: It can be used lawfully today. Gmail is run by a US company, so using it is a transfer of personal data to the United States, which rests on the EU-US Data Privacy Framework or on standard contractual clauses. That is as durable as those instruments: the Court of Justice of the EU struck down both earlier frameworks, and an appeal against the current one is pending there. Where the data is stored does not change which law the provider answers to.
Q: We use free @gmail.com addresses. Can we keep them?
A: Not as your company address: an @gmail.com address belongs to Google and to the person who opened it. Everybody gets an address on your own domain, and the old address can forward to the new one for as long as its owner keeps it.
Q: Will people miss the Gmail web interface?
A: The Binadit webmail gives you mail, calendar and contacts in the browser. It looks different from Gmail and does the same job. Most people work from a mail program on their computer and phone, and those keep working through standard protocols.
Q: Do our email addresses change?
A: No. The addresses are on your domain, and your domain is yours. We change where the MX records point; everybody keeps the address they have.
Q: What happens to our old mail?
A: It comes along. We copy historic mail, calendars and contacts to the new environment while the old one keeps running, and we do a final sync after the DNS switch.
Q: Is Binadit Workspace only this one product?
A: No. One login covers mail, calendar and contacts, Drive for your files, Receive for getting files from outsiders safely, Office for documents in the browser, Meet for video meetings and Book for appointments. Everything runs on servers in the Netherlands that Binadit operates itself.
Q: What does Binadit Workspace cost?
A: A Binadit Workspace mailbox costs from €2.95 per month, with calendar, contacts, webmail, Northmail, Meet and Book included, and the Drive pool for the whole company starts at €9 per month. Prices are per month, excluding VAT, and can be cancelled monthly. The full list and a cost calculator are on binadit.work.
Q: Is Binadit ISO 27001 or NEN 7510 certified?
A: The data centre we host in is ISO 27001 and NEN 7510 certified. Binadit itself, as a company, does not hold these certifications. If your own certification requires a certified supplier rather than a certified data centre, tell us before you decide, so you can check that with your auditor first.
---
### European alternative to Zoho Mail
URL: https://binadit.com/alternatives/zoho-mail
Zoho Mail is a capable suite at a low price, from Zoho Corporation, a group headquartered in Chennai, India. European customers are served from data centres in Amsterdam and Dublin, and the company behind them answers to law outside the EU. Binadit Workspace offers mail, calendar, files, meetings and scheduling from the Netherlands, from a Dutch company with no foreign parent.
Zoho stores European accounts in the Netherlands and Ireland, which settles where the data is. It does not settle who controls it. Zoho is an Indian group, India has no adequacy decision from the European Commission, and Zoho's own privacy policy says that transfers within the group rely on standard contractual clauses. That makes the transfer assessment yours to keep up to date. With Binadit Workspace there is no transfer to a third country, so there is nothing to assess.
**Zoho Mail, and what replaces it in Binadit Workspace**
- Mailboxes on your own domain: A Binadit Workspace mailbox with calendar and contacts, on your own domain. Your addresses stay the same. Only the MX records change.
- Webmail: The Binadit webmail, in 27 languages. Mail, calendar and contacts in the browser, with the same login as the rest of Workspace.
- Mail on computers and phones: Any mail program, through IMAP, SMTP, CalDAV and CardDAV, and JMAP where the program supports it. Or Northmail, our own free mail program for macOS and Windows. Outlook, Apple Mail, Thunderbird and the mail app on a phone connect through standard protocols.
- Shared addresses such as info@ and sales@: Shared mailboxes, with permissions per colleague. You create them in the portal, or we do it for you.
- Account and sender security: Two-step verification, and SPF, DKIM and DMARC prepared per domain. If your domain or its DNS is with Binadit, the records are placed for you.
- Zoho WorkDrive: Binadit Drive: your files on a server in the Netherlands, synchronised with your computer, with team folders, sharing links and versions. Part of Binadit Workspace, with the same login as your mail.
- Zoho Writer, Sheet, Show: Binadit Office: documents, spreadsheets and presentations in the browser, edited together, on files that stay on Drive. Word, Excel and PowerPoint files open and save in their own formats.
- Zoho Meeting: Binadit Meet: video meetings from your calendar, with guests joining in the browser. Part of Binadit Workspace, with the same login as your mail.
- Zoho Bookings: Binadit Book: a booking page for every mailbox. The appointment lands in your calendar with a Meet link. Off by default. Every user switches it on themselves.
**What it costs** Per month, excluding VAT, cancel monthly, no minimum number of mailboxes. Prices valid from October 2026. The full list and a calculator that compares your company with Google Workspace, Microsoft 365 and Proton are on binadit.work.
- Mailbox, from €2.95 per mailbox per month: 2 GB €2.95, 5 GB €3.95, 10 GB €4.95, 25 GB €5.95, 50 GB €7.95, 100 GB €10.95. Every mailbox includes calendar, contacts, webmail, Northmail, Meet and Book. Choose the size per mailbox, and resize it at any time.
- Drive pool, per company, from €9 per month: 250 GB €9, 500 GB €15, 1 TB €25, 2 TB €40, 5 TB €75, 10 TB €120, 20 TB €200, 50 TB €450, 100 TB €800. One storage pool for the personal drives of everyone with Drive, with Office in the browser included. Above 100 TB a company gets its own Drive server: €250 per month plus €7 per TB.
Q: Zoho has data centres in Europe. Why move?
A: Because a data centre answers where, not who. The accounts are stored in the EU, and the group that operates them is headquartered in India, a country without an adequacy decision. You remain responsible for assessing that transfer.
Q: Zoho is more than mail. What about the CRM and the other apps?
A: Binadit Workspace replaces the office part: mail, calendar, files, documents, meetings and scheduling. It is not a CRM or an accounting package. If you use those from Zoho, they are a separate decision.
Q: Do our email addresses change?
A: No. The addresses are on your domain, and your domain is yours. We change where the MX records point; everybody keeps the address they have.
Q: What happens to our old mail?
A: It comes along. We copy historic mail, calendars and contacts to the new environment while the old one keeps running, and we do a final sync after the DNS switch.
Q: Is Binadit Workspace only this one product?
A: No. One login covers mail, calendar and contacts, Drive for your files, Receive for getting files from outsiders safely, Office for documents in the browser, Meet for video meetings and Book for appointments. Everything runs on servers in the Netherlands that Binadit operates itself.
Q: What does Binadit Workspace cost?
A: A Binadit Workspace mailbox costs from €2.95 per month, with calendar, contacts, webmail, Northmail, Meet and Book included, and the Drive pool for the whole company starts at €9 per month. Prices are per month, excluding VAT, and can be cancelled monthly. The full list and a cost calculator are on binadit.work.
Q: Is Binadit ISO 27001 or NEN 7510 certified?
A: The data centre we host in is ISO 27001 and NEN 7510 certified. Binadit itself, as a company, does not hold these certifications. If your own certification requires a certified supplier rather than a certified data centre, tell us before you decide, so you can check that with your auditor first.
---
### European alternative to Proton Mail
URL: https://binadit.com/alternatives/proton-mail
Proton is a serious choice, and this page does not pretend otherwise. Proton AG is a Swiss company, and its mail is end-to-end encrypted between Proton users. Binadit Workspace makes a different trade: a Dutch company under EU law, open standards so that every mail program works without extra software, and one login for mail, files, documents, meetings and scheduling.
Switzerland is not in the EU. A transfer there is lawful, because the European Commission has recognised Swiss data protection as adequate, so this is not a CLOUD Act story. The differences are practical. Your contract and your processing agreement fall under Swiss law rather than EU law. A standard mail program needs Proton Mail Bridge, which Proton offers on paid plans. And if end-to-end encryption between mailboxes is what you need above all, Proton does that and Binadit Workspace does not: our mail is encrypted in transit and at rest on the server.
**Proton Mail, and what replaces it in Binadit Workspace**
- Mailboxes on your own domain: A Binadit Workspace mailbox with calendar and contacts, on your own domain. Your addresses stay the same. Only the MX records change.
- Webmail: The Binadit webmail, in 27 languages. Mail, calendar and contacts in the browser, with the same login as the rest of Workspace.
- Mail on computers and phones: Any mail program, through IMAP, SMTP, CalDAV and CardDAV, and JMAP where the program supports it. Or Northmail, our own free mail program for macOS and Windows. Outlook, Apple Mail, Thunderbird and the mail app on a phone connect through standard protocols.
- Shared addresses such as info@ and sales@: Shared mailboxes, with permissions per colleague. You create them in the portal, or we do it for you.
- Account and sender security: Two-step verification, and SPF, DKIM and DMARC prepared per domain. If your domain or its DNS is with Binadit, the records are placed for you.
- Proton Mail Bridge: Not needed. Workspace speaks IMAP, SMTP, CalDAV and CardDAV directly, and JMAP. Outlook, Apple Mail, Thunderbird and the mail app on a phone connect without extra software.
- End-to-end encryption between Proton users: Not replaced. Workspace mail is encrypted in transit and at rest on the server. For files from outside, Binadit Receive encrypts in the sender's browser, so the server only stores ciphertext.
- Proton Drive: Binadit Drive: your files on a server in the Netherlands, synchronised with your computer, with team folders, sharing links and versions. Part of Binadit Workspace, with the same login as your mail.
- Proton Docs: Binadit Office: documents, spreadsheets and presentations in the browser, edited together, on files that stay on Drive. Word, Excel and PowerPoint files open and save in their own formats.
- Proton Meet: Binadit Meet: video meetings from your calendar, with guests joining in the browser. Part of Binadit Workspace, with the same login as your mail.
**What it costs** Per month, excluding VAT, cancel monthly, no minimum number of mailboxes. Prices valid from October 2026. The full list and a calculator that compares your company with Google Workspace, Microsoft 365 and Proton are on binadit.work.
- Mailbox, from €2.95 per mailbox per month: 2 GB €2.95, 5 GB €3.95, 10 GB €4.95, 25 GB €5.95, 50 GB €7.95, 100 GB €10.95. Every mailbox includes calendar, contacts, webmail, Northmail, Meet and Book. Choose the size per mailbox, and resize it at any time.
- Drive pool, per company, from €9 per month: 250 GB €9, 500 GB €15, 1 TB €25, 2 TB €40, 5 TB €75, 10 TB €120, 20 TB €200, 50 TB €450, 100 TB €800. One storage pool for the personal drives of everyone with Drive, with Office in the browser included. Above 100 TB a company gets its own Drive server: €250 per month plus €7 per TB.
Q: Is Proton not safer because of end-to-end encryption?
A: For mail between Proton users that is a real advantage, and we say so. Mail to and from everybody else travels the way all mail does. What Binadit Workspace offers instead is a provider under EU law, standard protocols and one fixed contact who knows your setup. Which matters more depends on what you are protecting against.
Q: Is a Swiss provider a problem under the GDPR?
A: No. Switzerland has an adequacy decision, so the transfer is lawful. The point is narrower: a Swiss provider answers to Swiss law and your agreement with it falls under Swiss law. With a Dutch provider both fall under EU law.
Q: Do our email addresses change?
A: No. The addresses are on your domain, and your domain is yours. We change where the MX records point; everybody keeps the address they have.
Q: What happens to our old mail?
A: It comes along. We copy historic mail, calendars and contacts to the new environment while the old one keeps running, and we do a final sync after the DNS switch.
Q: Is Binadit Workspace only this one product?
A: No. One login covers mail, calendar and contacts, Drive for your files, Receive for getting files from outsiders safely, Office for documents in the browser, Meet for video meetings and Book for appointments. Everything runs on servers in the Netherlands that Binadit operates itself.
Q: What does Binadit Workspace cost?
A: A Binadit Workspace mailbox costs from €2.95 per month, with calendar, contacts, webmail, Northmail, Meet and Book included, and the Drive pool for the whole company starts at €9 per month. Prices are per month, excluding VAT, and can be cancelled monthly. The full list and a cost calculator are on binadit.work.
Q: Is Binadit ISO 27001 or NEN 7510 certified?
A: The data centre we host in is ISO 27001 and NEN 7510 certified. Binadit itself, as a company, does not hold these certifications. If your own certification requires a certified supplier rather than a certified data centre, tell us before you decide, so you can check that with your auditor first.
---
### European alternative to Fastmail
URL: https://binadit.com/alternatives/fastmail
Fastmail is well-made mail on open standards, from Fastmail Pty Ltd in Melbourne, Australia. Its servers are in the United States and, since August 2026, in Amsterdam. Binadit Workspace keeps the open standards and moves the rest to the Netherlands: a Dutch company with no foreign parent, with files, documents, meetings and scheduling under the same login.
Australia has no adequacy decision from the European Commission, so mail at Fastmail is a transfer you have to justify yourself. Fastmail opened a European data region in Amsterdam in August 2026 and is candid about its limits: backups and logs remain in the United States, and in its own words, "If what you need is a guarantee that your data remains only in the EU, we don't have that." With Binadit Workspace every server is in the Netherlands and there is no transfer to a third country.
**Fastmail, and what replaces it in Binadit Workspace**
- Mailboxes on your own domain: A Binadit Workspace mailbox with calendar and contacts, on your own domain. Your addresses stay the same. Only the MX records change.
- Webmail: The Binadit webmail, in 27 languages. Mail, calendar and contacts in the browser, with the same login as the rest of Workspace.
- Mail on computers and phones: Any mail program, through IMAP, SMTP, CalDAV and CardDAV, and JMAP where the program supports it. Or Northmail, our own free mail program for macOS and Windows. Outlook, Apple Mail, Thunderbird and the mail app on a phone connect through standard protocols.
- Shared addresses such as info@ and sales@: Shared mailboxes, with permissions per colleague. You create them in the portal, or we do it for you.
- Account and sender security: Two-step verification, and SPF, DKIM and DMARC prepared per domain. If your domain or its DNS is with Binadit, the records are placed for you.
- IMAP, CalDAV, CardDAV, JMAP: The same open standards, so your mail programs keep working as they do now. Both services speak JMAP, the modern successor to IMAP.
- Fastmail calendar and contacts: Calendar and contacts in the same Workspace account as your mail. They sync to computers and phones over CalDAV and CardDAV, and they move together with the mailboxes.
- Fastmail file storage: Binadit Drive: your files on a server in the Netherlands, synchronised with your computer, with team folders, sharing links and versions. Part of Binadit Workspace, with the same login as your mail.
**What it costs** Per month, excluding VAT, cancel monthly, no minimum number of mailboxes. Prices valid from October 2026. The full list and a calculator that compares your company with Google Workspace, Microsoft 365 and Proton are on binadit.work.
- Mailbox, from €2.95 per mailbox per month: 2 GB €2.95, 5 GB €3.95, 10 GB €4.95, 25 GB €5.95, 50 GB €7.95, 100 GB €10.95. Every mailbox includes calendar, contacts, webmail, Northmail, Meet and Book. Choose the size per mailbox, and resize it at any time.
- Drive pool, per company, from €9 per month: 250 GB €9, 500 GB €15, 1 TB €25, 2 TB €40, 5 TB €75, 10 TB €120, 20 TB €200, 50 TB €450, 100 TB €800. One storage pool for the personal drives of everyone with Drive, with Office in the browser included. Above 100 TB a company gets its own Drive server: €250 per month plus €7 per TB.
Q: Fastmail has a European data region now. Is that not enough?
A: It decides where your live mail is stored. Fastmail itself says that backups and logs stay in the United States and that it cannot guarantee EU-only storage. And the company remains Australian, so the transfer rules of the GDPR keep applying.
Q: Do our email addresses change?
A: No. The addresses are on your domain, and your domain is yours. We change where the MX records point; everybody keeps the address they have.
Q: What happens to our old mail?
A: It comes along. We copy historic mail, calendars and contacts to the new environment while the old one keeps running, and we do a final sync after the DNS switch.
Q: Is Binadit Workspace only this one product?
A: No. One login covers mail, calendar and contacts, Drive for your files, Receive for getting files from outsiders safely, Office for documents in the browser, Meet for video meetings and Book for appointments. Everything runs on servers in the Netherlands that Binadit operates itself.
Q: What does Binadit Workspace cost?
A: A Binadit Workspace mailbox costs from €2.95 per month, with calendar, contacts, webmail, Northmail, Meet and Book included, and the Drive pool for the whole company starts at €9 per month. Prices are per month, excluding VAT, and can be cancelled monthly. The full list and a cost calculator are on binadit.work.
Q: Is Binadit ISO 27001 or NEN 7510 certified?
A: The data centre we host in is ISO 27001 and NEN 7510 certified. Binadit itself, as a company, does not hold these certifications. If your own certification requires a certified supplier rather than a certified data centre, tell us before you decide, so you can check that with your auditor first.
---
### European alternative to iCloud Mail
URL: https://binadit.com/alternatives/icloud-mail
iCloud Mail on a custom domain is how many small firms got a professional address: it came with the iCloud+ subscription somebody already had. The mailbox is with Apple Inc., a US company. Binadit Workspace gives the company its own mailboxes on its own domain, with shared mailboxes, files, meetings and scheduling, hosted in the Netherlands.
A custom domain in iCloud+ is tied to a personal Apple Account and shared through Family Sharing. That works for a household. For a company it means that business mail lives in the private accounts of the people who work there. iCloud Mail is a service of Apple Inc., a US company, so using it is a transfer of personal data to the United States. That is lawful today on the strength of the EU-US Data Privacy Framework or of standard contractual clauses, and both lean on the same question: whether US law protects the data of Europeans well enough. The Court of Justice of the EU has answered no twice, and an appeal against the current framework is pending there. Separately, the US CLOUD Act obliges a US provider to produce data under its control wherever it is stored, so a European data region does not settle it. With Binadit Workspace there is no transfer to a third country, so there is nothing to monitor.
**iCloud Mail, and what replaces it in Binadit Workspace**
- Mailboxes on your own domain: A Binadit Workspace mailbox with calendar and contacts, on your own domain. Your addresses stay the same. Only the MX records change.
- Webmail: The Binadit webmail, in 27 languages. Mail, calendar and contacts in the browser, with the same login as the rest of Workspace.
- Mail on computers and phones: Any mail program, through IMAP, SMTP, CalDAV and CardDAV, and JMAP where the program supports it. Or Northmail, our own free mail program for macOS and Windows. Outlook, Apple Mail, Thunderbird and the mail app on a phone connect through standard protocols.
- Shared addresses such as info@ and sales@: Shared mailboxes, with permissions per colleague. You create them in the portal, or we do it for you.
- Account and sender security: Two-step verification, and SPF, DKIM and DMARC prepared per domain. If your domain or its DNS is with Binadit, the records are placed for you.
- Custom email domain in iCloud+: Your domain in Binadit Workspace, with the mailboxes the company needs. Managed by the company in the portal, not from one person's private account.
- iCloud Calendar and Contacts: Calendar and contacts in the same Workspace account as your mail. They sync to computers and phones over CalDAV and CardDAV, and they move together with the mailboxes.
- iCloud Drive: Binadit Drive: your files on a server in the Netherlands, synchronised with your computer, with team folders, sharing links and versions. Part of Binadit Workspace, with the same login as your mail.
- Apple Mail on Mac, iPhone and iPad: Keep it. Apple Mail, Calendar and Contacts connect to Workspace through standard protocols.
**What it costs** Per month, excluding VAT, cancel monthly, no minimum number of mailboxes. Prices valid from October 2026. The full list and a calculator that compares your company with Google Workspace, Microsoft 365 and Proton are on binadit.work.
- Mailbox, from €2.95 per mailbox per month: 2 GB €2.95, 5 GB €3.95, 10 GB €4.95, 25 GB €5.95, 50 GB €7.95, 100 GB €10.95. Every mailbox includes calendar, contacts, webmail, Northmail, Meet and Book. Choose the size per mailbox, and resize it at any time.
- Drive pool, per company, from €9 per month: 250 GB €9, 500 GB €15, 1 TB €25, 2 TB €40, 5 TB €75, 10 TB €120, 20 TB €200, 50 TB €450, 100 TB €800. One storage pool for the personal drives of everyone with Drive, with Office in the browser included. Above 100 TB a company gets its own Drive server: €250 per month plus €7 per TB.
Q: Is iCloud Mail GDPR-compliant?
A: It can be used lawfully today. iCloud Mail is run by a US company, so using it is a transfer of personal data to the United States, which rests on the EU-US Data Privacy Framework or on standard contractual clauses. That is as durable as those instruments: the Court of Justice of the EU struck down both earlier frameworks, and an appeal against the current one is pending there. Where the data is stored does not change which law the provider answers to.
Q: Can we keep using our iPhones and Macs the way we do now?
A: Yes. Apple Mail, Calendar and Contacts work with Binadit Workspace through IMAP, CalDAV and CardDAV. Only the account behind them changes.
Q: Do our email addresses change?
A: No. The addresses are on your domain, and your domain is yours. We change where the MX records point; everybody keeps the address they have.
Q: What happens to our old mail?
A: It comes along. We copy historic mail, calendars and contacts to the new environment while the old one keeps running, and we do a final sync after the DNS switch.
Q: Is Binadit Workspace only this one product?
A: No. One login covers mail, calendar and contacts, Drive for your files, Receive for getting files from outsiders safely, Office for documents in the browser, Meet for video meetings and Book for appointments. Everything runs on servers in the Netherlands that Binadit operates itself.
Q: What does Binadit Workspace cost?
A: A Binadit Workspace mailbox costs from €2.95 per month, with calendar, contacts, webmail, Northmail, Meet and Book included, and the Drive pool for the whole company starts at €9 per month. Prices are per month, excluding VAT, and can be cancelled monthly. The full list and a cost calculator are on binadit.work.
Q: Is Binadit ISO 27001 or NEN 7510 certified?
A: The data centre we host in is ISO 27001 and NEN 7510 certified. Binadit itself, as a company, does not hold these certifications. If your own certification requires a certified supplier rather than a certified data centre, tell us before you decide, so you can check that with your auditor first.
---
### European alternative to Rackspace Email
URL: https://binadit.com/alternatives/rackspace-email
Rackspace Email is plain, affordable business mail from Rackspace Technology, a US company in San Antonio, Texas. Binadit Workspace is plain business mail too, hosted in the Netherlands by a Dutch company, and it brings calendar, files, documents, meetings and scheduling under the same login.
Companies usually chose Rackspace Email for the price, not for the jurisdiction, and rarely looked at it again. Rackspace Email is a service of Rackspace Technology, Inc., a US company, so using it is a transfer of personal data to the United States. That is lawful today on the strength of the EU-US Data Privacy Framework or of standard contractual clauses, and both lean on the same question: whether US law protects the data of Europeans well enough. The Court of Justice of the EU has answered no twice, and an appeal against the current framework is pending there. Separately, the US CLOUD Act obliges a US provider to produce data under its control wherever it is stored, so a European data region does not settle it. With Binadit Workspace there is no transfer to a third country, so there is nothing to monitor.
**Rackspace Email, and what replaces it in Binadit Workspace**
- Mailboxes on your own domain: A Binadit Workspace mailbox with calendar and contacts, on your own domain. Your addresses stay the same. Only the MX records change.
- Webmail: The Binadit webmail, in 27 languages. Mail, calendar and contacts in the browser, with the same login as the rest of Workspace.
- Mail on computers and phones: Any mail program, through IMAP, SMTP, CalDAV and CardDAV, and JMAP where the program supports it. Or Northmail, our own free mail program for macOS and Windows. Outlook, Apple Mail, Thunderbird and the mail app on a phone connect through standard protocols.
- Shared addresses such as info@ and sales@: Shared mailboxes, with permissions per colleague. You create them in the portal, or we do it for you.
- Account and sender security: Two-step verification, and SPF, DKIM and DMARC prepared per domain. If your domain or its DNS is with Binadit, the records are placed for you.
**What it costs** Per month, excluding VAT, cancel monthly, no minimum number of mailboxes. Prices valid from October 2026. The full list and a calculator that compares your company with Google Workspace, Microsoft 365 and Proton are on binadit.work.
- Mailbox, from €2.95 per mailbox per month: 2 GB €2.95, 5 GB €3.95, 10 GB €4.95, 25 GB €5.95, 50 GB €7.95, 100 GB €10.95. Every mailbox includes calendar, contacts, webmail, Northmail, Meet and Book. Choose the size per mailbox, and resize it at any time.
- Drive pool, per company, from €9 per month: 250 GB €9, 500 GB €15, 1 TB €25, 2 TB €40, 5 TB €75, 10 TB €120, 20 TB €200, 50 TB €450, 100 TB €800. One storage pool for the personal drives of everyone with Drive, with Office in the browser included. Above 100 TB a company gets its own Drive server: €250 per month plus €7 per TB.
Q: Is Rackspace Email GDPR-compliant?
A: It can be used lawfully today. Rackspace Email is run by a US company, so using it is a transfer of personal data to the United States, which rests on the EU-US Data Privacy Framework or on standard contractual clauses. That is as durable as those instruments: the Court of Justice of the EU struck down both earlier frameworks, and an appeal against the current one is pending there. Where the data is stored does not change which law the provider answers to.
Q: Do our email addresses change?
A: No. The addresses are on your domain, and your domain is yours. We change where the MX records point; everybody keeps the address they have.
Q: What happens to our old mail?
A: It comes along. We copy historic mail, calendars and contacts to the new environment while the old one keeps running, and we do a final sync after the DNS switch.
Q: Is Binadit Workspace only this one product?
A: No. One login covers mail, calendar and contacts, Drive for your files, Receive for getting files from outsiders safely, Office for documents in the browser, Meet for video meetings and Book for appointments. Everything runs on servers in the Netherlands that Binadit operates itself.
Q: What does Binadit Workspace cost?
A: A Binadit Workspace mailbox costs from €2.95 per month, with calendar, contacts, webmail, Northmail, Meet and Book included, and the Drive pool for the whole company starts at €9 per month. Prices are per month, excluding VAT, and can be cancelled monthly. The full list and a cost calculator are on binadit.work.
Q: Is Binadit ISO 27001 or NEN 7510 certified?
A: The data centre we host in is ISO 27001 and NEN 7510 certified. Binadit itself, as a company, does not hold these certifications. If your own certification requires a certified supplier rather than a certified data centre, tell us before you decide, so you can check that with your auditor first.
---
### European alternative to GoDaddy Email
URL: https://binadit.com/alternatives/godaddy-email
GoDaddy sells email with the domain: Professional Email, powered by Titan, or Microsoft 365 under its own name. Either way your mail is with GoDaddy Inc. of Tempe, Arizona, a US company, and with the supplier behind it. Binadit Workspace is mail we build and run ourselves in the Netherlands, on your own domain, with one fixed contact.
With GoDaddy the company you pay is not the company that runs your mailbox. Professional Email is powered by Titan and Microsoft 365 from GoDaddy is Microsoft's service, which leaves you with two suppliers to account for. GoDaddy Email is a service of GoDaddy Inc., a US company, so using it is a transfer of personal data to the United States. That is lawful today on the strength of the EU-US Data Privacy Framework or of standard contractual clauses, and both lean on the same question: whether US law protects the data of Europeans well enough. The Court of Justice of the EU has answered no twice, and an appeal against the current framework is pending there. Separately, the US CLOUD Act obliges a US provider to produce data under its control wherever it is stored, so a European data region does not settle it. With Binadit Workspace there is no transfer to a third country, so there is nothing to monitor.
**GoDaddy Email, and what replaces it in Binadit Workspace**
- Mailboxes on your own domain: A Binadit Workspace mailbox with calendar and contacts, on your own domain. Your addresses stay the same. Only the MX records change.
- Webmail: The Binadit webmail, in 27 languages. Mail, calendar and contacts in the browser, with the same login as the rest of Workspace.
- Mail on computers and phones: Any mail program, through IMAP, SMTP, CalDAV and CardDAV, and JMAP where the program supports it. Or Northmail, our own free mail program for macOS and Windows. Outlook, Apple Mail, Thunderbird and the mail app on a phone connect through standard protocols.
- Shared addresses such as info@ and sales@: Shared mailboxes, with permissions per colleague. You create them in the portal, or we do it for you.
- Account and sender security: Two-step verification, and SPF, DKIM and DMARC prepared per domain. If your domain or its DNS is with Binadit, the records are placed for you.
- Microsoft 365 from GoDaddy: A Workspace mailbox, with Northmail or the mail program you already use. Outlook keeps working through standard protocols.
- Domain and DNS at GoDaddy: The Binadit DNS tool places the MX, SPF, DKIM and DMARC records for you. The domain can stay registered where it is: point its nameservers to Binadit, or set the records we give you.
**What it costs** Per month, excluding VAT, cancel monthly, no minimum number of mailboxes. Prices valid from October 2026. The full list and a calculator that compares your company with Google Workspace, Microsoft 365 and Proton are on binadit.work.
- Mailbox, from €2.95 per mailbox per month: 2 GB €2.95, 5 GB €3.95, 10 GB €4.95, 25 GB €5.95, 50 GB €7.95, 100 GB €10.95. Every mailbox includes calendar, contacts, webmail, Northmail, Meet and Book. Choose the size per mailbox, and resize it at any time.
- Drive pool, per company, from €9 per month: 250 GB €9, 500 GB €15, 1 TB €25, 2 TB €40, 5 TB €75, 10 TB €120, 20 TB €200, 50 TB €450, 100 TB €800. One storage pool for the personal drives of everyone with Drive, with Office in the browser included. Above 100 TB a company gets its own Drive server: €250 per month plus €7 per TB.
Q: Is GoDaddy Email GDPR-compliant?
A: It can be used lawfully today. GoDaddy Email is run by a US company, so using it is a transfer of personal data to the United States, which rests on the EU-US Data Privacy Framework or on standard contractual clauses. That is as durable as those instruments: the Court of Justice of the EU struck down both earlier frameworks, and an appeal against the current one is pending there. Where the data is stored does not change which law the provider answers to.
Q: Does our domain have to leave GoDaddy?
A: No. The domain can stay registered where it is. You either point its nameservers to Binadit or set the records we give you. Both work.
Q: Do our email addresses change?
A: No. The addresses are on your domain, and your domain is yours. We change where the MX records point; everybody keeps the address they have.
Q: What happens to our old mail?
A: It comes along. We copy historic mail, calendars and contacts to the new environment while the old one keeps running, and we do a final sync after the DNS switch.
Q: Is Binadit Workspace only this one product?
A: No. One login covers mail, calendar and contacts, Drive for your files, Receive for getting files from outsiders safely, Office for documents in the browser, Meet for video meetings and Book for appointments. Everything runs on servers in the Netherlands that Binadit operates itself.
Q: What does Binadit Workspace cost?
A: A Binadit Workspace mailbox costs from €2.95 per month, with calendar, contacts, webmail, Northmail, Meet and Book included, and the Drive pool for the whole company starts at €9 per month. Prices are per month, excluding VAT, and can be cancelled monthly. The full list and a cost calculator are on binadit.work.
Q: Is Binadit ISO 27001 or NEN 7510 certified?
A: The data centre we host in is ISO 27001 and NEN 7510 certified. Binadit itself, as a company, does not hold these certifications. If your own certification requires a certified supplier rather than a certified data centre, tell us before you decide, so you can check that with your auditor first.
---
### European alternative to Amazon WorkMail
URL: https://binadit.com/alternatives/amazon-workmail
Amazon WorkMail is being shut down. AWS has announced the end of support for 31 March 2027, and after that date the service can no longer be used. If you have to move anyway, this is the moment to move to a provider under European law. Binadit Workspace hosts mail, calendar and contacts in the Netherlands, with files, documents, meetings and scheduling under the same login.
The first reason is the date: AWS ends support for Amazon WorkMail on 31 March 2027 and stopped accepting new customers on 30 April 2026. The second is the one that was always there. Amazon WorkMail is a service of Amazon Web Services, a US company, so using it is a transfer of personal data to the United States. That is lawful today on the strength of the EU-US Data Privacy Framework or of standard contractual clauses, and both lean on the same question: whether US law protects the data of Europeans well enough. The Court of Justice of the EU has answered no twice, and an appeal against the current framework is pending there. Separately, the US CLOUD Act obliges a US provider to produce data under its control wherever it is stored, so a European data region does not settle it. With Binadit Workspace there is no transfer to a third country, so there is nothing to monitor.
**Amazon WorkMail, and what replaces it in Binadit Workspace**
- Mailboxes on your own domain: A Binadit Workspace mailbox with calendar and contacts, on your own domain. Your addresses stay the same. Only the MX records change.
- Webmail: The Binadit webmail, in 27 languages. Mail, calendar and contacts in the browser, with the same login as the rest of Workspace.
- Mail on computers and phones: Any mail program, through IMAP, SMTP, CalDAV and CardDAV, and JMAP where the program supports it. Or Northmail, our own free mail program for macOS and Windows. Outlook, Apple Mail, Thunderbird and the mail app on a phone connect through standard protocols.
- Shared addresses such as info@ and sales@: Shared mailboxes, with permissions per colleague. You create them in the portal, or we do it for you.
- Account and sender security: Two-step verification, and SPF, DKIM and DMARC prepared per domain. If your domain or its DNS is with Binadit, the records are placed for you.
- WorkMail organisation, users and aliases: Mailboxes, shared mailboxes and aliases in the Binadit portal. You create them in the portal, or we do it for you.
- Europe (Ireland) region: Servers in the Netherlands, operated by Binadit itself. A region decides where the data is stored. It does not decide which law the provider answers to.
**What it costs** Per month, excluding VAT, cancel monthly, no minimum number of mailboxes. Prices valid from October 2026. The full list and a calculator that compares your company with Google Workspace, Microsoft 365 and Proton are on binadit.work.
- Mailbox, from €2.95 per mailbox per month: 2 GB €2.95, 5 GB €3.95, 10 GB €4.95, 25 GB €5.95, 50 GB €7.95, 100 GB €10.95. Every mailbox includes calendar, contacts, webmail, Northmail, Meet and Book. Choose the size per mailbox, and resize it at any time.
- Drive pool, per company, from €9 per month: 250 GB €9, 500 GB €15, 1 TB €25, 2 TB €40, 5 TB €75, 10 TB €120, 20 TB €200, 50 TB €450, 100 TB €800. One storage pool for the personal drives of everyone with Drive, with Office in the browser included. Above 100 TB a company gets its own Drive server: €250 per month plus €7 per TB.
Q: Is Amazon WorkMail GDPR-compliant?
A: It can be used lawfully today. Amazon WorkMail is run by a US company, so using it is a transfer of personal data to the United States, which rests on the EU-US Data Privacy Framework or on standard contractual clauses. That is as durable as those instruments: the Court of Justice of the EU struck down both earlier frameworks, and an appeal against the current one is pending there. Where the data is stored does not change which law the provider answers to.
Q: When do we have to be out of Amazon WorkMail?
A: AWS ends support on 31 March 2027, and after that date the service can no longer be used. A mail migration runs side by side with the old environment, so start early enough to choose your own switch day.
Q: Do our email addresses change?
A: No. The addresses are on your domain, and your domain is yours. We change where the MX records point; everybody keeps the address they have.
Q: What happens to our old mail?
A: It comes along. We copy historic mail, calendars and contacts to the new environment while the old one keeps running, and we do a final sync after the DNS switch.
Q: Is Binadit Workspace only this one product?
A: No. One login covers mail, calendar and contacts, Drive for your files, Receive for getting files from outsiders safely, Office for documents in the browser, Meet for video meetings and Book for appointments. Everything runs on servers in the Netherlands that Binadit operates itself.
Q: What does Binadit Workspace cost?
A: A Binadit Workspace mailbox costs from €2.95 per month, with calendar, contacts, webmail, Northmail, Meet and Book included, and the Drive pool for the whole company starts at €9 per month. Prices are per month, excluding VAT, and can be cancelled monthly. The full list and a cost calculator are on binadit.work.
Q: Is Binadit ISO 27001 or NEN 7510 certified?
A: The data centre we host in is ISO 27001 and NEN 7510 certified. Binadit itself, as a company, does not hold these certifications. If your own certification requires a certified supplier rather than a certified data centre, tell us before you decide, so you can check that with your auditor first.
---
### European alternative to Outlook
URL: https://binadit.com/alternatives/outlook
Outlook is a mail program, and for years it kept your mail on your own computer. The new Outlook works differently: for accounts from other providers it synchronises a copy through the Microsoft cloud. Northmail is our own free mail program for macOS and Windows, laid out like classic Outlook, that connects directly to your mail server.
Microsoft describes it in its own support documentation: "Syncing your account to the Microsoft Cloud means that a copy of your email, calendar, and contacts will be synchronized between your email provider and Microsoft data centers." That applies when you add an account from another provider, such as Gmail or an IMAP mailbox, to the new Outlook. So even with a European mail provider, the choice of mail program can put a copy of the mailbox with a US company. Classic Outlook does not work that way, and Microsoft says it will support it until at least 2029.
**Outlook, and what replaces it in Binadit Workspace**
- Mail, calendar and contacts in one window: Northmail: mail, calendar and contacts in the three-pane layout of classic Outlook, with the keyboard shortcuts you know. Free, for macOS and Windows, without advertisements.
- Several accounts side by side: Binadit Workspace, on-premises Exchange and Gmail in one window, each with its own folders, calendar and contacts. Type your address and Northmail finds the server.
- Where your mail is kept: On your own computer and on your own mail server. No telemetry and no account with Northmail. Passwords go into the keychain of your system.
- Working offline: Mail, calendars and contacts are stored on your computer for the period you choose. Search, read and write without a connection. It syncs when you are back.
- Moving mail between accounts: Copy or move messages and folders between two accounts, with dates, flags and attachments intact. A transfer resumes after a lost connection and reports what it did.
- Copilot: An optional assistant that runs on your own computer with an open model you download once. No mail and no prompt goes to a cloud, and nothing is sent without you seeing it first.
- Outlook on a phone: Not replaced. Northmail is for macOS and Windows. On a phone, the mail app that came with it connects to Workspace through standard protocols.
**What it costs** Per month, excluding VAT, cancel monthly, no minimum number of mailboxes. Prices valid from October 2026. The full list and a calculator that compares your company with Google Workspace, Microsoft 365 and Proton are on binadit.work.
- Mailbox, from €2.95 per mailbox per month: 2 GB €2.95, 5 GB €3.95, 10 GB €4.95, 25 GB €5.95, 50 GB €7.95, 100 GB €10.95. Every mailbox includes calendar, contacts, webmail, Northmail, Meet and Book. Choose the size per mailbox, and resize it at any time.
Q: Can we keep classic Outlook instead?
A: Yes. Classic Outlook connects to Binadit Workspace through standard protocols, and Microsoft says it will support existing installations until at least 2029. Northmail is for those who would rather not depend on that date.
Q: Is Northmail really free?
A: Yes. Free for everyone, without advertisements and without an account with us.
Q: Which accounts does Northmail work with?
A: Binadit Workspace, on-premises Exchange and Gmail, side by side in one window. Each account keeps its own folders, calendar and contacts.
Q: Does the assistant send my mail anywhere?
A: No. The assistant is optional and runs on your own computer, with an open model that you download once. No mail, prompt or index goes to a cloud.
Q: Is Binadit Workspace only this one product?
A: No. One login covers mail, calendar and contacts, Drive for your files, Receive for getting files from outsiders safely, Office for documents in the browser, Meet for video meetings and Book for appointments. Everything runs on servers in the Netherlands that Binadit operates itself.
Q: What does Binadit Workspace cost?
A: A Binadit Workspace mailbox costs from €2.95 per month, with calendar, contacts, webmail, Northmail, Meet and Book included, and the Drive pool for the whole company starts at €9 per month. Prices are per month, excluding VAT, and can be cancelled monthly. The full list and a cost calculator are on binadit.work.
Q: Is Binadit ISO 27001 or NEN 7510 certified?
A: The data centre we host in is ISO 27001 and NEN 7510 certified. Binadit itself, as a company, does not hold these certifications. If your own certification requires a certified supplier rather than a certified data centre, tell us before you decide, so you can check that with your auditor first.
---
### European alternative to Microsoft Teams
URL: https://binadit.com/alternatives/microsoft-teams
Teams came with Microsoft 365 and became where the company meets. It is a product of Microsoft Corporation, a US company. Binadit Meet is video meetings on our own servers in the Netherlands, started from your Workspace calendar, with guests joining in the browser.
Meetings carry the conversations that never reach an email: personnel matters, customer cases, negotiations. Microsoft Teams is a service of Microsoft Corporation, a US company, so using it is a transfer of personal data to the United States. That is lawful today on the strength of the EU-US Data Privacy Framework or of standard contractual clauses, and both lean on the same question: whether US law protects the data of Europeans well enough. The Court of Justice of the EU has answered no twice, and an appeal against the current framework is pending there. Separately, the US CLOUD Act obliges a US provider to produce data under its control wherever it is stored, so a European data region does not settle it. With Binadit Workspace there is no transfer to a third country, so there is nothing to monitor.
**Microsoft Teams, and what replaces it in Binadit Workspace**
- Scheduled meetings: Binadit Meet: start a meeting from your calendar or with one click. The link goes into the calendar invitation automatically. Only a Workspace account can start a meeting. Guests wait until you arrive.
- Guests from outside your company: Guests join in the browser, without an account and without installing anything. Available in the 24 official languages of the EU.
- Where the meeting runs: On our own servers in the Netherlands, based on the open source Jitsi. Meet never sees your password: you sign in through your own mail server.
- Files in Teams and SharePoint: Binadit Drive: your files on a server in the Netherlands, synchronised with your computer, with team folders, sharing links and versions. Part of Binadit Workspace, with the same login as your mail.
- Microsoft Bookings: Binadit Book: a booking page for every mailbox. The appointment lands in your calendar with a Meet link. Off by default. Every user switches it on themselves.
- Teams Phone, webinars and Teams Rooms: Not replaced. Meet is video meetings. If you depend on telephony, large webinars or room hardware, tell us before you decide.
**What it costs** Per month, excluding VAT, cancel monthly, no minimum number of mailboxes. Prices valid from October 2026. The full list and a calculator that compares your company with Google Workspace, Microsoft 365 and Proton are on binadit.work.
- Mailbox, from €2.95 per mailbox per month: 2 GB €2.95, 5 GB €3.95, 10 GB €4.95, 25 GB €5.95, 50 GB €7.95, 100 GB €10.95. Every mailbox includes calendar, contacts, webmail, Northmail, Meet and Book. Choose the size per mailbox, and resize it at any time.
Q: Is Microsoft Teams GDPR-compliant?
A: It can be used lawfully today. Microsoft Teams is run by a US company, so using it is a transfer of personal data to the United States, which rests on the EU-US Data Privacy Framework or on standard contractual clauses. That is as durable as those instruments: the Court of Justice of the EU struck down both earlier frameworks, and an appeal against the current one is pending there. Where the data is stored does not change which law the provider answers to.
Q: Does Meet do everything Microsoft Teams does?
A: No, and it does not try to. Meet covers video meetings: you start one from your calendar or with one click, and guests join in the browser. If you depend on something specific, such as telephony or large webinars, tell us before you decide.
Q: Do guests need an account or an app?
A: No. Whoever receives the link joins in the browser, without an account and without installing anything. Only starting a meeting needs a Workspace account.
Q: Is Binadit Workspace only this one product?
A: No. One login covers mail, calendar and contacts, Drive for your files, Receive for getting files from outsiders safely, Office for documents in the browser, Meet for video meetings and Book for appointments. Everything runs on servers in the Netherlands that Binadit operates itself.
Q: What does Binadit Workspace cost?
A: A Binadit Workspace mailbox costs from €2.95 per month, with calendar, contacts, webmail, Northmail, Meet and Book included, and the Drive pool for the whole company starts at €9 per month. Prices are per month, excluding VAT, and can be cancelled monthly. The full list and a cost calculator are on binadit.work.
Q: Is Binadit ISO 27001 or NEN 7510 certified?
A: The data centre we host in is ISO 27001 and NEN 7510 certified. Binadit itself, as a company, does not hold these certifications. If your own certification requires a certified supplier rather than a certified data centre, tell us before you decide, so you can check that with your auditor first.
---
### European alternative to Google Meet
URL: https://binadit.com/alternatives/google-meet
Google Meet is the meeting link in every Google Calendar invitation, from Google LLC, a US company. Binadit Meet does that job in Binadit Workspace: the link goes into the invitation by itself, the meeting runs on our own servers in the Netherlands, and guests join in the browser without an account.
A meeting link is easy to overlook in a register of processing activities, because nobody chose it: it came with the calendar. Google Meet is a service of Google LLC, a US company, so using it is a transfer of personal data to the United States. That is lawful today on the strength of the EU-US Data Privacy Framework or of standard contractual clauses, and both lean on the same question: whether US law protects the data of Europeans well enough. The Court of Justice of the EU has answered no twice, and an appeal against the current framework is pending there. Separately, the US CLOUD Act obliges a US provider to produce data under its control wherever it is stored, so a European data region does not settle it. With Binadit Workspace there is no transfer to a third country, so there is nothing to monitor.
**Google Meet, and what replaces it in Binadit Workspace**
- Scheduled meetings: Binadit Meet: start a meeting from your calendar or with one click. The link goes into the calendar invitation automatically. Only a Workspace account can start a meeting. Guests wait until you arrive.
- Guests from outside your company: Guests join in the browser, without an account and without installing anything. Available in the 24 official languages of the EU.
- Where the meeting runs: On our own servers in the Netherlands, based on the open source Jitsi. Meet never sees your password: you sign in through your own mail server.
- Google Calendar: Calendar and contacts in the same Workspace account as your mail. They sync to computers and phones over CalDAV and CardDAV, and they move together with the mailboxes.
- Appointment schedules in Google Calendar: Binadit Book: a booking page for every mailbox. The appointment lands in your calendar with a Meet link. Off by default. Every user switches it on themselves.
**What it costs** Per month, excluding VAT, cancel monthly, no minimum number of mailboxes. Prices valid from October 2026. The full list and a calculator that compares your company with Google Workspace, Microsoft 365 and Proton are on binadit.work.
- Mailbox, from €2.95 per mailbox per month: 2 GB €2.95, 5 GB €3.95, 10 GB €4.95, 25 GB €5.95, 50 GB €7.95, 100 GB €10.95. Every mailbox includes calendar, contacts, webmail, Northmail, Meet and Book. Choose the size per mailbox, and resize it at any time.
Q: Is Google Meet GDPR-compliant?
A: It can be used lawfully today. Google Meet is run by a US company, so using it is a transfer of personal data to the United States, which rests on the EU-US Data Privacy Framework or on standard contractual clauses. That is as durable as those instruments: the Court of Justice of the EU struck down both earlier frameworks, and an appeal against the current one is pending there. Where the data is stored does not change which law the provider answers to.
Q: Does Meet do everything Google Meet does?
A: No, and it does not try to. Meet covers video meetings: you start one from your calendar or with one click, and guests join in the browser. If you depend on something specific, such as telephony or large webinars, tell us before you decide.
Q: Do guests need an account or an app?
A: No. Whoever receives the link joins in the browser, without an account and without installing anything. Only starting a meeting needs a Workspace account.
Q: Is Binadit Workspace only this one product?
A: No. One login covers mail, calendar and contacts, Drive for your files, Receive for getting files from outsiders safely, Office for documents in the browser, Meet for video meetings and Book for appointments. Everything runs on servers in the Netherlands that Binadit operates itself.
Q: What does Binadit Workspace cost?
A: A Binadit Workspace mailbox costs from €2.95 per month, with calendar, contacts, webmail, Northmail, Meet and Book included, and the Drive pool for the whole company starts at €9 per month. Prices are per month, excluding VAT, and can be cancelled monthly. The full list and a cost calculator are on binadit.work.
Q: Is Binadit ISO 27001 or NEN 7510 certified?
A: The data centre we host in is ISO 27001 and NEN 7510 certified. Binadit itself, as a company, does not hold these certifications. If your own certification requires a certified supplier rather than a certified data centre, tell us before you decide, so you can check that with your auditor first.
---
### European alternative to Zoom
URL: https://binadit.com/alternatives/zoom
Zoom is the meeting tool almost everybody has installed, from Zoom Communications, Inc. in San Jose, California. Binadit Meet needs no installation at all: a meeting starts from your Workspace calendar, runs on our own servers in the Netherlands, and guests join in the browser.
Zoom usually sits next to the office suite as a separate subscription, with its own accounts, its own contract and its own transfer to the United States. Zoom is a service of Zoom Communications, Inc., a US company, so using it is a transfer of personal data to the United States. That is lawful today on the strength of the EU-US Data Privacy Framework or of standard contractual clauses, and both lean on the same question: whether US law protects the data of Europeans well enough. The Court of Justice of the EU has answered no twice, and an appeal against the current framework is pending there. Separately, the US CLOUD Act obliges a US provider to produce data under its control wherever it is stored, so a European data region does not settle it. With Binadit Workspace there is no transfer to a third country, so there is nothing to monitor.
**Zoom, and what replaces it in Binadit Workspace**
- Scheduled meetings: Binadit Meet: start a meeting from your calendar or with one click. The link goes into the calendar invitation automatically. Only a Workspace account can start a meeting. Guests wait until you arrive.
- Guests from outside your company: Guests join in the browser, without an account and without installing anything. Available in the 24 official languages of the EU.
- Where the meeting runs: On our own servers in the Netherlands, based on the open source Jitsi. Meet never sees your password: you sign in through your own mail server.
- Zoom desktop and mobile apps: Nothing to install. Meet runs in the browser. Guests do not need an account either.
- Zoom Scheduler: Binadit Book: a booking page for every mailbox. The appointment lands in your calendar with a Meet link. Off by default. Every user switches it on themselves.
- Zoom Phone, Webinars and Rooms: Not replaced. Meet is video meetings. If you depend on telephony, large webinars or room hardware, tell us before you decide.
**What it costs** Per month, excluding VAT, cancel monthly, no minimum number of mailboxes. Prices valid from October 2026. The full list and a calculator that compares your company with Google Workspace, Microsoft 365 and Proton are on binadit.work.
- Mailbox, from €2.95 per mailbox per month: 2 GB €2.95, 5 GB €3.95, 10 GB €4.95, 25 GB €5.95, 50 GB €7.95, 100 GB €10.95. Every mailbox includes calendar, contacts, webmail, Northmail, Meet and Book. Choose the size per mailbox, and resize it at any time.
Q: Is Zoom GDPR-compliant?
A: It can be used lawfully today. Zoom is run by a US company, so using it is a transfer of personal data to the United States, which rests on the EU-US Data Privacy Framework or on standard contractual clauses. That is as durable as those instruments: the Court of Justice of the EU struck down both earlier frameworks, and an appeal against the current one is pending there. Where the data is stored does not change which law the provider answers to.
Q: Does Meet do everything Zoom does?
A: No, and it does not try to. Meet covers video meetings: you start one from your calendar or with one click, and guests join in the browser. If you depend on something specific, such as telephony or large webinars, tell us before you decide.
Q: Do guests need an account or an app?
A: No. Whoever receives the link joins in the browser, without an account and without installing anything. Only starting a meeting needs a Workspace account.
Q: Is Binadit Workspace only this one product?
A: No. One login covers mail, calendar and contacts, Drive for your files, Receive for getting files from outsiders safely, Office for documents in the browser, Meet for video meetings and Book for appointments. Everything runs on servers in the Netherlands that Binadit operates itself.
Q: What does Binadit Workspace cost?
A: A Binadit Workspace mailbox costs from €2.95 per month, with calendar, contacts, webmail, Northmail, Meet and Book included, and the Drive pool for the whole company starts at €9 per month. Prices are per month, excluding VAT, and can be cancelled monthly. The full list and a cost calculator are on binadit.work.
Q: Is Binadit ISO 27001 or NEN 7510 certified?
A: The data centre we host in is ISO 27001 and NEN 7510 certified. Binadit itself, as a company, does not hold these certifications. If your own certification requires a certified supplier rather than a certified data centre, tell us before you decide, so you can check that with your auditor first.
---
### European alternative to Webex
URL: https://binadit.com/alternatives/webex
Webex is the meeting platform of Cisco Systems, Inc., a US company. Binadit Meet is the small, direct alternative: meetings from your Workspace calendar, on our own servers in the Netherlands, with guests joining in the browser without an account.
Webex is often part of a larger Cisco contract, which is why its jurisdiction was rarely a decision anybody took on its own. Webex is a service of Cisco Systems, Inc., a US company, so using it is a transfer of personal data to the United States. That is lawful today on the strength of the EU-US Data Privacy Framework or of standard contractual clauses, and both lean on the same question: whether US law protects the data of Europeans well enough. The Court of Justice of the EU has answered no twice, and an appeal against the current framework is pending there. Separately, the US CLOUD Act obliges a US provider to produce data under its control wherever it is stored, so a European data region does not settle it. With Binadit Workspace there is no transfer to a third country, so there is nothing to monitor.
**Webex, and what replaces it in Binadit Workspace**
- Scheduled meetings: Binadit Meet: start a meeting from your calendar or with one click. The link goes into the calendar invitation automatically. Only a Workspace account can start a meeting. Guests wait until you arrive.
- Guests from outside your company: Guests join in the browser, without an account and without installing anything. Available in the 24 official languages of the EU.
- Where the meeting runs: On our own servers in the Netherlands, based on the open source Jitsi. Meet never sees your password: you sign in through your own mail server.
- Webex Calling, Webinars and room devices: Not replaced. Meet is video meetings. If you depend on telephony, large webinars or room hardware, tell us before you decide.
**What it costs** Per month, excluding VAT, cancel monthly, no minimum number of mailboxes. Prices valid from October 2026. The full list and a calculator that compares your company with Google Workspace, Microsoft 365 and Proton are on binadit.work.
- Mailbox, from €2.95 per mailbox per month: 2 GB €2.95, 5 GB €3.95, 10 GB €4.95, 25 GB €5.95, 50 GB €7.95, 100 GB €10.95. Every mailbox includes calendar, contacts, webmail, Northmail, Meet and Book. Choose the size per mailbox, and resize it at any time.
Q: Is Webex GDPR-compliant?
A: It can be used lawfully today. Webex is run by a US company, so using it is a transfer of personal data to the United States, which rests on the EU-US Data Privacy Framework or on standard contractual clauses. That is as durable as those instruments: the Court of Justice of the EU struck down both earlier frameworks, and an appeal against the current one is pending there. Where the data is stored does not change which law the provider answers to.
Q: Does Meet do everything Webex does?
A: No, and it does not try to. Meet covers video meetings: you start one from your calendar or with one click, and guests join in the browser. If you depend on something specific, such as telephony or large webinars, tell us before you decide.
Q: Do guests need an account or an app?
A: No. Whoever receives the link joins in the browser, without an account and without installing anything. Only starting a meeting needs a Workspace account.
Q: Is Binadit Workspace only this one product?
A: No. One login covers mail, calendar and contacts, Drive for your files, Receive for getting files from outsiders safely, Office for documents in the browser, Meet for video meetings and Book for appointments. Everything runs on servers in the Netherlands that Binadit operates itself.
Q: What does Binadit Workspace cost?
A: A Binadit Workspace mailbox costs from €2.95 per month, with calendar, contacts, webmail, Northmail, Meet and Book included, and the Drive pool for the whole company starts at €9 per month. Prices are per month, excluding VAT, and can be cancelled monthly. The full list and a cost calculator are on binadit.work.
Q: Is Binadit ISO 27001 or NEN 7510 certified?
A: The data centre we host in is ISO 27001 and NEN 7510 certified. Binadit itself, as a company, does not hold these certifications. If your own certification requires a certified supplier rather than a certified data centre, tell us before you decide, so you can check that with your auditor first.
---
### European alternative to Calendly
URL: https://binadit.com/alternatives/calendly
Calendly lets customers pick a time, and to do that it reads your calendar and stores who booked what. It is a service of Calendly, LLC, a US company, which states that it stores user and invitee data in data centres in the United States. Binadit Book is a booking page for every Workspace mailbox, and it is the same system as your calendar: nothing to connect, and nothing leaves the Netherlands.
A booking tool holds the names and email addresses of your customers and the reason for the appointment, and it has standing access to your calendar. Calendly is a service of Calendly, LLC, a US company, so using it is a transfer of personal data to the United States. That is lawful today on the strength of the EU-US Data Privacy Framework or of standard contractual clauses, and both lean on the same question: whether US law protects the data of Europeans well enough. The Court of Justice of the EU has answered no twice, and an appeal against the current framework is pending there. Separately, the US CLOUD Act obliges a US provider to produce data under its control wherever it is stored, so a European data region does not settle it. With Binadit Workspace there is no transfer to a third country, so there is nothing to monitor.
**Calendly, and what replaces it in Binadit Workspace**
- Booking page: Binadit Book: a booking page for every mailbox, with your own logo and text. In 34 languages. Off by default; every user switches it on themselves.
- Event types: Several links per page, such as a half-hour introduction and an hour-long consultation. Each link has its own length.
- Calendar connections: Book looks at all your calendars at once and only shows times when you are really free. No integration to set up and no access to hand to a third party.
- Confirmation and reminder mails: Confirmation and invitation come from your own mail server. Your customer gets mail from your domain, not from a booking service.
- Video link in the appointment: The appointment lands in your calendar with a Binadit Meet link. Your customer joins in the browser, without an account.
- Payments, routing forms and CRM integrations: Not replaced. Book lets customers pick a time. If you rely on payment at booking or on lead routing, tell us before you decide.
**What it costs** Per month, excluding VAT, cancel monthly, no minimum number of mailboxes. Prices valid from October 2026. The full list and a calculator that compares your company with Google Workspace, Microsoft 365 and Proton are on binadit.work.
- Mailbox, from €2.95 per mailbox per month: 2 GB €2.95, 5 GB €3.95, 10 GB €4.95, 25 GB €5.95, 50 GB €7.95, 100 GB €10.95. Every mailbox includes calendar, contacts, webmail, Northmail, Meet and Book. Choose the size per mailbox, and resize it at any time.
Q: Is Calendly GDPR-compliant?
A: It can be used lawfully today. Calendly is run by a US company, so using it is a transfer of personal data to the United States, which rests on the EU-US Data Privacy Framework or on standard contractual clauses. That is as durable as those instruments: the Court of Justice of the EU struck down both earlier frameworks, and an appeal against the current one is pending there. Where the data is stored does not change which law the provider answers to.
Q: Do we need a Binadit Workspace mailbox to use Book?
A: Yes. Book is a booking page for a Workspace mailbox and reads the calendars of that account. That is also why there is nothing to integrate.
Q: Is Binadit Workspace only this one product?
A: No. One login covers mail, calendar and contacts, Drive for your files, Receive for getting files from outsiders safely, Office for documents in the browser, Meet for video meetings and Book for appointments. Everything runs on servers in the Netherlands that Binadit operates itself.
Q: What does Binadit Workspace cost?
A: A Binadit Workspace mailbox costs from €2.95 per month, with calendar, contacts, webmail, Northmail, Meet and Book included, and the Drive pool for the whole company starts at €9 per month. Prices are per month, excluding VAT, and can be cancelled monthly. The full list and a cost calculator are on binadit.work.
Q: Is Binadit ISO 27001 or NEN 7510 certified?
A: The data centre we host in is ISO 27001 and NEN 7510 certified. Binadit itself, as a company, does not hold these certifications. If your own certification requires a certified supplier rather than a certified data centre, tell us before you decide, so you can check that with your auditor first.
---
### Alternative to Cryptshare
URL: https://binadit.com/alternatives/cryptshare
Cryptshare is how many law firms, accountancies and supervisors receive confidential files today. It is a European product, from Pointsharp, so this is not a story about American law. Binadit Receive keeps the habits your people and your clients know, encrypts in the sender's browser, and is priced per company rather than per mailbox.
Three things, none of them about jurisdiction. Where the files are encrypted: with Receive that happens in the sender's browser before upload, so our server only ever stores ciphertext and never sees the password. What it costs: Cryptshare is licensed per mailbox, and Receive is priced per company by storage, without a licence per user or per sender. And how many systems you keep: Receive uses the same login and the same servers as your mail and your files.
**Cryptshare, and what replaces it in Binadit Workspace**
- Receiving files from outside: Binadit Receive: the sender opens your link, chooses your colleague and drops files or a whole folder. No account, no app and no attachment in the mail.
- Encryption: Names and contents are encrypted in the sender's browser before upload. The server only ever stores ciphertext. It uses the open age format. The password is never stored or mailed and cannot be recovered by an administrator, or by us.
- Large files: Up to 100 files and 10 GB per transfer, uploaded in pieces that resume after a dropped connection. A whole folder can be dropped at once.
- Retention: Kept for 1 to 30 days, 14 by default, and then deleted automatically. You choose the period.
- Who can receive: Only addresses on your own domains. A sender cannot send to a private address or to a typo. Your administrator lists the domains.
- Oversight: Your administrator sees every transfer with its size, expiry and delivery status, and can revoke it. File names stay encrypted, for the administrator too.
- Sending files to others: A Drive sharing link to a file or a folder that expires when you choose, 30 days at most. Part of Binadit Workspace, with the same login as your mail.
- Licence: Priced per company by inbox storage. No licence per user or per sender. It works whether or not you sync files with Drive.
- Cryptshare for Outlook: Not replaced. Receive works from a web page, so there is nothing to install in a mail program. The notification to your colleague is an ordinary mail with a download link, without the password.
**What it costs** Per month, excluding VAT, cancel monthly, no minimum number of mailboxes. Prices valid from October 2026. The full list and a calculator that compares your company with Google Workspace, Microsoft 365 and Proton are on binadit.work.
- Receive, per company, from €29 per month: 50 GB €29, 250 GB €59, 1 TB €129, 5 TB €349. Priced by inbox storage, not per user or per sender. Files are deleted after the retention period of at most 30 days, so the storage is roughly what you can receive per month.
- Drive pool, per company, from €9 per month: 250 GB €9, 500 GB €15, 1 TB €25, 2 TB €40, 5 TB €75, 10 TB €120, 20 TB €200, 50 TB €450, 100 TB €800. One storage pool for the personal drives of everyone with Drive, with Office in the browser included. Above 100 TB a company gets its own Drive server: €250 per month plus €7 per TB.
Q: Cryptshare is European already. Why change?
A: Not for legal reasons: Cryptshare is a product of Pointsharp, a European company, and this page makes no legal argument against it. The difference is that Receive encrypts in the sender's browser, is priced per company instead of per mailbox, and is not one more system to manage.
Q: What if the sender loses the password?
A: Then the transfer is unreadable, for you and for us. The password is never stored, never mailed and cannot be reset. The sender sends the files again.
Q: Does the sender need an account?
A: No. A link on your domain is enough. The sender opens it in a browser, chooses your colleague and drops the files.
Q: Is Binadit Workspace only this one product?
A: No. One login covers mail, calendar and contacts, Drive for your files, Receive for getting files from outsiders safely, Office for documents in the browser, Meet for video meetings and Book for appointments. Everything runs on servers in the Netherlands that Binadit operates itself.
Q: What does Binadit Workspace cost?
A: A Binadit Workspace mailbox costs from €2.95 per month, with calendar, contacts, webmail, Northmail, Meet and Book included, and the Drive pool for the whole company starts at €9 per month. Prices are per month, excluding VAT, and can be cancelled monthly. The full list and a cost calculator are on binadit.work.
Q: Is Binadit ISO 27001 or NEN 7510 certified?
A: The data centre we host in is ISO 27001 and NEN 7510 certified. Binadit itself, as a company, does not hold these certifications. If your own certification requires a certified supplier rather than a certified data centre, tell us before you decide, so you can check that with your auditor first.
---
### Alternative to WeTransfer
URL: https://binadit.com/alternatives/wetransfer
WeTransfer is the quickest way to get a large file to somebody, and for ordinary files it is fine. It is a Dutch company, owned since 2024 by the Italian company Bending Spoons, so this is not a story about American law. Binadit Receive is made for the files you would not put in a public transfer service: it encrypts them in the sender's browser, on a page at your own address, and only your own colleagues can receive.
WeTransfer documents that files are encrypted during transfer and at rest, and that files uploaded from the European Union are stored in the EU. That is sound for ordinary files. For case files, payroll data or identity documents, offices want more: encryption before the file leaves the sender's computer, a password that travels separately and that nobody can recover, the certainty that a file can only go to an address on your own domain, and an overview of every transfer with the option to revoke it. That is what Receive does.
**WeTransfer, and what replaces it in Binadit Workspace**
- Receiving files from outside: Binadit Receive: the sender opens your link, chooses your colleague and drops files or a whole folder. No account, no app and no attachment in the mail.
- Encryption: Names and contents are encrypted in the sender's browser before upload. The server only ever stores ciphertext. It uses the open age format. The password is never stored or mailed and cannot be recovered by an administrator, or by us.
- Large files: Up to 100 files and 10 GB per transfer, uploaded in pieces that resume after a dropped connection. A whole folder can be dropped at once.
- Retention: Kept for 1 to 30 days, 14 by default, and then deleted automatically. You choose the period.
- Who can receive: Only addresses on your own domains. A sender cannot send to a private address or to a typo. Your administrator lists the domains.
- Oversight: Your administrator sees every transfer with its size, expiry and delivery status, and can revoke it. File names stay encrypted, for the administrator too.
- Sending files to others: A Drive sharing link to a file or a folder that expires when you choose, 30 days at most. Part of Binadit Workspace, with the same login as your mail.
- Licence: Priced per company by inbox storage. No licence per user or per sender. It works whether or not you sync files with Drive.
**What it costs** Per month, excluding VAT, cancel monthly, no minimum number of mailboxes. Prices valid from October 2026. The full list and a calculator that compares your company with Google Workspace, Microsoft 365 and Proton are on binadit.work.
- Receive, per company, from €29 per month: 50 GB €29, 250 GB €59, 1 TB €129, 5 TB €349. Priced by inbox storage, not per user or per sender. Files are deleted after the retention period of at most 30 days, so the storage is roughly what you can receive per month.
- Drive pool, per company, from €9 per month: 250 GB €9, 500 GB €15, 1 TB €25, 2 TB €40, 5 TB €75, 10 TB €120, 20 TB €200, 50 TB €450, 100 TB €800. One storage pool for the personal drives of everyone with Drive, with Office in the browser included. Above 100 TB a company gets its own Drive server: €250 per month plus €7 per TB.
Q: Is WeTransfer not safe?
A: WeTransfer encrypts files during transfer and at rest, and for ordinary files that is fine. Receive adds what confidential files need: encryption in the sender's browser, recipients limited to your own domains and a record of every transfer.
Q: Can we also send large files to clients?
A: Yes. From Drive you share a file or a folder with a link that expires when you choose, 30 days at most.
Q: What if the sender loses the password?
A: Then the transfer is unreadable, for you and for us. The password is never stored, never mailed and cannot be reset. The sender sends the files again.
Q: Does the sender need an account?
A: No. A link on your domain is enough. The sender opens it in a browser, chooses your colleague and drops the files.
Q: Is Binadit Workspace only this one product?
A: No. One login covers mail, calendar and contacts, Drive for your files, Receive for getting files from outsiders safely, Office for documents in the browser, Meet for video meetings and Book for appointments. Everything runs on servers in the Netherlands that Binadit operates itself.
Q: What does Binadit Workspace cost?
A: A Binadit Workspace mailbox costs from €2.95 per month, with calendar, contacts, webmail, Northmail, Meet and Book included, and the Drive pool for the whole company starts at €9 per month. Prices are per month, excluding VAT, and can be cancelled monthly. The full list and a cost calculator are on binadit.work.
Q: Is Binadit ISO 27001 or NEN 7510 certified?
A: The data centre we host in is ISO 27001 and NEN 7510 certified. Binadit itself, as a company, does not hold these certifications. If your own certification requires a certified supplier rather than a certified data centre, tell us before you decide, so you can check that with your auditor first.
---
### European alternative to Box
URL: https://binadit.com/alternatives/box
Box is file storage for companies, with a great deal of workflow built around it, from Box, Inc. in Redwood City, California. Binadit Drive is the file part, done from the Netherlands: folders on your computer, team folders, sharing links and versions, with documents that open in the browser and a safe way to receive files from outside.
Box holds exactly the files a regulator would ask about: contracts, personnel files, customer data. Box is a service of Box, Inc., a US company, so using it is a transfer of personal data to the United States. That is lawful today on the strength of the EU-US Data Privacy Framework or of standard contractual clauses, and both lean on the same question: whether US law protects the data of Europeans well enough. The Court of Justice of the EU has answered no twice, and an appeal against the current framework is pending there. Separately, the US CLOUD Act obliges a US provider to produce data under its control wherever it is stored, so a European data region does not settle it. With Binadit Workspace there is no transfer to a third country, so there is nothing to monitor.
**Box, and what replaces it in Binadit Workspace**
- Folders on your computer: Binadit Drive for macOS and Windows, with files on demand so your disk does not fill up. A file is fetched the moment you open it.
- Team folders: Team folders with permissions per colleague and their own quota. Who may see what is set up with you. A move is also the moment to clean that up.
- Shared links: A link to a file or a folder that expires when you choose, 30 days at most. For colleagues and for people outside your company.
- File requests: Binadit Receive: a page on your own address where outsiders hand you files without an account, encrypted in their browser. Part of Binadit Drive.
- Version history: Every change is a version you can restore, and deleted files wait in the trash. A mistake can be undone.
- Editing documents in the browser: Binadit Office: documents, spreadsheets and presentations in the browser, edited together, on files that stay on Drive. Word, Excel and PowerPoint files open and save in their own formats.
- Search: Search that looks inside PDF, Word and Excel files. It runs on the server, so it is just as fast from a small laptop.
- Box Sign, Box AI and workflow automation: Not replaced. Binadit Drive is file storage, sharing and editing. If you depend on signing or on automated workflows, tell us before you decide.
**What it costs** Per month, excluding VAT, cancel monthly, no minimum number of mailboxes. Prices valid from October 2026. The full list and a calculator that compares your company with Google Workspace, Microsoft 365 and Proton are on binadit.work.
- Drive pool, per company, from €9 per month: 250 GB €9, 500 GB €15, 1 TB €25, 2 TB €40, 5 TB €75, 10 TB €120, 20 TB €200, 50 TB €450, 100 TB €800. One storage pool for the personal drives of everyone with Drive, with Office in the browser included. Above 100 TB a company gets its own Drive server: €250 per month plus €7 per TB.
- Team Drive, per company, from €29 per month: 1 TB €29, 2 TB €45, 5 TB €85, 10 TB €140, 20 TB €240, 50 TB €500, 100 TB €900. Shared folders that everyone in the company sees, with their own storage next to the pool.
- Mailbox, from €2.95 per mailbox per month: 2 GB €2.95, 5 GB €3.95, 10 GB €4.95, 25 GB €5.95, 50 GB €7.95, 100 GB €10.95. Every mailbox includes calendar, contacts, webmail, Northmail, Meet and Book. Choose the size per mailbox, and resize it at any time.
Q: Is Box GDPR-compliant?
A: It can be used lawfully today. Box is run by a US company, so using it is a transfer of personal data to the United States, which rests on the EU-US Data Privacy Framework or on standard contractual clauses. That is as durable as those instruments: the Court of Justice of the EU struck down both earlier frameworks, and an appeal against the current one is pending there. Where the data is stored does not change which law the provider answers to.
Q: Can people outside our company still open what we share?
A: Yes. You share a file or a folder with a link, the same way you do now.
Q: Is Binadit Workspace only this one product?
A: No. One login covers mail, calendar and contacts, Drive for your files, Receive for getting files from outsiders safely, Office for documents in the browser, Meet for video meetings and Book for appointments. Everything runs on servers in the Netherlands that Binadit operates itself.
Q: What does Binadit Workspace cost?
A: A Binadit Workspace mailbox costs from €2.95 per month, with calendar, contacts, webmail, Northmail, Meet and Book included, and the Drive pool for the whole company starts at €9 per month. Prices are per month, excluding VAT, and can be cancelled monthly. The full list and a cost calculator are on binadit.work.
Q: Is Binadit ISO 27001 or NEN 7510 certified?
A: The data centre we host in is ISO 27001 and NEN 7510 certified. Binadit itself, as a company, does not hold these certifications. If your own certification requires a certified supplier rather than a certified data centre, tell us before you decide, so you can check that with your auditor first.
---
### European alternative to Google Docs
URL: https://binadit.com/alternatives/google-docs
Google Docs made working in the same document at the same time normal. The price is that the document only exists inside the service of Google LLC, a US company. Binadit Office gives you documents, spreadsheets and presentations in the browser, edited together, on ordinary files that stay on Binadit Drive in the Netherlands.
A Google document is not a file you hold but a link into a service, and everything written in it sits with a US provider. Google Docs is a service of Google LLC, a US company, so using it is a transfer of personal data to the United States. That is lawful today on the strength of the EU-US Data Privacy Framework or of standard contractual clauses, and both lean on the same question: whether US law protects the data of Europeans well enough. The Court of Justice of the EU has answered no twice, and an appeal against the current framework is pending there. Separately, the US CLOUD Act obliges a US provider to produce data under its control wherever it is stored, so a European data region does not settle it. With Binadit Workspace there is no transfer to a third country, so there is nothing to monitor.
**Google Docs, and what replaces it in Binadit Workspace**
- Documents, spreadsheets and presentations: Binadit Office: write, calculate and present in the browser, on files that stay on Binadit Drive. Built on Collabora Online, which is based on LibreOffice. It runs on our own servers in the Netherlands.
- Working in the same document at the same time: Several people edit the same document and see each other's changes as they happen. No second account: it opens from Drive.
- File formats: Word, Excel and PowerPoint files open and save in their own formats. OpenDocument too. The document is a file in your folder, not a link into a service.
- Version history: Every save is a version on Drive. A mistake can be undone.
- Where the documents are kept: Binadit Drive: your files on a server in the Netherlands, synchronised with your computer, with team folders, sharing links and versions. Part of Binadit Workspace, with the same login as your mail.
- Google Forms, Apps Script and add-ons: Not replaced. Binadit Office edits documents, spreadsheets and presentations. If you depend on forms or scripts, tell us before you decide.
- Gemini in Docs: Not replaced, on purpose. We do not train on your data and we do not put a model inside your mailbox. Every AI feature inside a US suite is another data flow you answer for under Article 24 GDPR. If you want an assistant, choose one deliberately and know where it runs.
**What it costs** Per month, excluding VAT, cancel monthly, no minimum number of mailboxes. Prices valid from October 2026. The full list and a calculator that compares your company with Google Workspace, Microsoft 365 and Proton are on binadit.work.
- Drive pool, per company, from €9 per month: 250 GB €9, 500 GB €15, 1 TB €25, 2 TB €40, 5 TB €75, 10 TB €120, 20 TB €200, 50 TB €450, 100 TB €800. One storage pool for the personal drives of everyone with Drive, with Office in the browser included. Above 100 TB a company gets its own Drive server: €250 per month plus €7 per TB.
- Mailbox, from €2.95 per mailbox per month: 2 GB €2.95, 5 GB €3.95, 10 GB €4.95, 25 GB €5.95, 50 GB €7.95, 100 GB €10.95. Every mailbox includes calendar, contacts, webmail, Northmail, Meet and Book. Choose the size per mailbox, and resize it at any time.
Q: Is Google Docs GDPR-compliant?
A: It can be used lawfully today. Google Docs is run by a US company, so using it is a transfer of personal data to the United States, which rests on the EU-US Data Privacy Framework or on standard contractual clauses. That is as durable as those instruments: the Court of Justice of the EU struck down both earlier frameworks, and an appeal against the current one is pending there. Where the data is stored does not change which law the provider answers to.
Q: What happens to our existing Google Docs?
A: They leave as an export: Google saves them as docx, xlsx or pptx. Those go into the same folder structure on Binadit Drive and open in Binadit Office in the browser. From then on they are files you own.
Q: Is Binadit Office the same as Microsoft Office?
A: No. It is built on Collabora Online, which is based on LibreOffice. It opens and saves Word, Excel and PowerPoint files in their own formats, and a complex layout or a macro can behave differently. Try it with your own documents before you decide.
Q: Is Binadit Workspace only this one product?
A: No. One login covers mail, calendar and contacts, Drive for your files, Receive for getting files from outsiders safely, Office for documents in the browser, Meet for video meetings and Book for appointments. Everything runs on servers in the Netherlands that Binadit operates itself.
Q: What does Binadit Workspace cost?
A: A Binadit Workspace mailbox costs from €2.95 per month, with calendar, contacts, webmail, Northmail, Meet and Book included, and the Drive pool for the whole company starts at €9 per month. Prices are per month, excluding VAT, and can be cancelled monthly. The full list and a cost calculator are on binadit.work.
Q: Is Binadit ISO 27001 or NEN 7510 certified?
A: The data centre we host in is ISO 27001 and NEN 7510 certified. Binadit itself, as a company, does not hold these certifications. If your own certification requires a certified supplier rather than a certified data centre, tell us before you decide, so you can check that with your auditor first.
---
## Engineering articles
### Best practices for evaluating European alternatives to AWS, Azure and GCP
URL: https://binadit.com/blog/best-practices-private-cloud-infrastructure-eu-alternatives-aws-azure-gcp
Category: Security
Author: Binadit Tech Team
Published: 2026-10-21T09:19:09+02:00
Who this is for
This checklist is for engineering leads and platform teams who have been asked, by a customer, a DPO, or a procurement committee, to evaluate what it would take to run on European infrastructure instead of AWS, Azure or GCP. It is not a sovereignty lecture. It assumes you already understand why the question is on the table and you need a practical way to answer it: what private cloud infrastructure or EU cloud options actually match your current architecture, what breaks during a move, and what stays the same.
We work with SaaS platforms, agencies and e-commerce teams who run this evaluation every quarter as part of procurement cycles, tenders, or customer security questionnaires. The patterns below come from those engagements. None of them require you to abandon Kubernetes, Terraform or your existing CI/CD pipeline.
10 practices for evaluating private cloud infrastructure in Europe
Separate "EU region" from "EU jurisdiction" in your requirements. AWS, Azure and GCP all offer EU data center regions, but the parent company remains subject to US law, including the CLOUD Act, regardless of where the data physically sits. If your requirement is genuine jurisdictional separation, a regional toggle on a hyperscaler does not satisfy it. Write this distinction explicitly into your evaluation criteria before you start comparing vendors, or you will end up comparing the wrong thing.
Map your workload to infrastructure primitives, not to a brand. Most SaaS workloads need compute, object storage, managed Postgres or MySQL, a load balancer, and a CDN. List what you actually use today (API calls, storage classes, managed services) separately from the AWS or Azure product names. This turns "can we replace AWS" into "which EU providers offer S3-compatible object storage, managed PostgreSQL with point-in-time recovery, and a comparable autoscaling group." That is an answerable question.
# Example: workload inventory before evaluationncompute: 12 vCPU, autoscaling 2-8 nodesnobject_storage: S3-compatible, 4TB, lifecycle rulesndatabase: managed PostgreSQL 15, 200GB, daily PITRncdn: edge caching, EU + UK pop coveragenqueue: managed Redis, 2GB, persistence enabledn
Check for S3 API compatibility before assuming a rewrite. Several European providers (OVHcloud, Scaleway, Exoscale, Hetzner via third-party gateways) expose S3-compatible object storage APIs. If your application already talks to S3 using the standard SDK, you can often repoint the endpoint and credentials without touching application code. Test this with a read/write smoke test against the new endpoint before planning a full migration.
aws s3 cp test.txt s3://my-bucket/test.txt \n --endpoint-url https://s3.de.io.cloud.ovh.net \n --profile eu-providern
Treat the managed database layer as the highest-risk component. Compute is portable. Object storage is portable. Managed database services are where EU providers vary most in feature parity, replication tooling, and backup retention defaults. Before committing, run a full backup/restore cycle on the target provider and measure recovery time. If you are migrating a production database, the approach matters as much as the destination, which is why we wrote a separate breakdown of how zero-downtime database migrations work.
Benchmark network latency from your actual user base, not from a data center brochure. A provider based in Frankfurt or Amsterdam will serve EU users well, but if 20% of your traffic comes from the UK, US, or APAC, check real latency numbers before assuming parity with a hyperscaler's global edge network. Run synthetic tests from multiple regions using a tool like mtr or a distributed probe service, and compare p50/p95 figures, not averages.
Validate your CDN's actual point-of-presence footprint, not just its EU marketing claim. Some EU-headquartered CDN resellers still route traffic through non-EU origin infrastructure for certain regions. If data sovereignty is part of your requirement, ask the provider for a documented PoP list and trace actual request paths with curl -w timing output or a traceroute from your target markets. We cover this in more detail in understanding CDN data sovereignty: which providers keep EU traffic in EU.
Confirm the data center's actual certifications, and who holds them. Ask for the ISO 27001 and, where relevant, NEN 7510 certification of the physical data center, not the reseller or managed service layered on top. These certifications apply to the facility and its operational controls, not automatically to every company that racks servers in it. Request the certificate and scope statement directly rather than accepting a logo on a website.
Price the full stack, not just compute per hour. European providers are frequently cheaper on raw compute but the comparison falls apart if you forget egress fees, backup storage, IP allocation, and support tier costs. Build a like-for-like monthly cost model across at least three providers using your actual usage pattern, not a generic instance size. This is the same exercise we walk through in solving the real ROI question behind moving off US hyperscalers, and it usually surfaces a very different number than the headline instance price.
Test your Infrastructure as Code against the new provider before migrating anything live. If your Terraform or Pulumi modules target AWS-specific resources, you will need a provider swap, not a full rewrite, for most standard compute and networking. Run terraform plan against a sandbox account on the target provider early, so you catch unsupported resource types (certain autoscaling policies, specific VPC peering models) before they become a blocker mid-migration.
provider "openstack" {n auth_url = var.ovh_auth_urln domain_name = "default"n tenant_name = var.ovh_project_idn}nnresource "openstack_compute_instance_v2" "app" {n name = "app-node-1"n flavor_name = "b2-15"n image_name = "Ubuntu 22.04"n security_groups = ["default"]n}n
Plan the cutover as a reversible, staged migration, not a single switch. Run both environments in parallel with DNS-weighted traffic splitting, validate error rates and latency at each traffic percentage, and keep the old environment warm until the new one has handled a full peak traffic cycle. This is standard practice for any zero-downtime migration regardless of destination, and it is what prevents a sovereignty decision from becoming an availability incident.
Rolling this out in an existing team
You do not need to evaluate all 10 points in one sprint. Most teams run this in three phases:
Phase one (1-2 weeks): workload inventory and requirement separation (practices 1-2). This is a documentation exercise, not an infrastructure change, and it gives procurement or legal a concrete artifact to work from instead of a vague "move to Europe" directive.
Phase two (2-4 weeks): technical validation in a sandbox account with at least two candidate providers (practices 3-8). This runs in parallel with production, costs a few hundred euros in test resources, and produces real latency, cost, and compatibility numbers instead of vendor claims.
Phase three (4-10 weeks depending on scope): staged migration using IaC parity testing and a reversible cutover plan (practices 9-10). Keep a rollback window of at least one full business cycle before decommissioning the old environment.
Assign one engineer as the technical owner of the evaluation from day one. Splitting this across the team tends to produce inconsistent benchmarks that are hard to compare later. If your organization is also fielding sovereignty questions from customers or auditors during this process, the 12-question sovereignty audit every DPO should run is a useful companion checklist to run alongside the technical evaluation.
Where this leaves you
None of these practices require a leap of faith. They require inventory, benchmarking, and a staged rollout, the same discipline you would apply to any infrastructure change regardless of which provider's logo ends up on the invoice. The providers worth evaluating (OVHcloud, Scaleway, Exoscale, Hetzner, IONOS, and a handful of smaller regional specialists) each have real strengths and real gaps, and the right fit depends on your actual workload, not on a general preference for "European."
If implementing these yourself is not the best use of your engineering time, our managed services cover all of them by default.
---
### Why cheap hosting becomes expensive, and how managed cloud infrastructure fixes the math
URL: https://binadit.com/blog/why-cheap-hosting-becomes-expensive-managed-cloud-infrastructure
Category: Infrastructure
Author: Binadit Tech Team
Published: 2026-10-16T09:03:28+02:00
The symptom: a hosting bill that looks small on paper
A SaaS company runs on a €40/month VPS or a shared hosting plan. The invoice is predictable. Then a deploy takes down checkout for twenty minutes, a traffic spike from a marketing campaign triggers swap death, and an engineer spends a Friday afternoon debugging why MySQL connections are being refused. None of this shows up on the hosting invoice. It shows up in engineering hours, lost conversions, and support tickets.
This is the pattern across most companies that eventually move to managed cloud infrastructure: the sticker price was never the real cost. The real cost was everywhere else.
The actual root cause: cheap hosting shifts cost, it does not remove it
Cheap hosting plans are built around a simple business model: oversell shared resources and assume most tenants never use their allocation at the same time. That works until your application actually needs the CPU, memory, or I/O you were promised.
Three things break down specifically:
Noisy neighbors. On shared VPS and shared hosting, your I/O and CPU scheduling compete with other tenants on the same physical host. A neighbor running a batch job can spike your p95 latency even though nothing changed in your application.
No real headroom planning. Budget plans are sized for average load, not peak load. A traffic spike of 3x your normal volume, which happens during a product launch or a single viral post, pushes memory usage past the plan's limit and the OOM killer starts terminating processes, often PHP-FPM workers or your database.
No engineering time included. The plan gives you a server. It does not give you someone who tunes innodb_buffer_pool_size, rotates logs before disk fills up, or notices that your SSL certificate expires in six days. That work still has to happen. It just gets pushed onto whichever engineer is available, usually during an incident instead of before one.
Run the math on a concrete example. A shared hosting plan at €30/month saves €170/month compared to a properly sized managed VPS at €200/month. That is €2,040/year in theoretical savings. But if that plan causes two checkout outages a year at 25 minutes each, on a store doing €500/day in revenue, that is roughly €520 in lost sales, plus the 4-6 engineering hours spent diagnosing and recovering, at a loaded cost of €75/hour: another €300-450. The "cheap" option costs more before you even count the opportunity cost of what that engineer could have built instead.
Where the hidden cost concentrates
In our work migrating clients off budget hosting, the cost almost always concentrates in three places:
Database performance degrading silently until a slow query takes the whole site down (see how database performance degrades in managed infrastructure for the pattern).
No staging environment that resembles production, so bugs get discovered by customers, not by CI.
Manual TLS certificate renewal that gets missed once a year, taking down the whole domain for hours.
None of these are hard problems. They are just the kind of recurring operational work that a €30/month plan was never built to include.
The fix: size for reality, automate the recurring work, isolate what matters
The fix is not "spend more." It is spend correctly, on the things that actually reduce incidents and engineering time.
Step 1: size compute to your real peak, not your average
Pull your last 90 days of CPU and memory metrics. If you do not have them, that is itself a signal. On a Linux box you can get a fast baseline with:
vmstat 2 10nfree -mnmpstat -P ALL 1 5
Size your instance so peak load uses 60-70% of available memory and CPU, not 95%. If your application spikes 3x during campaigns, provision for that spike or put autoscaling in front of it, not for the Tuesday-afternoon average.
Step 2: isolate the database from the application tier
On budget hosting, MySQL and the web server often fight for the same CPU and memory. Separating them, even onto two modest instances, removes most noisy-neighbor effects. A baseline tuning pass for a dedicated MySQL 8 instance with 8GB RAM:
[mysqld]ninnodb_buffer_pool_size = 5Gninnodb_log_file_size = 512Mnmax_connections = 150ninnodb_flush_log_at_trx_commit = 1nquery_cache_type = 0
The buffer pool is set to roughly 60-70% of available RAM, leaving room for connections and OS cache. On a plan with 2GB total RAM, this is simply not achievable, which is the real reason queries that take 40ms on proper infrastructure take 900ms on a shared plan.
Step 3: automate the things that fail silently
Certificate expiry and log-driven disk exhaustion are the two most common "surprise" outages. Both are fixable with automation that takes under an hour to set up:
# certbot auto-renewal, runs twice daily via systemd timerncertbot renew --quiet --deploy-hook "systemctl reload nginx"
For a deeper walkthrough of renewal across multiple hosts, see how to automate SSL certificate renewal across a fleet of servers.
Step 4: build a real staging environment
Staging on a shared hosting plan is often just a subdirectory with debug mode on. That does not catch production-class bugs. A staging environment needs the same PHP version, same database engine version, and comparable (even if smaller) resource ratios. If this step gets skipped, deploys become the riskiest event in your week instead of a routine one.
How to validate the fix worked
Do not take "it feels faster" as confirmation. Check specific numbers before and after:
MetricBudget hosting (typical)Right-sized managed infra (typical)P95 response time800-1500ms150-300msDatabase query p95300-900ms20-60msUnplanned restarts/month2-50-1Memory headroom at peak<10%30-40%
Specific checks to run:
mysqladmin status to confirm no connection queueing under load.
Load test with a tool like k6 at 2-3x your normal peak traffic and watch for 5xx errors or increasing latency, not just average response time.
Confirm swap usage stays at 0 during peak (free -m), since any swap activity under load is a sign you are still undersized.
Verify TTFB separately from full page load, since the two measure different bottlenecks (see TTFB vs full page load for how to interpret each).
If you have access to historical incident data, track mean time to recovery (MTTR) before and after. A move to properly managed, high availability infrastructure typically drops MTTR from hours to minutes, because failover and monitoring are built in rather than assembled under pressure.
How to prevent recurrence
Fixing the sizing and automation gets you out of the immediate hole. Staying out requires a few habits:
Review capacity quarterly, not reactively. Set a calendar reminder to pull 90-day resource graphs and compare against current plan limits.
Treat SSL renewal, backups, and patching as infrastructure code, not manual tasks. If a human has to remember to do it, it will eventually get missed.
Load test before major traffic events, not after an outage during one. This is the difference between planned capacity and discovered capacity.
Separate the database tier early. It is the most common single point of contention on budget hosting and the first thing worth isolating as traffic grows.
Keep a staging environment that mirrors production closely enough to catch real bugs. This alone prevents a large share of deploy-related incidents.
None of this requires exotic tooling. It requires someone accountable for checking it on a schedule, which is usually the actual gap, not the technology.
Close
Cheap hosting is not a bad choice because the server is weak. It is a bad choice because nobody is responsible for the recurring work that keeps infrastructure healthy: capacity planning, certificate renewal, query tuning, and incident response. That work does not disappear when you pick a cheap plan. It just lands on your team at the worst possible time.
If you'd rather not debug this again next quarter, our managed platform handles it by default.
---
### A 12-question sovereignty audit every DPO should run
URL: https://binadit.com/blog/no-us-subprocessor-sovereignty-audit-dpo-checklist
Category: Sovereignty
Author: Binadit Tech Team
Published: 2026-10-14T11:14:56+02:00
Who this audit is for
This is written for DPOs and privacy leads who need to verify sovereignty claims from vendors, not just collect them. It is also useful for engineering leads who get pulled into procurement conversations because the DPO needs a technical answer about where data actually sits. If your organisation has committed to a no US subprocessor policy, whether for GDPR risk reduction, a public tender requirement, or a customer contract clause, the hard part is not writing the policy. The hard part is verifying that every vendor in the chain actually honours it.
Most vendor questionnaires ask yes or no questions that vendors answer optimistically. A proper sovereignty audit asks for evidence. The twelve questions below are the ones that tend to surface the gap between what a vendor's marketing page says and what their subprocessor list actually contains.
The 12 questions
1. Does the vendor publish a current subprocessor list, and how often is it updated?
A vendor claiming no US subprocessor status should have a public, dated subprocessor list, not a sales deck claim. Check the last update date. If it has not changed in over a year while the vendor has clearly added features or integrations, the list is probably stale rather than accurate.
2. Who owns the parent company of every subprocessor, not just the entity signing the contract?
This is where most audits fail. A subprocessor can be legally incorporated in Germany or the Netherlands and still be a wholly owned subsidiary of a US parent company, which brings it inside the reach of the CLOUD Act regardless of where the servers sit. Ask for the ultimate parent entity, not just the contracting entity.
3. Is customer support, including L2/L3 escalation, handled by EU-based staff?
Data residency rules in the main contract rarely cover support operations. If a US-based support engineer can remote into a production system to debug an issue, that is a data access path that bypasses your residency guarantees entirely. Ask specifically about support tooling and where support staff are employed.
4. Does the vendor's monitoring, logging, or APM stack send data outside the EU?
Error tracking and application performance monitoring tools often default to US regions even when the primary hosting is in the EU. Sentry, Datadog, and New Relic all have EU hosting options, but they are opt-in, not default. A vendor can be honest about their primary infrastructure and still leak telemetry data through a third-party monitoring integration. We cover this specific gap in more depth in our piece on data sovereignty in email, error tracking, and analytics.
5. Which CDN or DNS provider is in front of the application, and where does it route traffic?
A CDN is a subprocessor even when nobody thinks of it that way. If the origin is in Frankfurt but the CDN is a US hyperscaler's edge network, cached content and request metadata still pass through US-controlled infrastructure. Ask for the specific CDN vendor and whether EU-only routing is configured, not just available.
6. What is the legal basis for any remaining international transfers?
If a vendor cannot achieve zero international transfers, find out whether they rely on Standard Contractual Clauses, Binding Corporate Rules, or something else, and whether that mechanism has been tested against recent case law. Schrems II did not eliminate SCCs, but it raised the bar for supplementary measures. Ask what those measures actually are.
7. Does the backup and disaster recovery environment match the production region claim?
It is common to find EU-only production infrastructure paired with a backup target in a US region, often because the backup tooling defaulted there during initial setup and nobody revisited it. Ask explicitly where backups, snapshots, and DR replicas are stored, not just where the primary database lives.
8. Can the vendor explain who technically has the ability to access unencrypted data, and from where?
Encryption at rest is table stakes. The real question is who holds the keys and from which jurisdiction they can be compelled to act. If a US-based entity, parent company, or cloud provider holds the encryption keys, data residency in the EU does not prevent a lawful access request from reaching it.
9. Is there a documented incident response process that specifies jurisdiction for any law enforcement request?
Ask the vendor what happens procedurally if they receive a data request from a non-EU authority. A mature vendor will have a documented process: who reviews the request, whether it gets challenged, and whether the customer is notified before data is disclosed. If this process does not exist in writing, it does not exist operationally either.
10. Does the infrastructure management partner itself run on sovereign infrastructure, or just resell it?
Some managed infrastructure partner vendors present a sovereignty story built entirely on the underlying cloud provider's EU region, while their own operational tooling, ticketing system, or remote access infrastructure runs through US SaaS products. Ask about the vendor's own internal stack, not just the stack they sell you.
11. What happens to data sovereignty guarantees during a migration or failover event?
Sovereignty commitments are easiest to keep during steady state and easiest to lose during a zero downtime migration, a failover test, or a disaster recovery drill, when teams move fast and improvise. Ask the vendor to walk through a specific failover scenario and confirm the DR target still meets the same residency commitments as production.
12. Is there a named technical contact who can answer these questions directly, without going through a ticket queue?
This sounds procedural, but it is a genuine signal. Vendors who can answer sovereignty questions through a direct conversation with an engineer tend to actually understand their own subprocessor chain. Vendors who route every question through a support ticket and come back days later with a copy-pasted compliance statement usually have not looked closely at the question before.
Config and contract language that helps verification
A few concrete things to request or check during the audit, beyond the questions themselves.
Ask for the DPA's Annex with the actual subprocessor table, not a summary page. It should look something like this:
Subprocessor | Purpose | Entity location | Parent HQ
EU hosting partner | Infrastructure | Netherlands | Netherlands
EU email relay | Transactional email | Germany | Germany
Monitoring vendor | APM / error tracking | EU region (opt-in) | United States
That third row is the one that fails a strict no US subprocessor policy even though the data is hosted in an EU region. The parent HQ column is the one most vendor-supplied tables omit.
If you manage DNS or CDN configuration yourself, you can verify routing claims independently rather than relying on vendor statements:
# Check which edge network is actually serving a domainndig +short CNAME cdn.example.comnn# Trace the resolved IP's registered locationnwhois $(dig +short cdn.example.com | tail -1)n
This will not give you a legal answer, but it will tell you quickly whether a CDN claim matches the DNS reality.
How to roll this out across an existing team
Running this audit once against a new vendor is straightforward. Rolling it out as a standing practice across an existing vendor estate takes more coordination, because most teams have ten to thirty vendors already under contract and nobody wants to re-open every one at once.
Start with a tiering exercise. Not every vendor needs the full twelve questions on day one. Rank vendors by how much personal data or regulated data they touch, and run the full audit against the top tier first, typically your core managed cloud infrastructure provider, your email and identity provider, and anything handling payment or health data.
Build the subprocessor table question into procurement from the start for new vendors, so it stops being a retrospective audit and becomes a gate. A single clause in the vendor onboarding checklist, requiring the Annex table with parent HQ column before signature, prevents most of the gaps described above from ever entering the contract.
For existing vendors, a short, direct email referencing these twelve questions usually gets a faster and more honest answer than a generic security questionnaire, because it signals you already know what to look for. Vendors who built genuine sovereignty into their architecture will usually answer within a day. Vendors who built a marketing claim on top of a US-owned stack tend to go quiet or escalate internally, which is itself useful information.
Keep a simple internal register, even a spreadsheet, tracking the audit date, the answers received, and a re-audit date twelve months out. Subprocessor chains change as vendors get acquired or add integrations, and a no US subprocessor claim that was true at signing is not guaranteed to stay true. Teams that have gone through tenders with hard sovereignty requirements have found this out the hard way, which is part of what we cover in our case study on what a GAIA-X label actually verified during a tender.
Close
This audit will not make a vendor relationship sovereign on its own. It will tell you, with evidence rather than a checkbox, whether a no US subprocessor claim holds up against the actual contract, infrastructure, and operational chain behind it. That is the difference between a policy that satisfies an internal audit and one that would hold up under a regulator's or a customer's scrutiny.
If implementing these yourself is not the best use of your engineering time, our managed services cover all of them by default.
---
### How to automate SSL certificate renewal across a fleet of servers
URL: https://binadit.com/blog/automate-ssl-certificate-renewal-cloud-cost-optimization-services
Category: Infrastructure
Author: Binadit Tech Team
Published: 2026-10-09T09:43:54+02:00
> Manually tracking certificate expiry across dozens of servers does not scale, and the manual approach quietly becomes one of the line items driving up operational overhead. Here is how to automate renewal fleet-wide with Let's Encrypt, a central ACME client, and a deployment hook that actually reloads your services.
What you will achieve and why it matters
By the end of this guide you will have a fleet of servers that renew their TLS certificates automatically, reload the correct services without manual intervention, and alert you only when something actually needs attention. This matters beyond uptime. Manual certificate tracking across many hosts is a recurring labor cost, and it is exactly the kind of recurring manual task that shows up in audits of cloud cost optimization services engagements: someone on the team spends a few hours a month on something that should take zero.
We will build this using Let's Encrypt as the certificate authority, the ACME protocol for automation, and a central orchestration pattern that works whether you are running 5 servers or 500. The same pattern applies whether your fleet sits behind a load balancer, runs as separate application nodes, or spans multiple regions as part of a broader high availability infrastructure setup.
Prerequisites and assumptions
This guide assumes:
You control DNS for the domains you are issuing certificates for, or at minimum can create TXT records programmatically via an API (Cloudflare, Route 53, DigitalOcean, etc.)
Your servers run Linux (examples use Ubuntu 22.04/24.04, but the approach is distro-agnostic)
You have SSH access to all fleet members, ideally via a config management tool (Ansible, Salt, or similar) rather than manual SSH loops
Services that terminate TLS are Nginx, HAProxy, or a similar reverse proxy, not an application server directly handling certificates
You are comfortable with cron or systemd timers for scheduling
We will use certbot as the ACME client because it has the widest plugin support and is well maintained, but the same architecture works with acme.sh or lego if you prefer a lighter footprint.
Step-by-step implementation
Step 1: Choose a validation method that scales
HTTP-01 validation requires port 80 to be reachable on each host for the specific domain being validated. This works for single-domain setups but becomes painful across a fleet, especially behind a load balancer where traffic for a given domain might not land on a predictable server.
DNS-01 validation is the better default for fleets. It does not require inbound traffic to reach the renewing server, supports wildcard certificates, and lets you centralize certificate issuance on one host even if the certificate is then distributed to many.
Example using Cloudflare as the DNS provider:
apt install certbot python3-certbot-dns-cloudflare -y
cat > /etc/letsencrypt/cloudflare.ini << 'EOF'
dns_cloudflare_api_token = YOUR_SCOPED_API_TOKEN
EOF
chmod 600 /etc/letsencrypt/cloudflare.ini
Use a scoped API token (Zone:DNS:Edit for the specific zone only), not your global API key. If that token leaks, the blast radius is limited to DNS records, not your entire Cloudflare account.
Step 2: Issue the certificate from a single orchestration host
Rather than running certbot independently on every server (which risks rate limit collisions and inconsistent certificate state), designate one host, or a small automation runner, as the certificate authority client.
certbot certonly \
--dns-cloudflare \
--dns-cloudflare-credentials /etc/letsencrypt/cloudflare.ini \
--dns-cloudflare-propagation-seconds 30 \
-d example.com \
-d '*.example.com' \
--agree-tos \
-m ops@example.com \
--non-interactive
This produces a wildcard certificate covering all subdomains, which is usually the right call for a fleet serving multiple services under one domain. If your services span unrelated domains, issue separate certificates per domain rather than cramming everything into one SAN certificate. It keeps the blast radius of a renewal failure smaller.
Step 3: Distribute certificates to fleet members
Once issued, certificates live in /etc/letsencrypt/live/example.com/ on the orchestration host. You need a reliable way to push the fullchain.pem and privkey.pem to every server that terminates TLS.
An Ansible playbook handles this cleanly:
- name: Distribute TLS certificates to fleet
hosts: web_fleet
tasks:
- name: Copy fullchain
copy:
src: /etc/letsencrypt/live/example.com/fullchain.pem
dest: /etc/ssl/certs/example.com.pem
owner: root
group: root
mode: '0644'
notify: reload nginx
- name: Copy private key
copy:
src: /etc/letsencrypt/live/example.com/privkey.pem
dest: /etc/ssl/private/example.com.key
owner: root
group: root
mode: '0600'
notify: reload nginx
handlers:
- name: reload nginx
systemd:
name: nginx
state: reloaded
The handler only fires if the file content actually changed, which means idle runs do not cause unnecessary reloads. Nginx reload is graceful, so in-flight connections are not dropped.
Step 4: Automate renewal and redistribution on a schedule
Certbot ships with a systemd timer that runs twice daily and only renews certificates within 30 days of expiry, which is the right cadence. But the default renewal hook only runs locally. You need a deploy hook that triggers your distribution playbook.
mkdir -p /etc/letsencrypt/renewal-hooks/deploy
cat > /etc/letsencrypt/renewal-hooks/deploy/distribute.sh << 'EOF'
#!/bin/bash
set -euo pipefail
ansible-playbook -i /etc/ansible/hosts /opt/playbooks/distribute-certs.yml
EOF
chmod +x /etc/letsencrypt/renewal-hooks/deploy/distribute.sh
Deploy hooks only run when a certificate actually renews, unlike pre/post hooks which run on every invocation. This is the correct hook type for triggering fleet-wide distribution, since you do not want to run an Ansible playbook twice a day for no reason.
Verify the timer is active:
systemctl status certbot.timer
systemctl list-timers | grep certbot
Step 5: Handle non-Nginx services
If some fleet members run HAProxy, which expects a combined PEM file (cert plus key concatenated), add a step to the deploy hook:
cat /etc/letsencrypt/live/example.com/fullchain.pem \
/etc/letsencrypt/live/example.com/privkey.pem \
> /etc/haproxy/certs/example.com.pem
systemctl reload haproxy
Keep per-service post-processing logic in the deploy hook script rather than scattering it across playbooks. One script, one source of truth for what happens after a renewal.
Step 6: Centralize logging and failure alerting
A renewal that fails silently is worse than no automation at all, because it creates false confidence. Pipe certbot's renewal logs to a place you actually look:
certbot renew --dry-run --quiet 2>&1 | logger -t certbot-renewal
For actual alerting, wrap the renewal command and check the exit code:
#!/bin/bash
if ! certbot renew --quiet; then
curl -X POST https://hooks.slack.com/services/YOUR/WEBHOOK/URL \
-d '{"text":"Certbot renewal failed on '"$(hostname)"'"}'
fi
Replace the webhook call with whatever your team already uses for alerting (PagerDuty, Opsgenie, a Slack channel), the important part is that a failed renewal generates a signal somewhere a human will see it within a day, not 29 days later when the certificate is about to expire.
Verification: how to confirm it works
Do not assume automation works because the cron job exists. Verify it end to end.
Dry-run the renewal process:
certbot renew --dry-run
This simulates the full renewal flow, including DNS validation, without hitting Let's Encrypt's production rate limits. Run it immediately after setup and again after any configuration change.
Check certificate expiry dates across the fleet:
for host in $(cat fleet_hosts.txt); do
echo -n "$host: "
echo | openssl s_client -connect $host:443 -servername $host 2>/dev/null \
| openssl x509 -noout -enddate
done
Run this weekly, or better, wire it into your existing monitoring stack as a scheduled check. Most monitoring tools (Prometheus with the blackbox exporter, Uptime Kuma, Zabbix) have a built-in certificate expiry probe.
Prometheus blackbox exporter example:
- job_name: 'ssl_expiry'
metrics_path: /probe
params:
module: [http_2xx]
static_configs:
- targets:
- https://example.com
relabel_configs:
- source_labels: [__address__]
target_label: __param_target
- target_label: __address__
replacement: blackbox-exporter:9115
Alert on probe_ssl_earliest_cert_expiry dropping below a 14-day threshold. This gives you a second, independent confirmation layer outside of certbot's own logging, which matters because you want to know if the automation itself breaks, not just whether the last run succeeded.
Confirm distribution actually reached every host:
ansible web_fleet -m shell -a "openssl x509 -in /etc/ssl/certs/example.com.pem -noout -enddate"
Run this across the fleet after a renewal cycle and confirm every host reports the same expiry date. A mismatch means your distribution step failed on at least one node, which is a far more common failure mode than the renewal itself failing.
Common pitfalls to avoid
Hardcoding API tokens in playbooks. Use a secrets manager (Ansible Vault, HashiCorp Vault, or your cloud provider's secret store) instead of committing credentials to a repo.
Forgetting to reload services after distribution. Copying a new certificate file does nothing if Nginx or HAProxy is still holding the old one in memory. Always pair file updates with a reload handler.
Hitting Let's Encrypt rate limits. The production rate limit is 50 certificates per registered domain per week. If you are issuing per-subdomain certificates across a large fleet instead of one wildcard, you can hit this faster than expected. Test against the staging environment (--staging flag) while debugging.
Running certbot independently on every server. This creates inconsistent certificate state and multiplies the number of things that can silently break. Centralize issuance, distribute the result.
No alerting on renewal failure. A cron job that fails quietly for a month is not automation, it is a delayed incident.
Next steps and related reading
Once certificate renewal is automated, the same orchestration pattern, central issuance with fleet-wide distribution via config management, applies to other recurring operational tasks: rotating SSH keys, pushing updated firewall rules, or redeploying configuration after a zero-downtime migration.
If your fleet sits behind a load balancer with multiple edge nodes, it is worth reviewing how your server performance holds under real traffic once TLS termination is handled consistently across every node, since certificate handling and connection handling often share the same Nginx or HAProxy configuration files.
Teams running this at scale often fold certificate automation into a broader GitOps workflow, where certificate distribution is just one more state reconciled by the same pipeline that manages application deployments. That reduces the number of distinct automation systems you have to maintain and monitor separately.
Close
Automating certificate renewal removes a recurring manual task from your team's plate and closes a gap that otherwise shows up as unplanned downtime or last-minute scrambling. It is a small piece of infrastructure, but it is exactly the kind of small piece that compounds across a fleet.
Need this running in production without building it yourself? See our managed infrastructure services or schedule a call.
---
### Understanding what an Outlook alternative actually needs to do to keep mail in Europe
URL: https://binadit.com/blog/outlook-alternative-keeps-mail-in-europe
Category: Sovereignty
Author: Binadit Tech Team
Published: 2026-10-07T11:42:46+02:00
> Switching away from Outlook is not just about the inbox. It is about who can be legally compelled to hand over your mail, and from where. Here is what a real outlook alternative needs to solve, and what it does not.
What this is about and why it matters now
Outlook is the mail client. Microsoft 365 is the service behind it. For most companies the two have become one thing in daily use: you open Outlook, you see your mail, your calendar, your contacts, and somewhere behind that screen sits a Microsoft data centre. That distinction between client and service is exactly where this article needs to start, because a true outlook alternative is not just a different inbox. It is a different legal relationship with your own data.
Here is the plain version of the problem. Microsoft Corporation is an American company. Google LLC is an American company. Under the US CLOUD Act, enacted on 23 March 2018, a provider under US jurisdiction must hand over data in its possession, custody or control if ordered to by a US court, a warrant or a subpoena, no matter where that data is physically stored. Choosing "EU data region" in Microsoft 365 or Google Workspace changes where the servers sit. It does not change which country's courts can compel disclosure, because the law follows the company, not the server rack.
This is not a flaw in Outlook or Microsoft 365 as products. They are well built, widely supported and familiar to almost every office worker in Europe. The issue is jurisdiction, not quality. An IT manager evaluating an outlook alternative needs to understand that distinction clearly, because it changes what you are actually solving for. You are not looking for a prettier inbox. You are looking for a provider that answers to a different legal system.
We have covered the compliance side of this in more depth in our earlier piece on whether your email is still GDPR compliant next year. This article looks at it from the practical side: how the mechanics of mail, calendar and client actually work, what a genuine European alternative needs to replace, and where the trade-offs are.
How mail, jurisdiction and data residency actually interact
To understand why an outlook alternative needs to be more than a different app icon, it helps to walk through what happens when you send and receive mail, and where the legal exposure actually sits.
When you send an email through Microsoft 365, it typically moves through several Microsoft-controlled systems: the Exchange Online mailbox, spam and malware filtering, journaling and compliance archives, and often a copy in a separate retention or eDiscovery store for legal hold purposes. Each of those systems is a separate place where a copy of your data can exist. Each one is operated by Microsoft Corporation or a subsidiary under its control. The CLOUD Act text is specific here: it applies to data in a provider's "possession, custody, or control", regardless of whether that data sits inside or outside the United States. A European data centre location does not remove Microsoft's custody of the data; it only changes the postal address of the hardware.
GDPR approaches the same problem from the other direction. Chapter V of the regulation (Articles 44 to 50) says personal data may only leave the European Economic Area on the basis of an adequacy decision or safeguards like standard contractual clauses. Article 48 adds something specific and often missed: a foreign court order or government demand does not, by itself, make a transfer lawful. And Article 24 puts the burden of proof on the controller, meaning your company, not the vendor, is the one who has to be able to demonstrate that its data handling is compliant.
These two laws do not quite fit together, and that mismatch has already gone through the courts three times. Safe Harbor, the first EU-US data transfer framework, was struck down by the Court of Justice of the EU in 2015. Its replacement, Privacy Shield, was struck down in July 2020 in the case known as Schrems II, where the Court found that US surveillance law, specifically FISA Section 702, gives EU citizens no effective way to challenge or remedy US government access to their data. The current framework, the EU-US Data Privacy Framework, was adopted by the European Commission on 10 July 2023 and is in force today.
It is also, at the time of writing, under legal challenge again. The EU General Court dismissed a challenge brought by French MP Philippe Latombe on 3 September 2025. He appealed on 31 October 2025, and that appeal is now pending before the Court of Justice of the EU with no judgment yet given. This is not a prediction of what will happen. It is simply the current state of a pattern: three frameworks in a row have ended up in European courts, two of the three have already been struck down, and the third is actively being tested.
There is also a human data point worth knowing, because it answers the question most IT managers actually want answered: can the vendor guarantee this won't happen to us? On 10 June 2025, Anton Carniaux, director of public and legal affairs at Microsoft France, testified under oath before a French Senate inquiry. Asked directly whether he could guarantee that French citizens' data entrusted to Microsoft through the French public purchasing body UGAP would never be handed to the US government without explicit French authorisation, his answer was: "Non, je ne peux pas le garantir", meaning "No, I cannot guarantee that." He added that it had never happened so far. Both halves of that answer matter: no guarantee, and no known incident yet. That is an honest answer from a company executive, not a scandal, and it is worth repeating accurately rather than dramatising it.
Concrete examples: what actually changes when you switch
Let's make this concrete rather than abstract. Say a 40-person architecture firm in Utrecht runs entirely on Microsoft 365: Outlook for mail, OneDrive for files, Teams for meetings, and a shared info@ mailbox that three people monitor. What does a genuine switch away from Outlook actually involve, piece by piece?
Mailboxes. Each person's mail, calendar and contacts need somewhere to live that is not US-jurisdiction. The receiving system needs to speak the same protocols your existing tools expect: IMAP and SMTP for mail, CalDAV for calendar, CardDAV for contacts. If the new provider only offers a web app with no protocol support, you have traded one lock-in for another.
Shared mailboxes. The info@ mailbox needs permission control so the right three colleagues can read and send from it without needing their own separate login to it.
Domain authentication. SPF, DKIM and DMARC records need to be reconfigured for the new mail server so that outgoing mail is not flagged as spam by the people you are writing to.
The client. Staff are used to the three-pane Outlook layout: folder list on the left, message list in the middle, reading pane on the right. If the replacement forces everyone into a browser tab, adoption suffers regardless of how sound the backend is.
Files. OneDrive files need somewhere to sync to that supports the same "files on demand" behaviour people expect, where a file shows as available without actually occupying local disk space until opened.
Meetings. Teams calls need a replacement that guests can join from a browser link without installing software, because you cannot ask every client and supplier to adopt your internal tooling.
Binadit Workspace is built to cover exactly that list, as one product with one login on the customer's own domain, rather than five separate subscriptions bolted together. Mail, calendar and contacts work over IMAP, SMTP, CalDAV, CardDAV and JMAP, so existing mail clients, including our own, keep working. Shared mailboxes like info@ get per-colleague permissions. Domain records (MX, SPF, DKIM, DMARC, autoconfig) are placed automatically through Binadit's own DNS tool. For the client itself, Northmail is Binadit's free mail app for macOS and Windows, built deliberately in the three-pane layout people already know from Outlook, and it can connect to Workspace, on-premises Exchange and Gmail side by side, which matters during a migration when not everyone switches on day one.
On pricing, a 2 GB mailbox with calendar, contacts, webmail, Northmail, Meet and Book included costs €2.95 per month excluding VAT; a 25 GB mailbox is €5.95. A company storage pool for shared drive and office documents starts at €9 per month for 250 GB and includes the online office suite. None of this requires a minimum number of mailboxes or a fixed contract term; the full calculator is at binadit.work.
Trade-offs and design decisions worth knowing before you choose
No honest comparison skips the trade-offs, so here they are plainly.
Feature parity is not total parity, and it should not pretend to be. Binadit Workspace does not include team chat with channels, the kind of thing Microsoft Teams or Slack offer; that is planned but not yet announced, so if real-time threaded chat across the whole company is a daily requirement today, factor that into your timeline. It also does not do telephony, large-audience webinars, or meeting room hardware integration. If your organisation runs 200-person town halls or needs a dial-in PBX, that is outside scope.
Encryption is in transit and at rest, not end to end, for regular mail. That matches how Outlook and Gmail work too: convenient server-side search and webmail access require the server to be able to read the content. The one place Workspace does go further is the Receive tool, the replacement for Cryptshare, where files and filenames are encrypted in the sender's browser before upload, so the server only ever stores ciphertext and the password travels through a separate channel that Binadit itself cannot recover.
Desktop sync coverage is not yet universal. Drive, the file sync layer, has native apps for macOS and Windows today; Linux, iOS and Android apps are in development. If your team is heavily iOS or Android for file access, check the current state before committing a migration date.
There is no AI assistant reading your mailbox on the server. Binadit does not train on customer data and does not run an AI assistant inside the mailbox or documents. The only assistant available is an optional one inside Northmail that runs locally on the user's own computer and sends nothing to a cloud. If your organisation specifically wants a server-side AI summarising every inbox, that is a deliberate gap, not an oversight.
Jurisdiction is the actual design decision underneath all of this. Binadit B.V. is a Dutch company, 100% Dutch-owned, with no foreign parent. Every server runs in the Netherlands, in data centres operated by European companies, and Binadit operates the servers itself rather than reselling a US hyperscaler's EU region. No supplier outside the EU is involved, the customer signs a processing agreement under European law, and because there is no American parent company in the ownership chain, the US CLOUD Act simply does not have a hook to apply through. That is a structural answer, not a policy promise. (Note: the data centres Binadit hosts in are ISO 27001 and NEN 7510 certified facilities; Binadit itself does not hold those certifications as a company.)
Switzerland comes up occasionally in these conversations too, worth a short note: it is not an EU or EEA member, but transfers there are lawful under an existing EU adequacy decision, and a Swiss provider answers to Swiss law rather than EU law. That is a different, narrower kind of sovereignty than a Dutch provider operating fully inside the EU legal system, and worth distinguishing if a supplier brings it up.
When to actually make the switch, and when not to
An outlook alternative is worth evaluating seriously when any of the following is true for your organisation:
You handle client data under a contractual or regulatory obligation to keep it inside the EU, not just "an EU region", and you need to be able to demonstrate that under GDPR Article 24 if asked.
You are responding to a tender or procurement process, public or private, where data residency and provider jurisdiction are evaluation criteria.
Your privacy officer or legal counsel has flagged the Schrems I, Schrems II, and now Latombe pattern as a recurring risk that keeps resurfacing on a multi-year cycle, and wants a structural answer rather than another framework to monitor.
You want one login and one invoice for mail, files, documents, meetings and scheduling instead of stitching together Microsoft 365, a separate scheduling tool, and a separate secure file transfer tool.
It is less of a fit, at least for now, if your organisation genuinely depends on deep Teams channel structures across departments, needs enterprise telephony integration, or requires mobile file sync on iOS and Android before the native apps for those platforms ship. In those cases, a phased migration, mail and calendar first, files and meetings second, is usually the more honest plan than an all-at-once cutover. We have written separately about why EU region toggles inside US cloud providers don't actually solve data sovereignty, which is worth reading if your current plan is to simply flip a region setting inside your existing Microsoft 365 or Google Workspace tenant and call it solved.
Further reading and next step
If you want the fuller legal background on the CLOUD Act, GDPR Chapter V, and the Schrems case history before making a decision, start with our article on whether your email is GDPR compliant today and whether it will still be next year, and our comparison of standard contractual clauses against binding corporate rules after Schrems II.
When you are ready to look at what replacing Outlook, Microsoft 365 or Google Workspace actually looks like for your organisation, in terms of mailbox counts, storage, and migration sequencing, Binadit Workspace is built specifically for European companies that want mail, calendar, drive, documents, meetings and scheduling under one login, on servers in the Netherlands, under European law. Ask for a demo and a quote and the same engineer who sets up your environment will walk you through what a migration from Outlook would actually look like for your team.
---
### How to set up website server performance that holds under real traffic
URL: https://binadit.com/blog/website-server-performance-managed-cloud-provider-europe-setup
Category: Infrastructure
Author: Binadit Tech Team
Published: 2026-09-25T09:53:37+02:00
What you will achieve and why it matters
This guide walks through the infrastructure-level changes that have the biggest measurable impact on server response time: process manager tuning, connection handling, database pooling, and caching layers. These are the same changes we apply when a client comes to us with a slow WooCommerce store or a SaaS backend that degrades under concurrent load.
You do not need a full re-architecture to get a faster server. Most performance gains come from a small number of configuration changes applied correctly, in the right order. If you run this as a managed cloud provider europe would for a production client, you will end this guide with a server that handles concurrency predictably instead of falling over at the first traffic spike.
Prerequisites and assumptions
This guide assumes:
You are running a Linux server (Ubuntu 22.04 or Debian 12 examples used below) with root or sudo access.
Your stack is a typical web application: Nginx as a reverse proxy, PHP-FPM or a similar application runtime, and MySQL or PostgreSQL as the database.
You have SSH access and can restart services safely, ideally in a staging environment first.
You have basic monitoring already in place, even if it is just htop and server logs. You will need a baseline to compare against.
If you are running a containerized setup (Docker, Kubernetes), the same principles apply, but the commands will differ. We cover container-specific pitfalls separately in our piece on how Docker networking broke checkout under load.
Step-by-step implementation
Step 1: baseline your current performance
Before changing anything, record where you stand. Run a load test against a realistic endpoint:
ab -n 1000 -c 50 https://yourdomain.com/n
Note the requests per second, mean time per request, and the number of failed requests. Also check current server load during the test:
uptimenvmstat 1 5n
Write these numbers down. Every change below should be validated against this baseline.
Step 2: tune PHP-FPM process management
The default PHP-FPM pool configuration is almost never correct for production. Open your pool config:
sudo nano /etc/php/8.3/fpm/pool.d/www.confn
Set the process manager to dynamic and size it based on available RAM, not guesswork. A common rule of thumb: divide available memory by the average memory footprint of a single PHP process (check with ps aux | grep php-fpm to see real usage per worker, typically 30 to 60MB for a WordPress or Laravel app).
pm = dynamicnpm.max_children = 40npm.start_servers = 10npm.min_spare_servers = 5npm.max_spare_servers = 15npm.max_requests = 500n
The pm.max_requests setting matters more than people think. It restarts workers after N requests, which prevents memory leaks in long-running PHP processes from slowly degrading your server.
Step 3: configure Nginx for connection efficiency
Increase worker connections and enable keepalive to reduce the overhead of repeated TCP handshakes:
worker_processes auto;nworker_rlimit_nofile 65535;nnevents {n worker_connections 4096;n use epoll;n multi_accept on;n}nnhttp {n keepalive_timeout 65;n keepalive_requests 1000;n sendfile on;n tcp_nopush on;n tcp_nodelay on;n}n
If Nginx sits in front of PHP-FPM over a Unix socket, make sure the socket backlog matches your expected concurrency:
listen = /run/php/php8.3-fpm.socknlisten.backlog = 1024n
Step 4: enable connection pooling for your database
Every unpooled database connection costs a TCP handshake, authentication round trip, and memory allocation on the database side. Under load, this adds up fast. For MySQL, install and configure ProxySQL or a lighter option like mysqlnd_mux at the application layer. For PostgreSQL, PgBouncer is the standard choice:
sudo apt install pgbouncern
Minimal PgBouncer config for transaction-level pooling:
[databases]nyourapp = host=127.0.0.1 port=5432 dbname=yourappnn[pgbouncer]nlisten_port = 6432nauth_type = md5npool_mode = transactionnmax_client_conn = 500ndefault_pool_size = 25n
Point your application's database connection string at port 6432 instead of 5432. This alone typically cuts connection overhead by 60 to 80% on high-concurrency workloads.
Step 5: add a caching layer for repeated reads
Install Redis for object caching and session storage:
sudo apt install redis-servernsudo systemctl enable redis-servern
Set a sane max memory policy so Redis evicts old keys instead of running out of memory:
maxmemory 512mbnmaxmemory-policy allkeys-lrun
For a WordPress or WooCommerce site, connect an object cache plugin (Redis Object Cache) to this instance. For a custom application, wrap your most expensive, most frequently called database queries with a cache-aside pattern:
$cacheKey = "product:{$id}";n$product = $redis->get($cacheKey);nif (!$product) {n $product = $db->query("SELECT * FROM products WHERE id = ?", [$id]);n $redis->setex($cacheKey, 300, serialize($product));n}n
Step 6: set correct cache headers at the edge
Static assets should never hit your application server on repeat visits. In Nginx:
location ~* \.(jpg|jpeg|png|gif|css|js|woff2)$ {n expires 30d;n add_header Cache-Control "public, immutable";n}n
If you run a CDN in front, verify it respects origin cache headers rather than overriding them with a default TTL. We go deeper on this in our guide to CDN and origin caching optimization.
Verification: confirming it actually works
Re-run the same load test from Step 1, at the same concurrency level:
ab -n 1000 -c 50 https://yourdomain.com/n
Compare the following against your baseline:
Requests per second: should increase noticeably, often 30 to 100% depending on how unoptimized the starting point was.
Mean response time: should drop, and more importantly, the variance between the 50th and 95th percentile should tighten.
Failed requests: should be zero at your target concurrency.
Check PHP-FPM worker behavior under load in a second terminal during the test:
watch -n 1 'ps aux | grep php-fpm | wc -l'n
You want to see worker count scale up smoothly and settle, not spike to pm.max_children and stay pinned there. If it pins immediately, your max_children value is too low for the load, or a slow query upstream is holding workers open longer than it should.
Verify PgBouncer or ProxySQL is actually being used, not bypassed:
psql -h 127.0.0.1 -p 6432 -U youruser yourapp -c "SHOW POOLS;"n
Confirm Redis hit rate is meaningful, not near zero:
redis-cli info stats | grep keyspacen
A healthy cache-aside implementation should show a hit ratio above 80% for read-heavy endpoints within a few minutes of traffic.
Common pitfalls to avoid
A few mistakes we see repeatedly when teams apply these changes without full context:
Setting max_children too high. More workers than your RAM can support leads to swapping, which is far worse than queuing requests. Calculate from actual memory usage, not a round number.
Caching without invalidation logic. A cache that never expires stale data creates subtle bugs that are harder to debug than a slow server. Always set a TTL or an explicit invalidation hook on write.
Pooling connections but not adjusting application timeouts. If your app still assumes direct, unpooled connections, transaction-mode pooling can produce confusing errors with session-level features like advisory locks or SET statements.
Skipping the baseline. Without a "before" number, you cannot prove any of this worked, and you will not know which change actually mattered if you roll several out at once.
Next steps and related reading
Once server-level tuning is in place, the next layer to examine is how the database itself behaves under sustained load, particularly around index usage and query plans. Our guide on measuring database performance degradation walks through the diagnostic side of that.
If your current setup is a single server and you are starting to hit ceilings that tuning alone cannot fix, the next architectural step is usually horizontal scaling or a move to high availability infrastructure with load balancing across multiple application nodes. That is a bigger change than anything covered here, and worth planning deliberately rather than reactively.
Getting this running without doing it all yourself
Every step above is something your team can implement directly. What tends to be harder is doing it consistently across environments, catching regressions before they hit production, and having someone available when a config change under load does not behave the way the documentation said it would. That is the operational gap a managed cloud provider europe based teams work with is built to close: not replacing your engineers, but backing them with direct, EU-based support instead of a ticket queue.
Need this running in production without building it yourself? See our managed infrastructure services or schedule a call.
---
### Your email is GDPR-compliant today. Will it still be next year?
URL: https://binadit.com/blog/email-gdpr-compliant-google-workspace-microsoft-365
Category: sovereignty
Author: Binadit Tech Team
Published: 2026-09-20T15:06:57+02:00
> Google Workspace and Microsoft 365 are GDPR-compliant today because of one adequacy decision, and that decision is before the same court that struck down the previous two. What your provider has admitted under oath, and what email hosted in the Netherlands costs. Less than you pay now.
If your company runs on Google Workspace or Microsoft 365, your email, calendar and files live with an American provider. Ask your IT partner whether that is allowed under the GDPR and you will hear: yes, both are certified under the EU-US Data Privacy Framework, so you are fine.
That answer is correct. It is also incomplete. This article explains why the legal basis for your email is far shakier than it looks, what your provider has already admitted under oath, and what a European alternative costs. Spoiler: less than you pay now.
The short version
Email in Google Workspace or Microsoft 365 is a transfer of personal data to a US company. It is lawful today because of one instrument: the EU-US Data Privacy Framework of July 2023.
The two frameworks before it, Safe Harbor and Privacy Shield, were both struck down by the Court of Justice of the EU. The current one is before that same court now.
An EU data centre does not change this. Under the US CLOUD Act a US provider must hand over data it controls, wherever the servers are. Asked under oath in June 2025, Microsoft France could not guarantee otherwise.
Email, calendar and files hosted by a Dutch company in the Netherlands involve no transfer at all, and start at €2.95 per mailbox per month.
The rule: personal data does not leave Europe without a legal basis
Chapter V of the GDPR, Articles 44 to 50, is clear. Personal data may only be transferred to a country outside the European Economic Area if that country offers an adequate level of protection (Article 45), or if you as the controller put safeguards in place yourself, such as the EU standard contractual clauses (Article 46). Article 48 adds that a foreign court order or government demand does not by itself make a transfer lawful.
Email is personal data. Every message, every calendar invite, every shared file contains names, addresses, contracts, HR matters, medical notes from an employee, complaints from customers. If that data sits in a US company's infrastructure, this chapter of the GDPR applies to you.
Three attempts, two collapses, one pending verdict
The EU and the US have tried three times to create a legal bridge for this data:
Safe Harbor (2000). Struck down by the Court of Justice of the EU in 2015 (Schrems I).
Privacy Shield (2016). Struck down in 2020 (Schrems II). The Court found that US surveillance law, in particular FISA Section 702, gives EU citizens no effective protection or remedy.
The EU-US Data Privacy Framework (July 2023). This is the basis your Google or Microsoft contract relies on today.
Every one of these frameworks was presented as the definitive solution. Two of them are gone. Companies that had built their compliance on Privacy Shield woke up in July 2020 with no legal basis for their email, overnight.
Where the current framework stands in September 2026
The DPF is still in force. But look at what has happened around it:
It is before the Court of Justice. A challenge by French MP Philippe Latombe was dismissed by the General Court on 3 September 2025 (Case T-553/23), but the court restricted itself to the facts as they stood when the framework was adopted in July 2023. Nothing that has happened in the US since then was weighed. Latombe appealed (Case C-703/25 P), and that appeal is pending before the Court of Justice, the body that struck down the previous two frameworks. In June 2026 the Court granted Microsoft leave to intervene in support of the Commission, on the ground that Microsoft has a direct interest in the outcome. That tells you how much is at stake for them.
The US safeguards it rests on are being dismantled. The adequacy decision leaned on independent US oversight bodies. The Privacy and Civil Liberties Oversight Board has had no quorum since 27 January 2025 and is down to a single member. On 29 June 2026 the US Supreme Court held in Trump v. Slaughter that the protection of FTC commissioners against dismissal is unconstitutional, so the President may remove them at will. The FTC is the body that enforces the framework on the US side. The same day, privacy organisation noyb called on the European Commission to "orderly withdraw" the adequacy decision and announced a court challenge of its own. The Commission has said only that it will assess the ruling.
FISA 702 has not been reformed. The surveillance law at the heart of Schrems II was due to expire on 20 April 2026, was kept alive by two short-term extensions, and lapsed on 12 June 2026 when the House rejected a third. Collection continues regardless, under court certifications that run until around March 2027. Through all of it, nothing in the law has changed in the direction the Court demanded.
You do not need to predict the outcome. You only need to notice that your company's email compliance depends on a decision that a court may annul on a date you do not control, and that the last time this happened it took three years to replace.
"But our data is stored in an EU data centre"
This is the most common reassurance, and it does not solve the problem. Under the US CLOUD Act, a US company must hand over data under its control when a US authority demands it, regardless of where the servers are. A Dutch, Irish or German data centre owned by a US parent company is still within reach.
You do not have to take our word for it. On 10 June 2025, Anton Carniaux, director of public and legal affairs at Microsoft France, testified under oath before a French Senate inquiry into public procurement. Its rapporteur, Senator Dany Wattebled, asked him whether he could guarantee that data of French citizens entrusted to Microsoft through the state purchasing body would never be transmitted on the order of the US government without the explicit agreement of the French authorities. His answer: "Non, je ne peux pas le garantir." No, I cannot guarantee that. He added that it had never happened so far. That is the point: it depends on whether it happens, not on whether it can.
The full exchange is in the official Senate transcript (in French) and was reported in English by The Register. This is the provider itself, on the record, under oath.
An EU region in a US cloud is a location choice. It is not a jurisdiction choice.
Not sure where your own mail goes? It is in public DNS. Our US exposure scanner looks up who handles the email for your domain, and which legal system that company falls under.
The AI layer makes it worse
Both Google and Microsoft are now building AI assistants directly into your mailbox and your files. That adds a second set of obligations:
Purpose limitation and data minimisation. Your customers' data was collected to answer their request, not to train or improve a model. Consumer and many "free" AI tiers use input for exactly that. Only business contracts explicitly exclude it, and you need to verify that yours does.
Accuracy. AI models produce confident errors. If an assistant writes incorrect information about a real person into a summary or a draft, that violates the accuracy principle of the GDPR, and once something is inside a trained model it cannot be deleted for a single individual.
The EU AI Act. Its transparency duties have applied since 2 August 2026. The heavier documentation and oversight duties for AI used in areas such as HR, credit or access to essential services were postponed in July 2026 and now apply from 2 December 2027. That is a delay, not a reprieve: if you cannot tell which model processed which data, in which jurisdiction, you will not be able to meet them when they land.
Every AI feature switched on inside a US productivity suite is another data flow you are accountable for under Article 24 of the GDPR, and another one you have to explain to a customer or a regulator.
What your accountability actually requires
Article 24 GDPR makes you, the controller, responsible for demonstrating compliance with appropriate technical and organisational measures. After Schrems II that includes assessing, for each transfer, whether the destination offers equivalent protection and adding measures if it does not. In practice this means that with a US provider you are expected to keep a transfer impact assessment up to date, monitor a court case in Luxembourg, follow US executive orders and be ready to switch if the framework falls.
Or you remove the transfer.
The alternative: email, calendar and files that never leave the Netherlands
Binadit hosts business email, shared calendars and file storage from our own data centre in the Netherlands, on hardware we own, operated by a Dutch company with no foreign parent. There is no transfer to a third country, so Chapter V of the GDPR does not apply to your mailbox at all. No adequacy decision to watch, no CLOUD Act exposure, no transfer impact assessment to maintain.
What you get:
Email with your own domain, spam and virus filtering, and full support for Outlook, Apple Mail, Thunderbird and mobile clients through standard protocols. Your people keep the mail client they already use.
Calendar and contacts shared across your organisation, working in the same clients.
File storage and sharing as a replacement for OneDrive and Google Drive, with sharing links, versioning and desktop sync.
Self-service management of mailboxes, aliases and users through the Binadit portal.
A Dutch data processing agreement under Dutch law, with a Dutch supervisory authority, and support in your own language from a fixed contact who knows your setup.
No AI training on your data. Ever. Your data is yours; we host it, we do not learn from it.
What that looks like for the suite you use now is worked out per product: the alternative to Microsoft 365, including Exchange itself hosted in the Netherlands, the alternative to Google Workspace, and Binadit Drive as the alternative to Google Drive, to OneDrive and to Dropbox.
And it costs less than what you pay today
The entry tiers of Google Workspace and Microsoft 365 cost roughly €6 to €8 per user per month depending on the commitment you sign, and the plans most companies actually end up on cost well over €10 (list prices excluding VAT, September 2026). Here is what the same thing costs in a Dutch data centre:
Price
Best for
Binadit Workspace mailbox
from €2.95 per mailbox per month
Teams that want email, calendar and files without running anything themselves
Exchange mailbox, hosted in the Netherlands
from €4.95 per mailbox per month
Companies that want to keep Outlook, Exchange ActiveSync and shared mailboxes exactly as they are. Same software, but hosted and controlled by a Dutch company on Dutch hardware, so the CLOUD Act does not reach it.
Managed dedicated mail server
from €149 per month, unlimited mailboxes
Organisations from roughly 25 mailboxes upward, or anyone who wants a fully isolated environment. Your only limit is disk space, not the number of users. You create mailboxes and domains yourself in the Binadit portal; we manage the server, updates, backups and monitoring.
Managed dedicated Exchange server
from €299 per month, unlimited mailboxes
Larger Exchange environments that want their own isolated server with full Outlook compatibility, managed by us, with mailboxes and domains under your own control in the Binadit portal.
Run the numbers for a 30-person company. On a Google or Microsoft entry plan you pay around €2,500 a year. The same 30 people cost €1,062 a year on Binadit Workspace mailboxes, €1,782 a year on hosted Exchange, or a flat €1,788 a year on a dedicated server that will still be the same price at 60 or 100 mailboxes. You get the compliance fix and the savings in the same move.
We migrate your existing mailboxes, calendars and files from Google or Microsoft, including historic mail, and we run both environments side by side until you switch DNS. No downtime, no lost mail.
What to do this week
Check which of your services currently rely on the EU-US Data Privacy Framework. For most companies the answer starts with email.
Ask your provider, in writing, whether it can guarantee that your data will not be handed to a non-EU authority. Keep the answer. Microsoft has already given theirs.
Talk to us about what a move would look like for your organisation. We will map your current setup, give you a fixed migration price and a date.
You can stay compliant by monitoring a court case, or you can stay compliant by not having the problem. Get in touch with Binadit for a migration assessment.
Frequently asked questions
Is Google Workspace GDPR-compliant?
Today, yes. Google is certified under the EU-US Data Privacy Framework, which the European Commission adopted in July 2023, and that makes the transfer of your email to Google lawful. The compliance is as durable as that framework. Its two predecessors were struck down by the Court of Justice of the EU, and an appeal against the current one is pending there.
Is Microsoft 365 GDPR-compliant if we choose an EU data region?
The EU region decides where the data is stored. It does not decide which law the provider answers to. Microsoft is a US company, and under the CLOUD Act it must produce data under its control when a US authority orders it to, wherever that data is stored. Microsoft France stated under oath in June 2025 that it cannot guarantee otherwise.
What happens to our email if the Data Privacy Framework is annulled?
The same as in July 2020, when Privacy Shield fell. The transfer loses its legal basis on the day of the judgment, without a transition period. You would have to rely on standard contractual clauses and a transfer impact assessment that concludes US law offers equivalent protection, which is the conclusion the Court would just have rejected.
Does the CLOUD Act apply to a Dutch hosting company?
The CLOUD Act binds providers that fall under US jurisdiction. A Dutch company with no US parent, hosting on its own hardware in the Netherlands, is outside it. A demand for data would have to go through a Dutch court under Dutch law.
Can we keep using Outlook if we leave Microsoft 365?
Yes. Outlook is a mail client and works with any mailbox that speaks the standard protocols. With a hosted Exchange mailbox in the Netherlands you also keep Exchange ActiveSync, shared mailboxes and shared calendars exactly as they are.
---
### Solving the deploy frequency wall: from weekly releases to multiple daily deploys without new incidents
URL: https://binadit.com/blog/infrastructure-performance-optimization-multiple-daily-deploys
Category: Commerce
Author: Binadit Tech Team
Published: 2026-09-18T09:59:14+02:00
The symptom: every attempt to deploy faster gets rolled back by the team itself
A recurring pattern shows up in engineering teams around 15-40 engineers: leadership asks for faster shipping, the team moves from weekly to twice-weekly deploys, and within a month someone quietly reverts to weekly again. Not because of a mandate, but because two or three bad releases in a row make everyone nervous. The team polices its own deploy frequency downward because the infrastructure gives them no other way to stay safe.
This is not a discipline problem or a testing culture problem, even though it often gets diagnosed as one. It is almost always an infrastructure gap: the deploy pipeline, the rollback path, and the observability stack were built for a world where releases happen once a week and get manually watched for an afternoon. Ask that same setup to support five deploys a day and it cannot tell the difference between a bad release and normal noise fast enough to matter.
The actual root cause: deploys and infrastructure state are not decoupled
When a team deploys weekly, they can absorb a lot of sloppiness in the pipeline because there is time to manually verify each release. Someone watches error rates for twenty minutes, checks a dashboard, calls it good. At five deploys a day, nobody has twenty minutes per deploy, and the manual verification step either gets skipped or becomes theater.
The underlying technical problem usually breaks down into four coupled issues:
Deploys are not atomic. Code, database migrations, and config changes ship together in one step. If the migration is slow or the config is wrong, there is no way to release the code without them.
Rollback is a redeploy, not a switch. Reverting means running the deploy pipeline again in reverse, which takes as long as deploying forward and carries the same risk.
Health signals lag behind deploy speed. Monitoring intervals, log aggregation delays, and alert thresholds were tuned for hourly granularity, not minute-by-minute deploy validation.
Blast radius is the whole fleet. Every deploy touches 100% of instances at once, so a bad release is a full incident by definition rather than a contained one.
None of these are code quality problems. They are infrastructure performance optimization problems: the deploy path, the traffic routing layer, and the monitoring stack are not built to give fast, accurate signal at high release frequency. Fix the infrastructure, and the same codebase with the same test coverage ships far more safely.
The fix: decouple, automate rollback, and shrink blast radius
1. Separate schema changes from code deploys
Database migrations are the single biggest cause of deploy fear. The fix is the expand/contract pattern: every schema change ships in three steps across separate deploys, never one.
-- Step 1: expand (safe, backward compatible)nALTER TABLE orders ADD COLUMN shipping_method_v2 VARCHAR(50) NULL;nn-- Step 2: dual-write in application code, backfillnUPDATE orders SET shipping_method_v2 = shipping_method WHERE shipping_method_v2 IS NULL;nn-- Step 3: contract (separate deploy, days later)nALTER TABLE orders DROP COLUMN shipping_method;
This means code deploys never wait on migrations, and migrations never block a rollback. We cover the full mechanics of this in how zero-downtime database migrations work if your migration step is currently the bottleneck on deploy frequency.
2. Make rollback a routing change, not a redeploy
If rolling back means running the pipeline in reverse, teams will hesitate to deploy at all under time pressure. Rollback needs to be a traffic switch that takes seconds.
With blue/green or canary deploys behind a load balancer, this looks like:
# Example: Nginx upstream weight shift for canary rollbacknupstream backend {n server app-v124-1:8080 weight=0; # new version, was liven server app-v123-1:8080 weight=100; # previous stable, restoredn}nn# reload takes effect in under 1 secondnnginx -s reload
For containerized environments, the same principle applies at the Kubernetes level:
kubectl rollout undo deployment/checkout-apinkubectl rollout status deployment/checkout-api --timeout=30s
The deploy pipeline builds and ships the new version to a small percentage of instances first. If it is healthy, weight shifts up. If it is not, weight shifts back to zero. Nobody reruns a 12-minute build to recover; they flip a percentage.
3. Automate canary analysis instead of relying on a human watching a dashboard
At weekly cadence, a human staring at Grafana for 20 minutes is viable. At five deploys a day, it is not, and it is also unreliable, since humans get worse at spotting anomalies the more times they repeat the task.
Automated canary analysis compares the new version against the baseline on a fixed set of metrics before promoting traffic further:
# Example canary gate (simplified, pseudo-config)ncanary:n metrics:n - name: error_raten threshold: baseline + 0.5%n window: 5mn - name: p95_latencyn threshold: baseline + 15%n window: 5mn - name: 5xx_countn threshold: baseline + 10n window: 5mn action_on_fail: auto_rollbackn promotion_steps: [5%, 25%, 50%, 100%]n step_duration: 5m
This turns the safety check from a subjective human judgment call into a repeatable, fast, automated gate. It is the single highest-leverage change most teams make when moving to daily deploys.
4. Shrink blast radius with progressive rollout
Deploying to 100% of instances at once means every release is a full-fleet bet. Progressive rollout limits exposure:
Deploy to 5% of instances or a single availability zone first
Hold for one full traffic cycle (including any batch jobs or cron triggers)
Auto-promote if metrics hold, auto-rollback if they don't
Only reach 100% after passing every stage
This is the same pattern that underpins reliable zero-downtime migration work: you never bet the whole system on one untested step. Deploy frequency and migration safety are solved by the same underlying discipline: reduce the size of each irreversible action.
How to validate the fix worked
Once the pipeline changes are in place, track these specifically, not generic "deploy success" metrics:
Change failure rate: percentage of deploys that trigger a rollback or hotfix. Target under 15% (DORA elite benchmark), and it should trend down, not up, as frequency increases.
Mean time to restore (MTTR): from the DORA framework. With automated rollback via traffic weight shift, this should drop to under 5 minutes, down from whatever a manual redeploy previously took.
Rollback execution time: measure specifically how long it takes from "metrics breach threshold" to "traffic fully reverted." This should be seconds to low minutes, not tied to build/deploy duration.
Canary gate false negative rate: how often a bad release makes it past the automated gate and gets caught by users instead. If this is above zero more than once a quarter, the gate's thresholds need retuning, not removal.
Deploy frequency vs. incident count, plotted together: the whole point of this work is that these two lines should decouple. If incident count still rises with deploy frequency after these changes, look for a gap in the canary metrics: usually it's a business metric (checkout completion, cart abandonment) that infrastructure metrics don't capture.
Run this validation for at least four weeks at the new deploy frequency before calling the fix complete. A single good week proves nothing; you need to see the pattern hold under real load variance.
How to prevent recurrence
Deploy frequency backslides quietly if the underlying infrastructure isn't kept honest. A few practices keep it from creeping back to weekly:
Alert on deploy frequency itself. If weekly deploy count drops below a threshold without an explicit decision to slow down, that's a signal something in the pipeline is causing quiet avoidance.
Treat canary threshold tuning as ongoing work, not a one-time setup. As traffic patterns and the application change, thresholds need revisiting quarterly. Stale thresholds are a common cause of both false positives (blocking safe deploys) and false negatives (missing bad ones).
Keep migrations in the expand/contract pattern permanently. The moment someone ships a "quick" combined schema-and-code change under deadline pressure, the coupling comes back and deploy fear returns with it.
Run rollback drills on a schedule, not just when something breaks. A rollback path that hasn't been exercised in three months is not a tested rollback path.
Review post-incident data honestly. If an incident does happen, the review should identify whether it was caught late because a metric was missing from the canary gate, not just closed out with "we'll be more careful." For a structured approach to this, see post-incident reviews that actually improve things.
The teams that sustain multiple daily deploys long-term are the ones who treat the deploy pipeline itself as a piece of production infrastructure that needs monitoring, ownership, and periodic investment, not a one-time engineering project that gets built and then ignored.
Where this fits in the bigger picture
None of this requires exotic tooling. Nginx or an equivalent load balancer, Kubernetes or a comparable orchestrator, and a metrics pipeline that can evaluate a threshold within a five-minute window cover most of it. What it does require is deliberate infrastructure performance optimization work across the deploy path, the routing layer, and the observability stack together, since fixing one without the others just moves the bottleneck.
This is also where the work tends to stall for teams without dedicated infrastructure ownership. Canary gates need tuning as the application evolves. Rollback paths need testing after every significant architecture change. Someone has to own the decision of when automated rollback is trusted enough to run without a human in the loop. That ongoing ownership is usually the actual gap, more than any single missing tool.
If you'd rather not debug this again next quarter
Moving from weekly to daily deploys without a corresponding rise in incidents is a solvable infrastructure problem, not a team discipline problem. The fix lives in decoupling schema changes from code, making rollback a routing decision instead of a redeploy, and replacing manual verification with automated canary analysis.
If you'd rather not debug this again next quarter, our managed platform handles it by default.
---
### How to stabilize a system while it is actively failing
URL: https://binadit.com/blog/stabilize-system-active-failure-infrastructure-management-services
Category: Infrastructure
Author: Binadit Tech Team
Published: 2026-09-16T09:31:45+02:00
What you will achieve and why it matters
This guide walks through the sequence for stabilizing a production system that is degrading right now, before you know the root cause. The goal is not to fix the underlying bug during the incident. The goal is to reduce user impact, buy diagnostic time, and avoid the two most common mistakes teams make under pressure: changing too many things at once, and diagnosing before stabilizing.
This applies whether you are running a single application server or a full high availability infrastructure setup with load balancers and read replicas. The steps below come from real incident response patterns we use across managed infrastructure for SaaS platforms and e-commerce stores.
Prerequisites and assumptions
Before you can stabilize anything, you need visibility and control. This guide assumes:
You have shell access to the affected hosts (SSH, or a bastion into your cloud provider's console)
You have basic monitoring in place: at minimum, CPU, memory, disk I/O, and application error rates. Prometheus, Datadog, or even htop and application logs count
You have a load balancer or reverse proxy in front of your application (Nginx, HAProxy, or a cloud load balancer)
You know how to roll back your last deployment
You have at least one other person who can act as a second pair of eyes, even if that is just a Slack thread
If none of this exists yet, the real prerequisite is building it before the next incident. That is a separate project, covered in the next steps section.
Step by step: stabilizing the system
Step 1: freeze changes
The first action is not technical. It is a decision: stop deploying, stop running migrations, stop anyone from touching config. Post in your team channel:
INCIDENT: elevated error rate on checkout-api since 14:32 UTC.
No deploys, no config changes, no manual DB edits until stabilized.
Incident channel: #inc-2024-checkout
This single step prevents the most common failure mode in live incidents: a well-meaning engineer pushes a "quick fix" that interacts badly with whatever is already going wrong, and now you have two problems layered on top of each other with no clean way to tell them apart.
Step 2: identify what changed recently
Most active failures trace back to a change in the last 24 hours: a deploy, a config edit, a traffic spike, or an upstream dependency update. Check in this order:
# Recent deploys
git log --since="24 hours ago" --oneline
# Recent config changes (if using version-controlled infra)
git -C /etc/nginx log --since="24 hours ago"
# System-level changes
last -x | head -20
cat /var/log/dpkg.log | grep "$(date +%Y-%m-%d)"
If a deploy or config change lines up with the onset of the problem, rolling it back is your fastest stabilization path. Do this before deep diagnosis, not after.
# Example: rolling back a bad deploy with a tagged release process
git checkout tags/v2.14.1
./deploy.sh production
# Or with a container-based setup
kubectl rollout undo deployment/checkout-api
Step 3: shed load before you shed correctness
If there is no clear recent change, or the rollback does not resolve it, the next lever is reducing load on the failing component. This is where you trade some functionality for stability, temporarily.
Common load-shedding techniques, roughly in order of how disruptive they are:
Rate limit at the edge. Add a rate limit in Nginx or your load balancer to cap requests per IP or per token
Disable non-critical endpoints. Turn off recommendation engines, analytics beacons, or background report generation
Enable a maintenance page for non-essential routes while keeping checkout or login alive
Scale horizontally if the bottleneck is compute, not a shared resource like a database
Example: adding an emergency rate limit in Nginx:
limit_req_zone $binary_remote_addr zone=emergency:10m rate=5r/s;
server {
location /api/ {
limit_req zone=emergency burst=10 nodelay;
limit_req_status 503;
proxy_pass http://backend;
}
}
Reload without dropping connections:
nginx -t && nginx -s reload
Step 4: protect the database first
In most stacks we manage, the database is the resource that fails last and recovers slowest. If connections are maxing out or queries are queueing, address this before anything else, because a saturated database will keep the rest of the system unstable no matter what you do upstream.
# Check active connections vs max on PostgreSQL
psql -c "SELECT count(*) FROM pg_stat_activity;"
psql -c "SHOW max_connections;"
# Find long-running queries
psql -c "SELECT pid, now() - query_start AS duration, query
FROM pg_stat_activity
WHERE state = 'active'
ORDER BY duration DESC LIMIT 10;"
If a handful of runaway queries are holding locks, killing them is often the fastest stabilization move available:
SELECT pg_terminate_backend(pid)
FROM pg_stat_activity
WHERE pid = 14832;
If connection exhaustion is the pattern, a connection pooler like PgBouncer absorbs this without touching application code. If you do not have one running, this incident is a strong signal to add it once things settle.
Step 5: fail over, do not fight in place
If a single node is the problem, whether from disk pressure, a memory leak, or a bad kernel state, it is usually faster to remove it from rotation than to debug it live. This is the core value of high availability infrastructure: you have somewhere to fail over to.
# Remove a node from the load balancer pool (HAProxy example)
echo "disable server backend/web03" | socat stdio /var/run/haproxy.sock
# Or drain it gracefully in Kubernetes
kubectl cordon node-3
kubectl drain node-3 --ignore-daemonsets --delete-emptydir-data
Once removed, that node becomes a forensic artifact, not a liability. You can debug it at your own pace without affecting live traffic.
Step 6: communicate a stable status, not a guess
Once error rates flatten and the change freeze is holding, post a status update with what you know and do not know:
STATUS 15:10 UTC: error rate back to baseline (0.2%) after rolling back
deploy v2.14.2 and draining node-3. Root cause under investigation.
Next update in 30 min or on change.
Resist the urge to declare the incident closed the moment metrics look normal. Stability under reduced load or a smaller traffic window does not confirm the fix.
Verification: how to confirm the system is actually stable
Do not rely on a single green dashboard. Confirm stability across at least three signals held over a meaningful window, typically 30 to 60 minutes at representative traffic:
Error rate: back to baseline, not just trending down. Check 5xx rate over 5-minute buckets, not instant values
Latency percentiles: p50, p95, and p99 all recovered, not just the average. A recovered average can hide a p99 that is still five times normal
Resource headroom: CPU, memory, and connection pools have margin, not just "not maxed"
Queue depth: if you use background jobs, confirm the backlog is draining, not just growing more slowly
Useful spot checks:
# Tail error rate from Nginx access logs
tail -n 5000 /var/log/nginx/access.log | awk '{print $9}' | sort | uniq -c
# Watch active connections in real time
watch -n 2 "ss -s"
# Confirm database connection headroom
psql -c "SELECT count(*), max_conn FROM pg_stat_activity, (SELECT setting::int AS max_conn FROM pg_settings WHERE name='max_connections') s GROUP BY max_conn;"
If you have historical dashboards, compare the current traffic-normalized error rate to the same time last week, not just to five minutes ago. A quiet period after load shedding can look deceptively healthy.
Common pitfalls to avoid
Fixing root cause mid-incident. Diagnosing while the system is unstable adds risk and rarely speeds up recovery. Stabilize first, root-cause after
Rolling back and rolling forward in the same window. Give the rollback time to prove itself before trying anything else
Declaring victory too early. A dashboard that looks calm for five minutes is not the same as a stable system under real traffic
Skipping the change freeze. Uncoordinated fixes during an incident are one of the most common causes of a second, unrelated outage layered on top of the first
Not documenting what you tried. If the same instability recurs, you want a timestamped record of every action taken, not a reconstructed memory two days later
Next steps and related reading
Once the immediate instability is resolved, the real work is making sure the pattern does not repeat. A few natural follow-ups:
Run a proper post-incident review while details are fresh. See post-incident reviews that actually improve things for a format that produces real action items instead of vague summaries
If load shedding was your main lever, it is worth reviewing whether your architecture has enough redundancy built in. Best practices for horizontal scaling in high availability infrastructure covers how to build that headroom in advance
If a deploy caused the instability, tightening your rollback process pays for itself the next time this happens. Our 6-phase zero-downtime migration playbook outlines how to structure changes so rollback is always a single command
If database exhaustion was the bottleneck, look at connection pooling and query-level monitoring before the next spike
Teams that handle this well tend to invest ahead of time in the boring parts: alerting thresholds tuned to catch drift early, a documented rollback procedure, and a database that has headroom before it is needed. That groundwork is most of what separates a five-minute blip from a multi-hour incident.
Getting this right consistently
Stabilizing an actively failing system is a skill that improves with repetition, but most teams only get the practice during real incidents, which is an expensive way to learn. This is also where dedicated infrastructure management services earn their keep: having engineers who have run this exact sequence dozens of times, on dozens of different stacks, means the freeze, the rollback, and the load-shedding decisions happen in minutes instead of being figured out live. A managed infrastructure partner brings that muscle memory as a standing capability rather than something your team assembles under pressure.
Need this running in production without building it yourself? See our managed infrastructure services or schedule a call.
---
### From 4.2s to 380ms: debugging latency in a high availability infrastructure setup
URL: https://binadit.com/blog/debugging-latency-high-availability-infrastructure-case-study
Category: Performance
Author: Binadit Tech Team
Published: 2026-09-11T09:05:28+02:00
The situation: a slow, gradual degradation with no obvious cause
The platform was a B2B SaaS product used for scheduling and resource planning, with around 40,000 active users across several hundred customer accounts. Traffic was steady during business hours in European time zones, with predictable spikes on Monday mornings.
The engineering team had not shipped anything unusual. No major feature releases, no big schema changes, no new integrations. But over roughly six months, p95 API response times had drifted from around 400ms to over 4.2 seconds. Support tickets mentioned the dashboard 'feeling laggy' long before anyone had hard numbers to point to.
This is a common pattern in distributed systems: nothing breaks outright, so nothing triggers an incident. Latency just creeps until it becomes a business problem. In this case, the business impact was concrete. Trial-to-paid conversion had dropped 11% over the same period, and the sales team was fielding questions about performance during renewal conversations.
The team had already tried the obvious fixes: bumping instance sizes, adding a read replica, restarting services on a schedule. None of it moved the needle for long. That is usually a sign the bottleneck is architectural, not capacity related, which is why they brought us in to audit the stack before committing to a bigger infrastructure overhaul.
What we found during the audit
We spent the first week purely on measurement, not fixing anything. Changing things before you understand the system just adds noise to your data. We instrumented request tracing across the API gateway, application servers, database, and cache layer, and let it run for five business days under normal load.
Five distinct issues emerged, and none of them alone explained the full 4.2 seconds. That is often how latency problems in high availability infrastructure actually work: it is rarely one dramatic bottleneck, it is several smaller ones stacking on top of each other.
N+1 queries in a hot path. A dashboard endpoint that loaded a customer's active projects was making one query for the project list, then a separate query per project to fetch its status. At 40 projects per account, that was 41 round trips to the database for a single page load.
Connection pool exhaustion under load. The application was running with a PostgreSQL connection pool of 20 per app server across 8 servers, but the database's max_connections was set to 100. During peak hours, requests were queueing for a connection slot before they ever reached a query.
Cache invalidation was too aggressive. A Redis cache in front of the project status data had a 30 second TTL, which sounds reasonable, but a background job was also flushing entire cache namespaces on any write, including unrelated writes. Cache hit ratio was measured at 34%, far lower than the team assumed.
Cross-AZ chatter between services. The application tier and the cache cluster were not pinned to the same availability zone. Roughly 40% of Redis calls were crossing zones, adding 2 to 4ms per call that, at high call volumes, added up to hundreds of milliseconds per request.
A synchronous third-party call in the request path. A usage-tracking webhook to an external analytics provider was being called synchronously during the request cycle. When that provider was slow, and it averaged 800ms to 1.5s on a bad day, every request downstream waited for it.
None of these were exotic. They were the kind of accumulated technical debt you find in a system that has grown organically for a few years without a dedicated performance review. That is worth naming clearly, because a lot of teams assume a latency problem this size must mean rearchitecting everything. It usually does not.
The approach we took and why
We prioritized fixes by expected impact versus risk, not by ease of implementation. It is tempting to fix the easiest thing first, but that is not always the thing costing you the most milliseconds.
We ranked the five issues like this:
Synchronous third-party call: highest impact, low risk to fix (move to async queue)
N+1 queries: high impact, low risk (query batching, well-understood pattern)
Connection pool exhaustion: high impact, medium risk (requires careful tuning to avoid overloading the database)
Cache invalidation strategy: medium impact, medium risk (requires understanding all write paths)
Cross-AZ placement: medium impact, low risk once the migration window was scheduled
We deliberately did not touch instance sizing again. The team had already scaled vertically twice without improvement, which was itself a useful data point: if adding CPU and memory does not help, the bottleneck is not compute, it is architecture or I/O. This is a distinction we cover in more depth in our piece on when high availability infrastructure becomes a bottleneck: past a certain point, more servers just mean more machines waiting on the same slow dependency.
We also made a call early on to fix things incrementally and measure after each change, rather than bundling everything into one release. Distributed systems have enough moving parts that a single combined deploy makes it nearly impossible to know which change actually helped, or whether one change quietly made something else worse.
Implementation details
Moving the analytics call off the request path
The synchronous webhook call was replaced with a message queue (Redis-backed, using the existing infrastructure rather than adding a new dependency). The request handler now pushes a lightweight event onto the queue and returns immediately. A separate worker process consumes the queue and handles delivery to the analytics provider, with retry logic and a dead-letter queue for failures.
// before: synchronous call blocking the response
await analyticsClient.track(event); // 800ms-1.5s on slow days
return response;
// after: fire-and-forget via queue
await queue.push('analytics.track', event); // ~2ms
return response;
This single change removed 800ms to 1.5 seconds from the median request in the affected endpoints. It also meant that if the analytics provider had an outage, which happened twice during our engagement, it no longer affected the user-facing application at all.
Fixing the N+1 pattern
The project status lookup was rewritten to use a single query with a join, rather than a loop of per-project queries.
-- before: 1 query for projects, then N queries for status
SELECT * FROM projects WHERE account_id = $1;
-- then, per project:
SELECT * FROM project_status WHERE project_id = $1;
-- after: one query
SELECT p.*, s.status, s.updated_at
FROM projects p
JOIN project_status s ON s.project_id = p.id
WHERE p.account_id = $1;
For an account with 40 active projects, this took the endpoint from 41 database round trips to 1. Query time for that endpoint dropped from an average of 620ms to 45ms.
Retuning the connection pool
We reduced per-server pool size from 20 to 12 (8 servers x 12 = 96, leaving headroom under the 100 connection limit) and added PgBouncer in transaction pooling mode in front of PostgreSQL. This let us actually raise the effective connection ceiling the application could use, since PgBouncer multiplexes many client connections onto a smaller number of real database connections.
[databases]
app_db = host=127.0.0.1 port=5432 dbname=app_production
[pgbouncer]
pool_mode = transaction
max_client_conn = 500
default_pool_size = 25
reserve_pool_size = 5
reserve_pool_timeout = 3
Connection wait time, which had been averaging 180ms during peak hours, dropped to under 5ms.
Rebuilding the cache invalidation logic
Rather than flushing entire namespaces on write, we moved to key-level invalidation tied to the specific project being updated. TTL stayed at 30 seconds as a safety net, but the primary invalidation mechanism became explicit and targeted.
Cache hit ratio moved from 34% to 91% within the first week after deployment. This is the kind of change that looks small in a code diff but has an outsized effect on database load, because every cache miss becomes a query.
Fixing AZ placement
We moved the Redis cluster and pinned application servers to the same availability zone, using zone-aware routing at the load balancer level so requests preferentially stayed local. This was the most operationally sensitive change, since it touched the load balancing and failover configuration, so we scheduled it during a low-traffic window and kept the previous configuration ready to roll back for 48 hours.
Results: before and after
MetricBeforeAfterp95 API response time4.2s380msp50 API response time1.1s95msCache hit ratio34%91%DB connection wait time (peak)180ms avg<5ms avgDatabase round trips (dashboard endpoint)41 per load1 per loadMonthly infrastructure costBaseline-18% (smaller instances, no over-provisioning)Uptime (90 day rolling)99.91%99.97%
The cost reduction is worth noting specifically. Once the actual bottlenecks were fixed, the team no longer needed the oversized instances they had added earlier in an attempt to outrun the latency problem. Right-sizing after the fix, rather than before it, meant the infrastructure spend went down while performance went up.
Trial-to-paid conversion recovered over the following quarter, though we are careful not to claim a direct causal number there. Latency was one factor among several, and conversion metrics have too many inputs to attribute cleanly to an infrastructure change. What we can say with confidence is the technical numbers above, measured with the same tracing setup before and after.
What we'd do differently next time
A few things we would change in hindsight.
We should have set up distributed tracing before the audit even started, as a standing part of the client's monitoring, rather than as a temporary measurement tool for the engagement. The team's existing monitoring showed server-level metrics (CPU, memory, request count) but nothing that connected a slow user-facing request to the specific downstream call causing it. That gap is what let the problem compound silently for six months. We have written before about how monitoring can give you a false sense of security when it only tracks infrastructure health and not request-level behavior, and this engagement was a clean example of exactly that gap.
We also would have pushed harder, earlier, to separate the AZ migration from the other changes. It carried more operational risk than the others, and while it went smoothly, bundling the planning for it alongside four other workstreams stretched the team thinner than it needed to be during that week.
Finally, we would have proposed load testing the fixes against production-like traffic patterns before the final rollout, rather than relying on staged rollout and close monitoring. It worked out, but synthetic load testing ahead of time would have caught the connection pool sizing question with more confidence and less babysitting during the rollout window.
Facing something similar
Latency problems like this rarely have one cause, and they rarely show up in a single dashboard metric. They show up as a slow accumulation of small inefficiencies that compound under real traffic, in ways that are hard to see until you trace individual requests end to end. If you want the methodology in more detail, we have written a broader walkthrough on how to trace performance bottlenecks end-to-end.
Facing a similar challenge? Tell us about your setup and we will outline an approach.
---
### How Docker networking broke checkout under load: a container ecommerce infrastructure case study
URL: https://binadit.com/blog/docker-networking-production-ecommerce-infrastructure-case-study
Category: Performance
Author: Binadit Tech Team
Published: 2026-09-09T09:02:38+02:00
The situation: a growing marketplace runs into container networking limits
The platform was a multi-vendor marketplace built on a PHP monolith, containerized about a year before we got involved. Roughly 40,000 daily active users, peak traffic around 900 requests per second during evening hours, and a catalog large enough that search and checkout both hit the database hard.
The engineering team had done the container migration themselves: application, Redis, and a background worker queue all running as Docker services on a single host, orchestrated with Docker Compose, sitting behind a managed load balancer. It worked. Deploys got faster. Local development matched production more closely. Nobody was unhappy with the decision to containerize.
The problem showed up six months later, and only under load. Checkout latency was fine most of the day and then, during traffic spikes, p95 response times on the checkout endpoint would climb from around 280ms to over 2.1 seconds. Support tickets mentioned "the site freezing" during flash sales. The team had already added more application containers, assuming it was a CPU or memory ceiling. It wasn't. Adding containers made things marginally worse.
This is a common pattern: teams scale the thing they can see (container count, CPU graphs) while the actual bottleneck sits in the network layer connecting those containers, where default tooling gives almost no visibility.
What we found during the audit
We started with the assumption that this was a database problem, because checkout latency almost always looks like a database problem. It wasn't, or at least not primarily. Query times on Postgres were stable, averaging 8-14ms even during the spike windows.
The actual issue was in how Docker's default bridge network was handling inter-container traffic at scale. Three specific findings:
Every packet between containers was traversing userland proxying. The Compose setup used the default bridge driver without any tuning. Under low traffic this is invisible. Under 900 req/s with each request making 3-4 internal calls (app to Redis, app to Postgres, app to a search service), the overhead compounded. We measured an average of 1.8ms added latency per hop through the default bridge, which sounds small until you multiply it by every internal call on every request.
Connection tracking (conntrack) was hitting its table limit. The host's nf_conntrack_max was left at the kernel default of 65,536 entries. With containers opening and closing short-lived connections to Redis and Postgres rapidly, the conntrack table filled up during spikes. Once full, new connections got dropped or delayed, and the kernel started logging "table full, dropping packet" without anyone noticing because syslog wasn't being shipped anywhere useful.
DNS resolution inside the Docker network was adding unpredictable latency. Docker's embedded DNS server (127.0.0.11) was resolving service names for every new connection rather than the app caching resolved addresses. Under normal load this is a rounding error. Under spike load, with connection churn, DNS lookups were queuing behind each other. We've written about this resolution path in more depth in how DNS resolution works under the hood, and the same fundamentals apply inside a container network, not just at the edge.
None of these three things alone would have caused a 2-second checkout. Together, under concurrent load, they created exactly the kind of compounding latency that's invisible in a staging environment with ten test users and very visible with nine hundred real ones.
The approach we took and why
We ruled out two tempting options early.
The first was "just move to Kubernetes." It would not have fixed anything here. The problems were in networking fundamentals (bridge driver behavior, conntrack limits, DNS caching), not in orchestration. Kubernetes has its own version of these same issues (CNI plugin choice, kube-proxy mode, CoreDNS caching) and swapping orchestrators without understanding the root cause would have just relocated the problem.
The second was rewriting the application to reduce internal network calls. Also unnecessary. The call pattern (app to Redis, app to Postgres, app to search) is normal and not excessive. The network layer needed to handle that pattern efficiently, not the other way around.
Our approach was to fix the network path itself, in three layers, matched to the audit findings: kernel-level tuning for conntrack, a driver change for inter-container traffic, and application-level DNS caching. This kept the change surface small and testable, and it meant zero application code changes, which mattered because the team's next planned release was already in QA and we didn't want to block it.
We also treated this as a chance to move the checkout path toward genuine high availability without a rewrite, since the same single-host container setup was also a single point of failure. That became phase two of the work.
Implementation details
1. Conntrack and kernel network tuning
We raised the connection tracking table and tuned the timeout for established connections, since most of the churn was short-lived Redis and Postgres connections that didn't need to linger in the table:
net.netfilter.nf_conntrack_max = 262144
net.netfilter.nf_conntrack_tcp_timeout_established = 600
net.ipv4.tcp_tw_reuse = 1
net.core.somaxconn = 4096
These went into /etc/sysctl.d/99-docker-network.conf and were applied with sysctl --system. We also added conntrack table utilization as a monitored metric going forward, since "table full" is exactly the kind of failure that produces no obvious symptom in application logs.
2. Replacing the default bridge with a tuned macvlan setup for internal services
For inter-container east-west traffic (app to Redis, app to Postgres proxy), we moved from Docker's default bridge to a dedicated user-defined bridge network with jumbo frame support enabled and ICC (inter-container communication) explicitly configured, rather than relying on default NAT-based routing for every hop:
docker network create \
--driver bridge \
--opt com.docker.network.bridge.enable_icc=true \
--opt com.docker.network.driver.mtu=9000 \
--subnet 172.28.0.0/16 \
internal-services
Services that only talked to each other internally (app containers, Redis, the search sidecar) were attached to this network instead of the default bridge, keeping their traffic off the NAT path Docker uses for the default network. The load balancer facing traffic still terminated on a separate public-facing network, so we weren't changing the external attack surface, just the internal routing.
3. Static internal DNS with local caching
Rather than resolving service names through Docker's embedded DNS on every new connection, we added a lightweight local DNS cache (dnsmasq running as a sidecar) in front of the Docker embedded resolver, with short but non-zero TTLs:
# dnsmasq.conf
no-resolv
server=127.0.0.11
cache-size=1000
local-ttl=10
neg-ttl=5
A 10-second local TTL was enough to absorb the connection churn during spikes without causing stale resolution problems when containers were replaced during deploys. We tested failover behavior specifically to confirm that a container restart was still picked up within one TTL window.
4. Application-side connection pooling
The last piece was at the application layer, but it was a configuration change, not a rewrite: enabling persistent connection pooling to Postgres via PgBouncer, which reduced the connection churn that was stressing conntrack in the first place.
[databases]
marketplace = host=postgres-primary port=5432 dbname=marketplace
[pgbouncer]
pool_mode = transaction
max_client_conn = 2000
default_pool_size = 50
This mattered more than it might look. Every new short-lived connection is a new conntrack entry and a new DNS resolution. Pooling connections at the application boundary reduced both problems simultaneously, which is why we sequenced it after the network fixes rather than before: it made the underlying fixes more effective, but it wasn't a substitute for them.
Results with real numbers
We rolled this out over two maintenance windows, three weeks apart, to isolate the impact of each layer.
MetricBeforeAfterCheckout p95 latency (peak load)2,140ms310msCheckout p95 latency (normal load)280ms195msConntrack table utilization at peak98-100%34%Failed/dropped internal connections per hour (peak)~1,900~12DNS resolution time (internal, p95)41ms2msTTFB, checkout page (peak)1,780ms240msSupport tickets mentioning slow checkout (monthly)~65~4
The most telling number is the gap between peak and normal load before the fix: 280ms normal, 2,140ms peak, almost an 8x difference under the same application code and same database. After the fix, that gap closed to roughly 1.6x, which is what you'd expect from legitimate resource contention rather than a network layer choking on connection churn.
The conntrack utilization number explains why the team's earlier fix (adding more app containers) made things worse. More containers meant more concurrent connections to Redis and Postgres, which meant the conntrack table filled up faster, which meant more dropped packets. They were scaling the exact resource that was already the bottleneck.
Infrastructure cost didn't change meaningfully. This wasn't a "add servers" fix; it was a "use the network correctly" fix. The only new cost was the monitoring for conntrack utilization and DNS cache hit rate, which added about €40/month in observability overhead across the fleet.
What we'd do differently next time
A few things we'd change if we ran this again from day one.
We would check conntrack limits before looking at anything else. It's a five-minute check (sysctl net.netfilter.nf_conntrack_count versus nf_conntrack_max) and it should be step one for any "intermittent slowness under load" ticket involving containers, ahead of database profiling. We spent longer than we'd like ruling out the database first, mostly because that's where the client's own monitoring pointed.
We would push harder, earlier, on separating the single-host container topology from the network fix. Fixing conntrack and DNS caching solved the immediate latency problem, but the underlying architecture still had a single Docker host as a single point of failure for the entire checkout path. We did eventually move this to a proper multi-host setup with the app tier load balanced across two hosts, but that happened as a phase two project rather than being bundled into the initial fix. In hindsight, given how closely the two problems were related, we'd scope both from the start rather than treating high availability as a separate conversation.
We would also set up conntrack and DNS resolution metrics as day-one monitoring for any container-based ecommerce infrastructure, rather than something added after an incident. These are cheap to monitor and expensive to debug blind. The default Docker and Compose setup gives you almost no visibility into either, which is exactly why this kind of problem tends to surface in production during a sale event rather than in a load test three weeks earlier.
Closing thoughts
Docker networking works well by default for development and for low-to-moderate production traffic. It stops working well by default once you have real concurrency: enough simultaneous connections to fill a conntrack table, enough churn to make DNS resolution matter, enough internal hops to make bridge overhead visible. None of that shows up in a staging environment with a handful of test requests, which is exactly why it tends to surface for the first time during a traffic spike that actually matters to the business.
The fix is rarely "add more containers" or "move to Kubernetes." It's usually tuning the kernel-level and network-level defaults that nobody set intentionally in the first place, and pairing that with connection pooling so the application isn't generating more network churn than it needs to.
Facing a similar challenge? Tell us about your setup and we will outline an approach.
---
### Measuring the boardroom case for sovereign infrastructure management services (with template)
URL: https://binadit.com/blog/measuring-boardroom-case-sovereign-infrastructure-management-services-template
Category: Infrastructure
Author: Binadit Tech Team
Published: 2026-09-04T09:11:00+02:00
> We modeled the cost, latency, and compliance tradeoffs of moving from a US hyperscaler to EU-based infrastructure management services across three real client profiles. Here are the actual numbers, the methodology behind them, and a template to build your own board case.
The question and why it matters commercially
Every board conversation about sovereign infrastructure eventually collapses into the same unanswered question: does this cost us money or save us money, and what happens to performance? Engineering teams usually know the compliance argument. Finance wants a number. Legal wants a risk classification. Nobody in the room wants to guess.
We built this analysis because we kept getting asked the same thing by CTOs preparing to pitch infrastructure changes internally: 'what do I actually put in the slide?' So we measured it. We took three anonymized client migrations from US hyperscaler setups to EU-based infrastructure management services, normalized the workloads, and pulled the real numbers: latency, cost per compute unit, migration downtime, and compliance overhead.
This is not a sovereignty scare piece. Data residency and jurisdiction exposure are real considerations, and we cover them elsewhere in detail (see our breakdown of CLOUD Act impact on managed cloud infrastructure). This article is about what the spreadsheet looks like when you're done arguing principle and start arguing numbers.
Methodology: setup, hardware, software versions, load profile
We selected three client environments that migrated from US hyperscaler regions (two on AWS eu-west-1, one on Azure West Europe) to EU-owned infrastructure management services with EU-domiciled entities operating the hardware. All three retained functionally equivalent architecture: same application code, same database engine, same caching layer, only the underlying infrastructure and operator changed.
Client A: B2B SaaS platform, 40,000 monthly active users, PostgreSQL 15 primary/replica, Redis 7.2 cache layer, Kubernetes 1.28 (EKS to self-managed K8s on bare metal)
Client B: WooCommerce marketplace, 18,000 daily sessions average, MySQL 8.0, Nginx 1.25, PHP 8.2-FPM
Client C: Fintech API platform, 2.4M requests/day, PostgreSQL 15, HAProxy 2.8, Docker Swarm to Kubernetes migration during the same window
Load testing used k6 v0.49 against staging environments configured to mirror production specs, run in three phases: baseline (pre-migration), cutover week, and 60-day post-migration steady state. We ran 30-minute sustained load tests at production-representative RPS for each client, repeated five times per phase to control for network variance, and report medians across runs.
Cost data comes from actual invoices: hyperscaler billing exports for the 6 months pre-migration, and Binadit infrastructure management services invoices for the 6 months post-migration, normalized to per-vCPU-hour and per-GB-transfer equivalents where billing models differed (reserved instances vs dedicated hardware allocation).
Compliance overhead was measured as legal and engineering hours spent on data processing agreements, sub-processor audits, and transfer impact assessments, tracked via time logs both client-side and on our side during the 90 days surrounding migration.
Results
Latency (p50/p95/p99, milliseconds)
ClientMetricPre-migration (hyperscaler)Post-migration (EU infra mgmt)A (SaaS)p5084ms71msp95310ms245msp99890ms612msB (WooCommerce)p50120ms98msp95480ms340msp991,150ms780msC (Fintech API)p5042ms39msp95190ms155msp99520ms410ms
All three clients saw p95/p99 improvements, primarily because the new infrastructure was provisioned on dedicated hardware sized specifically for their workload, rather than shared multi-tenant instance classes. This is a provisioning effect, not a jurisdiction effect. Client C's improvement was smallest because its original hyperscaler setup was already well-tuned with reserved capacity.
Cost per vCPU-hour equivalent
ClientPre-migration ($/vCPU-hr)Post-migration (EUR/vCPU-hr)ChangeA$0.048EUR 0.031-35%B$0.041EUR 0.029-29%C$0.052EUR 0.036-31%
Cost reductions cluster tightly between 29-35%. This matches broader trends we've documented in our ROI analysis of moving off US hyperscalers: the savings mostly come from eliminating egress fees, dropping reserved-instance markup, and moving from metered billing to flat dedicated-hardware pricing.
Migration downtime and compliance overhead
ClientCutover downtimeLegal/eng hours on compliance (90-day window)A0 minutes (blue-green cutover)62 hoursB4 minutes (DNS TTL-bound)38 hoursC0 minutes (rolling cutover)91 hours
Two of three clients achieved zero-downtime cutover using blue-green or rolling deployment patterns. Client B accepted a brief DNS-propagation window rather than invest in a fully parallel run, a tradeoff decision made jointly with their team based on traffic patterns (low-traffic Tuesday 3am cutover).
Analysis: what the numbers mean in production
The latency story is not really about sovereignty. It's about provisioning discipline. When you migrate to dedicated infrastructure management services, someone re-architects your resource allocation from scratch. Noisy-neighbor effects on shared hyperscaler instances disappear. That's where the p95/p99 gains come from, not from data crossing fewer borders.
This matters for how you present the case internally. If you tell your CFO 'moving to EU infrastructure makes us faster,' a sharp finance lead will ask why jurisdiction would affect milliseconds. It doesn't, directly. What affects milliseconds is dedicated capacity, tuned network paths, and an operator who actually looks at your metrics instead of routing you through a generic support ticket queue. Sovereignty and performance improvements happened together in our data, but they are two separate variables that a good procurement conversation should separate.
The cost numbers are more directly attributable. Egress fees on hyperscalers are punitive by design, they exist to keep you locked in. Client A alone was paying $4,200/month in cross-region egress before migration, a cost category that simply doesn't exist in a flat-rate dedicated hosting model. When you build your board deck, put egress and reserved-instance markup in their own line items. That's the part of the savings story finance will actually believe, because it's auditable from existing invoices without needing to trust a vendor's marketing math.
Compliance overhead is the number most boards underweight. Client C spent 91 hours of legal and engineering time in a 90-day window managing sub-processor documentation, standard contractual clauses, and internal audit prep, largely because their original setup involved four separate US-based sub-processors across compute, CDN, logging, and email. Post-migration, three of those four consolidated under a single EU-based infrastructure management services contract with one data processing agreement instead of four. That consolidation is where the real operational relief shows up, not in a compliance checkbox but in fewer vendor relationships to audit every renewal cycle.
Building your own board template
Structure the case around four rows, using your own invoices and monitoring data rather than vendor estimates:
Row 1, direct cost: current $/vCPU-hr and $/GB-transfer from your last two hyperscaler invoices, compared against quoted flat-rate dedicated pricing
Row 2, performance: p95/p99 from your existing APM tool, with a caveat that gains are provisioning-driven, not jurisdiction-driven
Row 3, compliance hours: legal and engineering time spent in the last audit cycle on sub-processor documentation, divided by number of sub-processors, to show the marginal cost of consolidation
Row 4, migration risk: your planned cutover strategy (blue-green, rolling, or DNS-bound) with expected downtime in minutes, not a vague 'minimal disruption' claim
If you're building this template for a regulated business, it's worth pairing it with a walkthrough of how EU region toggles differ from actual data sovereignty, since boards frequently confuse the two and a sharp legal reviewer will ask the difference.
Caveats and what we'd do differently
These are three data points, not a statistically significant sample. Workload shape matters enormously: a read-heavy SaaS app with aggressive caching will show different latency deltas than a write-heavy fintech ledger. Don't extrapolate our exact percentages to your stack without running your own load tests first.
We also didn't control for the fact that migration itself often forces overdue cleanup. All three clients did some amount of query optimization, index rebuilding, or cache-layer tuning during the migration project. Some of the p95/p99 improvement is migration-adjacent housekeeping, not purely infrastructure change. A cleaner experiment would freeze application-layer changes entirely during the measurement window, which we didn't fully enforce for Client A.
The cost comparison uses a snapshot of hyperscaler list pricing at time of migration. Hyperscaler pricing changes, and if you negotiate enterprise discounts or committed-use contracts, your baseline percentage will shift. Run the comparison against your actual negotiated rate, not public list price.
Finally, compliance hour tracking is inherently fuzzy. Legal teams don't always log time cleanly against a single project, and some of Client C's 91 hours likely overlapped with unrelated SOC 2 renewal work happening in the same quarter. Treat that number as directionally useful, not a precise unit cost.
If we ran this again, we'd add a fourth client with a genuinely write-heavy, low-cache workload to stress-test whether the p95/p99 gains hold up outside read-optimized architectures, and we'd instrument compliance hours with a dedicated time-tracking category from day one instead of reconstructing it after the fact.
Takeaways
The numbers that actually move a board: 29-35% direct cost reduction from eliminating egress and reserved-instance markup, zero-downtime cutover achievable with proper blue-green or rolling deployment planning, and meaningful reduction in compliance overhead when consolidating sub-processors under a single infrastructure management services contract. The latency improvements are real but attribute them correctly, to dedicated provisioning, not jurisdiction, or you'll lose credibility with a technical board member.
Build your board case on your own invoices and your own APM data. Vendor benchmarks, including this one, are directional. Your CFO will trust a number pulled from your own AWS billing export far more than any case study, ours included.
Want these kinds of numbers for your own stack? Request a performance audit.
---
### Solving the real ROI question behind moving off US hyperscalers
URL: https://binadit.com/blog/cloud-cost-optimization-services-roi-moving-off-us-hyperscalers
Category: Infrastructure
Author: Binadit Tech Team
Published: 2026-09-02T09:22:49+02:00
The symptom: a migration business case that never quite closes
An engineering lead runs the numbers on moving off a US hyperscaler. Compute looks cheaper elsewhere. Storage looks cheaper elsewhere. But when the spreadsheet goes to finance, someone asks about egress fees, migration engineering hours, and the risk of downtime during cutover, and the business case stalls. Six months later the same conversation happens again, with the same result.
This is not a failure of the migration plan. It is a failure of the cost model. Most ROI calculations for leaving a hyperscaler only compare list prices for compute and storage. That comparison is incomplete, and it consistently underestimates the actual return.
The actual root cause: hyperscaler pricing is designed to hide the real unit cost
US hyperscalers do not compete primarily on compute price. AWS, Azure, and Google Cloud compete on making it expensive to leave. The mechanism is architectural, not just contractual.
Egress fees are the anchor, not the compute bill
Data transfer out of AWS typically runs $0.09 per GB after the first GB, with regional variation. For a SaaS platform serving 50TB of monthly egress (video, API responses, backups, CDN origin pulls), that is roughly $4,500 per month just to move data out, before you have optimized anything else. This fee does not exist to cover bandwidth cost; bandwidth is cheap. It exists to make multi-cloud and cloud-exit financially uncomfortable.
Managed service lock-in compounds the exit cost
Teams that build on RDS, DynamoDB, SQS, and Lambda are not just using compute, they are using proprietary APIs. Migrating off means re-architecting data access layers, not just moving VMs. This is real engineering time, and it is the line item most ROI spreadsheets underestimate by 3-5x.
Reserved instance and savings plan commitments distort the comparison
Finance teams often compare an EU provider's list price against a hyperscaler's 3-year reserved instance rate, and conclude the hyperscaler is cheaper. But that reserved rate assumes flat, predictable usage for 36 months. Most SaaS and e-commerce workloads do not have flat usage. They have seasonal spikes, growth curves, and product pivots that make long commitments a liability, not a discount.
Support cost is invisible until you need it
Hyperscaler support tiers that include a named technical account manager start around $15,000/month minimum, and still route you through a ticket queue for anything beyond account management. Compare that to a managed infrastructure partner where a senior engineer answers directly, and the "free" basic support tier on a hyperscaler starts looking a lot less free once you factor in engineering hours lost to self-service troubleshooting.
The fix: a complete cost model, with the line items that actually matter
A real ROI calculation for cloud cost optimization services needs five cost categories, not one. Here is the framework we use with clients.
1. Baseline the current total cost of ownership
Pull 12 months of billing data, not one month. Seasonal spikes and reserved instance renewals distort single-month snapshots.
# AWS Cost Explorer CLI example: pull 12 months grouped by servicenaws ce get-cost-and-usage \n --time-period Start=2024-11-01,End=2025-11-01 \n --granularity MONTHLY \n --metrics 'UnblendedCost' \n --group-by Type=DIMENSION,Key=SERVICE
Break this into: compute, storage, egress, managed services (RDS, ElastiCache, managed Kubernetes), support tier cost, and reserved instance amortization. Egress alone is usually 8-15% of total spend for content-heavy or API-heavy platforms, and it is almost always excluded from the first draft of a migration business case.
2. Model the true migration cost, not just the target infrastructure cost
This includes:
Engineering hours for re-architecting managed-service dependencies (DynamoDB to PostgreSQL, Lambda to containerized workers, SQS to a self-hosted queue)
Data transfer cost to move the dataset out, one time
Parallel-run cost: running both environments during validation, typically 4-8 weeks
Downtime risk cost, budgeted as a contingency line, even if the migration is designed for zero downtime
For a mid-sized SaaS platform with a 2TB database and 15 microservices, this typically lands between 200 and 450 engineering hours. At a loaded cost of 80 euros/hour, that is 16,000 to 36,000 euros in one-time migration cost. This number needs to be in the model, not treated as a rounding error.
3. Recalculate egress and data transfer under the new architecture
If the target environment is EU-based and colocated with your CDN's EU PoPs, egress costs typically drop 40-70% because you are no longer paying hyperscaler egress rates and you have shortened the network path for the bulk of your traffic.
# Example: comparing egress cost modelsn# AWS us-east-1 to internet: $0.09/GB after first 1GB freen# EU-based provider with peering to major EU IXPs: $0.01-0.02/GB typicalnn# 50TB/month egress comparison:n# AWS: 50,000GB * $0.09 = $4,500/monthn# EU provider: 50,000GB * $0.015 = $750/monthn# Annual difference: $45,000
4. Rebuild the architecture around open standards, not proprietary APIs
The single highest-leverage decision in a hyperscaler exit is replacing managed proprietary services with open-source equivalents that run anywhere. This is what actually enables cloud cost optimization services to deliver a durable return, rather than a one-time discount that erodes as you re-adopt vendor-specific tooling.
DynamoDB to PostgreSQL or self-managed MongoDB, both portable
Lambda to containerized workers on Kubernetes or Nomad
SQS/SNS to self-hosted RabbitMQ or Kafka
CloudWatch to Prometheus and Grafana
This step is also where a decision between an open-source sovereign stack and managed public cloud gets made concretely, service by service, rather than as an abstract policy choice.
5. Execute the cutover as a zero downtime migration
The migration itself should follow a staged pattern: replicate data continuously, run both environments in parallel, cut over DNS with a low TTL, and keep the old environment warm for rollback for at least one billing cycle.
# Example: lowering DNS TTL ahead of cutovern# Set 48-72 hours before the cutover windownexample.com. 300 IN A 203.0.113.10nn# During cutover, monitor both originsncurl -s -o /dev/null -w '%{http_code} %{time_total}s\n' https://old-origin.example.com/healthncurl -s -o /dev/null -w '%{http_code} %{time_total}s\n' https://new-origin.example.com/health
For database-backed applications, this typically means logical replication running for days before cutover, not a single export/import window. We cover the mechanics of this in detail in our 6-phase zero downtime migration playbook.
How to validate the fix worked
ROI claims need to be measured against baseline, not assumed. Track these for at least two full billing cycles post-migration:
Total infrastructure spend per month, compared against the 12-month hyperscaler baseline, normalized for traffic growth
Egress cost as a percentage of total spend, expect this to drop from 8-15% to 2-4% if the new provider has favorable peering
p95 and p99 latency for the top 10 API endpoints, measured before and after cutover, to confirm the new environment did not trade cost for performance
Error rate and uptime during the 30 days following cutover, compared to the 30 days preceding it
Engineering hours spent on infrastructure operations per month, this often drops significantly when proprietary managed services are replaced with standard tooling that the team already understands
A useful sanity check: if total spend dropped 30% but p99 latency also degraded 25%, that is not a win, it is a cost shift into a metric that eventually costs you conversions. Reference our guide on choosing the right performance metric if you are validating this on a customer-facing platform.
Build a simple monthly reconciliation dashboard
MetricPre-migration baselineMonth 1 post-cutoverMonth 3 post-cutoverTotal infra spend€18,400€13,900€12,100Egress cost€2,760€890€610p99 latency (ms)340310280Uptime99.95%99.96%99.98%
This kind of table, filled in with your actual numbers, is what converts an ROI argument from a projection into evidence, useful both internally and for board-level reporting.
How to prevent recurrence
The reason this ROI conversation repeats every quarter at many companies is that nobody owns the cost model after the migration decision is made. A few structural changes fix that.
Assign cost ownership to engineering, not just finance
Cost regressions creep back in through architectural decisions: a new feature that reintroduces a proprietary managed service, a data pipeline that reintroduces cross-region egress. Finance cannot catch these at the point they are introduced. Engineering can, if cost is part of the architecture review checklist.
Set egress and compute budgets per service, with alerting
# Example Prometheus alert rule for cost anomaly detectionn# (paired with a cost-exporter feeding billing data as metrics)n- alert: EgressCostSpiken expr: increase(egress_cost_eur_total[24h]) > 150n for: 1hn labels:n severity: warningn annotations:n summary: 'Daily egress cost increased by more than 150 EUR'
Re-run the full TCO model annually, not just at migration time
Hyperscaler pricing changes, your traffic patterns change, and new services get adopted. A cost model that was accurate at migration time will drift within 12-18 months if nobody revisits it.
Avoid re-introducing proprietary lock-in through convenience adoption
The most common way teams undo their own cost optimization is adopting a new proprietary managed service because it is fast to integrate, then discovering 18 months later that it is now load-bearing and expensive to remove. Every new managed service should be evaluated against one question: can we replace this without a re-architecture if pricing changes?
The bottom line, and what to do next
The real ROI of moving off a US hyperscaler is rarely visible in a one-line compute price comparison. It shows up in egress costs that quietly consume 8-15% of spend, in the engineering hours saved by avoiding proprietary re-architecture cycles, and in support costs that stop scaling with your headcount. A complete model, built on 12 months of actual billing data and validated post-migration with real metrics, turns this from a recurring debate into a decision you make once.
If you'd rather not debug this again next quarter, our managed platform handles it by default.
---
### How to move ecommerce infrastructure from a single VPS to HA without rewriting the application
URL: https://binadit.com/blog/single-vps-to-ha-ecommerce-infrastructure-without-rewrite
Category: Commerce
Author: Binadit Tech Team
Published: 2026-08-26T09:22:43+02:00
What you will achieve and why it matters
Most stores start on a single VPS: web server, database, and file storage all on one box. It works, until that box reboots for a kernel update, runs out of memory during a campaign, or the disk fills up at 2am. This guide walks through moving that same application to a highly available architecture, load balancer, multiple app nodes, a managed or replicated database, shared storage, without rewriting the codebase.
This matters because the business risk of a single VPS is not exotic, it is a maintenance window, a traffic spike, or a hardware fault. The fix is architectural, not a code migration. If your application runs on PHP, Node, or Python with a relational database, the pattern below applies whether you run WooCommerce, Magento, or a custom stack.
Prerequisites and assumptions
Before starting, confirm the following:
Your application is stateless enough to run on more than one server, or can be made so (sessions, uploaded files, and cache are the usual blockers)
You have root access to provision new servers, not just an FTP account
You can schedule at least one maintenance window for DNS and database cutover, even if the goal is near-zero downtime
You have a recent, tested backup before touching anything
You know your current traffic pattern: peak requests per second, database connections, and average payload size
This guide assumes a typical LAMP or LEMP-style stack (Nginx or Apache, PHP-FPM or Node, MySQL or PostgreSQL, Redis for cache/sessions). The same principles apply to other stacks with different tooling.
Step-by-step implementation
Step 1: Separate state from compute
The single biggest blocker to horizontal scaling is state stored on local disk. Before adding servers, move these off the application server:
Sessions: move from file-based sessions to Redis
Uploaded media: move to object storage (S3-compatible) or a shared NFS volume
Cache: move from local disk/OPcache-only setups to Redis or Memcached
Example PHP session config change:
; php.ininsession.save_handler = redisnsession.save_path = "tcp://10.0.0.5:6379"
For uploaded files in a PHP application, most frameworks support a storage abstraction. For WooCommerce specifically, plugins like WP Offload Media handle this without custom code. For custom applications, replace direct filesystem writes with an S3-compatible SDK call:
$s3->putObject([n 'Bucket' => 'store-uploads',n 'Key' => $filename,n 'Body' => fopen($tmpPath, 'r'),n]);
Step 2: Introduce a load balancer
Provision a small VM or use a managed load balancer in front of your current VPS. At this stage you are running one backend, but the load balancer becomes the fixed point that DNS points to going forward, so future scaling does not require another DNS change.
Example Nginx upstream config as a starting load balancer:
upstream app_servers {n server 10.0.0.10:80 max_fails=3 fail_timeout=30s;n server 10.0.0.11:80 max_fails=3 fail_timeout=30s backup;n}nnserver {n listen 443 ssl;n server_name shop.example.com;n location / {n proxy_pass http://app_servers;n proxy_set_header Host $host;n proxy_set_header X-Real-IP $remote_addr;n }n}
Point DNS at the load balancer's IP now, even before the second app server exists. This decouples DNS from server count for every future change.
Step 3: Clone the application server
With sessions and media externalized, the app server itself should now be close to stateless. Build a second node from the same base image or provisioning script (Ansible, a Docker image, or a simple shell script that installs the same package versions).
Keep configuration in version control from this point forward, this is also the moment to start treating servers as replaceable rather than hand-tuned.
# minimal provisioning checknphp -vnnginx -vncomposer --versionncat /etc/php/8.2/fpm/pool.d/www.conf | grep pm.max_children
Add the second server to the load balancer's upstream block, remove the backup flag, and confirm both nodes are serving traffic under normal load before moving to the database.
Step 4: Replicate the database
This is the step with the most risk, and where zero-downtime database migration techniques matter most. The goal is a primary-replica setup, or a managed database cluster, instead of a single MySQL/PostgreSQL instance on the original VPS.
For MySQL, set up binary log replication:
# On primary my.cnfnserver-id = 1nlog_bin = /var/log/mysql/mysql-bin.lognbinlog_do_db = shop_productionnn# On replica my.cnfnserver-id = 2nrelay-log = /var/log/mysql/mysql-relay-bin.log
CHANGE MASTER TOn MASTER_HOST='10.0.0.5',n MASTER_USER='replicator',n MASTER_PASSWORD='***',n MASTER_LOG_FILE='mysql-bin.000003',n MASTER_LOG_POS=154;nSTART SLAVE;
Verify replication lag before cutover:
SHOW SLAVE STATUS\Gn-- check: Seconds_Behind_Master = 0
Once the replica is caught up and stable, promote it or switch the application's write connection using a virtual IP or a proxy like ProxySQL, so failover does not require editing config files under pressure.
Step 5: Add health checks and failover
A load balancer with two backends is not yet highly available if it does not detect failure. Add active health checks:
upstream app_servers {n server 10.0.0.10:80 max_fails=2 fail_timeout=10s;n server 10.0.0.11:80 max_fails=2 fail_timeout=10s;n}nn# with nginx plus or haproxy, use active checksnoption httpchk GET /healthnhttp-check expect status 200
Build a real /health endpoint that checks database connectivity and Redis, not just that Nginx is running:
connect('10.0.0.5', 6379);n http_response_code(200);n echo 'ok';n} catch (Exception $e) {n http_response_code(503);n echo 'unhealthy';n}
Verification: how to confirm it works
Do not assume HA works because the config is deployed. Test it deliberately:
Kill one app node during low traffic and confirm the load balancer routes around it within your configured fail_timeout window (check response times stay flat, not just that requests succeed)
Check replication lag under real load: SHOW SLAVE STATUS should show Seconds_Behind_Master near zero even during peak checkout traffic
Run a load test against both nodes and confirm even distribution: ab -n 5000 -c 50 https://shop.example.com/ and compare access logs on each server
Simulate a database failover in a staging environment first, and measure how long the application takes to reconnect, this number should be seconds, not minutes
Monitor TTFB separately from full page load post-migration, since load balancer hops can add latency if health checks are misconfigured, see our guide on choosing the right performance metric for ecommerce infrastructure
A useful baseline: after this migration, a well-configured setup should sustain the loss of one app node with zero customer-visible errors, and a database failover should complete in under 30 seconds with connection retry logic in place.
Common pitfalls to avoid
Sticky sessions as a shortcut: using session affinity instead of externalizing sessions works until one node goes down and half your logged-in users are logged out
Forgetting cron jobs: scheduled tasks (order processing, cache warming) running on both nodes will duplicate work; move them to a single designated node or a job queue
Skipping the health check depth: a health check that only pings port 80 will report "healthy" even when the database connection is dead
Assuming replication equals backup: replication protects against hardware failure, not against a bad DELETE statement replicating instantly to the replica
Not testing failover before you need it: the first time your database failover logic runs should not be during an actual outage
Next steps and related reading
Once the core HA pattern is in place, the next layers worth investing in are DNS resilience and caching strategy. Our guide on how DNS resolution works under the hood covers how TTLs and resolver behavior affect failover speed, which matters once you have multiple regions or providers in play.
If your store is WooCommerce specifically, pair this HA setup with the practices in how to scale WooCommerce infrastructure without downtime, since plugin behavior and object caching add constraints that generic HA guides do not cover.
Longer term, treat this as the foundation for infrastructure management services thinking: version-controlled provisioning, tested failover, and monitoring that reflects real user experience rather than server uptime alone.
Ready when you are
Moving from a single VPS to high availability infrastructure does not require a rewrite, but it does require getting the sequence right: externalize state, add the load balancer, replicate the database, and test failover before you need it. Done in that order, the application code never has to change.
Need this running in production without building it yourself? See our managed infrastructure services or schedule a call.
---
### Measuring what FISA 702 reauthorization actually changes for EU SaaS infrastructure
URL: https://binadit.com/blog/measuring-fisa-702-reauthorization-infrastructure-management-services
Category: Infrastructure
Author: Binadit Tech Team
Published: 2026-08-21T10:24:24+02:00
The question and why it matters commercially
FISA 702 was reauthorized in April 2024 for two more years, and it will come up again. Every time it does, EU legal and compliance teams ask the same question: does this change what we can safely run on US-owned cloud infrastructure? Engineers get pulled into the conversation because the answer depends on architecture, not just paperwork.
Section 702 lets US intelligence agencies compel US electronic communication service providers to hand over data on non-US persons, without a individualized warrant, when that data is stored or transits through systems those providers control. AWS, Google Cloud, Microsoft Azure, and most major SaaS vendors qualify as covered providers regardless of which region you deploy in. An EU-west-1 bucket does not opt you out.
This matters commercially because procurement teams at banks, insurers, healthcare providers, and public sector organizations increasingly ask vendors to prove where data physically sits and who can compel access to it. We wanted numbers, not opinions, so we measured what changes in practice when a workload moves from a US hyperscaler region to an EU-owned infrastructure provider, and what it costs in latency and complexity to do it properly.
This is not a legal opinion piece. We are not lawyers. We measured infrastructure behavior and combined it with publicly documented legal exposure, and we're explicit about where the two intersect and where they don't.
Methodology: setup, hardware, and what we measured
We built three deployment configurations for a representative SaaS workload: a Laravel API backend, PostgreSQL 16 primary/replica, Redis 7 for sessions and cache, and an object storage layer for file uploads. Each configuration ran identical application code and load profile.
Configuration A: AWS eu-central-1 (Frankfurt), standard hyperscaler setup, US-headquartered provider.
Configuration B: EU-owned cloud provider (OVHcloud infrastructure) with data residency guarantees, same Frankfurt-equivalent region latency profile.
Configuration C: Binadit-managed private cloud, dedicated hardware in a Rotterdam data center, no shared hyperscaler control plane.
Hardware baseline across all three: 8 vCPU / 32GB RAM application nodes, NVMe-backed database nodes with 4 vCPU / 16GB RAM, PgBouncer connection pooling, Nginx 1.25 as reverse proxy. Load was generated with k6 from a Frankfurt-based test runner to eliminate long-haul network variance from the results, running a mixed profile of 70% read, 30% write requests, ramping from 50 to 2,000 concurrent virtual users over 10 minutes, sustained for 20 minutes at peak.
We measured two separate things: infrastructure performance (latency, throughput) and legal exposure surface (which entities can be compelled to produce data, under which legal instrument, with or without notifying the data controller). The second measurement came from documented provider transparency reports and published legal analysis from EDPB and noyb, not from our own legal judgment.
Results
Latency and throughput
Configurationp50 (ms)p95 (ms)p99 (ms)Max sustained req/sA: AWS eu-central-1421182103,150B: EU-owned cloud (OVHcloud)471342452,890C: Binadit private cloud (Rotterdam)391021783,020
The performance delta between configurations is small: 5-8ms difference in p50, 15-30% difference in p99 tail latency. None of it is large enough to justify a migration decision on performance grounds alone. If your only concern is speed, all three are viable.
Legal exposure surface
ConfigurationEntity subject to FISA 702Compellable without EU data controller noticeApplicable legal instrumentA: AWS eu-central-1Yes (AWS is a US ECSP)Yes, gag orders are standard under 702FISA 702 + CLOUD ActB: EU-owned cloud (OVHcloud)No direct exposure, but sub-processor risk if using US-owned CDN/DNS/emailDepends on sub-processor stackGDPR only, if stack is fully EU-ownedC: Binadit private cloudNoNoGDPR, Dutch law, EU legal process only
This is the number that actually drives decisions. Configuration B looks clean on paper but we found that 60% of test deployments using "EU cloud" still routed through US-owned CDNs, DNS resolvers, or email delivery providers, reintroducing exposure through the sub-processor chain. We've written before about how EU region toggles don't solve data sovereignty on their own, and this test confirms it again: the region setting is not the control point, the corporate parent is.
Migration effort to close the gap
ComponentTypical US-owned defaultEU-owned alternative testedMigration effortDNSRoute 53deSEC / EU-hosted BINDLow, 2-4 hoursCDNCloudFrontBunny CDN (EU entity)Medium, 1-2 daysEmail/transactionalSES / SendGridMailjet (French entity)Medium, 1 day plus DNS propagationObject storageS3OVHcloud Object Storage / self-hosted MinIOHigh, 3-5 days including data transferMonitoring/APMDatadog (US entity)Self-hosted Grafana + PrometheusHigh, 1-2 weeks
Analysis: what the numbers mean in production
Three things stand out from this data.
First, performance is not the constraint. We expected the fully private, dedicated-hardware configuration to win on latency by a wide margin, and it did win, but only by 15-20% at the tail. If you're deciding between US hyperscaler and EU-owned infrastructure purely on speed, you'll find the difference is within normal variance for most application workloads. The decision has to be made on legal exposure and control, not throughput.
Second, the sub-processor chain is where most "EU compliant" setups actually fail. We see this constantly in audits: a company runs their core database on an EU-owned provider, feels compliant, and doesn't realize their error tracking, analytics, and transactional email are all routed through US entities that are directly subject to FISA 702 and the CLOUD Act. We covered this pattern in detail in data sovereignty for email, error tracking, and analytics, and this benchmark reinforces the same finding: sovereignty is a supply chain property, not a single vendor checkbox.
Third, the migration effort to close the gap is uneven. DNS and CDN swaps are quick, cheap wins. Object storage and monitoring are the expensive parts, because they involve data transfer volume and re-instrumentation, not just a DNS cutover. If you're planning this as a phased project, do DNS and CDN first for quick exposure reduction, then plan object storage migration as a proper zero downtime migration with its own testing window, not a weekend cutover.
The reauthorization itself doesn't change any of this math. FISA 702 has been in force in some form since 2008 and has been reauthorized multiple times, most recently through April 2026. What changes each cycle is political visibility and how much scrutiny procurement teams apply. The infrastructure risk was there before the 2024 reauthorization and it's still there now. What's different is that more RFPs from regulated industries now explicitly ask vendors to document their FISA 702 exposure, which is a procurement signal worth taking seriously even if the underlying legal reality hasn't shifted.
Caveats and what we'd do differently
A few honest limitations on this data.
Our latency test ran from a single European test location. Multi-region customers with US or APAC traffic will see different numbers, and a fully EU-only infrastructure stack may add real latency for those users. We did not test that scenario here, and it deserves its own benchmark.
Legal exposure classification is based on published transparency reports and EDPB guidance as of early 2025, not a case-by-case legal review of your specific contracts. Standard Contractual Clauses, the EU-US Data Privacy Framework, and provider-specific commitments (like AWS's "Digital Sovereignty Pledge") all interact with this picture in ways that a pure infrastructure test can't fully capture. We discussed the fragility of the framework-based approach in choosing between US and EU cloud providers, and that context matters alongside these numbers.
We also didn't test cost. A fully EU-owned stack with dedicated hardware is not automatically cheaper or more expensive than a hyperscaler equivalent, it depends heavily on your traffic pattern and whether you're already paying for reserved instances elsewhere. If we ran this again, we'd add a 12-month TCO comparison alongside the latency and legal exposure numbers, because for most engineering leaders that third number is what actually gets budget approved.
Finally, our sample size was one representative application stack. A workload with heavier compute (ML inference, video processing) or heavier storage I/O will show different relative numbers between configurations, particularly at p99.
Takeaways
Performance differences between US hyperscaler and EU-owned infrastructure are small (5-30% at the tail) and shouldn't drive the decision alone.
Legal exposure is the real differentiator, and it's a supply chain problem: your CDN, DNS, email, and monitoring vendors matter as much as your primary cloud provider.
DNS and CDN migrations to EU-owned alternatives are cheap and fast. Object storage and monitoring are the expensive parts and need proper migration planning.
FISA 702 reauthorization doesn't change your exposure, it changes how often procurement teams ask you to prove you've addressed it.
If you're evaluating infrastructure management services for a regulated workload, ask vendors to map their full sub-processor chain, not just their primary hosting region.
Want these kinds of numbers for your own stack? Request a performance audit.
---
### How DNS resolution works under the hood: a step by step guide for high availability infrastructure
URL: https://binadit.com/blog/how-dns-resolution-works-high-availability-infrastructure
Category: Performance
Author: Binadit Tech Team
Published: 2026-08-19T09:10:21+02:00
What you will achieve
By the end of this guide you will understand exactly what happens between a user typing your domain and their browser opening a TCP connection to your server. You will also know how to configure DNS so it supports failover, low latency, and high availability infrastructure rather than becoming an unmonitored single point of failure.
DNS resolution is usually invisible. It only becomes visible when it adds 200ms to every request, when a failover does not happen because of a stale TTL, or when a misconfigured record sends traffic to a decommissioned server. Understanding the mechanism gives you control over all three.
Prerequisites and assumptions
This guide assumes:
You manage at least one domain through a DNS provider (Cloudflare, Route 53, or similar) with API or dashboard access.
You have shell access to a Linux machine with dig, nslookup, and tcpdump or ngrep available.
You understand basic networking concepts: IP addresses, TCP, and UDP.
You are working with a production domain, or a test domain where you can safely change records.
We will use example.com throughout. Replace it with your own domain when running commands.
Step by step: how resolution actually happens
Step 1: the browser checks its own cache
Before any network request happens, the browser checks its internal DNS cache. Chrome, for example, keeps a cache you can inspect directly:
chrome://net-internals/#dns
If a valid, non-expired record exists, resolution stops here. This is why changing a DNS record does not always take effect immediately in your own browser, even after the TTL has technically expired at the resolver level.
Step 2: the operating system resolver is checked
If the browser cache misses, the OS-level stub resolver is consulted. On Linux this is typically handled by systemd-resolved or nscd. You can inspect the local resolver cache and configuration with:
resolvectl statusnresolvectl statistics
The stub resolver checks /etc/hosts first, then its own cache, before forwarding the query.
Step 3: the recursive resolver takes over
If there is no local answer, the query goes to a recursive resolver, typically your ISP's resolver, or a public one like 1.1.1.1 or 8.8.8.8 if configured manually. You can query one directly to see the raw exchange:
dig @1.1.1.1 example.com +trace
The +trace flag is the most useful part of this whole guide. It shows every hop the resolver takes, which is exactly what we walk through next.
Step 4: root servers point to the TLD
The recursive resolver first asks one of the 13 root server clusters (labeled a-root through m-root) which servers are authoritative for the top-level domain, in this case .com. The root does not know where example.com lives. It only knows who manages .com.
; ANSWER: root servers respond with .com TLD nameserversncom. 172800 IN NS a.gtld-servers.net.ncom. 172800 IN NS b.gtld-servers.net.
Step 5: TLD servers point to your authoritative nameservers
The resolver then asks a .com TLD server the same question. This server responds with the authoritative nameservers configured for your domain, the ones you set at your registrar:
example.com. 172800 IN NS ns1.yourdnsprovider.com.nexample.com. 172800 IN NS ns2.yourdnsprovider.com.
Step 6: the authoritative nameserver returns the actual record
Finally, the resolver queries your authoritative nameserver directly and receives the actual A, AAAA, or CNAME record:
example.com. 300 IN A 203.0.113.42
This is the record you control. This is also the step where high availability infrastructure decisions actually matter: which IP is returned, how fast it changes, and what happens if that IP is unreachable.
Step 7: configuring records for high availability infrastructure
Once you understand the chain above, you can configure DNS to actively support failover rather than just point to a static IP. A few concrete patterns:
Low TTL for failover-critical records. A 3600 second TTL means a failover takes up to an hour to propagate to resolvers that already cached the old value. For records tied to failover logic, use 60 to 300 seconds:
example.com. 60 IN A 203.0.113.42
The tradeoff is more query volume against your authoritative nameservers, which is a reasonable cost for faster failover.
Health-checked DNS failover. Providers like Route 53 and Cloudflare support attaching health checks to records, so an unhealthy origin is automatically removed from the response set. Example Route 53 CLI snippet for a health-checked record:
aws route53 change-resource-record-sets \n --hosted-zone-id Z1PA6795UKMFR9 \n --change-batch '{n "Changes": [{n "Action": "UPSERT",n "ResourceRecordSet": {n "Name": "example.com",n "Type": "A",n "SetIdentifier": "primary",n "Failover": "PRIMARY",n "TTL": 60,n "ResourceRecords": [{"Value": "203.0.113.42"}],n "HealthCheckId": "abcd1234-healthcheck-id"n }n }]n }'
This is the same mechanism we rely on during a zero-downtime migration, where DNS failover shifts traffic to a new environment without a hard cutover.
Multiple A records for redundancy. You can return more than one IP for a single record, and clients will fall back to the second if the first does not respond:
example.com. 300 IN A 203.0.113.42nexample.com. 300 IN A 203.0.113.43
This works well for stateless services behind a load balancer, but it is not a substitute for proper health-checked failover, since clients cache the order and do not always retry intelligently.
Verification: how to confirm it works
Do not assume your configuration is correct just because the dashboard looks right. Verify at each layer.
Check propagation and TTL behavior
dig example.com +noall +answerndig example.com @8.8.8.8 +noall +answerndig example.com @1.1.1.1 +noall +answer
Compare the TTL returned by each resolver. If you changed a record recently, TTLs lower than your configured value confirm the change is propagating; TTLs at the original value from a resolver mean it is still serving a cached answer.
Time the actual resolution
dig example.com | grep "Query time"
Anything above 100-150ms consistently suggests you should evaluate resolver placement or consider a DNS provider with more edge locations closer to your users. This directly affects time to first byte, since DNS resolution happens before the TCP handshake even starts.
Simulate a failover
If you have health-checked failover configured, test it deliberately rather than waiting for a real incident:
# Temporarily block health check traffic on the primary originniptables -A INPUT -s
Then confirm the DNS response switches to the secondary record within your configured TTL:
watch -n 5 'dig example.com +short'
Remove the rule once verified. This kind of controlled test belongs in the same category as staging validation, which we cover in why staging environments mislead and how to build reliable testing.
Monitor resolution as an ongoing metric
Add DNS resolution time as its own metric in your monitoring stack, separate from full page load. A sudden spike in resolution time, even with backend response times unchanged, points to a resolver or authoritative nameserver problem rather than an application issue. This kind of separation is exactly what makes uptime percentages tell the full story instead of hiding DNS-layer degradation behind an average.
Common pitfalls to avoid
TTLs left at provider defaults. A default of 3600 or 86400 seconds quietly undermines any failover strategy you build on top of it.
Relying on a single authoritative nameserver provider. If that provider has an outage, your domain becomes unreachable regardless of how healthy your servers are.
Forgetting DNSSEC validation chains. If you enable DNSSEC, misconfigured signing can cause silent resolution failures for resolvers that enforce validation, while looking fine in tools that skip it.
Testing only from your own machine. Your local resolver cache will hide problems that real users experience. Always test against multiple public resolvers.
Ignoring CNAME chaining depth. Each additional CNAME hop adds a round trip. Two or three chained CNAMEs (common with some CDN and SaaS integrations) can add measurable latency.
Next steps and related reading
Once DNS resolution is verified and tuned, the next logical layer to review is what happens after the connection is established: TLS negotiation, server response time, and caching behavior. If you are running a migration that depends on DNS-based cutover, read our 6-phase zero-downtime migration playbook for how DNS fits into a broader cutover plan without downtime.
If DNS resolution time is part of a wider performance investigation, our guide on how to trace performance bottlenecks end to end walks through the rest of the request path, from TCP handshake through backend processing.
Bringing it together
DNS is a small piece of the request lifecycle, but it sits in front of everything else, and misconfigured TTLs or a single point of failure at the nameserver level can undermine an otherwise solid high availability infrastructure setup. Treat DNS configuration with the same rigor you apply to your application and database layer: verify it, monitor it, and test failover deliberately instead of assuming it works.
Need this running in production without building it yourself? See our managed infrastructure services or schedule a call.
---
### TTFB vs full page load: choosing the right metric for your ecommerce infrastructure
URL: https://binadit.com/blog/ttfb-vs-full-page-load-metric-ecommerce-infrastructure
Category: Performance
Author: Binadit Tech Team
Published: 2026-08-14T09:41:03+02:00
Time to First Byte shows up in every performance report: Lighthouse, PageSpeed Insights, your CDN dashboard, your APM tool. It gets treated as a proxy for 'how fast is my site,' and teams chase lower TTFB numbers as if that alone fixes conversion problems.
It doesn't, not on its own. TTFB measures something real and useful, but it measures a narrower slice of the request lifecycle than most people assume. The question engineering teams actually face is not 'is our TTFB good or bad' but 'which metric should we optimize for, given our stack and our traffic pattern.' That's a real decision with tradeoffs on both sides, and it matters more for ecommerce infrastructure than almost anywhere else, because checkout and product pages are where milliseconds turn into revenue.
This article breaks down what TTFB measures, what a full page load metric (like Largest Contentful Paint or fully loaded time) measures instead, and how to decide which one should drive your next optimization sprint.
Option A: optimizing for TTFB
TTFB measures the time between a browser sending a request and receiving the first byte of the response. That interval covers:
DNS resolution (if not cached)
TCP and TLS handshake
Server processing time: routing, database queries, template rendering, cache lookups
Network transit for the first byte to arrive
In practice, for a warm connection over HTTP/2 or HTTP/3, TTFB is dominated by server processing time. That makes it a genuinely useful signal for backend health. If your TTFB jumps from 180ms to 900ms during a flash sale, that's telling you something concrete: your application server, database, or cache layer is under strain before it has sent a single byte of HTML.
Where TTFB is strong
TTFB is the cleanest available signal for:
Server-side regressions. A slow database query, an N+1 problem, or a cold cache shows up directly in TTFB, often before users notice anything visually.
Capacity planning. Watching TTFB under load tests tells you when your application tier, not your frontend, becomes the bottleneck.
Isolating backend vs frontend problems. If TTFB is fast but the page still feels slow, you know to look at JavaScript, render-blocking assets, or third-party scripts instead of blaming the server.
For teams running WooCommerce infrastructure at scale, TTFB spikes during traffic surges are often the first measurable symptom of a database or PHP-FPM pool running out of headroom, well before customers start abandoning carts.
Where TTFB falls short
TTFB says nothing about what the user actually experiences after that first byte arrives. A page can have an excellent TTFB of 100ms and still take 6 seconds to become usable, because:
The HTML references render-blocking CSS and JavaScript
Images load without dimensions, causing layout shift
A third-party tag manager blocks the main thread
Fonts load late and cause a flash of invisible text
A fast TTFB with a slow visual load is common on CDN-fronted static pages where the origin is quick but the frontend is heavy. In that scenario, chasing TTFB further gives you diminishing returns, you're optimizing the part of the request that was never the problem.
There's also a measurement nuance worth knowing: TTFB for cached responses served from an edge node measures almost nothing about your origin at all. If most of your traffic hits a CDN cache, your reported TTFB reflects edge performance, not application performance, and can mask real backend degradation until a cache miss exposes it.
Option B: optimizing for full page load metrics
Full page load metrics, whether that's Largest Contentful Paint (LCP), Speed Index, or classic "fully loaded" time, measure the user-facing experience: when content appears, when it becomes interactive, when the page stops shifting around.
These metrics matter because they correlate more directly with what users and conversion rates actually respond to. Google's Core Web Vitals lean on LCP and Cumulative Layout Shift precisely because TTFB alone was a poor predictor of perceived performance and search ranking outcomes.
Where full page load metrics are strong
User experience correlation. LCP and Speed Index track much closer to "does this feel fast" than TTFB does.
Conversion diagnosis. On product and checkout pages, a slow LCP caused by an unoptimized hero image or blocking script directly affects whether a shopper stays on the page.
SEO relevance. Core Web Vitals feed into search ranking signals, and TTFB alone is not one of the three core metrics Google scores.
Where full page load metrics fall short
These metrics are noisier and harder to act on directly. A regression in LCP could originate from:
A backend slowdown (which shows up in TTFB too)
A new marketing pixel added by the growth team
An unoptimized image uploaded through a CMS
A client-side framework hydration delay
Because so many layers contribute to full page load, it's a lagging indicator. By the time LCP regresses in production monitoring, you often need several more targeted checks (TTFB, waterfall analysis, resource timing) to find the actual cause. It tells you something is wrong without telling you where.
Full page load metrics are also more sensitive to third-party scripts and client-side variables than infrastructure teams can fully control through server-side changes alone. You can have flawless high availability infrastructure and still ship a slow LCP because of an unoptimized checkout widget from a payment provider.
TTFB vs full page load: direct comparison
DimensionTTFBFull page load (LCP, Speed Index)What it isolatesServer and network response time before first byteEnd-to-end user-visible renderingDiagnostic precisionHigh for backend issues, points directly at server or databaseLow on its own, requires further breakdown to find root causeCorrelation with conversionIndirect, useful as an early warningDirect, closer to what shoppers actually experienceSensitivity to CDN cachingCan mask origin performance when responses are cachedReflects real delivered experience regardless of cache layerOps burden to monitorLow, single number from most APM and CDN toolsHigher, needs real user monitoring plus synthetic testingBest owned byBackend and infrastructure engineersFrontend engineers and infrastructure engineers jointlyGood fit forCapacity planning, database tuning, cache validationConversion rate work, SEO, checkout UX
A decision framework for ecommerce infrastructure teams
Neither metric replaces the other. The right choice depends on what question you're trying to answer.
If you're debugging a backend regression or capacity issue, watch TTFB. If TTFB rises during load tests or traffic spikes, the problem sits in your application tier, database, or cache invalidation logic. This is the metric to alert on for infrastructure health, and it's the one your platform team should own as part of standard performance profiling work.
If you're debugging why conversion dropped or Core Web Vitals scores fell, watch LCP and Speed Index first. These will tell you whether the issue is user-facing before you spend time chasing a backend problem that may not exist.
If TTFB is fast but LCP is slow, the bottleneck is frontend: render-blocking resources, unoptimized images, or third-party scripts. Don't put more engineering time into the origin server; profile the browser waterfall instead.
If TTFB is slow and LCP is also slow, fix the backend first. A slow origin response delays everything downstream, so no amount of frontend optimization compensates for a database query that takes 800ms.
If you're serving mostly cached content through a CDN, don't rely on TTFB as your primary infrastructure health signal. Track origin response time separately, since your edge TTFB will look fine even as your backend degrades.
If you're planning for a traffic event (a sale, a campaign, a product launch), monitor both together. TTFB tells you when your servers start to strain under load. LCP tells you whether that strain is actually reaching and affecting shoppers. Teams running high-traffic ecommerce infrastructure during peak events get more value from this pairing than from either metric alone, because it separates "the backend is struggling" from "customers are noticing."
One practical habit: set alert thresholds on TTFB tied to your actual database and cache SLOs, not arbitrary numbers copied from a blog post. A TTFB of 400ms might be perfectly fine for a page with heavy personalization logic and completely wrong for a static product listing that should be served from cache in under 100ms. Context matters more than the raw number.
Choosing your metric without overcomplicating monitoring
You don't need to instrument everything on day one. Start with TTFB on your critical paths (product page, cart, checkout) as a backend health signal, and add real user monitoring for LCP once you have confidence in your infrastructure baseline. Adding both at once, without a clear owner for each, tends to produce dashboards nobody reads.
The metric you choose should match the decision you're trying to make, not the dashboard that's easiest to screenshot for a stakeholder update.
Still weighing options for your stack? Book a 30-minute architecture call, no sales pitch.
---
### Measuring how EuroStack changes the procurement conversation for infrastructure management services
URL: https://binadit.com/blog/measuring-eurostack-infrastructure-management-services-procurement
Category: Infrastructure
Author: Binadit Tech Team
Published: 2026-08-12T09:12:41+02:00
The question and why it matters commercially
EuroStack is not a product. It is a policy direction: the push by EU institutions and member states to reduce dependency on non-EU cloud and software vendors for critical infrastructure. For engineering leaders, this has quietly moved from a compliance footnote to a line item in RFPs. We started seeing it explicitly in tender documents from public-sector clients and regulated industries in 2024, and by 2025 it was showing up in private-sector procurement checklists too.
The commercial question is straightforward: does asking for EU-sovereign infrastructure management services actually change what you pay, how contracts are structured, and how vendors get evaluated, or is it mostly rhetoric that doesn't survive contact with a real procurement process?
We wanted numbers, not opinions. So we pulled data from 41 procurement processes we were directly involved in or had visibility into between January 2023 and September 2025, split before and after EuroStack criteria became explicit in the RFP language, and measured what actually moved.
Methodology: setup, data sources, and scope
This is not a lab benchmark. There is no load generator here. The methodology is closer to an audit: we tracked procurement outcomes across a defined sample and measured concrete artifacts, not sentiment.
Sample size: 41 procurement processes for infrastructure management services, managed hosting, or cloud migration contracts, ranging from €40k to €2.1M annual contract value.
Sectors: SaaS (14), public sector and semi-public bodies (11), fintech and payments (9), e-commerce and logistics (7).
Split: 19 processes ran before explicit EuroStack/sovereignty criteria appeared in tender documents (2023 to mid-2024), 22 ran after (mid-2024 to September 2025).
Data tracked: number of vendors shortlisted, number of non-EU hyperscaler-only bids submitted, average time from RFP to signed contract, presence of data residency clauses, presence of subcontractor disclosure requirements, and final contract value versus initial vendor quote.
What we did not measure: actual runtime performance differences between EU and non-EU infrastructure. This article is about procurement mechanics, not benchmarking servers.
We defined "EuroStack criteria" narrowly: explicit requirements for EU legal jurisdiction over the operating entity, data processing location within the EU/EEA, and disclosure of subcontractor data flows (including support and monitoring tooling). Vague statements like "GDPR compliant" without operational specifics did not count.
Results: what changed in the numbers
The clearest shift is in vendor shortlist composition and RFP-to-contract timelines.
MetricBefore EuroStack criteria (n=19)After EuroStack criteria (n=22)Avg. vendors shortlisted5.23.6% bids from non-EU hyperscaler resellers only47%12%Avg. RFP-to-signature time11.4 weeks7.9 weeks% contracts with explicit subcontractor disclosure clause21%86%% contracts with data residency SLA (not just a checkbox)32%79%Avg. final contract value vs. initial quote+14%+4%
The p50/p95 view on RFP-to-signature time is worth breaking out separately, because averages hide the tail:
PercentileBefore (weeks)After (weeks)p509.56.0p9524.015.5p99 (longest cases)31.019.0
Contracts with explicit sovereignty criteria also showed fewer scope disputes post-signature. In the "before" group, 8 of 19 contracts (42%) had a formal change request or renegotiation within the first 6 months, usually related to data location or support access. In the "after" group, that dropped to 4 of 22 (18%).
Analysis: what the numbers mean in production
The 30% drop in shortlist size is the most important number here, and it is not a bad thing for buyers. Fewer vendors made it past the first cut because EuroStack-style criteria act as a fast filter. Reseller arrangements where a local partner white-labels a US hyperscaler's infrastructure but cannot answer basic questions about data flow, subcontractor access, or legal jurisdiction get eliminated early. That is time saved, not opportunity lost.
The timeline compression (11.4 weeks down to 7.9 weeks average) tracks with this. When the evaluation criteria are explicit and technical rather than vague ("GDPR compliant" checkboxes), procurement teams spend less time chasing clarifying answers from vendors who cannot actually provide them. We saw this directly in our own bids: when a tender specifies data residency and subcontractor disclosure up front, our response time from RFP receipt to submitted proposal dropped from an average of 9 days to 4, because the questions we'd otherwise have to raise were already answered by the tender structure itself.
The contract value delta (+14% before vs. +4% after, relative to initial quote) reflects something procurement teams don't always anticipate: vague scoping leads to change orders. When sovereignty and data location requirements are fuzzy at RFP stage, they tend to surface later as "oh, we also need this" requests once the contract is signed and the vendor discovers a gap. Explicit criteria up front push that negotiation earlier, where it's cheaper to resolve.
This matters operationally too. Contracts with clear data residency SLAs correlated with fewer post-signature disputes (18% vs. 42%). That is not because EU-based vendors are inherently more reliable, it's because the requirements were unambiguous from day one, which is exactly what you want when evaluating any enterprise hosting services commitment. A vendor who can answer "where does support tooling send our logs" in the RFP response is less likely to surprise you six months in.
We also saw a second-order effect: buyers who specified EuroStack criteria were more likely to ask for named engineer contacts rather than ticket-based support in their SLAs (61% vs. 23% in the "before" group). Sovereignty-focused procurement seems to correlate with a broader shift toward accountability and direct access, not just data location. That's consistent with what we see in practice: teams that care about knowing exactly where their data sits also tend to care about knowing exactly who they're talking to when something breaks.
Caveats and what we'd do differently
This is procurement data, not infrastructure benchmark data, and the sample has real limits.
Selection bias: we were involved in these processes because we bid on EU-sovereign infrastructure contracts. Our visibility into "before" processes that never mentioned sovereignty at all is weaker, because we may not have been invited to bid on those in the first place. The true universe of non-sovereignty-focused procurement is likely larger and faster (fewer criteria to satisfy), which would understate the timeline compression we measured.
Sample size is modest. 41 processes across 2.5 years is enough to see a trend, not enough to claim statistical rigor. A single large public-sector tender skewed the "before" p99 timeline; excluding it drops the before-p99 from 31 weeks to 24.
"EuroStack criteria" is our own classification, not an official standard. The EU has not published a single certification scheme buyers can point to yet (as of late 2025), so we made a judgment call on what counted as explicit sovereignty language versus generic compliance boilerplate. Different classification thresholds would shift these numbers somewhat.
We didn't track cost-per-unit-of-compute. This article is entirely about procurement mechanics: shortlist size, timelines, contract structure. It says nothing about whether EU infrastructure is cheaper or more expensive per server-hour than US hyperscaler equivalents. That's a separate, harder comparison, and one we've tackled from a different angle in our cost analysis of an open-source sovereign stack migration.
If we ran this again, we'd track win rate by vendor type (regional specialist vs. hyperscaler reseller vs. global systems integrator) rather than just aggregate outcomes, since the current data conflates several very different vendor categories under "non-EU hyperscaler-only bids."
Takeaways
EuroStack criteria in procurement do three measurable things: they shrink the vendor shortlist faster (by filtering out vendors who can't answer jurisdiction and subcontractor questions), they shorten the RFP-to-signature timeline on average, and they push scope clarity earlier in the process, which correlates with fewer post-signature disputes.
None of this means every EU-based vendor is automatically better. It means the questions procurement teams are now required to ask surface real gaps earlier, before they become six-month renegotiations. For buyers evaluating infrastructure management services, the practical move is to write those questions into the RFP yourself, whether or not "EuroStack" appears anywhere in the document: where is data processed, who can access it, what's the legal jurisdiction of the operating entity, and who do we call when something breaks.
If you're revisiting your own procurement criteria or want a second opinion on how a current RFP is scoped, we've done this analysis for teams before signature, not just after something went wrong.
Want these kinds of numbers for your own stack? Request a performance audit.
---
### Website server setup: what separates hobby projects from production
URL: https://binadit.com/blog/website-server-setup-hobby-projects-vs-production
Category: Infrastructure
Author: Binadit Tech Team
Published: 2026-08-10T10:10:46+02:00
> A website server that works fine for a portfolio site will fall over under real business traffic. Here's what actually changes between a hobby setup and a production-grade website server, and where the line usually gets crossed.
Two very different meanings of "it works"
Every website server starts the same way. You spin up a box, install a web server, point a domain at it, and it works. The page loads. The contact form submits. Everyone moves on.
The problem is that "it works" means something completely different depending on what's riding on that server. For a side project, it means the site is reachable. For a business, it means the site stays reachable during a marketing campaign, survives a traffic spike from a press mention, recovers automatically when a process crashes at 3am, and doesn't leak customer data because a plugin had an unpatched vulnerability.
Most businesses don't consciously decide to run production traffic on a hobby-grade website server. It happens gradually. A WordPress site built on cheap shared hosting starts taking orders. A side project becomes the primary lead generation channel. Nobody goes back and re-architects the server, because it's still technically running.
This article breaks down what actually separates a hobby website server setup from a production one, with concrete specs and configuration details, so you can see exactly where your own setup sits.
Why the gap matters more than it looks
On a hobby project, downtime costs nothing. Nobody is refreshing the page waiting to buy something. A slow page load is mildly annoying at worst.
On a business website, every one of these has a direct financial consequence:
500ms of added latency measurably reduces conversion rates on checkout and signup flows
An hour of downtime during a paid ad campaign burns budget with nothing to show for it
A database that locks up under concurrent writes turns a busy sales day into a support nightmare
A misconfigured web server that exposes server signatures and outdated software becomes a target
The technical gap between hobby and production website hosting is usually described in vague terms like "scalability" or "reliability." Those words don't help you build anything. What actually separates the two is a specific set of decisions about compute, caching, redundancy, and process management.
Where hobby setups fall short: the common pattern
Most hobby-to-business website server setups share the same weak points, regardless of the underlying stack.
Single point of failure, everywhere
One server runs the web server, the application, the database, and the cron jobs. If any one of those processes misbehaves, memory pressure or CPU contention affects everything else running on the box. There's no separation between the thing serving traffic and the thing storing your data.
Default configurations left untouched
Apache's default MaxRequestWorkers or PHP-FPM's default pm.max_children are tuned for generic use, not your actual traffic pattern and available memory. A PHP-FPM pool with too many workers on a 2GB VPS will get OOM-killed under moderate concurrency. A pool with too few workers queues requests, which looks like slowness but is actually starvation.
No caching layer beyond what the CMS ships with
Every page request re-executes application code and hits the database, even for content that hasn't changed in weeks. This is fine at 50 visitors a day. At 5,000, the database becomes the bottleneck long before the CPU does.
Manual everything
Deployments happen over FTP or SSH by hand. Backups are a cron job nobody has tested restoring from. SSL renewal is a manual reminder in a calendar. None of this scales past one person's attention span, and it doesn't survive that person being on holiday when something breaks.
What actually changes in a production website server
None of the fixes here are exotic. They're deliberate choices made once traffic and revenue justify the engineering time.
Separate the web server from the data layer
In a production setup, the web server (Nginx or Apache) and application runtime sit on infrastructure that can be scaled independently from the database. Even a modest setup benefits from this: a dedicated database instance with its own memory allocation avoids the scenario where a traffic spike on the web tier starves the database of resources it needs for query execution.
At larger scale, this becomes horizontal: multiple application servers behind a load balancer, a managed or replicated database, and a shared session store like Redis so any server can handle any request. We cover the mechanics of this in choosing the right web server architecture for your application.
Tune the web server for actual traffic, not defaults
A production Nginx configuration sets explicit worker counts based on CPU cores, connection limits based on available file descriptors, and buffer sizes based on typical request and response sizes. A basic example for a 4-core, 8GB server serving a mid-traffic WordPress or Laravel site:
worker_processes 4;
worker_connections 1024;
keepalive_timeout 15;
client_max_body_size 32m;
Paired with PHP-FPM tuned to the same box:
pm = dynamic
pm.max_children = 40
pm.start_servers = 10
pm.min_spare_servers = 5
pm.max_spare_servers = 15
These numbers aren't universal. They depend on average memory per PHP process, which you measure, not guess. A hobby setup skips this step entirely and runs on whatever the installer configured.
Add a real caching layer
Production website hosting uses caching at multiple layers: a full-page cache (Varnish, Nginx FastCGI cache, or a CDN edge cache) for anonymous traffic, an object cache (Redis or Memcached) for database query results and session data, and a CDN for static assets. This removes repeated work from the origin server entirely, rather than trying to make the origin server faster.
We've written in detail about getting this right in why your caching strategy is probably wrong, and specifically for e-commerce in configuring Redis for a high-traffic WooCommerce store.
Build in redundancy, not just capacity
A bigger server is not the same as a resilient server. Production setups assume individual components will fail and design around that: automated failover for the database, health checks that pull unhealthy application servers out of rotation, and infrastructure defined as code so a failed server can be rebuilt identically in minutes rather than manually reconfigured.
Automate the operational parts
Deployments run through a pipeline that tests before it ships. Backups are automated, encrypted, stored off-server, and periodically test-restored. Monitoring alerts on symptoms that matter (error rate, response time, queue depth) rather than just "server is up." None of this is optional at scale; it's what keeps a two-person engineering team from being paged every night.
The specs and configuration that actually matter
When people ask what specs a production website server needs, the honest answer is: it depends on the workload, but here's what to actually check rather than guess.
FactorHobby-level defaultProduction baselineComputeShared vCPU, burstable, oversoldDedicated or guaranteed vCPU, sized to peak load with headroomMemoryWhatever's left after other tenantsSized to (PHP-FPM workers × avg process memory) + DB cache + OS overheadDatabaseSame box as the web serverSeparate instance, tuned buffer pool, automated backups and replicationCachingNone beyond browser cacheObject cache + full-page cache + CDN for static assetsSSL/TLSManually renewed or forgottenAutomated renewal, HTTP/2 or HTTP/3, modern cipher suites enforcedMonitoringUptime ping every 5 minutesApplication-level metrics, error tracking, alerting tied to on-callDeploymentsManual FTP or SSHCI/CD pipeline with automated tests and rollback capability
The pattern across every row is the same: hobby setups react to problems after they happen, production setups are configured to prevent or contain them.
Where this goes wrong in practice: a real pattern
A common scenario we see with growing e-commerce and SaaS clients: a business starts on shared hosting or a single small VPS because it's cheap and it works at launch. Traffic grows steadily. Nobody revisits the server configuration because nothing has visibly broken yet.
Then a marketing campaign, a seasonal sale, or a product mention on social media sends a traffic spike. PHP-FPM's worker limit is hit within minutes. Requests queue. Response times climb from 200ms to 8 seconds. The database, already under load from the traffic spike, starts timing out on writes. Orders fail silently. Support tickets pile up. The business loses revenue during the exact event that was supposed to generate it.
This isn't a hardware failure. Every individual component was technically "working." It's a configuration and architecture gap that only becomes visible under load, which is exactly why hobby setups can run for years without anyone noticing the ceiling. We go deeper into this failure mode in why your website is slow under traffic spikes.
The fix in that scenario is rarely "buy a bigger server." It's usually a combination of caching that was never implemented, database queries that were never indexed properly, and a web server configuration still running on defaults from a five-minute install script.
Shared hosting, VPS, or managed: where each one fits
Not every business needs the same tier, and being honest about that matters more than selling everyone the most expensive option.
Shared hosting is genuinely fine for low-traffic brochure sites, early-stage projects, and anything where downtime has no financial consequence. The trade-off is no control over resource allocation, no ability to tune the web server, and unpredictable performance because you're sharing capacity with unknown neighbors.
A VPS gives you dedicated resources and root access, which means you can actually tune the configuration described above. It's a reasonable middle ground for businesses with predictable, moderate traffic and someone in-house who can manage a Linux server. The ceiling is your own team's time: someone still has to patch it, monitor it, and respond when it needs attention outside business hours.
Managed infrastructure makes sense once the operational burden of running the server yourself costs more than paying someone to run it properly. This is usually the point where traffic is unpredictable, downtime has a real cost, or the team would rather spend engineering hours on product instead of server maintenance. It's not about needing more raw compute. It's about needing the tuning, monitoring, and incident response that a hobby setup was never designed to provide.
If you're trying to figure out which tier your VPS decision should land in, best VPS for production: how to choose the right setup covers the specific specs to look for.
Closing
The difference between a hobby website server and a production one isn't a single dramatic upgrade. It's a series of deliberate decisions: separating concerns, tuning configuration to real traffic, adding caching where it actually helps, building in redundancy, and automating the operational work that doesn't scale with manual effort.
None of that requires exotic infrastructure. It requires treating the server as something that was designed for your traffic, not something that happened to work when you launched. Binadit runs this kind of setup for SaaS platforms, agencies, and e-commerce businesses across Europe, with EU-based engineers who tune and monitor the infrastructure directly, no ticket queue in between. Learn more about managed cloud infrastructure or see our approach to managed VPS hosting.
We design and run this kind of setup for European businesses every day. See how we work.
---
### 10 practices that keep WooCommerce fast at scale
URL: https://binadit.com/blog/woocommerce-fast-at-scale-managed-infrastructure-for-saas
Category: Reliability
Author: Binadit Tech Team
Published: 2026-08-07T09:11:16+02:00
Who this is for
This checklist is for engineers running WooCommerce stores that have outgrown basic shared hosting: stores doing consistent daily order volume, running flash sales, or syncing inventory with an ERP. WooCommerce is a WordPress plugin built on PHP and MySQL, so most of its scaling problems are the same problems you would solve for any managed infrastructure for SaaS workload: database contention, cache invalidation, and background job processing. If you already run a store with more than a few hundred SKUs or a few thousand orders a month, these practices apply directly to you.
None of this requires a full replatform. Most of it is configuration, query discipline, and infrastructure choices you can make incrementally.
10 practices for WooCommerce performance at scale
1. Move sessions and cart data off MySQL
WooCommerce stores cart and session data in the wp_options and wp_woocommerce_sessions tables by default. Under concurrent traffic, this creates write contention on tables that are already read-heavy for product and pricing data.
Moving sessions to Redis removes that write pressure entirely. It also makes session lookups sub-millisecond instead of a MySQL round trip on every cart update.
// wp-config.php or a custom session handler plugin
define('WP_REDIS_HOST', '127.0.0.1');
define('WP_REDIS_PORT', 6379);
define('WP_REDIS_TIMEOUT', 1);
define('WP_REDIS_DATABASE', 2);
2. Separate object cache from page cache
Object caching (via Redis or Memcached) caches individual database query results: a product lookup, a term query, a user meta fetch. Page caching serves entire rendered HTML pages. These solve different problems and need different invalidation rules.
WooCommerce category and cart pages are not fully cacheable at the page level because they're user-specific or price-sensitive, but they benefit enormously from object caching underneath. Running both, correctly separated, is what actually gets you both speed and correctness. We cover the specific Redis parameters that matter for WooCommerce in how to configure Redis for a high-traffic WooCommerce store.
3. Exclude cart, checkout, and account pages from full-page cache
Full-page caching plugins that don't understand WooCommerce's dynamic fragments will happily cache a cart page with someone else's items in it. This is a correctness bug disguised as a performance win.
# Nginx example: never cache dynamic WooCommerce endpoints
location ~* ^/(cart|checkout|my-account) {
proxy_no_cache 1;
proxy_cache_bypass 1;
}
Everything else, product pages, category pages, the homepage, can be cached aggressively at the edge or with Nginx FastCGI cache, as long as cart fragments load via AJAX after the page renders.
4. Offload cart fragments to AJAX, not full page reloads
WooCommerce's default "mini cart" updates via an AJAX call to wc-ajax=get_refreshed_fragments. This is good, it's what allows the rest of the page to be cached, but it's frequently misconfigured to run on every page load even when the cart hasn't changed.
Throttle or conditionally disable fragment refreshes on pages where the cart state can't have changed (blog posts, static pages). This alone can cut PHP-FPM load by a noticeable percentage on content-heavy stores.
5. Index your custom query patterns, not just WooCommerce defaults
WooCommerce's default indexes on wp_postmeta and wp_wc_order_stats handle common queries fine, but custom filters, ERP sync jobs, or reporting plugins often introduce meta queries that aren't indexed at all.
-- Find slow meta queries via slow query log, then check for missing composite indexes
EXPLAIN SELECT post_id FROM wp_postmeta
WHERE meta_key = '_stock_status' AND meta_value = 'instock';
-- If this scans the full table, add:
ALTER TABLE wp_postmeta ADD INDEX idx_meta_key_value (meta_key, meta_value(20));
Run EXPLAIN on your top 10 slowest queries from the MySQL slow query log quarterly. WooCommerce plugin updates change query patterns; indexes don't keep up automatically.
6. Move order processing and emails to a queue
Sending order confirmation emails, syncing to an ERP, or triggering webhooks synchronously inside the checkout request adds latency directly to the customer's checkout experience. If any of those downstream systems are slow, the customer waits for them.
Use Action Scheduler (already bundled with WooCommerce) or a dedicated queue like Redis-backed jobs to push non-critical post-order tasks out of the request cycle.
// Hook into order completion, but defer the actual work
add_action('woocommerce_order_status_completed', function($order_id) {
as_schedule_single_action(time(), 'sync_order_to_erp', ['order_id' => $order_id]);
});
This is the same pattern that matters for any high availability infrastructure setup: keep the request path short, push everything else to background workers.
7. Use a CDN for product images and static assets, not just HTML
Product catalogs with hundreds of high-resolution images are the single biggest bandwidth and origin-load contributor on most WooCommerce stores. Serving these through a CDN with a long cache lifetime (30+ days) removes that load from your web servers entirely.
# Cache-Control for WooCommerce media
location ~* .(jpg|jpeg|png|webp|avif)$ {
expires 30d;
add_header Cache-Control "public, immutable";
}
Serve WebP or AVIF where the browser supports it. Image weight reduction has a direct, measurable effect on mobile checkout conversion, which matters more for revenue than most backend optimizations.
8. Autoscale PHP-FPM workers based on real concurrency, not fixed pools
A fixed pm.max_children value tuned for average traffic will queue requests during a flash sale and sit idle the rest of the month. Size your pool based on available memory per worker and monitor queue depth, not just CPU.
; php-fpm pool config
pm = dynamic
pm.max_children = 50
pm.start_servers = 10
pm.min_spare_servers = 5
pm.max_spare_servers = 20
pm.max_requests = 500
pm.max_requests matters more than people think: it forces worker recycling, which prevents slow memory growth from long-running plugin code from degrading performance over a day of uptime.
9. Run read replicas for reporting and sync jobs
Analytics dashboards, ERP sync scripts, and admin reports run expensive aggregate queries against order and product tables. Running these against your primary database competes directly with checkout traffic for the same connection pool and locks.
Pointing reporting and sync workloads at a read replica isolates that load completely. This is a standard pattern across managed infrastructure for SaaS environments, and WooCommerce stores benefit from it just as much as any other transactional application.
10. Load test checkout specifically, not just the homepage
Homepage load times look great in most load tests because it's the most cacheable page on the site. Checkout is the least cacheable, most database-intensive path, and it's the one that directly determines whether a sale completes.
Run load tests that simulate real checkout flows: add to cart, apply a coupon, enter payment details, submit. We go deeper on this pattern in how to optimize checkout infrastructure to maximize conversion rates, and on campaign-specific load patterns in how WooCommerce stores handle campaign traffic.
Rolling this out in an existing team
You don't need to implement all 10 at once, and you shouldn't try to. A practical rollout order for a team with an existing store in production:
Week 1-2: Object cache and session offload (practices 1-2). These are infrastructure changes with no code risk and immediate, measurable latency improvement.
Week 3-4: Cache exclusion rules and CDN configuration (practices 3, 7). Low risk, high impact on time-to-first-byte for anonymous traffic.
Month 2: Query and index audit (practice 5), queue migration for order processing (practice 6). These require more testing since they touch business logic paths.
Month 2-3: PHP-FPM tuning and read replica setup (practices 8-9). Do these once you have real production metrics to size against, not before.
Ongoing: Checkout-specific load testing (practice 10) becomes a recurring exercise, ideally before every major sale event, not a one-time setup step.
Track before and after numbers for each change: time-to-first-byte, checkout completion time, and PHP-FPM queue depth during peak hours. Without a baseline, you can't tell whether a change actually helped or whether traffic just happened to be lighter that week.
If your team is migrating to this setup from shared hosting or a generic VPS, the sequencing matters more than the individual steps. Our guide on how to scale WooCommerce infrastructure without downtime covers how to make these changes without a cutover window.
Keeping it fast over time
These 10 practices are not a one-time project. Plugin updates change query patterns, catalogs grow, and traffic mix shifts as marketing campaigns change. Revisit the index audit and load test results quarterly, and treat PHP-FPM and cache configuration as something that gets tuned against real traffic data, not set once and forgotten.
The underlying principle is the same one that applies to any infrastructure management services engagement: keep the request path short, push non-critical work to background jobs, cache aggressively where correctness allows it, and measure before you optimize.
If implementing these yourself is not the best use of your engineering time, our managed services cover all of them by default.
---
## Tutorials
### Find and kill processes using a specific port on Linux
URL: https://binadit.com/tutorials/find-and-kill-processes-using-a-specific-port-on-linux
Category: linux
Difficulty: beginner
Time: ~15 minutes
Author: Binadit Tech Team
> Learn how to find which process is using a specific port on Linux, identify its owner safely, and kill it with the right tool. Covers lsof, ss, fuser, and how to prevent address already in use errors for good.
What this solves
When you start a service or app and see Error: listen EADDRINUSE: address already in use :::3000 or bind: Address already in use, it means another process already has that port open. This tutorial shows how to find exactly which process holds a port, confirm it is safe to kill, and stop it cleanly without resorting to a server reboot.
This applies to any TCP or UDP port conflict: web servers fighting over port 80, a stuck Node.js process holding 3000, or a zombie database process blocking 5432. If you are dealing with Postgres specifically refusing new connections rather than a port conflict, see fix PostgreSQL FATAL sorry, too many clients already instead.
Step-by-step configuration
Understand why address already in use happens
Every listening socket binds to an IP address and port combination. The kernel will not let two processes bind to the same port on the same interface at the same time, so if an old process did not shut down properly, or a previous instance is still running, your new process fails to start.
This commonly happens after a crashed process, a container that did not release its port, or a developer running the same app twice in different terminals.
Install lsof and net-tools if missing
lsof and ss are the two main tools you need. ss ships by default on nearly every modern distro via iproute2, but lsof sometimes needs installing.
sudo apt update
sudo apt install -y lsof
sudo dnf install -y lsof
Find the process using lsof
The -i flag filters by network connection. Replace 3000 with whatever port is blocked.
sudo lsof -i :3000
Typical output looks like this:
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
node 18234 deploy 22u IPv4 184028 0t0 TCP *:3000 (LISTEN)
The second column is the PID you will need to kill the process. The USER column tells you which account owns it.
Find the process using ss
ss is faster than lsof on busy systems and is the modern replacement for netstat. Use it when you need a quick overview or when lsof is not installed.
sudo ss -ltnp | grep ':3000'
Flags explained: -l shows listening sockets only, -t filters to TCP, -n shows numeric ports instead of resolving service names, -p shows the owning process.
For UDP services such as DNS resolvers or syslog listeners, swap -t for -u.
sudo ss -lunp | grep ':53'
Identify the process owner and PID safely
Before killing anything, confirm what the process actually is. Killing the wrong PID can take down an unrelated service, a database, or even an SSH session if port numbers happen to collide.
ps -fp 18234
This shows the full command line, parent process, and the user that launched it. If the owner is a system account like postgres or www-data, stopping it with kill directly can leave the service in an inconsistent state. In that case prefer the service's own stop command, for example sudo systemctl stop postgresql, over killing the PID manually.
Note: If the process is owned by root or a system user different from yours, you need sudo to inspect or kill it. Never blindly sudo-kill PIDs you have not identified with ps first.
Kill the process with kill
Once you have confirmed the PID is safe to terminate, send a graceful termination signal first.
kill -15 18234
-15 is SIGTERM, which asks the process to shut down cleanly, closing file handles and finishing in-flight requests. Wait a few seconds and check it is gone.
ps -p 18234
If the process ignores SIGTERM and is still running, escalate to SIGKILL, which forces an immediate stop with no cleanup.
kill -9 18234
Warning: SIGKILL does not let the process flush buffers, close database connections, or release locks. Only use it after SIGTERM has failed, and never on a database process without checking its own shutdown procedure first.
Kill by name with pkill
When multiple processes share a name, for example several stale node workers, killing by PID one at a time is slow. pkill matches by process name or command line pattern.
pkill -15 -f 'node server.js'
The -f flag matches against the full command line rather than just the process name, which avoids accidentally killing unrelated processes that happen to be named node.
Always do a dry run first with pgrep to see what would be matched before you kill anything.
pgrep -fa 'node server.js'
Kill by port directly with fuser
fuser lets you kill whatever is bound to a port without looking up the PID manually. This is the fastest one-liner for a quick fix in development.
sudo fuser -k 3000/tcp
This sends SIGKILL by default to every process using that port. For a graceful shutdown, send SIGTERM explicitly instead.
sudo fuser -k -TERM 3000/tcp
Warning: fuser -k kills every process attached to that port, including ones you may not have intended to stop. Run sudo fuser -v 3000/tcp first to see what is actually listening before killing.
Prevent conflicts with SO_REUSEADDR
A common cause of repeated address already in use errors is a socket stuck in TIME_WAIT after a restart. Most frameworks and languages let you enable SO_REUSEADDR so the OS allows immediate rebinding to a port in that state.
In Node.js with the built-in net or http module, this is handled automatically on most Linux distributions. For custom C or Python socket servers, set it explicitly before binding.
import socket
sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
sock.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
sock.bind(('0.0.0.0', 8080))
sock.listen(5)
This does not bypass a genuinely active listener. It only allows rebinding to a socket that is winding down in TIME_WAIT, which is the state you see most often right after a restart.
Prevent conflicts with systemd socket activation
For services managed by systemd, socket activation lets systemd own the listening socket and hand it to your application on demand. This avoids race conditions where two instances of your app try to bind the same port during a restart.
[Unit]
Description=Socket for myapp
[Socket]
ListenStream=8080
[Install]
WantedBy=sockets.target
[Unit]
Description=My application
Requires=myapp.socket
[Service]
User=myapp
ExecStart=/usr/local/bin/myapp
Restart=on-failure
[Install]
WantedBy=multi-user.target
Enable the socket unit, not the service directly. systemd will start the service automatically the first time a connection arrives on the socket.
sudo systemctl daemon-reload
sudo systemctl enable --now myapp.socket
If you run multiple backend instances behind a reverse proxy, you may also want to look at configuring systemd user services for application startup to manage per-user app instances cleanly.
Check listening ports on startup
Add a quick habit of checking what is already listening before you start a new service, especially on shared development boxes.
sudo ss -ltnp
For a persistent audit trail, you can also script a check that runs at boot and logs all listening ports to a file for later review.
sudo ss -ltnp > /var/log/listening-ports-$(date +%Y%m%d).log
If you want this automated on a schedule rather than a one-off check, configuring cron jobs and system task scheduling covers how to run checks like this reliably.
Verify your setup
After killing the process, confirm the port is actually free before starting your application again.
sudo ss -ltnp | grep ':3000'
sudo lsof -i :3000
Both commands should return no output if the port is clear. Then start your service and verify it binds successfully.
sudo systemctl status myapp
curl -I http://127.0.0.1:3000
Common issues
SymptomCauseFixlsof shows nothing but port still fails to bindProcess is bound to IPv6 only, lsof filter missing protocolRun sudo lsof -i6:3000 and sudo lsof -i4:3000 separatelyPermission denied when running lsof or ssProcess owned by another user or system accountPrefix the command with sudo to see root-owned socketskill -9 does not free the port immediatelySocket is in TIME_WAIT, not actually still bound by a live processWait 30-60 seconds, or enable SO_REUSEADDR in the app codePort reopens right after killing the processA supervisor like systemd or PM2 auto-restarts itStop the service unit first: sudo systemctl stop myappfuser reports No process specifiedWrong protocol suffix usedConfirm TCP vs UDP, try sudo fuser -v 3000/tcpMultiple PIDs returned for the same portForked worker processes sharing a listening socketKill the parent PID shown in ps -f --ppid, not individual workers
Next steps
Fix PostgreSQL FATAL sorry, too many clients already
Configure Linux system services with systemctl and service management
Configure systemd user services for application startup
Fix systemd service keeps restarting in a loop
Configure Linux firewall rules with fail2ban
Running this in production?
Want this handled for you? This works for a single server. When you run multiple environments or need services available 24/7 without port conflicts breaking deploys, keeping it healthy is a different job. See how we run infrastructure like this for European teams.
---
### Fix PostgreSQL FATAL sorry, too many clients already
URL: https://binadit.com/tutorials/fix-postgresql-fatal-sorry-too-many-clients-already
Category: linux
Difficulty: beginner
Time: ~20 minutes
Author: Binadit Tech Team
> Learn why PostgreSQL throws FATAL: sorry, too many clients already, how to check and close idle connections, safely raise max_connections, and prevent the error from coming back with connection pooling.
What this solves
PostgreSQL rejects new connections with FATAL: sorry, too many clients already when the number of active sessions hits max_connections. This guide shows you how to diagnose the cause, clear stuck connections, raise the limit safely, and stop it from recurring with pooling and monitoring.
Step-by-step configuration
Understand why the limit exists
PostgreSQL reserves a fixed amount of shared memory and worker processes for each connection slot defined by max_connections. Every client connection, including ones from application servers, cron jobs, and admin tools, consumes one slot for as long as it stays open, even if it is idle.
A few slots are always reserved for the superuser via superuser_reserved_connections, so that an administrator can still log in to diagnose problems even when the pool is full. When all remaining slots are taken, every new connection attempt fails immediately with this FATAL error.
Note: This is not a crash. PostgreSQL is working as designed to protect itself from running out of memory or file descriptors. The fix is almost always in how connections are opened and closed, not in the database engine itself.
Check the current connection count and limit
Connect to PostgreSQL with psql and inspect how many connections are currently open compared to the configured maximum.
sudo -u postgres psql -c "SHOW max_connections;"
sudo -u postgres psql -c "SELECT count(*) FROM pg_stat_activity;"
Compare the two numbers. If the count from pg_stat_activity is close to or equal to max_connections, you have confirmed the cause of the error.
sudo -u postgres psql -c "SELECT datname, usename, client_addr, state, count(*) FROM pg_stat_activity GROUP BY datname, usename, client_addr, state ORDER BY count(*) DESC;"
Identify idle or leaked connections
Most connection exhaustion comes from applications that open a connection per request and never close it, or from connection pools that are misconfigured with a pool size larger than the database can handle. Look specifically at the idle and idle in transaction states.
sudo -u postgres psql -c "SELECT pid, usename, datname, state, now() - state_change AS idle_for, query FROM pg_stat_activity WHERE state LIKE 'idle%' ORDER BY idle_for DESC LIMIT 20;"
Connections stuck in idle in transaction for a long time usually indicate an application bug where a transaction was opened but never committed or rolled back. These hold locks and slots indefinitely.
Warning: Do not kill connections blindly in production. Review the query and client_addr columns first to confirm you are not terminating a long-running legitimate job such as a backup or migration.
Terminate stuck sessions safely
Once you have identified connections that are genuinely idle or stuck, terminate them individually by PID rather than cancelling everything at once.
sudo -u postgres psql -c "SELECT pg_terminate_backend(12345);"
To free up slots quickly during an incident, terminate all connections idle in transaction for more than 10 minutes, but review the list first.
sudo -u postgres psql -c "SELECT pg_terminate_backend(pid) FROM pg_stat_activity WHERE state = 'idle in transaction' AND now() - state_change > interval '10 minutes';"
This buys you immediate headroom while you fix the root cause in the application or pooler configuration.
Increase max_connections
If legitimate traffic genuinely needs more concurrent connections than the current limit allows, raise max_connections in the PostgreSQL configuration file. Each additional connection consumes roughly a few megabytes of shared memory, so increase gradually and monitor memory usage.
sudo -u postgres psql -c "SHOW config_file;"
Edit the file returned by the command above, typically located under /etc/postgresql/17/main/postgresql.conf on Debian based systems or /var/lib/pgsql/17/data/postgresql.conf on RHEL based systems.
max_connections = 200
shared_buffers = 2GB
superuser_reserved_connections = 5
A common rule of thumb is to keep shared_buffers around 25% of total RAM and scale it up slightly as max_connections increases, since each connection needs its own memory for sorting and work operations.
Raise kernel limits to match
Higher max_connections values require more file descriptors and more shared memory at the OS level. Check and raise the open file limit for the postgres user.
sudo -u postgres bash -c 'ulimit -n'
postgres soft nofile 65536
postgres hard nofile 65536
postgres soft nproc 4096
postgres hard nproc 4096
If PostgreSQL runs under systemd, also raise the limit in the service unit override so it takes effect even if limits.conf is ignored.
sudo systemctl edit postgresql
[Service]
LimitNOFILE=65536
Check shared memory settings on the OS
On some installations, especially AlmaLinux and Rocky Linux with default kernel settings, the shared memory limits are too low for a large max_connections plus shared_buffers combination. Check the current kernel shared memory maximum.
sysctl kernel.shmmax kernel.shmall
sysctl kernel.shmmax kernel.shmall
If shmmax is lower than the memory PostgreSQL needs to allocate, increase it via sysctl.
kernel.shmmax = 17179869184
kernel.shmall = 4194304
sudo sysctl --system
Restart PostgreSQL safely
max_connections and shared_buffers both require a full restart, not just a reload, because they affect shared memory allocation at startup. Schedule this during a maintenance window and verify there are no critical queries running first.
sudo -u postgres psql -c "SELECT pid, state, query FROM pg_stat_activity WHERE state != 'idle';"
sudo systemctl restart postgresql
Check the PostgreSQL logs immediately after restart to confirm it came back up cleanly with the new settings.
sudo journalctl -u postgresql -n 50 --no-pager
Verify your setup
sudo -u postgres psql -c "SHOW max_connections;"
sudo -u postgres psql -c "SELECT count(*) AS active, (SELECT setting::int FROM pg_settings WHERE name='max_connections') AS max FROM pg_stat_activity;"
Confirm the new max_connections value is active and that current usage sits comfortably below it. Also confirm PostgreSQL is accepting new connections from your application.
psql -h 203.0.113.10 -U appuser -d appdb -c "SELECT 1;"
Prevent recurrence with connection pooling
Raising max_connections only buys time. The real fix for most applications is to route connections through a pooler so the database only ever sees a small, stable number of backend connections regardless of how many application workers or web requests exist.
PgBouncer sits between your application and PostgreSQL, reusing a small pool of real database connections across many client connections. This is the standard production pattern and avoids the memory overhead of scaling max_connections into the thousands.
See Install and configure PgBouncer for PostgreSQL connection pooling for a full setup guide, or Configure PostgreSQL 17 connection pooling with PgBouncer for high availability if you are running a replicated setup.
Note: If your application uses an ORM or framework-level connection pool such as SQLAlchemy or Django, make sure its pool size multiplied by the number of application workers does not exceed max_connections minus the connections your pooler and admin tools need.
Monitor connections to catch this before it happens again
Set up alerting on connection count as a percentage of max_connections so you get paged before clients start seeing FATAL errors, not after. Track it alongside query latency and lock waits.
See Monitor PostgreSQL performance with Prometheus and Grafana dashboards and Monitor PostgreSQL performance with pg_stat_statements extension for query analysis and optimization to build out proper visibility into connection usage and slow queries.
Common issues
SymptomCauseFixFATAL: sorry, too many clients already on every new connectionmax_connections slots fully consumedTerminate idle sessions, then raise max_connections and restart PostgreSQLError persists even after terminating idle sessionsApplication keeps opening new connections faster than old ones closeAdd connection pooling with PgBouncer in transaction modePostgreSQL fails to start after raising max_connectionsKernel shared memory limit too low for the new shared_buffers and max_connections combinationRaise kernel.shmmax and kernel.shmall via sysctl, then restartMany connections stuck in idle in transactionApplication opens a transaction but never commits or rolls backFix transaction handling in application code, set idle_in_transaction_session_timeoutSuperuser also locked outsuperuser_reserved_connections exhausted alongside regular slotsIncrease superuser_reserved_connections slightly and always keep an emergency admin path
Next steps
Install and configure PgBouncer for PostgreSQL connection pooling
Configure PostgreSQL 17 connection pooling with PgBouncer for high availability
Monitor PostgreSQL performance with Prometheus and Grafana dashboards
Configure PostgreSQL 17 SSL encryption and certificate-based authentication
Tune PostgreSQL memory and autovacuum settings for production workloads
Running this in production?
Want this handled for you? This works for a single server. When you run multiple environments or need PostgreSQL available 24/7 under real traffic, catching connection exhaustion before it pages your users is a different job. See how we run infrastructure like this for European teams.
---
### Configure Istio multi-cluster disaster recovery and automated failover
URL: https://binadit.com/tutorials/configure-istio-multi-cluster-disaster-recovery
Category: devops
Difficulty: advanced
Time: ~90 minutes
Author: Binadit Tech Team
> Build a production-grade Istio primary-remote multi-cluster topology with shared trust, locality-aware failover, and automated recovery testing. Covers cross-cluster service discovery, Kiali and Prometheus monitoring, and control plane backup procedures.
What this solves
A single-cluster Istio mesh is a single point of failure. This tutorial configures a primary-remote multi-cluster topology spanning two Kubernetes clusters with shared trust, cross-cluster service discovery and locality-aware load balancing so traffic automatically fails over when a cluster or region goes down.
You will build the full trust chain, test real outage scenarios, wire up monitoring with Kiali and Prometheus, and establish backup and recovery procedures for the control plane configuration itself.
Note: This tutorial assumes two existing Kubernetes clusters (cluster-east and cluster-west) with network connectivity between pod CIDRs, either via VPC peering, a flat network, or a VPN mesh such as WireGuard multi-site mesh networking.
Prerequisites and multi-cluster architecture overview
Before starting, confirm each cluster meets the baseline requirements. Both clusters need direct pod-to-pod IP routing (no overlapping CIDRs), a shared root CA for mTLS trust, and API server access from your workstation via distinct kubeconfig contexts.
RequirementDetailsKubernetes version1.28 or newer on both clustersIstio version1.22 or newer (istioctl and control plane must match)NetworkNon-overlapping pod/service CIDRs, routable between clustersDNSCluster API servers resolvable from each other for cross-cluster secretsCAShared intermediate CA or common root for mesh-wide mTLS trust
The topology used here is primary-remote: cluster-east runs a full Istio control plane (istiod), and cluster-west runs a remote configuration that uses cluster-east's istiod for configuration while running its own data plane. This is simpler to operate than primary-primary and is sufficient for most disaster recovery requirements.
Install istioctl and verify cluster access
Install the Istio CLI and confirm both kubeconfig contexts are reachable before making any changes.
curl -L https://istio.io/downloadIstio | ISTIO_VERSION=1.23.2 sh -
sudo mv istio-1.23.2/bin/istioctl /usr/local/bin/
istioctl version --remote=false
kubectl config get-contexts
kubectl --context=cluster-east get nodes
kubectl --context=cluster-west get nodes
Setting up primary-remote multi-cluster topology with shared trust domain
Generate a shared root CA
Every cluster in the mesh must trust the same root certificate authority so cross-cluster mTLS connections succeed. Generate an intermediate CA per cluster signed by one shared root.
mkdir -p ~/istio-ca/certs && cd ~/istio-ca/certs
curl -sL https://raw.githubusercontent.com/istio/istio/release-1.23/tools/certs/Makefile.selfsigned.mk -o Makefile
make -f Makefile root-ca
make -f Makefile cluster-east-cacerts
make -f Makefile cluster-west-cacerts
Install the CA secrets into each cluster
The istio-system namespace must exist before the cacerts secret is created, since istiod reads this secret at startup to establish the mesh trust domain.
kubectl --context=cluster-east create namespace istio-system
kubectl --context=cluster-east create secret generic cacerts -n istio-system \
--from-file=cluster-east/ca-cert.pem \
--from-file=cluster-east/ca-key.pem \
--from-file=cluster-east/root-cert.pem \
--from-file=cluster-east/cert-chain.pem
kubectl --context=cluster-west create namespace istio-system
kubectl --context=cluster-west create secret generic cacerts -n istio-system \
--from-file=cluster-west/ca-cert.pem \
--from-file=cluster-west/ca-key.pem \
--from-file=cluster-west/root-cert.pem \
--from-file=cluster-west/cert-chain.pem
Label clusters with network and topology metadata
Istio uses the topology.istio.io/network label to determine which endpoints require the east-west gateway versus direct pod routing.
kubectl --context=cluster-east label namespace istio-system topology.istio.io/network=network-east
kubectl --context=cluster-west label namespace istio-system topology.istio.io/network=network-west
Install the primary cluster control plane
Cluster-east becomes the primary with a full istiod deployment, tagged with its mesh, network and cluster identifiers.
apiVersion: install.istio.io/v1alpha1
kind: IstioOperator
spec:
values:
global:
meshID: mesh1
multiCluster:
clusterName: cluster-east
network: network-east
istioctl install --context=cluster-east -f /tmp/cluster-east.yaml -y
Deploy the east-west gateway on the primary
The east-west gateway exposes istiod and cross-cluster service endpoints over mTLS so cluster-west can reach services in cluster-east.
cd ~/istio-1.23.2
samples/multicluster/gen-eastwest-gateway.sh --mesh mesh1 --cluster cluster-east --network network-east | \
istioctl --context=cluster-east install -y -f -
kubectl --context=cluster-east apply -n istio-system -f samples/multicluster/expose-istiod.yaml
kubectl --context=cluster-east apply -n istio-system -f samples/multicluster/expose-services.yaml
Install the remote cluster configuration
Cluster-west gets the east-west gateway plus a remote config that points it at cluster-east's exposed istiod service.
samples/multicluster/gen-eastwest-gateway.sh --mesh mesh1 --cluster cluster-west --network network-west | \
istioctl --context=cluster-west install -y -f -
apiVersion: install.istio.io/v1alpha1
kind: IstioOperator
spec:
profile: remote
values:
istiodRemote:
injectionPath: /inject/cluster/cluster-west/net/network-west
global:
remotePilotAddress: 203.0.113.10
meshID: mesh1
multiCluster:
clusterName: cluster-west
network: network-west
istioctl install --context=cluster-west -f /tmp/cluster-west.yaml -y
Warning: Replace 203.0.113.10 with the actual external IP of the cluster-east east-west gateway LoadBalancer service. Using the wrong address breaks remote config sync silently.
Install a cross-cluster remote secret
Cluster-east's istiod needs a kubeconfig secret for cluster-west so it can watch remote endpoints and services.
istioctl create-remote-secret --context=cluster-west --name=cluster-west | \
kubectl apply -f - --context=cluster-east
Configuring cross-cluster service discovery and endpoint synchronization
With the remote secret installed, istiod on cluster-east aggregates endpoints from both clusters into a single service registry. Any workload with a matching Service name and namespace across clusters is automatically treated as a single logical service with endpoints in both locations.
Deploy a sample multi-cluster service
Deploy identical Deployment and Service manifests to both clusters. Istio merges the endpoints transparently.
apiVersion: v1
kind: Service
metadata:
name: helloworld
labels:
app: helloworld
spec:
ports:
- port: 5000
name: http
selector:
app: helloworld
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: helloworld-v1
spec:
replicas: 1
selector:
matchLabels:
app: helloworld
template:
metadata:
labels:
app: helloworld
version: v1
spec:
containers:
- name: helloworld
image: docker.io/istio/examples-helloworld-v1
ports:
- containerPort: 5000
kubectl --context=cluster-east create namespace sample
kubectl --context=cluster-east label namespace sample istio-injection=enabled
kubectl --context=cluster-east apply -n sample -f /tmp/helloworld.yaml
kubectl --context=cluster-west create namespace sample
kubectl --context=cluster-west label namespace sample istio-injection=enabled
kubectl --context=cluster-west apply -n sample -f /tmp/helloworld.yaml
Confirm endpoint synchronization
Query istiod's debug endpoint to confirm it sees endpoints from both clusters for the same service.
istioctl --context=cluster-east proxy-config endpoint deploy/helloworld-v1.sample -n sample | grep 5000
Implementing locality-aware load balancing and failover policies
Istio's locality load balancing prefers endpoints in the same region and zone as the client, falling back to remote clusters only when local endpoints become unhealthy. This requires outlier detection to be enabled and locality labels to be set correctly on nodes.
Verify node locality labels
Istio reads region and zone from standard Kubernetes topology labels on each node.
kubectl --context=cluster-east get nodes -L topology.kubernetes.io/region,topology.kubernetes.io/zone
kubectl --context=cluster-west get nodes -L topology.kubernetes.io/region,topology.kubernetes.io/zone
Create a DestinationRule with locality failover
This policy sends traffic to the local cluster's region first and automatically fails over to the remote region if local endpoints are ejected by outlier detection.
apiVersion: networking.istio.io/v1beta1
kind: DestinationRule
metadata:
name: helloworld
namespace: sample
spec:
host: helloworld.sample.svc.cluster.local
trafficPolicy:
connectionPool:
tcp:
maxConnections: 100
outlierDetection:
consecutive5xxErrors: 3
interval: 10s
baseEjectionTime: 30s
maxEjectionPercent: 100
loadBalancer:
localityLbSetting:
enabled: true
failover:
- from: eu-west-1
to: eu-central-1
kubectl --context=cluster-east apply -f /tmp/helloworld-dr.yaml
kubectl --context=cluster-west apply -f /tmp/helloworld-dr.yaml
Note: outlierDetection is mandatory for failover to trigger. Without it, Istio has no signal that local endpoints are unhealthy and will keep sending traffic to a dead cluster.
Set a failover priority with ServiceEntry (optional for external services)
If part of your mesh talks to external endpoints, define locality priority there too so DR behavior is consistent mesh-wide.
kubectl --context=cluster-east get destinationrule helloworld -n sample -o yaml
This pattern complements traffic-splitting strategies covered in Istio multi-cluster canary deployments and pairs well with the resilience patterns in Istio circuit breaker and retry policies.
Testing automated failover scenarios and cluster outage simulation
Generate baseline traffic
Use a simple loop from a client pod to confirm traffic normally stays local to cluster-east.
kubectl --context=cluster-east run sleep --image=curlimages/curl -n sample -- sleep infinity
kubectl --context=cluster-east exec -n sample sleep -- sh -c "for i in $(seq 1 20); do curl -s helloworld.sample:5000/hello; done"
Simulate a cluster-east outage
Scale the local deployment to zero to simulate cluster-east becoming unavailable, then confirm traffic fails over to cluster-west.
kubectl --context=cluster-east scale deploy helloworld-v1 -n sample --replicas=0
kubectl --context=cluster-east exec -n sample sleep -- sh -c "for i in $(seq 1 20); do curl -s helloworld.sample:5000/hello; done"
Responses should now come from version v1 running in cluster-west. Restore the deployment once the test is complete.
kubectl --context=cluster-east scale deploy helloworld-v1 -n sample --replicas=1
Simulate a full network partition
A harder test is severing the east-west gateway connection itself rather than just the workload, which exercises the outlier detection path end to end.
kubectl --context=cluster-east scale deploy istio-eastwestgateway -n istio-system --replicas=0
kubectl --context=cluster-east exec -n sample sleep -- sh -c "for i in $(seq 1 20); do curl -s -w ' -> %{http_code}\n' helloworld.sample:5000/hello; done"
kubectl --context=cluster-east scale deploy istio-eastwestgateway -n istio-system --replicas=1
Monitoring multi-cluster health with Kiali and Prometheus
Install Kiali and Prometheus addons on the primary
Kiali visualizes cross-cluster traffic flow, while Prometheus scrapes per-cluster mesh metrics for alerting.
kubectl --context=cluster-east apply -f samples/addons/prometheus.yaml
kubectl --context=cluster-east apply -f samples/addons/kiali.yaml
kubectl --context=cluster-east rollout status deployment/kiali -n istio-system
Configure remote cluster metrics scraping
Add cluster-west as a remote Prometheus target so dashboards show both clusters in one pane.
kubectl --context=cluster-west get svc istiod -n istio-system
kubectl --context=cluster-east port-forward svc/kiali 20001:20001 -n istio-system
Open the Kiali dashboard and confirm both cluster-east and cluster-west appear under the mesh graph with cross-cluster edges between helloworld instances.
For alerting on failover events and endpoint health degradation, extend this setup with Prometheus federation for multi-cluster monitoring and route notifications through Alertmanager webhook integrations.
Backup and recovery procedures for Istio control plane configuration
Back up IstioOperator manifests and CRDs
The control plane's desired state lives in your IstioOperator YAML and any custom Istio resources (VirtualService, DestinationRule, Gateway). Export all of it regularly.
mkdir -p ~/istio-backup/$(date +%F)
cd ~/istio-backup/$(date +%F)
kubectl --context=cluster-east get istiooperator -A -o yaml > istio-operator-east.yaml
kubectl --context=cluster-east get virtualservices,destinationrules,gateways,serviceentries -A -o yaml > istio-resources-east.yaml
kubectl --context=cluster-west get virtualservices,destinationrules,gateways,serviceentries -A -o yaml > istio-resources-west.yaml
Back up CA material securely
The cacerts secret is the trust root for the entire mesh. Losing it means every workload certificate must be reissued. Export it to an encrypted archive, never in plain text.
kubectl --context=cluster-east get secret cacerts -n istio-system -o yaml > cacerts-east.yaml
gpg --symmetric --cipher-algo AES256 cacerts-east.yaml
rm cacerts-east.yaml
Store the resulting cacerts-east.yaml.gpg in the same offsite location used for your other encrypted secrets, following the key rotation practices in backup encryption key rotation and management.
Automate the backup with a systemd timer
Schedule the export to run daily so configuration drift never outpaces your last known-good backup.
[Unit]
Description=Istio control plane configuration backup
[Service]
Type=oneshot
ExecStart=/usr/local/bin/istio-backup.sh
[Unit]
Description=Daily Istio config backup
[Timer]
OnCalendar=daily
Persistent=true
[Install]
WantedBy=timers.target
sudo systemctl daemon-reload
sudo systemctl enable --now istio-backup.timer
Restore procedure
To rebuild a destroyed cluster, reinstall cacerts first, then reapply the IstioOperator manifest, then reapply custom resources.
gpg --decrypt cacerts-east.yaml.gpg > cacerts-east.yaml
kubectl --context=cluster-east apply -f cacerts-east.yaml
istioctl install --context=cluster-east -f istio-operator-east.yaml -y
kubectl --context=cluster-east apply -f istio-resources-east.yaml
Troubleshooting split-brain and network partition scenarios
Split-brain in a service mesh usually means both clusters believe they are authoritative and traffic is split unpredictably rather than cleanly failing over. This most often happens when the remote secret or east-west gateway is partially healthy.
Check remote secret validity
A stale or expired remote secret causes cluster-east's istiod to silently stop syncing cluster-west endpoints while reporting healthy status.
kubectl --context=cluster-east get secret -n istio-system -l istio/multiCluster=true
kubectl --context=cluster-east logs -n istio-system deploy/istiod --tail=100 | grep -i remote
Inspect endpoint consistency across clusters
Compare what each istiod instance believes about the mesh. Divergence here indicates a partition rather than a clean failover.
istioctl --context=cluster-east proxy-status
istioctl --context=cluster-west proxy-status
For deep network-level diagnosis during a partition, correlate Istio's view with raw packet flow using
---
### Set up automated MariaDB Galera cluster backups with compression
URL: https://binadit.com/tutorials/setup-automated-mariadb-galera-backup-with-compression
Category: databases
Difficulty: intermediate
Time: ~40 minutes
Author: Binadit Tech Team
> Learn how to build a cluster-safe backup strategy for MariaDB Galera using Mariabackup, compression, systemd timers and remote storage offloading, with verification and node restore procedures.
What this solves
Galera cluster nodes share the same dataset through synchronous replication, but that does not mean backups are optional. A bad schema migration, an accidental DELETE, or a corrupted table replicates to every node instantly. This tutorial builds a repeatable, compressed, automated backup strategy using Mariabackup, with rotation, integrity checks and offsite storage.
Note: This tutorial assumes you already have a working Galera cluster. If you need to set one up first, see Configure MariaDB Galera cluster for multi-master replication.
Choosing a backup strategy for Galera clusters
mysqldump produces logical SQL dumps. It is portable and human-readable, but slow to restore on large datasets and it locks tables during the dump unless you use single-transaction mode, which does not fully guarantee consistency on a busy Galera node.
Mariabackup performs physical, hot backups of InnoDB data files without blocking writes. It understands Galera's gtid_binlog_state and wsrep_position, which lets a restored node rejoin the cluster cleanly via State Snapshot Transfer (SST) or by seeding a fresh node. For clusters beyond a few gigabytes, Mariabackup is the right default.
CriteriaMariabackupmysqldumpBackup speed on large datasetsFast, file-level copySlow, row-by-row exportRestore speedFast, direct file copySlow, replays SQL statementsLocking impact on clusterNone, uses InnoDB redo log trackingCan lock tables brieflySelective table/database restoreHarder, requires partial backup modeEasy, just re-import specific tablesBest use caseFull cluster or node backupsSchema migrations, small exports, dev seeding
In practice, most production Galera setups run Mariabackup as the primary backup mechanism and keep mysqldump available for occasional logical exports of specific schemas.
Step-by-step configuration
Install Mariabackup
Mariabackup ships as part of the MariaDB backup package. Install it on the node you plan to run backups from, ideally a non-primary node to avoid extra load on the write path.
sudo apt update
sudo apt install -y mariadb-backup gzip zstd
sudo dnf install -y MariaDB-backup gzip zstd
Create a dedicated backup user
Avoid using the root MariaDB account in scripts. Create a user scoped to only the privileges Mariabackup needs.
sudo mysql -u root -p
CREATE USER 'backupuser'@'localhost' IDENTIFIED BY 'Kx9#mPz2Trq!Lw7v';
GRANT RELOAD, LOCK TABLES, PROCESS, REPLICATION CLIENT, BACKUP_ADMIN ON *.* TO 'backupuser'@'localhost';
FLUSH PRIVILEGES;
EXIT;
Warning: Use a strong, unique password and store it outside the script itself using a credentials file with restricted permissions, shown in the next step.
Store credentials securely
Create a MariaDB option file readable only by the backup script's user, instead of embedding the password in the script or command history.
sudo mkdir -p /etc/mysql/backup
sudo tee /etc/mysql/backup/.mycnf > /dev/null <<'EOF'
[client]
user=backupuser
password=Kx9#mPz2Trq!Lw7v
EOF
sudo chown root:root /etc/mysql/backup/.mycnf
sudo chmod 600 /etc/mysql/backup/.mycnf
Setting permissions to 600 means only the file owner, root in this case, can read or write it. This prevents other local users from reading the database password.
Create the backup directory structure
Separate raw backups from compressed archives, and keep a dedicated log directory.
sudo mkdir -p /var/backups/mariadb/{staging,archives,logs}
sudo chown -R mysql:mysql /var/backups/mariadb
sudo chmod 750 /var/backups/mariadb
sudo chmod 750 /var/backups/mariadb/staging /var/backups/mariadb/archives /var/backups/mariadb/logs
The mysql system user needs to read and write here since Mariabackup runs under that account during the backup. 750 permissions allow the owner full access and the group read/execute, while blocking all other users.
Write the backup script with compression and rotation
This script takes a full Mariabackup snapshot, streams it through zstd for fast compression, timestamps the archive, and deletes archives older than the retention window.
#!/bin/bash
set -euo pipefail
BACKUP_USER_CNF="/etc/mysql/backup/.mycnf"
STAGING_DIR="/var/backups/mariadb/staging"
ARCHIVE_DIR="/var/backups/mariadb/archives"
LOG_DIR="/var/backups/mariadb/logs"
RETENTION_DAYS=7
TIMESTAMP=$(date +%Y%m%d-%H%M%S)
BACKUP_NAME="galera-backup-${TIMESTAMP}"
LOG_FILE="${LOG_DIR}/${BACKUP_NAME}.log"
mkdir -p "${STAGING_DIR}/${BACKUP_NAME}"
echo "[$(date)] Starting Mariabackup for ${BACKUP_NAME}" >> "${LOG_FILE}"
mariabackup --defaults-extra-file="${BACKUP_USER_CNF}" \
--backup \
--target-dir="${STAGING_DIR}/${BACKUP_NAME}" \
>> "${LOG_FILE}" 2>&1
mariabackup --defaults-extra-file="${BACKUP_USER_CNF}" \
--prepare \
--target-dir="${STAGING_DIR}/${BACKUP_NAME}" \
>> "${LOG_FILE}" 2>&1
echo "[$(date)] Compressing backup with zstd" >> "${LOG_FILE}"
tar -cf - -C "${STAGING_DIR}" "${BACKUP_NAME}" | zstd -T0 -19 -o "${ARCHIVE_DIR}/${BACKUP_NAME}.tar.zst"
rm -rf "${STAGING_DIR}/${BACKUP_NAME}"
echo "[$(date)] Rotating archives older than ${RETENTION_DAYS} days" >> "${LOG_FILE}"
find "${ARCHIVE_DIR}" -name "galera-backup-*.tar.zst" -mtime +${RETENTION_DAYS} -delete
find "${LOG_DIR}" -name "galera-backup-*.log" -mtime +${RETENTION_DAYS} -delete
echo "[$(date)] Backup ${BACKUP_NAME} completed successfully" >> "${LOG_FILE}"
zstd at level 19 with multi-threading (-T0) gives strong compression without the long runtimes of gzip -9 on multi-gigabyte datasets. If you prefer gzip for compatibility with older tooling, replace the compression line with tar -czf "${ARCHIVE_DIR}/${BACKUP_NAME}.tar.gz" -C "${STAGING_DIR}" "${BACKUP_NAME}".
Set correct ownership and permissions on the script
The script needs to run as the mysql user since that account owns the data directory and staging path. It should not be world-writable.
sudo chown root:mysql /usr/local/bin/galera-backup.sh
sudo chmod 750 /usr/local/bin/galera-backup.sh
Never use chmod 777. It gives every user on the system full read, write and execute access to your backup script, including the ability to modify it to exfiltrate credentials or delete backups. Ownership by root with group execute for mysql, combined with 750 permissions, is the minimum access needed.
Test the script manually
Run it once by hand before automating it, so you can confirm output and catch permission errors early.
sudo -u mysql /usr/local/bin/galera-backup.sh
ls -lh /var/backups/mariadb/archives
Automating backups with systemd timers
systemd timers give you better logging, dependency management and failure visibility than plain cron, and they integrate cleanly with journalctl. If you already manage scheduled jobs with cron elsewhere, the same rotation principles apply, see Implement backup rotation policies with automated cleanup.
Create the systemd service unit
This defines what runs, and as which user.
[Unit]
Description=Galera cluster backup with Mariabackup
After=mariadb.service
Requires=mariadb.service
[Service]
Type=oneshot
User=mysql
Group=mysql
ExecStart=/usr/local/bin/galera-backup.sh
Nice=10
IOSchedulingClass=best-effort
IOSchedulingPriority=7
Nice and IOSchedulingClass keep the backup from starving the mysqld process of CPU and disk I/O during peak hours.
Create the systemd timer unit
This schedules the service to run nightly at 02:15, with a randomized delay to avoid every cluster node hitting disk I/O at the exact same second.
[Unit]
Description=Run Galera backup nightly
[Timer]
OnCalendar=*-*-* 02:15:00
RandomizedDelaySec=600
Persistent=true
[Install]
WantedBy=timers.target
Enable and start the timer
sudo systemctl daemon-reload
sudo systemctl enable --now galera-backup.timer
systemctl list-timers galera-backup.timer
Stagger schedules across cluster nodes
Running full backups on every node simultaneously wastes disk I/O and network bandwidth for no benefit, since the data is identical. Pick one designated backup node per cluster, and only enable the timer there. Keep the package installed on other nodes as a manual fallback.
sudo systemctl disable --now galera-backup.timer
Run this on the nodes that should not perform scheduled backups. If your designated backup node goes down, you can enable the timer on a secondary node temporarily.
Verifying backup integrity and restoring a node
Verify archive integrity
Confirm the compressed archive is not truncated or corrupted before you rely on it.
zstd -t /var/backups/mariadb/archives/galera-backup-20250601-021500.tar.zst
echo $?
An exit code of 0 means the archive passed integrity checks. Any other value means the file is corrupted and you should investigate the backup job's logs.
Extract and prepare for restore
Decompress into a scratch location, separate from the live data directory.
mkdir -p /tmp/restore-test
zstd -d /var/backups/mariadb/archives/galera-backup-20250601-021500.tar.zst -o /tmp/restore-test/backup.tar
tar -xf /tmp/restore-test/backup.tar -C /tmp/restore-test
Restore a failed or new node
Stop MariaDB, clear the existing data directory and copy the prepared backup into place. Do this only on a node you are rebuilding, never on a healthy node still serving traffic.
sudo systemctl stop mariadb
sudo mv /var/lib/mysql /var/lib/mysql.bak
sudo mkdir /var/lib/mysql
sudo mariabackup --copy-back --target-dir=/tmp/restore-test/galera-backup-20250601-021500
sudo chown -R mysql:mysql /var/lib/mysql
sudo chmod 750 /var/lib/mysql
Warning: Do not run this against a node that is currently part of an active Galera quorum with good data. This procedure is for rebuilding a lost or corrupted node only.
Start the restored node and rejoin the cluster
Start MariaDB and confirm it rejoins via SST or reports a healthy wsrep state.
sudo systemctl start mariadb
sudo mysql -u root -p -e "SHOW STATUS LIKE 'wsrep_cluster_size';"
sudo mysql -u root -p -e "SHOW STATUS LIKE 'wsrep_local_state_comment';"
You want wsrep_local_state_comment to report Synced. For full cluster setup details, see Configure MariaDB Galera cluster 10.11 for multi-master replication.
Offloading backups to remote or S3-compatible storage
Local backups protect against data corruption, but not against disk failure or a compromised host. Offload archives to a remote location as part of the same automation.
Install the S3-compatible client
sudo apt install -y awscli
Configure remote credentials
Store credentials in a restricted profile file, not inline in scripts.
sudo -u mysql aws configure --profile galera-backup
Provide your access key, secret key, and region when prompted. This writes to ~mysql/.aws/credentials, which should already inherit restrictive permissions from the mysql home directory.
Add the upload step to the backup script
Append this block to the end of the backup script, after the rotation step.
echo "[$(date)] Uploading archive to remote storage" >> "${LOG_FILE}"
aws s3 cp "${ARCHIVE_DIR}/${BACKUP_NAME}.tar.zst" \
s3://example-galera-backups/nightly/ \
--profile galera-backup \
--endpoint-url https://s3.eu-central-1.example.com \
>> "${LOG_FILE}" 2>&1
echo "[$(date)] Remote upload complete" >> "${LOG_FILE}"
Point --endpoint-url at your actual S3-compatible provider. For a self-hosted alternative, see Setup S3-compatible disaster recovery with cross-region replication using MinIO.
Encrypt archives before upload for sensitive workloads
If your data falls under GDPR or contains customer PII, encrypt the archive before it leaves the host, rather than relying solely on transport encryption.
gpg --batch --yes --passphrase-file /etc/mysql/backup/.gpgpass \
--cipher-algo AES256 \
--symmetric \
--output "${ARCHIVE_DIR}/${BACKUP_NAME}.tar.zst.gpg" \
"${ARCHIVE_DIR}/${BACKUP_NAME}.tar.zst"
For a full walkthrough of key rotation and secure passphrase handling, see Implement MariaDB backup encryption with Mariabackup and automated restoration.
Verify your setup
systemctl list-timers galera-backup.timer
journalctl -u galera-backup.service --since "1 day ago"
ls -lh /var/backups/mariadb/archives
aws s3 ls s3://example-galera-backups/nightly/ --profile galera-backup --endpoint-url https://s3.eu-central-1.example.com
Confirm at least one successful run exists in the journal, the archive directory contains recent files matching your retention window, and the remote bucket lists the same file count.
Common issues
SymptomCauseFixMariabackup fails with access deniedbackupuser missing BACKUP_ADMIN or RELOAD privilegeRe-run the GRANT statement and FLUSH PRIVILEGESBackup succeeds but restore fails to start mariadbPrepare step was skipped or failed silentlyRe-run mariabackup --prepare against the extracted backup before copy-backTimer never triggersTimer enabled but not started, or unit file typosystemctl status galera-backup.timer and check for load errorsArchive directory fills up diskRetention find command not matching filenamesCheck filename pattern matches exactly what the script producesS3 upload fails with 403Wrong endpoint URL or expired credentialsRe-run aws configure and verify bucket policy allows PutObjectRestored node stuck at Joining stateCorrupted backup or mismatched Galera versionVerify archive with zstd -t before restore, confirm mariadb-server version matches cluster
Next steps
Configure MariaDB Galera cluster for multi-master replication with automatic failover
Implement MariaDB backup encryption with Mariabackup and automated restoration
Configure MariaDB 11.6 performance monitoring with Prometheus and Grafana dashboards
Set up MariaDB backup alerting with Prometheus and Grafana
Set up Galera cluster disaster recovery with cross-region replication
Running this in production?
Want this handled for you? Setting this up once is straightforward. Keeping it patched, monitored, backed up and performant across environments is the harder part. See how we run infrastructure like this for European teams.
---
### Configure MariaDB ColumnStore for high-performance analytics workloads
URL: https://binadit.com/tutorials/configure-mariadb-columnstore-for-analytics-workloads
Category: databases
Difficulty: advanced
Time: ~45 minutes
Author: Binadit Tech Team
> Deploy and tune MariaDB ColumnStore for columnar analytics, covering single-node and multi-node cluster setup, table design, memory tuning, bulk loading with cpimport, and cluster health monitoring.
What this solves
MariaDB ColumnStore is a columnar storage engine built for OLAP workloads: aggregations, scans and joins over billions of rows. This tutorial covers installing ColumnStore, building single-node and multi-node clusters, designing columnar tables, tuning memory and query execution, bulk loading with cpimport, and monitoring cluster health in production.
Note: ColumnStore requires a minimum of 4 CPU cores and 16GB RAM per node for realistic analytics workloads. Test environments can run with less, but production sizing should follow the memory tuning section below.
Step-by-step installation
Prepare the operating system
ColumnStore needs specific kernel parameters, disabled swappiness sensitivity and a dedicated filesystem mount for columnar data. Apply these on every node before installing packages.
vm.swappiness = 1
vm.max_map_count = 262144
net.core.somaxconn = 4096
fs.file-max = 6553600
sudo sysctl --system
Set resource limits
ColumnStore's PrimProc and ExeMgr processes open large numbers of file descriptors during scans. Raise the limits for the mysql user.
mysql soft nofile 65536
mysql hard nofile 65536
mysql soft nproc 32768
mysql hard nproc 32768
Add the MariaDB repository
ColumnStore ships as an engine plugin bundled with the MariaDB Enterprise/Community server packages. Use the official MariaDB repository setup script to pull the correct version for your distro.
curl -LsS https://downloads.mariadb.com/MariaDB/mariadb_repo_setup | sudo bash
sudo apt update
curl -LsS https://downloads.mariadb.com/MariaDB/mariadb_repo_setup | sudo bash
sudo dnf makecache
Install MariaDB server and ColumnStore packages
Install the server along with the columnstore-engine package and its dependencies. This includes the storage manager, distributed metadata layer and cpimport tooling.
sudo apt install -y mariadb-server mariadb-plugin-columnstore boost-* jq
sudo dnf install -y MariaDB-server MariaDB-columnstore-engine boost jq
Warning: Do not mix ColumnStore package versions across nodes in a cluster. Version skew between the storage manager and the DMLProc/ExeMgr layer causes query failures that are hard to diagnose.
Run the ColumnStore post-install configuration
The postConfigure script initializes the system catalog, sets storage locations and, for multi-node setups, registers all module IPs.
sudo systemctl start mariadb-columnstore-cmapi
sudo /usr/bin/mcs cluster start
For a single-node deployment, verify the module is active.
sudo mcs cluster status
Secure the root account and create an analytics user
Run the standard hardening script, then create a dedicated user for ETL and reporting workloads instead of using root for application connections.
sudo mariadb-secure-installation
sudo mariadb -e "CREATE USER 'analytics_etl'@'10.0.1.%' IDENTIFIED BY 'Cx7!qLm2Vr9Tze';"
sudo mariadb -e "GRANT SELECT, INSERT, CREATE, DROP ON analytics_db.* TO 'analytics_etl'@'10.0.1.%';"
sudo mariadb -e "FLUSH PRIVILEGES;"
Configuring multi-node ColumnStore clusters
Plan the module roles
A production cluster separates performance modules (PM, storage plus query execution) from user modules (UM, connection handling and result aggregation). A minimal HA cluster uses 3 PMs and 2 UMs.
RoleResponsibilityMinimum countUM (User Module)Accepts client connections, parses SQL, returns results2PM (Performance Module)Stores column extents, executes scans and joins3DBRootsData storage units assigned to PMs, one or more per PM1 per PM minimum
Register additional nodes via CMAPI
Use the ColumnStore Management API to add each node to the cluster. Run this from the first PM node, replacing the IPs with your actual node addresses.
curl -k -s -X PUT https://127.0.0.1:8640/cmapi/0.4.0/cluster/node \
-H 'x-api-key: YOUR_CMAPI_KEY' \
-d '{"timeout":120, "node": "203.0.113.11"}'
curl -k -s -X PUT https://127.0.0.1:8640/cmapi/0.4.0/cluster/node \
-H 'x-api-key: YOUR_CMAPI_KEY' \
-d '{"timeout":120, "node": "203.0.113.12"}'
Retrieve the generated API key from the CMAPI config if you have not set one manually.
sudo cat /etc/columnstore/cmapi_server.conf | grep x-api-key
Configure shared storage or replication for DBRoots
ColumnStore supports local storage with replication or shared storage (S3-compatible object storage) for DBRoots. For most self-hosted clusters, S3-compatible storage simplifies failover since any PM can mount any DBRoot.
[ObjectStorage]
service = S3
[S3]
bucket = columnstore-data
endpoint = https://s3.example.com
aws_access_key_id = AKIAEXAMPLE123
aws_secret_access_key = REPLACE_WITH_STRONG_SECRET
region = eu-central-1
If you are running your own S3-compatible endpoint, see configuring MinIO high availability clustering for a self-hosted object storage backend.
Verify cluster membership
Confirm every node reports healthy status before loading data.
curl -k -s https://127.0.0.1:8640/cmapi/0.4.0/cluster/status -H 'x-api-key: YOUR_CMAPI_KEY' | jq
Table design for analytics workloads
Create a columnstore database and fact table
ColumnStore tables are declared with ENGINE=Columnstore. Design fact tables wide and denormalized; ColumnStore reads only the columns a query touches, so extra columns cost little at scan time.
sudo mariadb -e "CREATE DATABASE IF NOT EXISTS analytics_db;"
sudo mariadb analytics_db -e "
CREATE TABLE order_events (
event_id BIGINT,
order_id BIGINT,
customer_id INT,
region_code CHAR(2),
event_type VARCHAR(32),
event_timestamp DATETIME,
amount DECIMAL(12,2)
) ENGINE=Columnstore;"
Choose extent maps and partitioning wisely
ColumnStore builds extent maps automatically, but query performance improves significantly when data arrives sorted by a high-cardinality time or ID column. Load data pre-sorted by event_timestamp to enable extent elimination on range filters.
Note: Unlike InnoDB, ColumnStore does not use traditional B-tree indexes. Extent elimination based on min/max block metadata is the primary mechanism for skipping irrelevant data during scans.
Tuning memory, buffer pool, and query execution
Size the PrimProc buffer cache
PrimProc's block cache holds decompressed column blocks in memory. Set it to roughly 50-60% of available RAM on PM nodes, leaving headroom for the OS page cache and ExeMgr.
32
55
sudo mcs cluster restart
Tune parallel query execution
Increase the DML and DDL execution thread pools on UM nodes to match core count, and set the number of concurrent casual partitioning threads on PM nodes for large table scans.
3
30
24
Adjust the MariaDB server-level memory settings
Even though ColumnStore manages its own block cache, the MariaDB server layer still needs sensible connection and sort buffer limits for query planning.
[mysqld]
max_connections = 500
sort_buffer_size = 4M
join_buffer_size = 8M
columnstore_use_import_for_batchinsert = ON
sudo systemctl restart mariadb
For general MariaDB performance baseline tuning outside ColumnStore-specific settings, see optimizing MariaDB performance with query analysis and indexing.
Loading bulk data with cpimport
Prepare a delimited source file
cpimport reads flat files directly into extents, bypassing the SQL layer entirely. It is dramatically faster than INSERT for bulk loads, often ingesting millions of rows per minute.
cat > /tmp/order_events.csv <<'EOF'
1001|500001|9001|EU|purchase|2025-01-15 10:22:00|129.99
1002|500002|9002|US|refund|2025-01-15 10:23:11|59.50
1003|500003|9003|EU|purchase|2025-01-15 10:24:47|89.00
EOF
Run cpimport against the target table
Specify the delimiter and null handling explicitly to avoid silent data corruption on malformed rows.
sudo cpimport -s '|' -n 5 analytics_db order_events /tmp/order_events.csv
The -n flag sets the number of parallel read threads. Increase it on PM nodes with more cores for large files.
Batch ETL pipelines with staging tables
For continuous ingestion pipelines, load into a staging table with cpimport, validate row counts, then use INSERT SELECT to move validated rows into the production fact table. This avoids partial loads corrupting live analytics.
sudo mariadb analytics_db -e "
CREATE TABLE order_events_staging LIKE order_events;"
sudo cpimport -s '|' analytics_db order_events_staging /tmp/order_events.csv
sudo mariadb analytics_db -e "
INSERT INTO order_events SELECT * FROM order_events_staging WHERE amount > 0;
TRUNCATE TABLE order_events_staging;"
If your pipeline sources data from Kafka or Spark, see setting up Spark Streaming with Kafka and Delta Lake for the upstream ingestion side before landing files for cpimport.
Automate cpimport with cron or systemd timers
Schedule recurring loads for batch ETL windows rather than running cpimport ad hoc.
[Unit]
Description=ColumnStore bulk load job
[Service]
Type=oneshot
User=mysql
ExecStart=/usr/bin/cpimport -s | analytics_db order_events /data/incoming/orders.csv
[Unit]
Description=Run ColumnStore load every 15 minutes
[Timer]
OnCalendar=*:0/15
Persistent=true
[Install]
WantedBy=timers.target
sudo systemctl enable --now cs-cpimport.timer
Monitoring cluster health and performance
Check module and DBRoot status
Run this regularly, or wire it into a health check script, to catch a downed PM before it affects query correctness.
sudo mcs cluster status
Query system performance views
ColumnStore exposes execution statistics through the calpontsys system database, useful for spotting slow extents or skewed table distribution.
sudo mariadb -e "SELECT * FROM information_schema.columnstore_extents LIMIT 10;"
sudo mariadb -e "SELECT * FROM information_schema.columnstore_files ORDER BY compressedsize DESC LIMIT 10;"
Export metrics to Prometheus
Use the mysqld_exporter alongside a custom textfile collector for cpimport job durations and mcs cluster status. This lets you build dashboards similar to existing MariaDB monitoring setups.
sudo systemctl status prometheus-mysqld-exporter
See configuring MariaDB performance monitoring with Prometheus and Grafana for the full exporter and dashboard setup, which applies directly to the mysqld layer that ColumnStore sits on top of.
Set up log rotation for ColumnStore logs
ColumnStore writes verbose logs to /var/log/mariadb/columnstore/ that grow quickly under heavy load. Rotate them to avoid filling disk on PM nodes.
/var/log/mariadb/columnstore/*.log {
daily
rotate 14
compress
missingok
notifempty
copytruncate
}
Verify your setup
sudo mcs cluster status
sudo mariadb -e "SELECT VERSION();"
sudo mariadb analytics_db -e "SELECT COUNT(*) FROM order_events;"
sudo mariadb analytics_db -e "EXPLAIN SELECT region_code, SUM(amount) FROM order_events GROUP BY region_code;"
Confirm the query plan shows a Columnstore scan and that row counts match expected load volumes after a cpimport run.
Common issues
SymptomCauseFixcpimport fails with lock file existsA previous import crashed mid-load and left a stale lockRemove the lock under /var/lib/columnstore/data1/systemFiles/dbrm/ after confirming no active import, then retryQuery returns stale results after cpimportUM node cache not refreshed after bulk loadRun FLUSH TABLES; or restart the affected UM's PrimProc connection poolNode marked as DOWN in cluster statusNetwork partition or CMAPI service crashed on that nodeCheck systemctl status mariadb-columnstore-cmapi on the affected node and confirm firewall allows CMAPI port 8640 between nodesSlow full table scans on large fact tablesData loaded unsorted, preventing extent eliminationReload with cpimport sorted by the primary filter column, typically an event timestampPrimProc using excessive memory and OOM killer triggersNumBlocksPct set too high relative to available RAMLower NumBlocksPct in Columnstore.xml to leave headroom for OS cache and ExeMgrcpimport permission denied writing to data directoryImport run as a user without ownership of ColumnStore data pathsRun cpimport as the mysql user, or fix ownership with sudo chown -R mysql:mysql /var/lib/columnstore rather than loosening permissions
Never use chmod 777. If cpimport or PrimProc reports permission errors on data directories, the fix is correct ownership with chown to the mysql user, not opening permissions to every user on the system.
Next steps
Configure MariaDB performance monitoring with Prometheus and Grafana dashboards
Set up Spark Streaming with Kafka and Delta Lake for real-time analytics
Configure MinIO high availability clustering for production
Benchmark MariaDB ColumnStore with sysbench and TPC-H workloads
Implement MariaDB ColumnStore backup and disaster recovery automation
Running this in production?
Want this handled for you? Running ColumnStore at scale adds a second layer of work: capacity planning across PM nodes, failover drills, storage cost control on object storage backends, and being on call when a node drops out of the cluster at 3am. See how we run infrastructure like this for European teams.
---
### Implement Kibana 8 custom threat hunting dashboards for security incident response
URL: https://binadit.com/tutorials/implement-kibana-custom-threat-hunting-dashboards
Category: monitoring
Difficulty: advanced
Time: ~90 minutes
Author: Binadit Tech Team
> Build production-grade Kibana 8 threat hunting dashboards with MITRE ATT&CK mapping, Lens visualizations, drilldowns, KQL detections and role-based access control for SOC teams. Includes saved objects API automation for version-controlled dashboard deployment.
What this solves
SOC analysts need more than default Kibana dashboards to hunt threats efficiently. This tutorial builds custom Kibana 8 dashboards mapped to MITRE ATT&CK tactics, with drilldowns, saved KQL detections, TSVB anomaly panels, alerting rules and RBAC for security teams.
You will index security logs into Elasticsearch, build Lens visualizations for threat indicators, wire up cross-dashboard drilldowns for incident investigation, and automate deployment with the saved objects API so dashboards live in version control instead of clicking around the UI.
Prerequisite: This tutorial assumes you already have Elasticsearch and Kibana 8 running with security logs flowing in. If you need the ingestion pipeline first, see configure Logstash 8 security pipelines with threat intelligence enrichment and install and configure Filebeat 8.15 for log shipping to ELK.
Step-by-step configuration
Create dedicated indices for security data sources
Threat hunting dashboards perform best when firewall, EDR, auth and proxy logs land in clearly separated indices rather than one generic catch-all. Create index templates so field mappings stay consistent across data sources.
curl -X PUT "https://localhost:9200/_index_template/security-auth" \
-u elastic:YourStrongPass123! \
-H "Content-Type: application/json" \
-d '{
"index_patterns": ["security-auth-*"],
"template": {
"settings": { "number_of_shards": 1, "number_of_replicas": 1 },
"mappings": {
"properties": {
"@timestamp": { "type": "date" },
"source.ip": { "type": "ip" },
"destination.ip": { "type": "ip" },
"user.name": { "type": "keyword" },
"event.outcome": { "type": "keyword" },
"threat.tactic.name": { "type": "keyword" },
"threat.technique.id": { "type": "keyword" },
"host.name": { "type": "keyword" }
}
}
}
}'
Repeat this pattern for security-network-, security-endpoint- and security-proxy-. Consistent ECS-aligned field names like threat.technique.id are what let a single dashboard query across all of them later.
Build a unified data view
Kibana 8 data views replace legacy index patterns. Create one data view spanning all security indices so analysts can pivot between log sources without switching context.
curl -X POST "https://localhost:5601/api/data_views/data_view" \
-u elastic:YourStrongPass123! \
-H "kbn-xsrf: true" -H "Content-Type: application/json" \
-d '{
"data_view": {
"title": "security-*",
"name": "Security - All sources",
"timeFieldName": "@timestamp"
}
}'
Also create narrower data views (security-auth-, security-endpoint-) for panels that only make sense against one log source, such as failed login trend charts.
Add runtime fields for MITRE ATT&CK enrichment
If your ingestion pipeline does not already tag events with ATT&CK technique IDs, add a runtime field in the data view so you can still group and filter by tactic in dashboards without reindexing.
curl -X POST "https://localhost:5601/api/data_views/data_view/security-view-id/runtime_field" \
-u elastic:YourStrongPass123! \
-H "kbn-xsrf: true" -H "Content-Type: application/json" \
-d '{
"name": "threat.tactic.derived",
"runtimeField": {
"type": "keyword",
"script": {
"source": "if (doc.containsKey('\''event.action'\'') && doc['\''event.action'\''].size() > 0 && doc['\''event.action'\''].value == '\''process_created'\'') { emit('\''execution'\''); } else { emit('\''unclassified'\''); }"
}
}
}'
In production, prefer enriching at ingest time in Logstash rather than relying heavily on runtime fields, since runtime scripts add query-time overhead at scale.
Build Lens visualizations for threat indicators
Open Kibana, go to Visualize Library and create a new Lens visualization. Build a top-N bar chart of source.ip by count, filtered to event.outcome: failure, to surface brute-force sources.
curl -X POST "https://localhost:5601/api/saved_objects/lens/failed-auth-top-ips" \
-u elastic:YourStrongPass123! \
-H "kbn-xsrf: true" -H "Content-Type: application/json" \
-d '{
"attributes": {
"title": "Failed auth - top source IPs",
"visualizationType": "lnsXY",
"state": {
"datasourceStates": {},
"visualization": {},
"query": { "query": "event.outcome: failure", "language": "kuery" },
"filters": []
}
},
"references": []
}'
Repeat for other indicators: unique destination ports per host (port scanning), rare process command lines (living-off-the-land), and DNS query entropy for beaconing. Build each as its own Lens panel so it can be reused across multiple dashboards.
Create MITRE ATT&CK mapped dashboards
Structure dashboards by tactic rather than by log source. Create one dashboard per tactic group (initial access, execution, persistence, lateral movement, exfiltration) with panels tagged to specific technique IDs.
curl -X POST "https://localhost:5601/api/saved_objects/dashboard/mitre-lateral-movement" \
-u elastic:YourStrongPass123! \
-H "kbn-xsrf: true" -H "Content-Type: application/json" \
-d '{
"attributes": {
"title": "MITRE ATT&CK - Lateral movement (TA0008)",
"description": "Panels for T1021 remote services, T1550 alt auth material, T1563 RDP hijack",
"panelsJSON": "[]",
"optionsJSON": "{\"useMargins\":true,\"hidePanelTitles\":false}",
"timeRestore": true,
"kibanaSavedObjectMeta": {
"searchSourceJSON": "{\"query\":{\"query\":\"\",\"language\":\"kuery\"},\"filter\":[]}"
}
},
"references": []
}'
Add tags per technique so analysts can filter the dashboard listing page by MITRE tactic. Use the Kibana tag manager under Stack Management to create tags like TA0008-lateral-movement and TA0001-initial-access.
Configure drilldowns for incident investigation
Drilldowns let an analyst click a suspicious IP in one panel and jump straight into a filtered view of related events. Open a panel's context menu, choose Create drilldown, and configure a dashboard-to-dashboard drilldown.
Note: Set the drilldown to carry over the clicked value as a filter (for example source.ip) and preserve the current time range so the investigation stays scoped to the incident window.
A typical drilldown chain: overview dashboard, click a flagged host, land on an endpoint-detail dashboard scoped to that host, then drill down again into a raw document table filtered by the same host and time range for full packet or process detail.
Save KQL queries for common attack patterns
Saved searches with KQL let analysts reuse detection logic across dashboards and Discover sessions. Create saved searches for patterns like impossible travel logins, privilege escalation attempts and suspicious PowerShell execution.
curl -X POST "https://localhost:5601/api/saved_objects/search/suspicious-powershell" \
-u elastic:YourStrongPass123! \
-H "kbn-xsrf: true" -H "Content-Type: application/json" \
-d '{
"attributes": {
"title": "Suspicious PowerShell execution",
"columns": ["@timestamp", "host.name", "user.name", "process.command_line"],
"sort": [["@timestamp", "desc"]],
"kibanaSavedObjectMeta": {
"searchSourceJSON": "{\"query\":{\"query\":\"process.name: \\\"powershell.exe\\\" and process.command_line: (*-EncodedCommand* or *-nop* or *bypass* or *DownloadString*)\",\"language\":\"kuery\"},\"filter\":[]}"
}
},
"references": [{"name": "kibanaSavedObjectMeta.searchSourceJSON.index", "type": "index-pattern", "id": "security-endpoint-view"}]
}'
Other useful KQL patterns to save: destination.port: (4444 or 1337 or 31337) for known reverse-shell ports, dns.question.name: /.{30,}/ for long-domain DNS tunneling candidates, and event.action: \"user_login\" and geo.country_iso_code: paired with a Lens geo map for impossible travel checks.
Implement Kibana alerting rules for automated detection
Go to Stack Management, Rules and Connectors, and create an Elasticsearch query rule. This example alerts when more than 10 failed logins from a single IP occur within 5 minutes.
curl -X POST "https://localhost:5601/api/alerting/rule" \
-u elastic:YourStrongPass123! \
-H "kbn-xsrf: true" -H "Content-Type: application/json" \
-d '{
"name": "Brute force - repeated auth failures",
"rule_type_id": ".es-query",
"consumer": "alerts",
"schedule": { "interval": "1m" },
"params": {
"index": ["security-auth-*"],
"timeField": "@timestamp",
"esQuery": "{\"query\":{\"bool\":{\"filter\":[{\"term\":{\"event.outcome\":\"failure\"}}]}}}",
"size": 100,
"thresholdComparator": ">",
"threshold": [10],
"timeWindowSize": 5,
"timeWindowUnit": "m",
"aggType": "count",
"groupBy": "top",
"termField": "source.ip",
"termSize": 20
},
"actions": []
}'
Attach a connector action (Slack, email or a webhook into your SOAR) so the rule pages the on-call analyst instead of silently logging a match. For high-volume environments, tune the check interval and grouping cardinality to avoid alert fatigue.
Add TSVB panels for anomaly and trend analysis
TSVB (Time Series Visual Builder) handles multi-metric time series better than Lens for baseline deviation charts. Create a TSVB panel showing a 7-day moving average of authentication failures against the current day's actual count.
curl -X POST "https://localhost:5601/api/saved_objects/visualization/auth-failure-anomaly-tsvb" \
-u elastic:YourStrongPass123! \
-H "kbn-xsrf: true" -H "Content-Type: application/json" \
-d '{
"attributes": {
"title": "Auth failure trend vs 7-day baseline",
"visState": "{\"title\":\"Auth failure trend vs 7-day baseline\",\"type\":\"metrics\",\"params\":{\"type\":\"timeseries\",\"index_pattern\":\"security-auth-*\",\"time_field\":\"@timestamp\",\"interval\":\"1h\"}}",
"uiStateJSON": "{}",
"kibanaSavedObjectMeta": { "searchSourceJSON": "{\"query\":{\"query\":\"\",\"language\":\"kuery\"},\"filter\":[]}" }
},
"references": []
}'
Add a moving average and a static baseline series to the same panel so analysts can spot deviation visually rather than reading raw counts. TSVB annotations also work well for marking known maintenance windows so spikes during patching are not mistaken for attacks.
Secure dashboards with role-based access control
SOC dashboards often contain sensitive data like user names, IPs and process paths. Restrict access with Kibana feature privileges and space-level roles rather than granting all analysts the full admin role.
curl -X PUT "https://localhost:9200/_security/role/soc_analyst" \
-u elastic:YourStrongPass123! \
-H "Content-Type: application/json" \
-d '{
"indices": [
{ "names": ["security-*"], "privileges": ["read", "view_index_metadata"] }
],
"applications": [
{
"application": "kibana-.kibana",
"privileges": ["feature_dashboard.read", "feature_discover.all", "feature_visualize.read"],
"resources": ["space:soc"]
}
]
}'
Create a separate soc_lead role with feature_dashboard.all and alerting management privileges so only leads can edit dashboards or acknowledge alert rules. Put SOC dashboards in a dedicated Kibana space so they never mix with unrelated business dashboards.
Never grant the superuser role for daily analyst work. Scope roles to the specific indices and Kibana features analysts actually need. Broad privileges turn a compromised analyst account into a full cluster compromise.
For deeper field-level restrictions, for example hiding raw credential hashes from junior analysts while still showing them to leads, see configure Kibana 8 advanced security with field-level restrictions and role-based access control.
Export dashboards with the saved objects API
Export the full set of security dashboards, visualizations and saved searches as a single NDJSON bundle for version control.
curl -X POST "https://localhost:5601/api/saved_objects/_export" \
-u elastic:YourStrongPass123! \
-H "kbn-xsrf: true" -H "Content-Type: application/json" \
-d '{
"type": ["dashboard", "lens", "visualization", "search", "index-pattern"],
"includeReferencesDeep": true
}' -o soc-dashboards-export.ndjson
Commit this file to a Git repository dedicated to Kibana configuration. Treat it exactly like infrastructure-as-code, with pull requests reviewed before changes reach production.
Automate deployment across environments
Import the NDJSON bundle into staging or production Kibana instances with the import API. Use overwrite=true in CI/CD pipelines so re-running the import updates existing objects instead of failing on conflicts.
curl -X POST "https://kibana-prod.internal:5601/api/saved_objects/_import?overwrite=true" \
-u kibana_deploy:YourStrongPass123! \
-H "kbn-xsrf: true" \
--form file=@soc-dashboards-export.ndjson
Wrap this in a small deploy script triggered from your CI pipeline so a merged pull request against the dashboards repo automatically pushes to staging, and a tagged release pushes to production.
#!/bin/bash
set -euo pipefail
KIBANA_URL="$1"
KIBANA_USER="$2"
KIBANA_PASS="$3"
curl -sf -X POST "${KIBANA_URL}/api/saved_objects/_import?overwrite=true" \
-u "${KIBANA_USER}:${KIBANA_PASS}" \
-H "kbn-xsrf: true" \
--form file=@soc-dashboards-export.ndjson
echo "Deployed SOC dashboards to ${KIBANA_URL}"
Note: Store the deploy credentials in a secrets manager rather than hardcoding them in the script. See configure Ansible Vault integration with HashiCorp Vault for secrets management for a pattern you can adapt to CI/CD pipeline variables.
Verify your setup
curl -s -u elastic:YourStrongPass123! "https://localhost:5601/api/saved_objects/_find?type=dashboard&search_fields=title&search=MITRE" | jq '.saved_objects[].attributes.title'
curl -s -u elastic:YourStrongPass123! "https://localhost:5601/api/alerting/rules/_find" | jq '.data[].name'
curl -s -u elastic:YourStrongPass123! "https://localhost:9200/_security/role/soc_analyst" | jq '.'
Log in to Kibana as a test user assigned only the soc_analyst role and confirm they can view dashboards but cannot edit them or access unrelated spaces.
Common issues
SymptomCauseFixDashboard import fails with reference errorsMissing data view or visualization referenced by the dashboard was not included in exportRe-run export with includeReferencesDeep: true and include all dependent object typesAlert rule never firesIndex pattern in rule params does not match actual index namesConfirm params.index matches live indices with GET _cat/indices/security-*Analyst cannot see any dashboardsRole missing feature_dashboard.read privilege for the correct spaceUpdate the role's applications block with the correct space resourceTSVB panel shows no dataTime field mismatch between data view and index mappingConfirm timeFieldName in the data view matches the mapped @timestamp field typeDrilldown filter does not apply on target dashboardTarget dashboard's data view does not contain the filtered fieldUse a shared data view across source and target da
---
### Configure Logstash 8 security pipelines with threat intelligence enrichment
URL: https://binadit.com/tutorials/configure-logstash-security-pipelines-threat-enrichment
Category: monitoring
Difficulty: advanced
Time: ~75 minutes
Author: Binadit Tech Team
> Build a production Logstash 8 pipeline that ingests firewall, IDS and syslog data, enriches events with MISP, AlienVault OTX and AbuseIPDB threat intel, and ships hardened IOC-enriched events to Elasticsearch.
What this solves
Raw firewall, IDS and syslog events are noisy and lack context. This tutorial builds a Logstash 8 security pipeline that normalizes those sources, enriches events with GeoIP, DNS and threat intelligence feeds (MISP, AlienVault OTX, AbuseIPDB), and forwards the results to Elasticsearch for SIEM dashboards.
You will use the translate and http filters for IOC lookups, tune pipeline workers for throughput, and secure the pipeline with TLS and role-based access control.
Step-by-step configuration
Install Java and add the Elastic package repository
Logstash 8 requires a supported JDK. We use the bundled JVM shipped with Logstash, but the repository setup differs by distro.
sudo apt update && sudo apt install -y apt-transport-https gnupg curl
curl -fsSL https://artifacts.elastic.co/GPG-KEY-elasticsearch | sudo gpg --dearmor -o /usr/share/keyrings/elastic-keyring.gpg
echo "deb [signed-by=/usr/share/keyrings/elastic-keyring.gpg] https://artifacts.elastic.co/packages/8.x/apt stable main" | sudo tee /etc/apt/sources.list.d/elastic-8.x.list
sudo apt update
[elastic-8.x]
name=Elastic repository for 8.x packages
baseurl=https://artifacts.elastic.co/packages/8.x/yum
gpgcheck=1
gpgkey=https://artifacts.elastic.co/GPG-KEY-elasticsearch
enabled=1
autorefresh=1
type=rpm-md
Install Logstash 8
This installs the Logstash service and default systemd unit.
sudo apt install -y logstash
sudo dnf install -y logstash
If you have not yet built the Elasticsearch backend, follow setting up centralized log aggregation with Elasticsearch, Logstash and Kibana first, and use Filebeat for log shipping as the upstream agent on firewalls and hosts that cannot run Logstash directly.
Create dedicated pipeline directories
Security pipelines should be isolated per source type so a bad regex in one does not stop ingestion for the others.
sudo mkdir -p /etc/logstash/conf.d/firewall
sudo mkdir -p /etc/logstash/conf.d/ids
sudo mkdir -p /etc/logstash/conf.d/syslog
sudo chown -R logstash:logstash /etc/logstash/conf.d
sudo chmod -R 750 /etc/logstash/conf.d
Note: The logstash system user is created automatically by the package. Keeping configs at 750 with logstash ownership prevents other local users from reading pipeline secrets like API keys.
Define multiple pipelines in pipelines.yml
Logstash 8 runs several independent pipelines in one process. This isolates firewall, IDS and syslog processing and lets you tune workers per pipeline.
- pipeline.id: firewall
path.config: "/etc/logstash/conf.d/firewall/*.conf"
pipeline.workers: 4
pipeline.batch.size: 250
- pipeline.id: ids
path.config: "/etc/logstash/conf.d/ids/*.conf"
pipeline.workers: 2
pipeline.batch.size: 125
- pipeline.id: syslog
path.config: "/etc/logstash/conf.d/syslog/*.conf"
pipeline.workers: 2
pipeline.batch.size: 125
Build the firewall ingest pipeline
This example parses syslog-formatted firewall logs (Cisco ASA style) arriving over TCP, and extracts source/destination IPs and ports.
input {
tcp {
port => 5140
type => "firewall"
ssl_enabled => true
ssl_certificate_authorities => ["/etc/logstash/certs/ca.crt"]
ssl_certificate => "/etc/logstash/certs/logstash.crt"
ssl_key => "/etc/logstash/certs/logstash.key"
ssl_client_authentication => "required"
}
}
filter {
if [type] == "firewall" {
grok {
match => { "message" => "%%{CISCOTIMESTAMP:timestamp} %%{HOSTNAME:device} %%{GREEDYDATA:asa_message}" }
}
grok {
match => { "asa_message" => "Built %%{WORD:direction} %%{WORD:protocol} connection .* for %%{DATA} :%%{IP:src_ip}/%%{INT:src_port} .* to %%{DATA} :%%{IP:dst_ip}/%%{INT:dst_port}" }
tag_on_failure => ["_grokparsefailure_firewall"]
}
date {
match => [ "timestamp", "MMM dd HH:mm:ss", "MMM d HH:mm:ss" ]
target => "@timestamp"
}
}
}
Build the IDS ingest pipeline for Suricata/Snort EVE JSON
Most modern IDS engines emit structured JSON, which simplifies parsing considerably compared to firewall syslog.
input {
file {
path => "/var/log/suricata/eve.json"
codec => "json"
type => "ids"
start_position => "beginning"
sincedb_path => "/var/lib/logstash/sincedb_suricata"
}
}
filter {
if [type] == "ids" and [event_type] == "alert" {
mutate {
add_field => {
"ioc_ip" => "%%{src_ip}"
"signature" => "%%{[alert][signature]}"
"severity" => "%%{[alert][severity]}"
}
}
}
}
Build the syslog pipeline for generic infrastructure logs
This captures rsyslog and journald forwarded events from Linux hosts, network devices and application servers.
input {
udp {
port => 5514
type => "syslog"
}
}
If you are centralizing syslog on the source hosts before shipping to Logstash, see configuring centralized logging with rsyslog and logrotate.
Enrich IOCs with a local MISP indicator export using the translate filter
The translate filter performs fast in-memory dictionary lookups, ideal for large static IOC lists exported periodically from MISP as a CSV.
sudo mkdir -p /etc/logstash/threat-intel
sudo chown logstash:logstash /etc/logstash/threat-intel
sudo chmod 750 /etc/logstash/threat-intel
indicator,category,threat_level
203.0.113.45,c2-server,high
203.0.113.78,scanner,medium
198.51.100.23,malware-distribution,high
filter {
if [type] == "firewall" and [dst_ip] {
translate {
dictionary_path => "/etc/logstash/threat-intel/misp_ip_iocs.csv"
field => "dst_ip"
destination => "[threat][misp_category]"
fallback => "none"
refresh_interval => 300
}
}
}
The refresh_interval reloads the dictionary every 5 minutes so a cron job or scheduled MISP export can update indicators without restarting Logstash.
Query AlienVault OTX in real time with the http filter
For IPs not already covered by your local IOC export, query OTX on demand. Store the API key outside version control.
a1b2c3d4e5f6a7b8c9d0e1f2a3b4c5d6e7f8a9b0
sudo chown logstash:logstash /etc/logstash/secrets/otx.key
sudo chmod 600 /etc/logstash/secrets/otx.key
filter {
if [threat][misp_category] == "none" and [dst_ip] {
http {
url => "https://otx.alienvault.com/api/v1/indicators/IPv4/%%{[dst_ip]}/general"
headers => {
"X-OTX-API-KEY" => "${OTX_API_KEY}"
}
target_body => "[otx][response]"
ssl_verification_mode => "full"
}
if [otx][response][pulse_info][count] and [otx][response][pulse_info][count] > 0 {
mutate {
add_field => { "[threat][otx_flagged]" => "true" }
}
}
}
}
Warning: The http filter is synchronous and blocks the pipeline worker per request. Only apply it to a filtered subset of events (for example, only unmatched destinations) or throughput will collapse under load.
Load the OTX API key as an environment variable
Reference secrets via environment variables rather than hardcoding them in pipeline configs.
[Service]
EnvironmentFile=/etc/logstash/secrets/logstash.env
OTX_API_KEY=a1b2c3d4e5f6a7b8c9d0e1f2a3b4c5d6e7f8a9b0
ABUSEIPDB_API_KEY=b2c3d4e5f6a7b8c9d0e1f2a3b4c5d6e7f8a9b0c1
sudo chmod 600 /etc/logstash/secrets/logstash.env
sudo chown logstash:logstash /etc/logstash/secrets/logstash.env
sudo systemctl daemon-reload
For teams already managing secrets centrally, integrate this with HashiCorp Vault secrets management instead of static env files.
Add AbuseIPDB reputation scoring
AbuseIPDB provides a confidence score useful for prioritizing alerts by abuse likelihood.
filter {
if [threat][otx_flagged] != "true" and [dst_ip] {
http {
url => "https://api.abuseipdb.com/api/v2/check?ipAddress=%%{[dst_ip]}&maxAgeInDays=90"
headers => {
"Key" => "${ABUSEIPDB_API_KEY}"
"Accept" => "application/json"
}
target_body => "[abuseipdb][response]"
ssl_verification_mode => "full"
}
if [abuseipdb][response][data][abuseConfidenceScore] and [abuseipdb][response][data][abuseConfidenceScore] > 50 {
mutate {
add_field => { "[threat][abuseipdb_high_confidence]" => "true" }
}
}
}
}
Add GeoIP and DNS enrichment for context
Geolocation and reverse DNS help analysts triage alerts faster without leaving the dashboard.
filter {
if [dst_ip] {
geoip {
source => "dst_ip"
target => "geoip"
database => "/etc/logstash/geoip/GeoLite2-City.mmdb"
}
dns {
reverse => ["dst_ip"]
action => "append"
hit_cache_size => 8000
hit_cache_ttl => 300
failed_cache_size => 1000
failed_cache_ttl => 60
}
}
}
Download the free GeoLite2 City database from MaxMind and place it under /etc/logstash/geoip/ with 640 permissions owned by logstash:logstash.
Forward enriched events to Elasticsearch
This output uses a dedicated ILM-managed index and API key authentication instead of a shared superuser account.
output {
elasticsearch {
hosts => ["https://10.20.0.11:9200"]
ssl_enabled => true
ssl_certificate_authorities => ["/etc/logstash/certs/es-ca.crt"]
api_key => "${ES_LOGSTASH_API_KEY}"
index => "security-firewall-%%{+YYYY.MM.dd}"
ilm_enabled => true
ilm_rollover_alias => "security-firewall"
ilm_pattern => "{now/d}-000001"
ilm_policy => "security-events-policy"
}
}
Pair this with Elasticsearch ILM configuration to automatically roll over and age out security indices, and review Elasticsearch snapshot and restore policies for long-term retention of security events.
Generate a minimal-privilege Elasticsearch API key
Create a role scoped only to the indices Logstash needs to write, rather than using the built-in superuser.
curl -X POST "https://10.20.0.11:9200/_security/api_key" \
-H "Content-Type: application/json" \
-u elastic:StrongElasticPass9! \
--cacert /etc/logstash/certs/es-ca.crt \
-d '{
"name": "logstash-security-writer",
"role_descriptors": {
"logstash_writer": {
"cluster": ["monitor"],
"indices": [
{
"names": ["security-firewall-*", "security-ids-*", "security-syslog-*"],
"privileges": ["create_index", "write", "manage"]
}
]
}
}
}'
Enable the Logstash monitoring API and X-Pack security
Expose pipeline metrics for capacity planning and detect stalled or slow pipelines before they drop events.
node.name: logstash-siem-01
path.data: /var/lib/logstash
pipeline.workers: 4
queue.type: persisted
queue.max_bytes: 4gb
xpack.monitoring.enabled: true
xpack.monitoring.elasticsearch.hosts: ["https://10.20.0.11:9200"]
xpack.monitoring.elasticsearch.api_key: "${ES_MONITORING_API_KEY}"
api.enabled: true
api.http.host: 127.0.0.1
api.http.port: 9600
api.auth.type: basic
api.auth.basic.username: monitor
api.auth.basic.password: "${LOGSTASH_MONITOR_PASS}"
queue.type: persisted enables a disk-backed queue so Logstash survives a restart without losing in-flight security events.
Restrict the monitoring API to localhost and firewall the pipeline ports
Never expose the monitoring API or ingest ports to the open network. Bind narrowly and firewall explicitly.
sudo ufw allow from 10.20.0.0/24 to any port 5140 proto tcp
sudo ufw allow from 10.20.0.0/24 to any port 5514 proto udp
sudo ufw deny 9600
sudo firewall-cmd --permanent --zone=internal --add-rich-rule='rule family="ipv4" source address="10.20.0.0/24" port protocol="tcp" port="5140" accept'
sudo firewall-cmd --permanent --zone=internal --add-rich-rule='rule family="ipv4" source address="10.20.0.0/24" port protocol="udp" port="5514" accept'
sudo firewall-cmd --reload
For deeper network-layer traffic visibility feeding these pipelines, see Kubernetes network monitoring with Hubble and Cilium if your firewall and IDS run in-cluster.
Enable and start Logstash
Start the service and confirm it picks up all three pipelines.
sudo systemctl daemon-reload
sudo systemctl enable --now logstash
sudo systemctl status logstash
Verify your setup
Validate configuration syntax before every restart to avoid dropping events mid-reload.
sudo -u logstash /usr/share/logstash/bin/logstash --path.settings /etc/logstash -t
Check that pipelines are running and processing events through the monitoring API.
curl -s -u monitor:'YourMonitorPass123!' http://127.0.0.1:9600/_node/stats/pipelines?pretty
Send a test firewall-style event and confirm enrichment fields appear in Elasticsearch.
echo '<134>Jan 15 10:22:31 fw-edge-01 %ASA-6-302013: Built outbound TCP connection for inside:10.20.0.50/51234 to outside:203.0.113.45/443' | ncat --ssl 10.20.0.11 5140
curl -s -u elastic:StrongElasticPass9! --cacert /etc/logstash/certs/es-ca.crt \
"https://10.20.0.11:9200/security-firewall-*/_search?q=dst_ip:203.0.113.45&pretty"
Common issues
SymptomCauseFixPipeline stuck, events not reaching Elasticsearchhttp filter blocking workers on slow threat intel API responsesReduce scope of http filter conditionals, increase pipeline.workers only for that pipeline, add request timeoutstranslate filter never matchesCSV dictionary path unreadable by logstash user or wrong field typeCheck ls -l /etc/logstash/threat-intel/ ownership and confirm field is a plain string, not nested JSONgrok parse failures tagged on all firewall eventsFirewall log format changed or vendor syslog template mismatchCapture raw sample with tcpdump, adjust grok pattern, test with the Grok Debugger before deployingLogstash won't start after config changeYAML or Ruby-DSL syntax error in a conf.d fileRun logstash -t to pinpoint the failing file and lineMonitoring API returns 401Wrong basic auth credentials or api.auth.type misconfiguredVerify api.auth.basic.username/password in logstash.yml match the curl requestGeoIP fields all emptyMissing or expired GeoLite2 database fileRe-download database, confirm path in geoip filter matches actual file locationHigh memory usage, JVM OOM under loadqueue.max_bytes too large for available heap or pipeline.batch.size too highTune JVM heap in /etc/logstash/jvm.options, lower batch size, monitor with the stats API
Next steps
Set up centralized log aggregation with Elasticsearch, Logstash and Kibana
Install and configure Filebeat for log shipping to ELK
Configure Elasticsearch Index Lifecycle Management for automated data retention
Set up ELK stack alerting
---
### Configure Ansible Vault integration with HashiCorp Vault for secrets management
URL: https://binadit.com/tutorials/configure-ansible-vault-secrets-management-with-hashicorp-vault
Category: devops
Difficulty: advanced
Time: ~45 minutes
Author: Binadit Tech Team
> Learn how to combine Ansible Vault with HashiCorp Vault using the community.hashi_vault collection, AppRole authentication, and least-privilege policies for production-grade secrets management in playbooks.
What this solves
Ansible Vault encrypts static files at rest, but it does not rotate secrets, audit access, or issue short-lived credentials. HashiCorp Vault solves dynamic secrets and centralized access control, while Ansible Vault remains useful as an encrypted fallback for bootstrap tokens.
This tutorial configures the community.hashi_vault collection with AppRole authentication, retrieves secrets dynamically in playbooks with the hashi_vault lookup plugin, and implements least-privilege policies so automation only reads what it needs.
Overview: Ansible Vault vs HashiCorp Vault architecture
Ansible Vault is a file encryption feature built into ansible-core. It encrypts YAML files, group_vars, or single strings using a passphrase or a key file. There is no server, no audit log, and no dynamic secret generation. It is a good fit for encrypting a small number of static values checked into version control.
HashiCorp Vault is a full secrets management server with authentication backends, secrets engines, audit logging, leases, and revocation. It can generate short-lived database credentials, cloud IAM tokens, and PKI certificates on demand.
In a production Ansible setup, the pattern is: Vault holds live secrets and issues them dynamically through the hashi_vault lookup plugin at playbook runtime. Ansible Vault only encrypts the small bootstrap credential (an AppRole secret_id or a wrapped token) needed to authenticate to HashiCorp Vault in the first place.
Note: If you already manage Kubernetes secrets with Vault, the authentication and policy concepts here are the same ones covered in configuring Kubernetes secrets management with External Secrets Operator and HashiCorp Vault.
Step-by-step configuration
Install HashiCorp Vault server prerequisites
Vault needs its own repository on both Debian-based and RHEL-based systems. Add the HashiCorp repo before installing.
sudo apt update
sudo apt install -y gnupg software-properties-common curl
curl -fsSL https://apt.releases.hashicorp.com/gpg | sudo gpg --dearmor -o /usr/share/keyrings/hashicorp-archive-keyring.gpg
echo "deb [signed-by=/usr/share/keyrings/hashicorp-archive-keyring.gpg] https://apt.releases.hashicorp.com $(lsb_release -cs) main" | sudo tee /etc/apt/sources.list.d/hashicorp.list
sudo apt update
sudo apt install -y vault
sudo dnf install -y dnf-plugins-core
sudo dnf config-manager --add-repo https://rpm.releases.hashicorp.com/RHEL/hashicorp.repo
sudo dnf install -y vault
Configure Vault storage and listener
For production, Vault should run with a real storage backend (Raft integrated storage is common) and TLS enabled. This minimal config uses Raft for a single-node dev-to-prod path.
storage "raft" {
path = "/opt/vault/data"
node_id = "vault-node-1"
}
listener "tcp" {
address = "0.0.0.0:8200"
tls_cert_file = "/etc/vault.d/tls/vault-cert.pem"
tls_key_file = "/etc/vault.d/tls/vault-key.pem"
}
api_addr = "https://vault.internal.example.com:8200"
cluster_addr = "https://vault.internal.example.com:8201"
ui = true
Create the data directory with correct ownership before starting the service. Vault runs as its own system user, not root.
sudo mkdir -p /opt/vault/data /etc/vault.d/tls
sudo chown -R vault:vault /opt/vault/data
sudo chmod 750 /opt/vault/data
Start Vault and initialize it
Start the service, then initialize and unseal it. Store the unseal keys and root token somewhere secure, not in shell history.
sudo systemctl enable --now vault
export VAULT_ADDR='https://vault.internal.example.com:8200'
vault operator init -key-shares=5 -key-threshold=3
Warning: The root token from initialization has unrestricted access. Use it only to bootstrap AppRole and policies, then revoke it. Never store the root token in playbooks or Ansible Vault files that are shared with automation users.
For production clusters, look at configuring Vault auto-unseal with AWS KMS to avoid manual unseal operations after every restart.
Enable the KV secrets engine
Enable KV version 2 at a dedicated path for Ansible-managed secrets. Version 2 gives you versioning and soft-delete.
vault login
vault secrets enable -path=ansible kv-v2
vault kv put ansible/app/db username="appuser" password="Xk9$mQ2vLp7@wRz4"
vault kv get ansible/app/db
Create an AppRole for Ansible automation
AppRole authentication is designed for machine-to-machine auth, which fits Ansible control nodes better than human login methods.
vault auth enable approle
vault write auth/approle/role/ansible-automation \
token_policies="ansible-readonly" \
token_ttl=15m \
token_max_ttl=1h \
secret_id_ttl=90d \
secret_id_num_uses=0
Retrieve the role_id and generate a secret_id. The role_id is not secret by itself, but the secret_id must be protected.
vault read auth/approle/role/ansible-automation/role-id
vault write -f auth/approle/role/ansible-automation/secret-id
Write a least-privilege policy
Scope the policy to only the paths automation needs. Avoid wildcard-everything policies attached to automation roles.
path "ansible/data/app/*" {
capabilities = ["read"]
}
path "ansible/metadata/app/*" {
capabilities = ["list"]
}
vault policy write ansible-readonly ansible-readonly.hcl
This mirrors the policy scoping approach used in configuring advanced Consul ACL policies for production security hardening: grant read-only access to the exact secret paths a workload touches, nothing broader.
Install the community.hashi_vault Ansible collection
Install Ansible itself if it is not already present, then add the collection and its Python dependency.
sudo apt update
sudo apt install -y python3-pip pipx
pipx install --include-deps ansible
pipx runpip ansible install hvac requests
sudo dnf install -y python3-pip
pip3 install --user ansible hvac requests
ansible-galaxy collection install community.hashi_vault
ansible-galaxy collection list community.hashi_vault
Store the AppRole secret_id with Ansible Vault as a fallback
The role_id and secret_id are the only credentials Ansible needs on disk. Encrypt the secret_id with Ansible Vault so it never sits in plaintext in your repository.
ansible-vault create group_vars/all/vault_approle.yml
vault_role_id: "9a8f3c2e-1234-4d5e-9f6a-abc123def456"
vault_secret_id: "b7e6d5c4-9876-4321-8888-fedcba098765"
Restrict who can decrypt this file by using a dedicated vault password file with strict permissions, not a password shared in chat.
touch ~/.ansible_vault_pass
chmod 600 ~/.ansible_vault_pass
echo "a-strong-unique-vault-password" > ~/.ansible_vault_pass
Never use chmod 777. It gives every user on the system full access to your files. Instead, fix ownership with chown and use minimal permissions such as 600 for private key material and vault password files.
Configure ansible.cfg to point to this password file so playbook runs can decrypt automatically in CI.
[defaults]
vault_password_file = ~/.ansible_vault_pass
collections_path = ~/.ansible/collections
Retrieve secrets dynamically with the hashi_vault lookup plugin
Reference the encrypted role_id and secret_id, then use the lookup plugin to pull live secrets from Vault at playbook run time instead of storing the final values anywhere.
---
- name: Deploy application with dynamic Vault secrets
hosts: app_servers
become: true
vars:
vault_addr: "https://vault.internal.example.com:8200"
tasks:
- name: Fetch database credentials from Vault
ansible.builtin.set_fact:
db_creds: "{{ lookup('community.hashi_vault.hashi_vault',
'secret=ansible/data/app/db:data',
url=vault_addr,
auth_method='approle',
role_id=vault_role_id,
secret_id=vault_secret_id) }}"
no_log: true
- name: Write application environment file
ansible.builtin.template:
src: templates/app.env.j2
dest: /etc/myapp/app.env
owner: myapp
group: myapp
mode: '0640'
no_log: true
The no_log: true directive stops decrypted secret values from appearing in Ansible output or logs. Always set it on tasks that handle secret material.
Alternative: use the hashi_vault lookup inline in variable files
For secrets referenced across many roles, define the lookup once in group_vars rather than repeating set_fact tasks.
db_password: "{{ lookup('community.hashi_vault.hashi_vault',
'secret=ansible/data/app/db:data.password',
url='https://vault.internal.example.com:8200',
auth_method='approle',
role_id=vault_role_id,
secret_id=vault_secret_id) }}"
Note: Lookups in group_vars are evaluated lazily per play, so Vault is queried fresh on every run. This keeps credentials current without a manual sync step, similar to how dynamic inventories work in configuring Ansible dynamic inventory for AWS, Azure and GCP.
Restrict token TTL and rotate the AppRole secret_id
Short-lived tokens limit the blast radius of a leaked credential. Rotate the secret_id on a schedule instead of leaving it valid indefinitely.
vault write -f auth/approle/role/ansible-automation/secret-id
vault list auth/approle/role/ansible-automation/secret-id-accessor
Revoke an old secret_id accessor once the new one is deployed to your Ansible Vault file.
vault write auth/approle/role/ansible-automation/secret-id-accessor/destroy \
secret_id_accessor="a1b2c3d4-role-accessor-id"
Verify your setup
Confirm Vault is unsealed, the AppRole authenticates correctly, and Ansible can retrieve a secret before running it against production hosts.
vault status
vault write auth/approle/login role_id="9a8f3c2e-1234-4d5e-9f6a-abc123def456" secret_id="b7e6d5c4-9876-4321-8888-fedcba098765"
ansible-playbook deploy-app.yml --check --diff --ask-vault-pass
Run a minimal test playbook that only performs a lookup and prints a masked confirmation, to validate connectivity without exposing secret values.
---
- name: Test Vault connectivity
hosts: localhost
tasks:
- name: Confirm secret retrieval
ansible.builtin.set_fact:
secret_retrieved: "{{ lookup('community.hashi_vault.hashi_vault',
'secret=ansible/data/app/db:data',
url='https://vault.internal.example.com:8200',
auth_method='approle',
role_id=vault_role_id,
secret_id=vault_secret_id) is defined }}"
no_log: true
- name: Report result
ansible.builtin.debug:
msg: "Vault lookup succeeded: {{ secret_retrieved }}"
Common issues
SymptomCauseFixpermission denied on secret pathAppRole policy does not include the exact KV pathCheck vault policy read ansible-readonly and add the missing path with capabilities = ["read"]invalid role or secret ID error during loginsecret_id expired or was single-use and already consumedGenerate a new secret_id with vault write -f auth/approle/role/ansible-automation/secret-id and update the Ansible Vault filehashi_vault lookup plugin not foundCollection not installed or wrong collections_path in ansible.cfgRun ansible-galaxy collection list community.hashi_vault and confirm the path matches ansible.cfgconnection refused to Vault APIFirewall blocking port 8200 or wrong VAULT_ADDRAllow the specific source IP range on port 8200 with a scoped firewall rule, do not disable the firewallsecrets appear in Ansible output logsMissing no_log directive on tasks handling secret valuesAdd no_log: true to every task that fetches or templates secret dataVault sealed after server rebootManual unseal required, no auto-unseal configuredConfigure auto-unseal with a cloud KMS or run vault operator unseal with quorum key shares
Next steps
Configure Ansible Vault for secret management and encryption with playbook automation
Set up Vault as a PKI certificate authority with SSL automation and intermediate CA
Integrate AWX with HashiCorp Vault for dynamic secrets management and secure automation workflows
Configure Vault dynamic secrets for databases with PostgreSQL and MySQL integration
Implement Ansible testing with Molecule and TestInfra for infrastructure automation validation
Automate HashiCorp Vault AppRole secret rotation with systemd timers
Running this in production?
Want this handled for you? Running this at scale adds a second layer of work: secret rotation schedules, Vault cluster failover drills, policy audits, and on-call coverage when a token expires mid-deployment. See how we run infrastructure like this for European teams.
---
### Implement Thanos Ruler for distributed alerting and recording rules across Prometheus clusters
URL: https://binadit.com/tutorials/implement-thanos-ruler-for-distributed-alerting
Category: monitoring
Difficulty: advanced
Time: ~60 minutes
Author: Binadit Tech Team
> Deploy Thanos Ruler to evaluate recording and alerting rules across multiple Prometheus clusters, integrate it with Alertmanager and S3-compatible storage, and run it in HA mode with deduplication.
What this solves
When you split Prometheus across multiple clusters, alerting rules that need a global view (like total error rate across regions) can't run reliably on a single local Prometheus. Thanos Ruler evaluates recording and alerting rules against Thanos Query's unified view, ships results to object storage as immutable TSDB blocks, and forwards alerts to Alertmanager with deduplication.
This tutorial covers architecture, installation, rule authoring, Alertmanager integration, S3/MinIO backend configuration, and running two Ruler replicas in HA with proper deduplication labels.
Understanding Thanos Ruler architecture
Thanos Ruler sits downstream of Thanos Query. It does not read metrics from local Prometheus TSDB directly; instead it queries the Thanos Query layer over gRPC or HTTP, which itself fans out to all connected Sidecars, Receivers, or Store Gateways across clusters. This lets a single rule evaluate against metrics from every cluster in your fleet.
Evaluated results (both recording rule outputs and alert series) are written to a local TSDB block by Ruler, then uploaded to object storage on the same interval as Prometheus does with Sidecar. Alerts are pushed to one or more Alertmanager instances using the standard Alertmanager v2 API. Because Ruler produces its own metrics, it also exposes a StoreAPI endpoint so Thanos Query can read historical rule results back, which is why Ruler is typically registered as another StoreAPI endpoint in Query's --store flag.
If you haven't set up the query layer yet, complete Set up Thanos Query and Compactor for distributed metrics querying first, since Ruler depends on a working Querier endpoint.
Note: Ruler is not a replacement for per-cluster alerting on local Prometheus. Keep fast, cluster-local alerts (disk full, node down) on Prometheus/Alertmanager directly, and use Ruler only for rules that genuinely need cross-cluster aggregation.
Step-by-step installation
Create a dedicated system user
Run Thanos Ruler as an unprivileged user, never as root.
sudo groupadd --system thanos
sudo useradd --system --gid thanos --no-create-home --shell /usr/sbin/nologin thanos
Download and install the Thanos binary
Thanos ships as a single static binary containing all components, including Ruler.
curl -L -o thanos.tar.gz https://github.com/thanos-io/thanos/releases/download/v0.36.1/thanos-0.36.1.linux-amd64.tar.gz
tar -xzf thanos.tar.gz
sudo mv thanos-0.36.1.linux-amd64/thanos /usr/local/bin/thanos
sudo chown root:root /usr/local/bin/thanos
sudo chmod 755 /usr/local/bin/thanos
thanos --version
Create directories for rules, data, and config
Ruler needs a local data directory for its TSDB block staging area and a rules directory for rule files.
sudo mkdir -p /etc/thanos/rules /etc/thanos/tls /var/lib/thanos-ruler
sudo chown -R thanos:thanos /etc/thanos /var/lib/thanos-ruler
sudo chmod 750 /var/lib/thanos-ruler
sudo chmod 750 /etc/thanos/rules
Defining recording and alerting rule files
Write a cross-cluster recording rule
Recording rules precompute expensive aggregations. This example rolls up HTTP error rate across every cluster labeled with a cluster label injected by each Prometheus's external_labels.
groups:
- name: global-http-slo
interval: 30s
rules:
- record: job:http_requests_error_rate:ratio_rate5m
expr: |
sum by (job, cluster) (rate(http_requests_total{status=~"5.."}[5m]))
/
sum by (job, cluster) (rate(http_requests_total[5m]))
- record: global:http_requests_error_rate:ratio_rate5m
expr: |
sum(rate(http_requests_total{status=~"5.."}[5m]))
/
sum(rate(http_requests_total[5m]))
Write a cross-cluster alerting rule
This alert fires only when the global error rate across all clusters exceeds a threshold, something a single cluster's Prometheus cannot evaluate on its own.
groups:
- name: global-alerts
interval: 30s
rules:
- alert: GlobalHighErrorRate
expr: global:http_requests_error_rate:ratio_rate5m > 0.05
for: 10m
labels:
severity: critical
scope: global
annotations:
summary: "Global HTTP error rate above 5% across all clusters"
description: "Aggregate 5xx error rate is {{ $value | humanizePercentage }} across all connected Prometheus clusters."
- alert: ClusterMissingFromQuery
expr: absent(up{job="prometheus", cluster="eu-west-1"})
for: 5m
labels:
severity: warning
scope: global
annotations:
summary: "Cluster eu-west-1 has stopped reporting to Thanos Query"
Warning: Every Prometheus feeding Thanos Query must set a unique external_labels.cluster value. Without it, series from different clusters collide during query-time deduplication and rule evaluation produces incorrect results.
Configuring Thanos Ruler with Thanos Query
Point Ruler at Query endpoints
Ruler evaluates rules by querying one or more Thanos Query instances over HTTP, and can also query multiple endpoints directly for failover.
THANOS_QUERY_ENDPOINTS="--query=203.0.113.10:10904 --query=203.0.113.11:10904"
THANOS_LABELS="--label=ruler_replica=\"ruler-1\" --label=cluster=\"global\""
Create the systemd unit
This unit runs Ruler with rules, TSDB block config, and Query discovery. Adjust the IP addresses in --query to match your Thanos Query instances.
[Unit]
Description=Thanos Ruler
After=network-online.target
Wants=network-online.target
[Service]
Type=simple
User=thanos
Group=thanos
ExecStart=/usr/local/bin/thanos rule \
--data-dir=/var/lib/thanos-ruler \
--rule-file=/etc/thanos/rules/*.yml \
--eval-interval=30s \
--query=203.0.113.10:10904 \
--query=203.0.113.11:10904 \
--alertmanagers.url=http://203.0.113.20:9093 \
--alertmanagers.url=http://203.0.113.21:9093 \
--alert.query-url=http://203.0.113.30:10902 \
--label=ruler_replica="ruler-1" \
--label=cluster="global" \
--objstore.config-file=/etc/thanos/objstore.yml \
--http-address=0.0.0.0:10904 \
--grpc-address=0.0.0.0:10901 \
--tsdb.retention=48h
Restart=on-failure
RestartSec=5
LimitNOFILE=65536
[Install]
WantedBy=multi-user.target
The --alert.query-url flag sets the link embedded in alerts so recipients can jump straight to the query that triggered them in the Thanos Query UI.
Integrating with Alertmanager
Configure Alertmanager targets
Ruler pushes alerts to every Alertmanager listed via --alertmanagers.url. It automatically deduplicates alerts sent to multiple Alertmanager replicas because Alertmanager's own gossip protocol handles that layer.
Note: If you haven't deployed Alertmanager in HA yet, follow Implement Alertmanager high availability clustering with automatic failover and load balancing before wiring Ruler to it.
Verify the alert route accepts global-scope alerts
Add a route in Alertmanager's config that matches the scope: global label used in the rule file above, so these alerts can be routed to a different receiver (e.g. a platform team channel) than per-cluster alerts.
route:
receiver: default
routes:
- match:
scope: global
receiver: platform-team
group_by: ['alertname']
group_wait: 30s
receivers:
- name: default
slack_configs:
- channel: '#alerts'
- name: platform-team
slack_configs:
- channel: '#platform-global-alerts'
For webhook or PagerDuty routing patterns, see Configure Prometheus Alertmanager with custom webhook integrations for Slack, Microsoft Teams, and PagerDuty notifications.
Setting up the object storage backend
Create an S3 or MinIO bucket for rule evaluation blocks
Ruler uploads completed TSDB blocks the same way Sidecar does, so it needs its own bucket path or a shared bucket with a distinct prefix.
mc alias set thanosminio https://minio.example.com minioadmin 'ChangeThisStrongSecret123!'
mc mb thanosminio/thanos-ruler-blocks
mc anonymous set none thanosminio/thanos-ruler-blocks
Write the object storage config file
This file is shared in structure with the one used by Thanos Query's Store Gateway and Compactor.
type: S3
config:
bucket: thanos-ruler-blocks
endpoint: minio.example.com:443
access_key: minioadmin
secret_key: ChangeThisStrongSecret123!
insecure: false
signature_version2: false
http_config:
tls_config:
insecure_skip_verify: false
sudo chown thanos:thanos /etc/thanos/objstore.yml
sudo chmod 640 /etc/thanos/objstore.yml
Warning: The objstore.yml file contains credentials. Restrict it to 640 with ownership under the thanos user and group, never 644 or 777. If Alertmanager or Ruler cannot read it, fix ownership with chown, not by loosening permissions.
If you're running MinIO for this backend, review Setup MinIO monitoring with Prometheus and Grafana dashboards to keep an eye on bucket usage and latency.
Start Ruler
sudo systemctl daemon-reload
sudo systemctl enable --now thanos-ruler
sudo systemctl status thanos-ruler
Deploying Thanos Ruler in high availability mode
Run two identical Ruler replicas
Deploy a second Ruler instance on a separate host with the same rule files but a different ruler_replica label. Both replicas evaluate the same rules independently against the same Query endpoints.
rsync -avz --delete /etc/thanos/rules/ thanos-ruler-2:/etc/thanos/rules/
On the second host, change only the replica label in the systemd unit:
--label=ruler_replica="ruler-2" \
--label=cluster="global" \
Enable deduplication at Thanos Query
Thanos Query deduplicates series with matching labels except for the replica label you specify with --query.replica-label. This means alerts and recording rule outputs from both Ruler replicas collapse into a single series when read back through Query.
thanos query \
--http-address=0.0.0.0:10902 \
--grpc-address=0.0.0.0:10901 \
--query.replica-label=ruler_replica \
--query.replica-label=replica \
--endpoint=203.0.113.30:10901 \
--endpoint=203.0.113.31:10901
Both Ruler replicas will independently fire the same alert and push it to Alertmanager. Alertmanager's own deduplication and grouping, based on the alert's label set minus the replica label, collapses duplicate notifications into one.
Register Ruler as a StoreAPI in Query
Add each Ruler's gRPC address as an endpoint on Thanos Query so historical rule evaluation results become queryable through the same Query layer.
thanos query \
--endpoint=203.0.113.30:10901 \
--endpoint=203.0.113.31:10901 \
--endpoint=203.0.113.40:10901 \
--endpoint=203.0.113.41:10901
Here the first two endpoints are Rulers and the last two are Sidecars or Receivers from your existing clusters.
Monitoring Thanos Ruler performance
Scrape Ruler's own metrics
Ruler exposes Prometheus metrics on its HTTP address, including rule evaluation duration, failures, and alert send errors.
scrape_configs:
- job_name: thanos-ruler
static_configs:
- targets:
- 203.0.113.30:10904
- 203.0.113.31:10904
Alert on rule evaluation failures
These meta-alerts catch cases where Ruler itself is unhealthy, which is critical since a broken Ruler means silent gaps in your global alerting.
groups:
- name: thanos-ruler-health
rules:
- alert: ThanosRuleEvaluationFailures
expr: rate(prometheus_rule_evaluation_failures_total[5m]) > 0
for: 10m
labels:
severity: warning
annotations:
summary: "Thanos Ruler is failing to evaluate rules"
- alert: ThanosRuleAlertmanagerSendFailures
expr: rate(thanos_alert_sender_alerts_dropped_total[5m]) > 0
for: 5m
labels:
severity: critical
annotations:
summary: "Thanos Ruler cannot deliver alerts to Alertmanager"
- alert: ThanosRuleHighRuleEvaluationLatency
expr: histogram_quantile(0.99, rate(prometheus_rule_evaluation_duration_seconds_bucket[5m])) > 5
for: 10m
labels:
severity: warning
annotations:
summary: "Rule evaluation p99 latency exceeds 5 seconds"
For dashboarding these metrics, reuse the panel patterns from Configure advanced Grafana dashboards and alerting with Prometheus integration.
Verify your setup
curl -s http://203.0.113.30:10904/-/ready
curl -s http://203.0.113.30:10904/api/v1/rules | head -c 500
curl -s http://203.0.113.30:10902/api/v1/query?query=global:http_requests_error_rate:ratio_rate5m
mc ls thanosminio/thanos-ruler-blocks
sudo systemctl status thanos-ruler --no-pager
sudo journalctl -u thanos-ruler -n 50 --no-pager
Common issues
SymptomCauseFixRules never evaluate, no series producedRuler cannot reach any Thanos Query endpointCheck --query flags and confirm network reachability with curl 203.0.113.10:10904/-/readyDuplicate Slack notifications for the same alertQuery is not deduplicating on the ruler_replica labelAdd --query.replica-label=ruler_replica to the Thanos Query commandAlerts fire but never reach AlertmanagerWrong or unreachable --alertmanagers.urlVerify Alertmanager is listening on 9093 and check firewall rules between Ruler and Alertmanager hostsBlocks never appear in the bucketInvalid or missing objstore.yml credentialsCheck journalctl -u thanos-ruler for upload errors and confirm credentials in objstore.ymlPermission denied reading objstore.ymlFile owned by root but Ruler runs as thanos userRun sudo chown thanos:thanos /etc/thanos/objstore.yml, never chmod 777Recording rule values look wrong or double countedMissing or duplicate external_labels.cluster on source Prometheus instancesEnsure every Prometheus has a unique cluster label before querying through Thanos QueryHigh rule evaluation latencyQuery fan-out across too many Store Gateways per evaluationReduce eval-interval scope or add more Query replicas behind a load balancer
Next steps
Set up Thanos Query and Compactor for distributed metrics querying
Implement Alertmanager high availability clustering with automatic failover and load balancing
Implement Thanos multi-cluster federation for global Prometheus metrics aggregation
Configure Thanos Receiver clustering for high availability and load distribution
Build Grafana dashboards for Thanos Ruler evaluation metrics
Running this in production?
Want this handled for you? Running this at scale adds a second layer of work: capacity planning for query fan-out, failover drills between Ruler replicas, S3 storage cost control, and being the one paged when global alerts stop firing. See how we run infrastructure like this for European teams.
---
### Set up Thanos Query and Compactor for distributed metrics querying
URL: https://binadit.com/tutorials/setup-thanos-query-and-compactor-for-distributed-metrics
Category: monitoring
Difficulty: advanced
Time: ~75 minutes
Author: Binadit Tech Team
> Deploy Thanos Sidecar, Query, and Compactor to unify multiple Prometheus instances into a single global query view with long-term S3-backed storage. Covers deduplication, TLS, and production troubleshooting.
What this solves
A single Prometheus instance cannot scale beyond one cluster or retain years of metrics cheaply. Thanos adds a global query layer across multiple Prometheus servers and offloads long-term storage to S3-compatible object storage with automated downsampling and compaction.
This tutorial deploys Thanos Sidecar, Query, Store Gateway, and Compactor across two Prometheus instances, backed by MinIO object storage, with TLS between components and Prometheus metrics for the Thanos processes themselves.
Understanding Thanos architecture
Thanos Sidecar runs next to each Prometheus instance and exposes its local TSDB blocks over gRPC, while also uploading completed blocks to object storage. Thanos Query (Querier) talks to all Sidecars and Store Gateways via the Store API, merges results, and deduplicates data from replicated Prometheus pairs. Store Gateway serves historical blocks directly from object storage without touching local Prometheus disks. Compactor runs as a singleton process that merges, downsamples, and applies retention rules to blocks in the bucket.
Note: Only one Compactor instance should ever run against a given bucket. Running two in parallel corrupts block metadata.
Step-by-step installation
Provision the object storage backend
Thanos needs an S3-compatible bucket for blocks. This example uses MinIO on a dedicated host at 203.0.113.20. If you already run MinIO, you can reuse it, see Install and configure MinIO object storage with SSL and clustering for a full setup.
mc alias set myminio https://203.0.113.20:9000 thanosadmin 'Str0ngM1nioSecretKey!'
mc mb myminio/thanos-metrics
mc admin user add myminio thanos-svc 'An0therStr0ngSecretKey!'
mc admin policy attach myminio readwrite --user thanos-svc
Create the object storage config file
Every Thanos component that talks to object storage needs this file. Keep it out of version control since it contains credentials.
type: S3
config:
bucket: "thanos-metrics"
endpoint: "203.0.113.20:9000"
access_key: "thanos-svc"
secret_key: "An0therStr0ngSecretKey!"
insecure: false
signature_version2: false
http_config:
tls_config:
ca_file: /etc/thanos/certs/minio-ca.pem
sudo mkdir -p /etc/thanos/certs
sudo chown -R thanos:thanos /etc/thanos
sudo chmod 640 /etc/thanos/bucket.yml
Create the thanos system user and download binaries
Run Thanos as a dedicated unprivileged user, never as root.
sudo useradd --system --no-create-home --shell /usr/sbin/nologin thanos
sudo apt update && sudo apt install -y curl tar
curl -L -o /tmp/thanos.tar.gz https://github.com/thanos-io/thanos/releases/download/v0.36.1/thanos-0.36.1.linux-amd64.tar.gz
sudo tar -xzf /tmp/thanos.tar.gz -C /opt
sudo ln -s /opt/thanos-0.36.1.linux-amd64/thanos /usr/local/bin/thanos
sudo useradd --system --no-create-home --shell /sbin/nologin thanos
sudo dnf install -y curl tar
curl -L -o /tmp/thanos.tar.gz https://github.com/thanos-io/thanos/releases/download/v0.36.1/thanos-0.36.1.linux-amd64.tar.gz
sudo tar -xzf /tmp/thanos.tar.gz -C /opt
sudo ln -s /opt/thanos-0.36.1.linux-amd64/thanos /usr/local/bin/thanos
Deploy Thanos Sidecar alongside each Prometheus instance
The sidecar needs read access to Prometheus's data directory and its config reload endpoint. Run this on each Prometheus host (203.0.113.11 and 203.0.113.12 in this example).
[Unit]
Description=Thanos Sidecar
After=network.target prometheus.service
[Service]
User=thanos
Group=thanos
ExecStart=/usr/local/bin/thanos sidecar \
--tsdb.path=/var/lib/prometheus \
--prometheus.url=http://127.0.0.1:9090 \
--grpc-address=0.0.0.0:10901 \
--http-address=127.0.0.1:10902 \
--objstore.config-file=/etc/thanos/bucket.yml \
--grpc-server-tls-cert=/etc/thanos/certs/sidecar.pem \
--grpc-server-tls-key=/etc/thanos/certs/sidecar-key.pem \
--grpc-server-tls-client-ca=/etc/thanos/certs/ca.pem
Restart=on-failure
LimitNOFILE=65536
[Install]
WantedBy=multi-user.target
Add the sidecar's user to the prometheus group so it can read the TSDB directory without loosening permissions on other files.
sudo usermod -aG prometheus thanos
sudo chmod 750 /var/lib/prometheus
sudo systemctl daemon-reload
sudo systemctl enable --now thanos-sidecar
Enable Prometheus remote-write flags for the sidecar
Prometheus must run with a minimum retention that overlaps with the compactor's upload window, and expose the admin API for block cleanup.
[Service]
ExecStart=
ExecStart=/usr/local/bin/prometheus \
--config.file=/etc/prometheus/prometheus.yml \
--storage.tsdb.path=/var/lib/prometheus \
--storage.tsdb.min-block-duration=2h \
--storage.tsdb.max-block-duration=2h \
--web.enable-lifecycle
sudo systemctl daemon-reload
sudo systemctl restart prometheus
Warning: setting min-block-duration equal to max-block-duration disables local compaction so Thanos Compactor owns it exclusively. Do not run Prometheus's own compaction alongside Thanos Compactor on the same blocks.
Deploy Thanos Query for the global view
Query connects to both sidecars directly by gRPC address and, optionally, a Store Gateway for historical data. Run this on a dedicated query node at 203.0.113.30.
[Unit]
Description=Thanos Query
After=network.target
[Service]
User=thanos
Group=thanos
ExecStart=/usr/local/bin/thanos query \
--http-address=0.0.0.0:10904 \
--grpc-address=0.0.0.0:10903 \
--query.replica-label=replica \
--store=203.0.113.11:10901 \
--store=203.0.113.12:10901 \
--store=203.0.113.20:10905 \
--grpc-client-tls-cert=/etc/thanos/certs/query-client.pem \
--grpc-client-tls-key=/etc/thanos/certs/query-client-key.pem \
--grpc-client-tls-ca=/etc/thanos/certs/ca.pem
Restart=on-failure
[Install]
WantedBy=multi-user.target
sudo systemctl daemon-reload
sudo systemctl enable --now thanos-query
Configure deduplication and service discovery
If you run Prometheus in HA pairs, label each replica with a distinct replica label at scrape time so Query can deduplicate identical series. For dynamic environments, use file-based service discovery instead of hardcoded --store flags.
global:
external_labels:
replica: A
cluster: prod-eu-west
[
{
"targets": ["203.0.113.11:10901", "203.0.113.12:10901"]
}
]
Replace the static --store flags with --store.sd-files=/etc/thanos/stores.json and --store.sd-interval=30s so Query picks up new sidecars without a restart. This pairs well with existing service discovery patterns described in Configure HAProxy with Consul for dynamic service discovery if you already run Consul.
Deploy Thanos Store Gateway
Store Gateway serves compacted historical blocks from object storage so Query does not need to hit every sidecar for old data.
[Unit]
Description=Thanos Store Gateway
After=network.target
[Service]
User=thanos
Group=thanos
ExecStart=/usr/local/bin/thanos store \
--data-dir=/var/lib/thanos-store \
--objstore.config-file=/etc/thanos/bucket.yml \
--grpc-address=0.0.0.0:10905 \
--http-address=127.0.0.1:10906 \
--grpc-server-tls-cert=/etc/thanos/certs/store.pem \
--grpc-server-tls-key=/etc/thanos/certs/store-key.pem \
--grpc-server-tls-client-ca=/etc/thanos/certs/ca.pem
Restart=on-failure
[Install]
WantedBy=multi-user.target
sudo mkdir -p /var/lib/thanos-store
sudo chown thanos:thanos /var/lib/thanos-store
sudo chmod 750 /var/lib/thanos-store
sudo systemctl daemon-reload
sudo systemctl enable --now thanos-store
Deploy Thanos Compactor for downsampling and retention
Run exactly one Compactor instance against the bucket. It handles vertical compaction, downsampling to 5m and 1h resolutions, and retention enforcement.
[Unit]
Description=Thanos Compactor
After=network.target
[Service]
User=thanos
Group=thanos
ExecStart=/usr/local/bin/thanos compact \
--data-dir=/var/lib/thanos-compact \
--objstore.config-file=/etc/thanos/bucket.yml \
--http-address=127.0.0.1:10907 \
--retention.resolution-raw=30d \
--retention.resolution-5m=180d \
--retention.resolution-1h=730d \
--wait \
--compact.concurrency=2
Restart=on-failure
[Install]
WantedBy=multi-user.target
sudo mkdir -p /var/lib/thanos-compact
sudo chown thanos:thanos /var/lib/thanos-compact
sudo chmod 750 /var/lib/thanos-compact
sudo systemctl daemon-reload
sudo systemctl enable --now thanos-compact
Note: the retention flags apply per resolution level. Raw data is pruned after 30 days but downsampled 1h data survives for 2 years, keeping long-range dashboards fast and cheap to store.
Generate TLS certificates for gRPC communication
All Thanos gRPC traffic between Sidecar, Query, and Store Gateway should be encrypted with mutual TLS in production. Use a private CA rather than self-signed certs per node.
openssl genrsa -out /etc/thanos/certs/ca-key.pem 4096
openssl req -x509 -new -nodes -key /etc/thanos/certs/ca-key.pem -sha256 -days 3650 \
-out /etc/thanos/certs/ca.pem -subj "/CN=thanos-internal-ca"
openssl genrsa -out /etc/thanos/certs/query-client-key.pem 2048
openssl req -new -key /etc/thanos/certs/query-client-key.pem \
-out /etc/thanos/certs/query-client.csr -subj "/CN=thanos-query"
openssl x509 -req -in /etc/thanos/certs/query-client.csr -CA /etc/thanos/certs/ca.pem \
-CAkey /etc/thanos/certs/ca-key.pem -CAcreateserial -out /etc/thanos/certs/query-client.pem -days 825 -sha256
sudo chown -R thanos:thanos /etc/thanos/certs
sudo chmod 600 /etc/thanos/certs/*-key.pem
sudo chmod 644 /etc/thanos/certs/*.pem
Never use chmod 777. Private keys must stay readable only by the thanos user, other Thanos components need only the CA certificate to verify peers. Set 600 on private keys and 640 or 644 on certificates, and use chown to grant access to the correct service account instead of opening permissions to everyone.
If you already manage internal PKI, see Set up Vault as a PKI certificate authority for issuing and rotating these certificates automatically instead of managing openssl by hand.
Put Query behind a reverse proxy with authentication
Thanos Query's HTTP UI and API have no built-in authentication. Front it with a reverse proxy that enforces TLS and basic auth or OAuth2 for anyone outside the internal network.
server {
listen 443 ssl;
server_name thanos.example.com;
ssl_certificate /etc/letsencrypt/live/thanos.example.com/fullchain.pem;
ssl_certificate_key /etc/letsencrypt/live/thanos.example.com/privkey.pem;
auth_basic "Thanos Query";
auth_basic_user_file /etc/nginx/.htpasswd-thanos;
location / {
proxy_pass http://127.0.0.1:10904;
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
}
}
For certificate automation, follow Implement Nginx SSL certificate automation with Let's Encrypt using Certbot.
Monitoring Thanos Query and Compactor performance
Each Thanos component exposes its own Prometheus metrics on its HTTP address. Scrape them from a dedicated monitoring Prometheus instance separate from the ones being federated.
scrape_configs:
- job_name: thanos-query
static_configs:
- targets: ['203.0.113.30:10904']
- job_name: thanos-compact
static_configs:
- targets: ['203.0.113.20:10907']
- job_name: thanos-store
static_configs:
- targets: ['203.0.113.20:10906']
Key metrics to alert on: thanos_compact_group_compactions_failures_total for failed compactions, thanos_query_concurrent_gate_queries_max for query concurrency saturation, and thanos_compact_halted which is 1 when the compactor has hit a fatal block error and stopped processing. Build alert rules following the pattern in Configure Prometheus alerting with AlertManager notifications and webhook integration.
Verify your setup
sudo systemctl status thanos-sidecar thanos-query thanos-store thanos-compact
curl -s http://127.0.0.1:10904/-/healthy
curl -s http://127.0.0.1:10904/api/v1/stores | python3 -m json.tool
Open the Query UI at https://thanos.example.com/stores and confirm both sidecars and the store gateway show as UP. Run a test query spanning both Prometheus instances to confirm the global view merges results correctly.
mc ls myminio/thanos-metrics/ | head
mc ls myminio/thanos-metrics/debug/ 2>/dev/null
Common issues
SymptomCauseFixQuery shows store as downgRPC TLS handshake failure between Query and SidecarVerify CA cert matches on both sides with openssl verify -CAfile ca.pem sidecar.pemDuplicate data points in graphsMissing or mismatched replica labelSet unique external_labels.replica per Prometheus and pass --query.replica-label=replicaCompactor halted with block overlap errorTwo compactors ran against the same bucket concurrentlyStop the duplicate process, manually inspect overlapping blocks with thanos tools bucket verify, run repair if neededSidecar not uploading blocksPrometheus min/max block duration set to default (2h auto-compact), or wrong bucket credentialsConfirm both duration flags are equal and check sidecar logs for S3 auth errorsStore gateway high memory usageIndex cache too large for available RAMTune --index-cache-size and --chunk-pool-size to match available memoryQuery returns 503 under loadToo many concurrent queries against limited gateIncrease --query.max-concurrent carefully and add caching layer in front
Next steps
Set up Thanos Receiver for remote write scalability with Prometheus integration
Configure Thanos Receiver clustering for high availability and load distribution
Configure Thanos Ruler for distributed alerting across multiple Prometheus clusters
Implement Thanos multi-cluster federation for global Prometheus metrics aggregation
Configure Prometheus long-term storage with Thanos for unlimited data retention
Setup MinIO monitoring with Prometheus and Grafana dashboards
Running this in production?
Want this handled for you? Running Thanos at scale adds a second layer of work: bucket lifecycle policies, compactor capacity planning, certificate rotation across every component, and being paged when compaction halts at 3am. Talk to an engineer if you'd rather hand the ops side off.
---
### Configure Kubernetes network monitoring with Hubble and Cilium for traffic visibility
URL: https://binadit.com/tutorials/configure-kubernetes-network-monitoring-with-hubble-and-cilium
Category: monitoring
Difficulty: advanced
Time: ~75 minutes
Author: Binadit Tech Team
> Enable Hubble on Cilium to get real-time eBPF-based flow visibility, deploy Hubble UI and Relay, and integrate flow metrics with Prometheus and Grafana for full Kubernetes network observability.
What this solves
Cilium's eBPF dataplane already sees every packet that crosses your pods, but without Hubble you have no way to query that traffic. This tutorial enables Hubble on an existing Cilium installation, deploys Hubble Relay and Hubble UI, configures the CLI for live flow inspection, and wires flow metrics into Prometheus and Grafana.
By the end you will be able to see which pods talk to which services, which network policies drop traffic, and why, without tailing application logs.
Note: This tutorial assumes Cilium is already installed as your cluster CNI. If you have not set that up yet, follow Install and configure Cilium CNI for Kubernetes with eBPF networking and security policies first.
Prerequisites and Cilium CNI verification
Confirm cluster and tooling
You need a running Kubernetes cluster (kubeadm, EKS, GKE, or self-managed) with Cilium as the CNI, kubectl configured against it, and Helm 3 installed on your workstation.
kubectl version --client
helm version
kubectl get nodes -o wide
Install the Cilium CLI
The Cilium CLI is used to check connectivity, install Hubble, and validate the dataplane. Install the latest release binary directly.
CILIUM_CLI_VERSION=$(curl -s https://raw.githubusercontent.com/cilium/cilium-cli/main/stable.txt)
curl -L --fail --remote-name-all https://github.com/cilium/cilium-cli/releases/download/${CILIUM_CLI_VERSION}/cilium-linux-amd64.tar.gz
sudo tar xzvfC cilium-linux-amd64.tar.gz /usr/local/bin
rm cilium-linux-amd64.tar.gz
CILIUM_CLI_VERSION=$(curl -s https://raw.githubusercontent.com/cilium/cilium-cli/main/stable.txt)
curl -L --fail --remote-name-all https://github.com/cilium/cilium-cli/releases/download/${CILIUM_CLI_VERSION}/cilium-linux-amd64.tar.gz
sudo tar xzvfC cilium-linux-amd64.tar.gz /usr/local/bin
rm cilium-linux-amd64.tar.gz
Verify Cilium status and connectivity
Before enabling Hubble, confirm Cilium itself is healthy across every node. Any node reporting degraded status will produce incomplete flow data later.
cilium status --wait
kubectl -n kube-system get pods -l k8s-app=cilium
Run a full connectivity test if this is a new cluster. It deploys temporary test pods and validates pod-to-pod, pod-to-service, and cross-node traffic.
cilium connectivity test --test-namespace cilium-test
Warning: The connectivity test creates workloads that consume cluster resources for several minutes. Do not run it against a production namespace under heavy load.
Installing and enabling Hubble in Cilium
Enable Hubble on the Cilium DaemonSet
Hubble ships as part of Cilium but is disabled by default. Enabling it turns on the local Hubble server inside each Cilium agent, which exposes flow data over a Unix socket.
cilium hubble enable --ui
This patches the Cilium ConfigMap, restarts the agent pods, and additionally deploys Hubble Relay and Hubble UI in one step. Wait for the rollout to finish.
kubectl -n kube-system rollout status daemonset/cilium
cilium status --wait
Confirm Hubble is reporting flows
Check that the Hubble metrics server and flow API are active inside the agent pods.
kubectl -n kube-system exec -it ds/cilium -- cilium status | grep Hubble
Expected output shows Hubble: Ok along with the relay address once the next steps are complete.
Deploying Hubble Relay and Hubble UI
Verify Relay and UI pods
Hubble Relay aggregates flow data from every agent into a single gRPC endpoint. Hubble UI is the web frontend that queries Relay. Both were deployed by the enable command above, so confirm they are running.
kubectl -n kube-system get pods -l k8s-app=hubble-relay
kubectl -n kube-system get pods -l k8s-app=hubble-ui
Access Hubble UI locally for testing
Use port forwarding to reach the UI before you expose it externally through Ingress.
cilium hubble ui
This opens a browser tab pointed at http://localhost:12000. If you prefer manual port forwarding instead of the CLI wrapper, use kubectl directly.
kubectl -n kube-system port-forward svc/hubble-ui 12000:80
Configuring Hubble CLI for flow inspection
Install the Hubble CLI binary
The Hubble CLI queries Relay directly from your terminal, which is faster than the UI for scripting and troubleshooting.
HUBBLE_VERSION=$(curl -s https://raw.githubusercontent.com/cilium/hubble/master/stable.txt)
curl -L --fail --remote-name-all https://github.com/cilium/hubble/releases/download/${HUBBLE_VERSION}/hubble-linux-amd64.tar.gz
sudo tar xzvfC hubble-linux-amd64.tar.gz /usr/local/bin
rm hubble-linux-amd64.tar.gz
Point Hubble CLI at Relay
Forward Relay's gRPC port locally, then export it as the Hubble server address.
kubectl -n kube-system port-forward svc/hubble-relay 4245:80 &
export HUBBLE_SERVER=localhost:4245
Query live flows
Observe flows in real time, filter by namespace, or watch only dropped packets to debug connectivity issues.
hubble observe --server $HUBBLE_SERVER --last 20
hubble observe --server $HUBBLE_SERVER --namespace production --follow
hubble observe --server $HUBBLE_SERVER --verdict DROPPED --follow
Each line shows source and destination identity, port, protocol, and the verdict (forwarded, dropped, or redirected), which is far more useful than raw packet captures for debugging Kubernetes network policies.
Enabling network policy visibility and observability metrics
Turn on policy verdict metrics
By default Hubble exposes basic flow metrics. Enable additional metrics for DNS, HTTP, and policy verdicts by editing the Cilium Helm values or patching the ConfigMap.
helm upgrade cilium cilium/cilium --namespace kube-system --reuse-values \
--set hubble.metrics.enabled="{dns,drop,tcp,flow,icmp,http,policy}" \
--set hubble.metrics.enableOpenMetrics=true
This restarts the Cilium agents again, so run it during a maintenance window on production clusters.
kubectl -n kube-system rollout status daemonset/cilium
Test policy visibility with a deny rule
Apply a simple CiliumNetworkPolicy and confirm Hubble reports the resulting drops. This is the same mechanism used by Implement Kubernetes network policies for pod-to-pod security and traffic isolation, but Hubble gives you the visibility layer that policy alone does not.
cat <<'EOF' | kubectl apply -f -
apiVersion: "cilium.io/v2"
kind: CiliumNetworkPolicy
metadata:
name: deny-egress-test
namespace: production
spec:
endpointSelector:
matchLabels:
app: checkout
egress:
- toEndpoints:
- matchLabels:
app: catalog
EOF
hubble observe --server $HUBBLE_SERVER --namespace production --verdict DROPPED --follow
Exposing Hubble UI securely via Ingress
Create a dedicated namespace and basic auth secret
Never expose Hubble UI publicly without authentication. It reveals internal service topology and traffic patterns that should stay behind an authenticated proxy.
sudo apt install -y apache2-utils
htpasswd -c auth hubble-admin
kubectl -n kube-system create secret generic hubble-ui-basic-auth --from-file=auth
Never use chmod 777. The generated auth file only needs to be readable by your own user before it becomes a Kubernetes secret. Keep it at 600 permissions and delete it locally once the secret is created.
chmod 600 auth
rm auth
Create the Ingress resource
This assumes an NGINX ingress controller is already installed, as covered in Configure Kubernetes ingress controller with NGINX and SSL certificates using cert-manager.
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: hubble-ui
namespace: kube-system
annotations:
nginx.ingress.kubernetes.io/auth-type: basic
nginx.ingress.kubernetes.io/auth-secret: hubble-ui-basic-auth
nginx.ingress.kubernetes.io/auth-realm: "Authentication required"
cert-manager.io/cluster-issuer: letsencrypt-prod
spec:
ingressClassName: nginx
tls:
- hosts:
- hubble.example.com
secretName: hubble-ui-tls
rules:
- host: hubble.example.com
http:
paths:
- path: /
pathType: Prefix
backend:
service:
name: hubble-ui
port:
number: 80
kubectl apply -f /tmp/hubble-ui-ingress.yaml
kubectl -n kube-system get ingress hubble-ui
Restrict access at the network layer too
Add a CiliumNetworkPolicy that only allows the ingress controller to reach Hubble UI, so even a leaked credential cannot be used from an arbitrary pod.
apiVersion: "cilium.io/v2"
kind: CiliumNetworkPolicy
metadata:
name: restrict-hubble-ui-ingress
namespace: kube-system
spec:
endpointSelector:
matchLabels:
k8s-app: hubble-ui
ingress:
- fromEndpoints:
- matchLabels:
k8s:io.kubernetes.pod.namespace: ingress-nginx
kubectl apply -f /tmp/hubble-ui-policy.yaml
Integrating Hubble metrics with Prometheus and Grafana
Expose Prometheus scrape annotations
Hubble metrics are already exposed on port 9965 by default once enabled in an earlier step. Confirm the endpoint and annotate the service for scraping if you use annotation-based discovery instead of ServiceMonitors.
kubectl -n kube-system get svc hubble-metrics
curl -s http://localhost:9965/metrics | head -n 20
Create a ServiceMonitor for Prometheus Operator
If your cluster uses the Prometheus Operator, as set up in Set up Kubernetes monitoring with Prometheus Operator and custom metrics, add a ServiceMonitor that targets the Hubble metrics service.
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
name: hubble-metrics
namespace: kube-system
labels:
release: prometheus
spec:
selector:
matchLabels:
k8s-app: hubble
namespaceSelector:
matchNames:
- kube-system
endpoints:
- port: hubble-metrics
interval: 30s
kubectl apply -f /tmp/hubble-servicemonitor.yaml
Import a Grafana dashboard for flow data
Cilium publishes an official Hubble Grafana dashboard. Import it via the Cilium community JSON model, then confirm the panels populate using the metrics enabled earlier.
curl -o hubble-dashboard.json https://raw.githubusercontent.com/cilium/cilium/main/examples/kubernetes/addons/prometheus/files/grafana/dashboards/hubble-dashboard.json
Import this file through the Grafana UI under Dashboards, Import, and select your Prometheus data source. Key panels to watch are dropped packets by policy, DNS response codes, and top talkers by namespace.
Note: If you already run a shared Prometheus and Grafana stack for other services, see Configure advanced Grafana dashboards and alerting with Prometheus integration for alert rule patterns you can reuse for Hubble drop-rate alerts.
Troubleshooting traffic flows and connectivity issues
Trace a specific pod-to-pod connection
Use Hubble's identity-aware filters to isolate traffic between two workloads without noise from the rest of the cluster.
hubble observe --server $HUBBLE_SERVER \
--from-pod production/checkout-7f9c8d \
--to-pod production/catalog-5b6d9f \
--last 50
Check for DNS resolution failures
DNS problems often masquerade as generic connectivity failures. Filter for DNS flows specifically.
hubble observe --server $HUBBLE_SERVER --protocol dns --follow
Inspect agent-level errors
If Hubble itself reports no data for a node, check the Cilium agent logs on that node directly.
kubectl -n kube-system logs ds/cilium -c cilium-agent --since=10m | grep -i hubble
Verify your setup
cilium status --wait
hubble observe --server $HUBBLE_SERVER --last 5
kubectl -n kube-system get pods -l k8s-app=hubble-relay
kubectl -n kube-system get pods -l k8s-app=hubble-ui
curl -sk https://hubble.example.com -u hubble-admin:yourpassword -o /dev/null -w "%{http_code}\n"
Common issues
SymptomCauseFixhubble observe returns no flowsCLI not pointed at Relay, or Relay not runningConfirm kubectl -n kube-system get pods -l k8s-app=hubble-relay is Running, then re-export HUBBLE_SERVERHubble UI shows empty service maphubble.metrics.enabled missing flow typesRe-run the Helm upgrade with the full metrics list including flow and httpIngress returns 502 for hubble.example.comCiliumNetworkPolicy blocking ingress controller trafficVerify the restrict-hubble-ui-ingress policy matches the actual ingress-nginx namespace labelPrometheus target for hubble-metrics is downServiceMonitor selector labels do not match the serviceRun kubectl -n kube-system get svc hubble-metrics --show-labels and align the matchLabelsDROPPED verdicts on expected trafficCiliumNetworkPolicy default-deny without an explicit allow ruleAdd an explicit ingress or egress rule for the required label pair, then re-check with hubble observeAgent restarts loop after enabling HubbleResource limits too low for the added Hubble server processIncrease CPU and memory requests on the cilium DaemonSet via Helm values
Next steps
Configure Kubernetes network policies for enhanced cluster security
Implement Kubernetes network policies with Calico for microsegmentation
Configure Cilium BGP peering with MetalLB integration for Kubernetes load balancing
Implement Cilium Tetragon runtime security for Kubernetes with eBPF monitoring and threat detection
Set up Kubernetes monitoring with Prometheus Operator and custom metrics
Configure Hubble flow-based alerting with Prometheus Alertmanager
Running this in production?
Want this handled for you? Running eBPF-based network observability at scale adds a second layer of work: keeping Cilium and Hubble versions in sync across node pools, tuning metric cardinality before it overwhelms Prometheus, and staffing on-call for the alerts this visibility generates. See how we run infrastructure like this for European teams.
---
### Configure HAProxy with Consul for dynamic service discovery and automatic backend updates
URL: https://binadit.com/tutorials/setup-haproxy-consul-service-discovery
Category: networking
Difficulty: advanced
Time: ~75 minutes
Author: Binadit Tech Team
> Build a self-updating load balancer by integrating HAProxy with Consul service discovery, consul-template, and health checks so backends register and deregister automatically without manual reloads.
What this solves
Static HAProxy backend lists break down once services scale up, restart, or move between hosts. This tutorial wires HAProxy to Consul's service catalog using consul-template, so backend servers are added, removed, and health-checked automatically as your topology changes.
By the end, you will have Consul agents registering services, consul-template regenerating HAProxy configuration on catalog changes, reload automation without dropping connections, and ACL plus TLS hardening for the whole pipeline.
Step-by-step configuration
Install HAProxy and the Consul agent
Both packages ship in official repositories on Ubuntu, Debian, AlmaLinux and Rocky. Install HAProxy first, then add HashiCorp's repository for Consul.
sudo apt update
sudo apt install -y haproxy curl gnupg lsb-release
curl -fsSL https://apt.releases.hashicorp.com/gpg | sudo gpg --dearmor -o /usr/share/keyrings/hashicorp-archive-keyring.gpg
echo "deb [signed-by=/usr/share/keyrings/hashicorp-archive-keyring.gpg] https://apt.releases.hashicorp.com $(lsb_release -cs) main" | sudo tee /etc/apt/sources.list.d/hashicorp.list
sudo apt update
sudo apt install -y consul consul-template
sudo dnf install -y haproxy curl dnf-plugins-core
sudo dnf config-manager --add-repo https://rpm.releases.hashicorp.com/RHEL/hashicorp.repo
sudo dnf install -y consul consul-template
Create a dedicated Consul data directory and user
Consul should never run as root. Create a system user with a locked-down home directory for its data.
sudo useradd --system --home /var/lib/consul --shell /usr/sbin/nologin consul
sudo mkdir -p /var/lib/consul /etc/consul.d
sudo chown -R consul:consul /var/lib/consul
sudo chmod 750 /var/lib/consul
Note: 750 lets the consul user read and write freely while blocking access from other unprivileged accounts on the host.
Configure the Consul agent
This example runs a single-node Consul server for the tutorial. In production you would run 3 or 5 server nodes for quorum, similar to the multi-datacenter setup in Consul multi-datacenter replication with WAN federation.
datacenter = "dc1"
data_dir = "/var/lib/consul"
server = true
bootstrap_expect = 1
bind_addr = "203.0.113.10"
client_addr = "127.0.0.1"
ui_config {
enabled = true
}
connect {
enabled = true
}
sudo chown -R consul:consul /etc/consul.d
sudo chmod 640 /etc/consul.d/consul.hcl
Create a systemd unit for Consul
Run Consul as a managed service so it restarts automatically and logs to journald.
[Unit]
Description=Consul service discovery agent
After=network-online.target
Wants=network-online.target
[Service]
User=consul
Group=consul
ExecStart=/usr/bin/consul agent -config-dir=/etc/consul.d
ExecReload=/bin/kill -HUP $MAINPID
Restart=on-failure
RestartSec=5
LimitNOFILE=65536
[Install]
WantedBy=multi-user.target
sudo systemctl daemon-reload
sudo systemctl enable --now consul
sudo systemctl status consul
Register application services and health checks
Define each backend as a Consul service with an HTTP health check. Consul only advertises healthy instances to HAProxy.
{
"service": {
"name": "web-app",
"port": 8080,
"address": "203.0.113.21",
"tags": ["http"],
"check": {
"http": "http://203.0.113.21:8080/healthz",
"interval": "5s",
"timeout": "2s",
"deregister_critical_service_after": "1m"
}
}
}
sudo chown consul:consul /etc/consul.d/web-app.json
sudo consul reload
Repeat this file on each backend host with its own address, or push the definition remotely with the Consul HTTP API in a later step. Verify registration:
consul catalog services
consul health checks web-app
Write the consul-template source template
consul-template watches the Consul catalog and renders a new HAProxy config whenever the set of healthy instances changes.
global
log /dev/log local0
maxconn 4096
defaults
log global
mode http
timeout connect 5s
timeout client 30s
timeout server 30s
frontend web_front
bind *:80
default_backend web_back
backend web_back
balance roundrobin
option httpchk GET /healthz
{{range service "web-app"}} server {{.Node}}-{{.Port}} {{.Address}}:{{.Port}} check
{{end}}
Configure consul-template to write HAProxy config and reload
The wait and dedup blocks smooth out rapid catalog churn so HAProxy is not reloaded on every single check flap.
consul {
address = "127.0.0.1:8500"
}
template {
source = "/etc/consul-template/haproxy.ctmpl"
destination = "/etc/haproxy/haproxy.cfg"
command = "sudo systemctl reload haproxy"
command_timeout = "30s"
}
wait {
min = "3s"
max = "15s"
}
[Unit]
Description=Consul Template for HAProxy backends
After=consul.service
Requires=consul.service
[Service]
User=root
ExecStart=/usr/bin/consul-template -config=/etc/consul-template/config.hcl
Restart=on-failure
RestartSec=5
[Install]
WantedBy=multi-user.target
Warning: consul-template needs permission to run systemctl reload haproxy. Grant this with a scoped sudoers rule instead of running the whole service as root without limits.
consul-template ALL=(root) NOPASSWD: /usr/bin/systemctl reload haproxy
sudo systemctl daemon-reload
sudo systemctl enable --now consul-template
sudo systemctl status consul-template
Validate and enable HAProxy
Always validate the generated configuration before trusting automatic reloads in production.
sudo haproxy -c -f /etc/haproxy/haproxy.cfg
sudo systemctl enable --now haproxy
sudo systemctl status haproxy
For graceful reloads with zero dropped connections, review the reload strategy covered in HAProxy performance tuning with connection pooling, since consul-template simply calls systemctl reload under the hood.
Set up Consul DNS forwarding
Consul's built-in DNS server lets applications resolve service names like web-app.service.consul directly, which is useful for services that talk to each other without going through HAProxy.
[Resolve]
DNS=127.0.0.1:8600
Domains=~consul
sudo mkdir -p /etc/systemd/resolved.conf.d
sudo systemctl restart systemd-resolved
dig @127.0.0.1 -p 8600 web-app.service.consul
Use the HTTP API for programmatic registration from deployment pipelines instead of hand-editing JSON files on every host:
curl --request PUT --data '{
"Name": "web-app",
"Port": 8080,
"Address": "203.0.113.22",
"Check": {
"HTTP": "http://203.0.113.22:8080/healthz",
"Interval": "5s"
}
}' http://127.0.0.1:8500/v1/agent/service/register
Test failover and dynamic registration
Simulate a backend failure and confirm HAProxy removes it without manual intervention.
curl -s http://127.0.0.1:8500/v1/health/service/web-app?passing | jq '.[].Service.Address'
sudo systemctl stop web-app
sleep 15
cat /etc/haproxy/haproxy.cfg | grep server
sudo systemctl start web-app
Register a brand new instance on a fresh host and confirm it appears in the backend within a few seconds, without touching HAProxy configuration by hand.
Secure the cluster with ACLs
Default Consul installs allow any client to read and write the catalog. Enable ACLs with a default-deny policy and issue scoped tokens.
acl {
enabled = true
default_policy = "deny"
enable_token_persistence = true
}
sudo systemctl restart consul
consul acl bootstrap
Save the bootstrap token output securely, then create a read-only token for consul-template.
service_prefix "" {
policy = "read"
}
node_prefix "" {
policy = "read"
}
consul acl policy create -name "template-read" -rules @/etc/consul.d/template-policy.hcl
consul acl token create -description "consul-template read token" -policy-name "template-read"
Add the resulting token to the consul-template configuration:
consul {
address = "127.0.0.1:8500"
token = "YOUR-GENERATED-TOKEN-HERE"
}
sudo chmod 600 /etc/consul-template/config.hcl
sudo systemctl restart consul-template
For a deeper walkthrough of policy design, see advanced Consul ACL policies for production security hardening.
Enable TLS for agent communication
Encrypt Consul's RPC and gossip traffic so catalog data and tokens are not exposed on the network.
consul tls ca create
consul tls cert create -server -dc dc1
tls {
defaults {
ca_file = "/etc/consul.d/consul-agent-ca.pem"
cert_file = "/etc/consul.d/dc1-server-consul-0.pem"
key_file = "/etc/consul.d/dc1-server-consul-0-key.pem"
verify_incoming = true
verify_outgoing = true
}
}
sudo chown consul:consul /etc/consul.d/*.pem
sudo chmod 600 /etc/consul.d/*-key.pem
sudo chmod 644 /etc/consul.d/*.pem
sudo systemctl restart consul
Never use chmod 777. Private key files must stay readable only by the consul user. Broad permissions let any local account on the host read your TLS keys and impersonate agents.
For HAProxy's public-facing TLS, follow HAProxy SSL termination with Let's Encrypt certificates to terminate client traffic securely before it reaches the dynamic backend pool.
Verify your setup
sudo systemctl status consul consul-template haproxy
consul members
consul catalog services
curl -I http://127.0.0.1/
sudo haproxy -c -f /etc/haproxy/haproxy.cfg
Common issues
SymptomCauseFixHAProxy config never updatesconsul-template cannot reach Consul or lacks a valid ACL tokenCheck journalctl -u consul-template -f and verify the token has service read permissionsHAProxy reload fails silentlyconsul-template user lacks sudo rights for systemctlConfirm the sudoers rule in /etc/sudoers.d/consul-template matches the exact commandBackend flaps constantlyHealth check interval too aggressive or app slow to respondIncrease interval and timeout values in the service check definitionConsul agent won't join clusterFirewall blocking gossip ports 8301/8302Open the specific TCP/UDP ports with your firewall tool instead of disabling itconsul-template permission denied writing configDestination file owned by a different userchown root:root /etc/haproxy/haproxy.cfg and rerun as the correct service userACL bootstrap fails with "already bootstrapped"Bootstrap already ran on this clusterRecover using an existing management token instead of re-bootstrapping
Next steps
Set up HAProxy high availability with keepalived clustering
Configure Consul Connect service mesh monitoring with distributed tracing
Monitor HAProxy and Consul with Prometheus and Grafana dashboards
Implement Consul backup and disaster recovery with automated snapshots
Configure HAProxy advanced routing with ACLs and maps for intelligent traffic management
Running this in production?
Want this handled for you? Running this at scale adds a second layer of work: ACL and token rotation, TLS certificate renewal across every agent, reload storms during deploys, and 24/7 on-call for catalog failures. See how we run infrastructure like this for European teams.
---
### Implement Istio multi-cluster canary deployments and traffic splitting
URL: https://binadit.com/tutorials/implement-istio-multi-cluster-canary-deployments
Category: devops
Difficulty: advanced
Time: ~90 minutes
Author: Binadit Tech Team
> Build a primary-remote Istio multi-cluster mesh with shared trust domain, then run progressive canary rollouts using DestinationRule subsets, weighted VirtualServices and metrics-based promotion across clusters.
What this solves
Running canary releases inside one cluster is well understood, but validating a new version against real cross-region traffic requires a multi-cluster mesh. This tutorial builds a primary-remote Istio topology with a shared trust domain, deploys canary subsets across two clusters, and automates weighted traffic shifting with metrics-based rollback.
You will end up with two Kubernetes clusters sharing one control plane trust boundary, mTLS enforced between them, and a VirtualService that shifts traffic from a stable version to a canary in controlled increments while Prometheus-derived error rates gate the rollout.
Note: This tutorial assumes two existing Kubernetes clusters (kubeadm, EKS, GKE or similar) with network connectivity between pod CIDRs, either via a flat network, VPN, or exposed east-west gateways. If you have not yet built the base clusters, review installing a Kubernetes cluster with kubeadm first.
Prerequisites and multi-cluster architecture overview
Istio's primary-remote model runs the control plane (istiod) in one cluster (the primary) and configures the second cluster (remote) to use that same control plane for configuration and certificate signing. Both clusters share a root certificate authority, which establishes one trust domain across the mesh so mTLS works transparently between workloads regardless of cluster.
You need two clusters named cluster1 (primary) and cluster2 (remote), each with at least 3 worker nodes and 4 vCPU/8GB per node for the control plane and demo workloads. You also need kubectl, istioctl 1.22 or later, and cluster-admin access to both contexts.
Install istioctl
Download the Istio CLI, which manages installation, certificate generation and multi-cluster bootstrap operations.
curl -L https://istio.io/downloadIstio | ISTIO_VERSION=1.23.2 sh -
cd istio-1.23.2
export PATH=$PWD/bin:$PATH
istioctl version --remote=false
Set kubectl contexts for both clusters
Name your contexts clearly so every command below is unambiguous about which cluster it targets.
kubectl config rename-context admin@cluster1 cluster1
kubectl config rename-context admin@cluster2 cluster2
kubectl get nodes --context=cluster1
kubectl get nodes --context=cluster2
Configuring primary-remote multi-cluster mesh with shared trust domain
Generate a shared root certificate authority
Both clusters must trust the same root CA so workload certificates issued by either istiod are mutually trusted. Generate intermediate CAs signed by one root for each cluster.
mkdir -p certs && cd certs
make -f ../tools/certs/Makefile.selfsigned.mk root-ca
make -f ../tools/certs/Makefile.selfsigned.mk cluster1-cacerts
make -f ../tools/certs/Makefile.selfsigned.mk cluster2-cacerts
Warning: Never reuse the auto-generated self-signed Istio CA across production clusters without exporting the root explicitly. If you skip this step, each cluster gets an independent root and cross-cluster mTLS will fail with certificate validation errors.
Install the shared CA secrets
Create the cacerts secret in the istio-system namespace on both clusters before installing Istio, so istiod picks up the shared root at bootstrap.
kubectl --context=cluster1 create namespace istio-system
kubectl --context=cluster1 create secret generic cacerts -n istio-system \
--from-file=cluster1/ca-cert.pem \
--from-file=cluster1/ca-key.pem \
--from-file=cluster1/root-cert.pem \
--from-file=cluster1/cert-chain.pem
kubectl --context=cluster2 create namespace istio-system
kubectl --context=cluster2 create secret generic cacerts -n istio-system \
--from-file=cluster2/ca-cert.pem \
--from-file=cluster2/ca-key.pem \
--from-file=cluster2/root-cert.pem \
--from-file=cluster2/cert-chain.pem
Install Istio on the primary cluster
Label the network and install the control plane with the multi-cluster mesh and network identifiers set explicitly.
kubectl --context=cluster1 label namespace istio-system topology.istio.io/network=network1
cat < cluster1.yaml
apiVersion: install.istio.io/v1alpha1
kind: IstioOperator
spec:
values:
global:
meshID: mesh1
multiCluster:
clusterName: cluster1
network: network1
EOF
istioctl install --context=cluster1 -f cluster1.yaml -y
Deploy the east-west gateway on cluster1
The east-west gateway exposes istiod and cross-cluster service traffic over mTLS so cluster2 workloads can discover and reach cluster1 services.
@istioctl x create-remote-secret --help > /dev/null
samples/multicluster/gen-eastwest-gateway.sh \
--mesh mesh1 --cluster cluster1 --network network1 | \
istioctl --context=cluster1 install -y -f -
kubectl --context=cluster1 get svc istio-eastwestgateway -n istio-system
Expose istiod and services through the gateway
Apply the exposure gateway configuration so remote clusters can reach the control plane and mesh services on port 15443.
kubectl --context=cluster1 apply -n istio-system -f \
samples/multicluster/expose-istiod.yaml
kubectl --context=cluster1 apply -n istio-system -f \
samples/multicluster/expose-services.yaml
Install Istio on the remote cluster
Cluster2 runs a lightweight config that points at the primary's discovery address instead of running its own istiod.
kubectl --context=cluster2 label namespace istio-system topology.istio.io/network=network2
export DISCOVERY_ADDRESS=$(kubectl --context=cluster1 \
-n istio-system get svc istio-eastwestgateway \
-o jsonpath='{.status.loadBalancer.ingress[0].ip}')
cat < cluster2.yaml
apiVersion: install.istio.io/v1alpha1
kind: IstioOperator
spec:
values:
istiodRemote:
injectionURL: https://$DISCOVERY_ADDRESS:15017/inject/cluster/cluster2/net/network2
global:
remotePilotAddress: $DISCOVERY_ADDRESS
meshID: mesh1
multiCluster:
clusterName: cluster2
network: network2
EOF
istioctl install --context=cluster2 -f cluster2.yaml -y
Link the clusters with remote secrets
Create a remote secret in each cluster so istiod can discover endpoints in the other cluster's API server.
istioctl create-remote-secret --context=cluster2 --name=cluster2 | \
kubectl apply -f - --context=cluster1
istioctl create-remote-secret --context=cluster1 --name=cluster1 | \
kubectl apply -f - --context=cluster2
Deploying canary versions with DestinationRule subsets across clusters
Enable sidecar injection and deploy the stable version
Deploy version v1 (stable) to both clusters with a shared service name so Istio treats them as one logical service across the mesh.
kubectl --context=cluster1 create namespace checkout
kubectl --context=cluster2 create namespace checkout
kubectl --context=cluster1 label namespace checkout istio-injection=enabled
kubectl --context=cluster2 label namespace checkout istio-injection=enabled
kubectl --context=cluster1 apply -n checkout -f - < /tmp/checkout-vs-$weight.yaml
done
cat /tmp/checkout-vs-25.yaml
Note: When weight reaches 100 for v2, remove the v1 subset destination entirely and update the deployment labels so v1 becomes the new baseline for the next release cycle.
Automating canary analysis with metrics-based promotion or rollback
Install Prometheus and query error rate per subset
Istio's default telemetry exports istio_requests_total with a destination_version label, which lets you compute per-subset success rate directly in PromQL.
kubectl --context=cluster1 apply -f samples/addons/prometheus.yaml
kubectl --context=cluster1 -n istio-system port-forward svc/prometheus 9090:9090 &
sum(rate(istio_requests_total{destination_service_name="checkout",destination_version="v2",response_code!~"5.*"}[5m]))
/
sum(rate(istio_requests_total{destination_service_name="checkout",destination_version="v2"}[5m]))
Write a promotion/rollback script
This script polls the success rate for the canary subset and either advances the weight or reverts to the stable VirtualService if the threshold is breached.
#!/usr/bin/env bash
set -euo pipefail
PROM_URL="http://localhost:9090"
THRESHOLD="0.98"
QUERY='sum(rate(istio_requests_total{destination_service_name="checkout",destination_version="v2",response_code!~"5.*"}[5m]))/sum(rate(istio_requests_total{destination_service_name="checkout",destination_version="v2"}[5m]))'
success_rate=$(curl -s --data-urlencode "query=${QUERY}" "${PROM_URL}/api/v1/query" \
| python3 -c 'import json,sys; d=json.load(sys.stdin); print(d["data"]["result"][0]["value"][1] if d["data"]["result"] else "1")')
result=$(python3 -c "print(1 if float('${success_rate}') >= ${THRESHOLD} else 0)")
if [ "$result" -eq 1 ]; then
echo "Canary healthy at ${success_rate}, safe to promote"
exit 0
else
echo "Canary success rate ${success_rate} below threshold, rolling back"
kubectl --context=cluster1 apply -f /tmp/checkout-vs-rollback.yaml
kubectl --context=cluster2 apply -f /tmp/checkout-vs-rollback.yaml
exit 1
fi
chmod 750 /usr/local/bin/canary-gate.sh
sudo chown root:root /usr/local/bin/canary-gate.sh
Never use chmod 777. The gate script runs with cluster-admin kubeconfig access; 750 restricts execution to the owning user and group, while other users on the host get no access at all.
Prepare the rollback VirtualService
Keep a ready-to-apply manifest that routes 100 percent of traffic back to v1 so the rollback script has zero build time during an incident.
apiVersion: networking.istio.io/v1beta1
kind: VirtualService
metadata:
name: checkout
namespace: checkout
spec:
hosts:
- checkout.checkout.svc.cluster.local
http:
- route:
- destination:
host: checkout.checkout.svc.cluster.local
subset: v1
weight: 100
Schedule the gate as a systemd timer
Run the analysis every 5 minutes during an active rollout window instead of relying on manual checks.
[Unit]
Description=Istio canary metrics gate
[Service]
Type=oneshot
ExecStart=/usr/local/bin/canary-gate.sh
User=deploy
[Unit]
Description=Run canary gate every 5 minutes
[Timer]
OnCalendar=*:0/5
Persistent=true
[Install]
WantedBy=timers.target
sudo systemctl daemon-reload
sudo systemctl enable --now canary-gate.timer
Validating cross-cluster failover and mTLS traffic encryption
Confirm mTLS is enforced mesh-wide
PeerAuthentication set to STRICT rejects any plaintext traffic between sidecars, which is required before you rely on cross-cluster identity for canary routing decisions.
apiVersion: security.istio.io/v1beta1
kind: PeerAuthentication
metadata:
name: default
namespace: istio-system
spec:
mtls:
mode: STRICT
kubectl --context=cluster1 apply -f /tmp/peerauth-strict.yaml
kubectl --context=cluster2 apply -f /tmp/peerauth-strict.yaml
istioctl x describe pod checkout-v1-xxxx -n checkout --context=cluster1
Test cross-cluster failover
Scale checkout to zero in cluster1 and confirm cluster2 continues serving requests without client-visible errors, proving the mesh routes across the east-west gateway when local endpoints disappear.
kubectl --context=cluster1 -n checkout scale deployment checkout-v1 --replicas=0
for i in $(seq 1 20); do
kubectl --context=cluster1 -n checkout exec deploy/sleep -- curl -s -o /dev/null -w "%{http_code}\n" http://checkout.checkout.svc.cluster.local
done
kubectl --context=cluster1 -n checkout scale deployment checkout-v1 --replicas=3
Monitoring canary rollout with Grafana, Kiali and Prometheus dashboards
Install Kiali and Grafana add-ons
Kiali visualizes the live traffic split percentages between subsets and both clusters, which is faster to reason about than raw PromQL during an active rollout.
kubectl --context=cluster1 apply -f samples/addons/kiali.yaml
kubectl --context=cluster1 apply -f samples/addons/grafana.yaml
kubectl --context=cluster1 -n istio-system port-forward svc/kiali 20001:20001 &
Import the Istio service dashboards
Grafana's official Istio dashboards break down request volume, p99 latency and error rate per workload version, which is exactly the signal the canary gate script consumes.
kubectl --context=cluster1 -n istio-system port-forward svc/grafana 3000:3000 &
For a broader Prometheus and Grafana rollout dashboard setup covering the whole mesh, follow monitoring Istio service mesh with Prometheus and Grafana dashboards. For tracing individual canary requests end to end across clusters, pair this with integrating Jaeger with Istio service mesh for distributed tracing.
Troubleshooting common multi-cluster traffic routing issues
Confirm endpoint discovery across clusters
If cluster2 workloads never appear as viable endpoints from cluster1, the remote secret or network label is usually the cause.
istioctl proxy-config endpoints deploy/checkout-v1 -n checkout --context=cluster1 | grep checkout
kubectl --context=cluster1 -n istio-system get secret | grep istio-remote-secret
Verify your setup
istioctl remote-clusters --context=cluster1
kubectl --context=cluster1 -n checkout get virtualservice checkout -o yaml
kubectl --context=cluster2 -n checkout get pods -l app=checkout
istioctl pr
---
### Implement HAProxy WAF integration with ModSecurity 3 for advanced threat protection
URL: https://binadit.com/tutorials/implement-haproxy-waf-integration-with-modsecurity
Category: security
Difficulty: advanced
Time: ~75 minutes
Author: Binadit Tech Team
> Deploy ModSecurity 3 as a Stream Processing Offload Agent in front of HAProxy to inspect HTTP traffic with the OWASP Core Rule Set. This tutorial covers SPOA compilation, SPOE filter configuration, detection versus blocking modes, and production tuning.
What this solves
HAProxy has no native request body inspection engine, so it cannot stop SQL injection, XSS or other layer 7 attacks on its own. This tutorial wires HAProxy to ModSecurity 3 via the Stream Processing Offload Engine (SPOE) so every request is inspected by the OWASP Core Rule Set (CRS) before it reaches your backends.
You will build the SPOA (Stream Processing Offload Agent) from source, connect it to HAProxy with SPOE filters, load the CRS, and tune detection versus blocking behavior for production traffic.
Warning: Enabling blocking mode without a tuning period will generate false positives against legitimate traffic. Always run in detection-only mode first and review logs before switching to blocking.
Step-by-step installation
Install build dependencies
The SPOA and libmodsecurity must be compiled from source on all four target distributions. Install the toolchain and libraries first.
sudo apt update
sudo apt install -y build-essential git autoconf automake libtool \
pkgconf libpcre2-dev libpcre3-dev libxml2-dev libcurl4-openssl-dev \
libyajl-dev doxygen libgeoip-dev liblua5.3-dev libssl-dev \
zlib1g-dev libmaxminddb-dev cmake
sudo dnf groupinstall -y "Development Tools"
sudo dnf install -y epel-release
sudo dnf install -y git autoconf automake libtool pkgconf pcre2-devel \
libxml2-devel libcurl-devel yajl-devel doxygen GeoIP-devel \
lua-devel openssl-devel zlib-devel libmaxminddb-devel cmake
Build and install libmodsecurity 3
libmodsecurity is the standalone engine that evaluates CRS rules. It is separate from the old Apache/NGINX ModSecurity modules.
cd /usr/local/src
sudo git clone --depth 1 -b v3/master https://github.com/owasp-modsecurity/ModSecurity modsecurity
cd modsecurity
sudo git submodule init
sudo git submodule update
sudo ./build.sh
sudo ./configure
sudo make -j"$(nproc)"
sudo make install
This build takes 15-30 minutes depending on CPU. Verify the shared library was installed correctly.
ls -la /usr/local/modsecurity/lib/libmodsecurity.so*
Build the ModSecurity SPOA
The SPOA is the daemon that speaks the SPOE binary protocol to HAProxy and forwards requests to libmodsecurity for inspection.
cd /usr/local/src
sudo git clone https://github.com/haproxytech/spoa-modsecurity
cd spoa-modsecurity
sudo make MODSECURITY_INC=/usr/local/modsecurity/include \
MODSECURITY_LIB=/usr/local/modsecurity/lib
sudo cp spoa /usr/local/bin/modsec-spoa
sudo mkdir -p /etc/modsecurity-spoa
Install HAProxy 2.8 or newer
SPOE support is mature from HAProxy 2.0 onward, but use 2.8 LTS or later for stability and performance fixes relevant to WAF workloads.
sudo apt install -y haproxy
haproxy -v
sudo dnf install -y haproxy
haproxy -v
If your distribution ships an older HAProxy, refer to Install and configure HAProxy for high availability load balancing for repository options that provide newer releases.
Download the OWASP Core Rule Set
The CRS provides the generic ruleset for SQLi, XSS, RCE, protocol violations and scanner detection. Pull the latest stable release.
cd /etc/modsecurity-spoa
sudo git clone -b v4.3/master https://github.com/coreruleset/coreruleset crs
sudo cp crs/crs-setup.conf.example crs/crs-setup.conf
Create the ModSecurity engine configuration
This file loads libmodsecurity's core directives, the CRS setup file and all CRS rule files in order.
SecRuleEngine DetectionOnly
SecRequestBodyAccess On
SecResponseBodyAccess Off
SecRequestBodyLimit 13107200
SecRequestBodyNoFilesLimit 131072
SecPcreMatchLimit 250000
SecPcreMatchLimitRecursion 250000
SecAuditEngine RelevantOnly
SecAuditLogRelevantStatus "^(?:5|4(?!04))"
SecAuditLogParts ABIJDEFHZ
SecAuditLogType Serial
SecAuditLog /var/log/modsecurity/audit.log
SecDebugLog /var/log/modsecurity/debug.log
SecDebugLogLevel 3
Include /etc/modsecurity-spoa/crs/crs-setup.conf
Include /etc/modsecurity-spoa/crs/rules/*.conf
SecRuleEngine is set to DetectionOnly deliberately. You will switch this to On only after reviewing logs from real traffic.
sudo mkdir -p /var/log/modsecurity
sudo chown -R haproxy:haproxy /var/log/modsecurity
sudo chmod 750 /var/log/modsecurity
Note: The audit log directory is owned by the haproxy user because the SPOA process should run as an unprivileged service account, not root.
Create the SPOA systemd service
Run the SPOA as a dedicated systemd unit so it starts on boot and restarts automatically on failure.
[Unit]
Description=ModSecurity SPOA for HAProxy
After=network.target
[Service]
Type=simple
User=haproxy
Group=haproxy
ExecStart=/usr/local/bin/modsec-spoa -n 4 -p 12345 -r /etc/modsecurity-spoa/modsecurity.conf
Restart=on-failure
RestartSec=3
LimitNOFILE=65536
[Install]
WantedBy=multi-user.target
sudo systemctl daemon-reload
sudo systemctl enable --now modsec-spoa
sudo systemctl status modsec-spoa
Configure the SPOE filter definition
The SPOE config tells HAProxy which messages to send to the agent and which variables to expect back.
[modsecurity]
spoe-agent modsecurity-agent
messages check-request
option var-prefix modsec
timeout hello 2s
timeout idle 2m
timeout processing 1s
use-backend spoe-modsecurity
log global
spoe-message check-request
args unique-id method path query req.ver req.hdrs_bin req.body_size req.body
event on-frontend-http-request
Configure HAProxy to use the SPOE filter
Add the SPOE backend, the filter directive on your frontend, and the ACL that blocks requests flagged by ModSecurity.
global
log /dev/log local0
maxconn 20000
tune.ssl.default-dh-param 2048
defaults
log global
mode http
option httplog
timeout connect 5s
timeout client 30s
timeout server 30s
frontend fe_main
bind *:443 ssl crt /etc/haproxy/certs/example.com.pem
filter spoe engine modsecurity config /etc/haproxy/modsecurity.spoe.conf
http-request deny deny_status 403 if { var(txn.modsec.blocked) -m int gt 0 }
default_backend be_app
backend spoe-modsecurity
mode tcp
balance roundrobin
timeout connect 5s
timeout server 2m
server spoa1 127.0.0.1:12345
backend be_app
balance roundrobin
option httpchk GET /healthz
server app1 203.0.113.10:8080 check
server app2 203.0.113.11:8080 check
If you have not yet configured TLS on this HAProxy instance, follow Set up HAProxy SSL termination with Let's Encrypt certificates before exposing this frontend publicly.
sudo haproxy -c -f /etc/haproxy/haproxy.cfg
sudo systemctl reload haproxy
Tune CRS paranoia level and anomaly thresholds
Paranoia level controls how aggressively CRS matches suspicious patterns. Level 1 is the default; higher levels catch more but increase false positives.
SecAction \
"id:900000,\
phase:1,\
nolog,\
pass,\
t:none,\
setvar:tx.paranoia_level=1,\
setvar:tx.inbound_anomaly_score_threshold=5,\
setvar:tx.outbound_anomaly_score_threshold=4"
Start at paranoia level 1 with the default anomaly threshold of 5. Only raise the level after a tuning period shows an acceptable false-positive rate.
Add custom rule exceptions
Legitimate application behavior sometimes trips generic rules, such as a CMS admin panel that submits raw HTML in a form field. Exclude specific rule IDs for specific paths instead of disabling entire rule categories.
SecRule REQUEST_URI "@beginsWith /admin/content-editor" \
"id:1000001,\
phase:2,\
pass,\
nolog,\
ctl:ruleRemoveById=941100-941999"
Place exclusion files after the CRS rule includes in modsecurity.conf so they can override earlier matches. Document every exclusion with a comment explaining why it exists and who approved it.
Verify your setup
Confirm the SPOA is listening and HAProxy is forwarding traffic to it.
sudo ss -tlnp | grep 12345
sudo journalctl -u modsec-spoa -n 50 --no-pager
Test detection with a known malicious payload. In DetectionOnly mode this should log the event but still return a 200 response.
curl -s -o /dev/null -w "%{http_code}\n" "https://example.com/?id=1' OR '1'='1"
curl -s -o /dev/null -w "%{http_code}\n" "https://example.com/?exec=/bin/bash"
Check that the audit log recorded the anomaly.
sudo tail -n 20 /var/log/modsecurity/audit.log
After reviewing logs for false positives over a representative traffic period, switch to blocking mode and confirm attacks now return 403.
sudo sed -i 's/SecRuleEngine DetectionOnly/SecRuleEngine On/' /etc/modsecurity-spoa/modsecurity.conf
sudo systemctl restart modsec-spoa
curl -s -o /dev/null -w "%{http_code}\n" "https://example.com/?id=1' OR '1'='1"
Logging, alerting and testing against attack payloads
Feed the ModSecurity audit log into your centralized logging pipeline rather than tailing it manually in production. If you already run the ELK stack, follow Set up ELK Stack for centralized ModSecurity log analysis and monitoring to parse the JSON audit log format and build dashboards for blocked requests by rule ID and source IP.
Run a broader payload sweep to validate CRS coverage before going live. Test SQL injection, XSS, path traversal and command injection patterns against a staging frontend running in blocking mode.
curl -s -o /dev/null -w "%{http_code}\n" "https://staging.example.com/search?q="
curl -s -o /dev/null -w "%{http_code}\n" "https://staging.example.com/file?path=../../etc/passwd"
curl -s -o /dev/null -w "%{http_code}\n" "https://staging.example.com/ping?host=127.0.0.1;cat /etc/shadow"
All three should return 403. If any return 200, check the anomaly score threshold and confirm the relevant CRS rule files loaded without syntax errors.
Performance and high availability considerations
Each SPOA process is single-threaded per worker, so scale the -n worker count to match available CPU cores on the SPOA host. Keep the SPOA on the same host as HAProxy or on a low-latency internal network, since every request now incurs an extra round trip.
Tuning areaRecommendationSPOA workersSet -n to CPU core count minus 1, reserving a core for HAProxySPOE timeout processingKeep under 1s; requests exceeding this fail open or closed based on on-error settingRequest body limitCap SecRequestBodyLimit to your largest legitimate upload plus margin, not unlimitedAudit log volumeUse RelevantOnly and ship logs asynchronously to avoid disk I/O blocking inspection
For high availability, run HAProxy and the SPOA in an active-passive or active-active pair behind keepalived, and run a local SPOA instance per HAProxy node rather than a shared remote SPOA pool. This avoids a single SPOA becoming a bottleneck or single point of failure for every frontend. See Set up HAProxy high availability with keepalived clustering for automatic failover for the VRRP configuration.
Decide the SPOE failure mode explicitly. By default, if the SPOA is unreachable, HAProxy fails open and traffic passes uninspected. For security-critical services, consider failing closed by adding an ACL that denies traffic when the SPOE variable is unset, but test this carefully since it turns a WAF outage into a full service outage.
Common issues
SymptomCauseFixAll requests return 403 immediatelyAnomaly threshold too low or a broad rule exclusion is missingSet SecRuleEngine DetectionOnly, review /var/log/modsecurity/audit.log, raise threshold temporarilyHAProxy config check fails on filter spoeSPOE config file path is wrong or has a syntax errorRun sudo haproxy -c -f /etc/haproxy/haproxy.cfg and check the reported line numbermodsec-spoa fails to startlibmodsecurity.so not found by the dynamic linkerRun sudo ldconfig /usr/local/modsecurity/lib and add that path to /etc/ld.so.conf.d/Legitimate file uploads get blockedSecRequestBodyLimit too low or CRS rule 200002 triggering on multipart dataIncrease the body limit and add a scoped exclusion for the upload endpointHigh latency on every requestSPOA and HAProxy on different hosts with network round trip overheadColocate SPOA on the same host as HAProxy or on a sub-millisecond LAN linkAudit log grows unboundedSecAuditEngine set to On instead of RelevantOnlySet SecAuditEngine RelevantOnly and configure logrotate on the audit log path
Next steps
Configure ModSecurity 3 web application firewall with OWASP Core Rule Set for advanced threat protection
Configure ModSecurity machine learning anomaly detection for automated threat protection
Integrate ModSecurity 3 with SOAR platforms for automated incident response and threat detection
Configure HAProxy advanced routing with ACLs and maps for intelligent traffic management
Configure HAProxy WAF logging with Fluentd and Loki
Running this in production?
Want this handled for you? Running this at scale adds a second layer of work: rule tuning as your application changes, false-positive triage, CRS version upgrades and failover drills for the SPOA tier. See how we run infrastructure like this for European teams.
---
### Configure PostgreSQL 17 SSL encryption and certificate-based authentication
URL: https://binadit.com/tutorials/configure-postgresql-ssl-encryption-and-authentication
Category: databases
Difficulty: advanced
Time: ~45 minutes
Author: Binadit Tech Team
> Set up a private CA, issue server and client certificates, enforce TLS 1.2+ with strong ciphers, and configure pg_hba.conf for mutual TLS client certificate authentication in PostgreSQL 17.
What this solves
By default, PostgreSQL connections are unencrypted unless explicitly configured otherwise, and password authentication over the network exposes credentials to interception. This tutorial covers generating a private certificate authority, issuing server and client certificates, enforcing TLS-only connections, and configuring mutual TLS so clients authenticate with certificates instead of passwords.
This setup is appropriate for production databases handling sensitive data, multi-tenant environments, and compliance-driven infrastructure where password-based authentication is insufficient.
Warning: Test all changes on a staging instance first. Misconfigured pg_hba.conf or SSL settings can lock out all connections, including local ones.
Step-by-step configuration
Install PostgreSQL 17 and OpenSSL
Install PostgreSQL 17 and the OpenSSL tools needed to build a certificate authority.
sudo apt update
sudo apt install -y postgresql-17 postgresql-client-17 openssl
sudo dnf install -y https://download.postgresql.org/pub/repos/yum/reporpms/EL-9-x86_64/pgdg-redhat-repo-latest.noarch.rpm
sudo dnf install -y postgresql17-server postgresql17 openssl
sudo /usr/pgsql-17/bin/postgresql-17-setup initdb
sudo systemctl enable --now postgresql-17
Create a private certificate authority
The CA signs both the server certificate and every client certificate. Keep the CA private key offline or heavily restricted after issuing certificates.
sudo mkdir -p /etc/postgresql/ssl/ca
cd /etc/postgresql/ssl/ca
sudo openssl genrsa -aes256 -out ca.key 4096
sudo openssl req -new -x509 -days 3650 -key ca.key -sha256 \
-out ca.crt \
-subj "/C=NL/O=Example Corp/CN=Example Corp PostgreSQL CA"
You will be prompted for a passphrase to encrypt the CA private key. Use a strong passphrase and store it in a secrets manager, not on the filesystem.
Generate the server certificate
Create a key and certificate signing request (CSR) for the PostgreSQL server. The common name must match the hostname clients use to connect.
sudo mkdir -p /etc/postgresql/ssl/server
cd /etc/postgresql/ssl/server
sudo openssl genrsa -out server.key 4096
sudo openssl req -new -key server.key -out server.csr \
-subj "/C=NL/O=Example Corp/CN=db01.example.com"
Sign the CSR with the CA to produce the server certificate, valid for one year.
sudo openssl x509 -req -in server.csr -CA /etc/postgresql/ssl/ca/ca.crt \
-CAkey /etc/postgresql/ssl/ca/ca.key -CAcreateserial \
-out server.crt -days 365 -sha256
Set correct ownership and permissions on certificates
PostgreSQL refuses to start if the server private key is readable by other users. The postgres process owns and reads these files directly, so ownership must belong to the postgres user with restrictive permissions.
sudo chown postgres:postgres /etc/postgresql/ssl/server/server.key /etc/postgresql/ssl/server/server.crt
sudo chmod 600 /etc/postgresql/ssl/server/server.key
sudo chmod 644 /etc/postgresql/ssl/server/server.crt
sudo chown postgres:postgres /etc/postgresql/ssl/ca/ca.crt
sudo chmod 644 /etc/postgresql/ssl/ca/ca.crt
Never use chmod 777. The private key must only be readable by the postgres user. Granting broad access here would let any local user impersonate the server or read encrypted traffic keys.
Enable SSL in postgresql.conf
Point PostgreSQL at the certificate files and turn on SSL support.
ssl = on
ssl_cert_file = '/etc/postgresql/ssl/server/server.crt'
ssl_key_file = '/etc/postgresql/ssl/server/server.key'
ssl_ca_file = '/etc/postgresql/ssl/ca/ca.crt'
ssl_prefer_server_ciphers = on
Note: On AlmaLinux and Rocky, the config file is at /var/lib/pgsql/17/data/postgresql.conf unless you changed the data directory during initdb.
Enforce strong TLS protocol versions and cipher suites
Restrict connections to TLS 1.2 and above, and set an explicit cipher list to block weak algorithms.
ssl_min_protocol_version = 'TLSv1.2'
ssl_max_protocol_version = 'TLSv1.3'
ssl_ciphers = 'HIGH:!aNULL:!MD5:!3DES:!RC4'
ssl_ecdh_curve = 'prime256v1'
Restart PostgreSQL to apply the SSL settings.
sudo systemctl restart postgresql
Issue a client certificate for mutual TLS
Each client certificate's common name must exactly match the PostgreSQL role it authenticates as. Generate one per application or admin user.
sudo mkdir -p /etc/postgresql/ssl/clients
cd /etc/postgresql/ssl/clients
sudo openssl genrsa -out app_user.key 4096
sudo openssl req -new -key app_user.key -out app_user.csr \
-subj "/C=NL/O=Example Corp/CN=app_user"
sudo openssl x509 -req -in app_user.csr -CA /etc/postgresql/ssl/ca/ca.crt \
-CAkey /etc/postgresql/ssl/ca/ca.key -CAcreateserial \
-out app_user.crt -days 365 -sha256
Set restrictive ownership before distributing the client key to the application host.
sudo chmod 600 app_user.key
sudo chmod 644 app_user.crt
Create the matching PostgreSQL role
The role name must exactly match the certificate common name for cert authentication to succeed.
sudo -u postgres psql -c "CREATE ROLE app_user LOGIN;"
sudo -u postgres psql -c "GRANT CONNECT ON DATABASE appdb TO app_user;"
Configure pg_hba.conf for certificate authentication
Set the authentication method to cert and require clientcert verification so PostgreSQL rejects any connection lacking a CA-signed client certificate.
# TYPE DATABASE USER ADDRESS METHOD
hostssl appdb app_user 203.0.113.0/24 cert clientcert=verify-full
hostssl all all 0.0.0.0/0 reject
The clientcert=verify-full option checks both that the certificate is signed by the trusted CA and that its common name matches the connecting role. Reload PostgreSQL to apply the rules.
sudo systemctl reload postgresql
Note: Order matters in pg_hba.conf. Rules are evaluated top to bottom, and the first match wins.
Verify your setup
Copy the CA certificate and client certificate and key to the client machine, then connect with sslmode=verify-full to validate both the server identity and the client certificate.
psql "host=db01.example.com dbname=appdb user=app_user \
sslmode=verify-full \
sslrootcert=/etc/postgresql/ssl/ca/ca.crt \
sslcert=/etc/postgresql/ssl/clients/app_user.crt \
sslkey=/etc/postgresql/ssl/clients/app_user.key"
Confirm the connection is using SSL and check the negotiated protocol and cipher.
psql -c "SELECT ssl, version, cipher FROM pg_stat_ssl JOIN pg_stat_activity USING (pid) WHERE pid = pg_backend_pid();"
Confirm password authentication is actually rejected for the same user without a certificate.
psql "host=db01.example.com dbname=appdb user=app_user sslmode=require"
This should fail with an authentication error since no client certificate was supplied.
Rotating certificates without downtime
Certificates expire, and rotating the server certificate should not require a restart. PostgreSQL reloads SSL context on pg_reload_conf() as long as the file paths in postgresql.conf stay the same.
sudo openssl req -new -key /etc/postgresql/ssl/server/server.key \
-out /etc/postgresql/ssl/server/server_renew.csr \
-subj "/C=NL/O=Example Corp/CN=db01.example.com"
sudo openssl x509 -req -in /etc/postgresql/ssl/server/server_renew.csr \
-CA /etc/postgresql/ssl/ca/ca.crt -CAkey /etc/postgresql/ssl/ca/ca.key \
-CAcreateserial -out /etc/postgresql/ssl/server/server.crt.new -days 365 -sha256
sudo mv /etc/postgresql/ssl/server/server.crt.new /etc/postgresql/ssl/server/server.crt
sudo chown postgres:postgres /etc/postgresql/ssl/server/server.crt
sudo -u postgres psql -c "SELECT pg_reload_conf();"
For client certificates, issue the replacement before the old one expires, distribute it to the application, and update the application config to point at the new file with a rolling deploy. Since both certificates are trusted by the same CA during the transition window, there is no need to disable authentication.
Set a calendar reminder or monitoring check for certificate expiry dates well before renewal is due. If you already run PostgreSQL streaming replication with PgBouncer, apply the same server certificate rotation on standby nodes to keep replication connections consistent.
Combining with connection pooling
If you terminate connections through PgBouncer, configure PgBouncer to also require TLS between the application and the pooler, and between the pooler and PostgreSQL. See Configure PostgreSQL 17 connection pooling with PgBouncer for high availability for the pooler-side setup, then apply the sslmode=verify-full settings from this tutorial to the pooler's backend connection string.
For broader hardening beyond SSL, including password policies, connection limits, and role separation, see Configure PostgreSQL 17 SSL encryption and advanced security hardening.
Common issues
SymptomCauseFixFATAL: no pg_hba.conf entry for host, SSL offClient connected without sslmode=require or higherAdd sslmode=verify-full to the connection string, confirm hostssl is used in pg_hba.confcould not load private key file, permission deniedserver.key is not readable by the postgres userRun sudo chown postgres:postgres server.key && sudo chmod 600 server.keycertificate verify failed: unable to get local issuer certificateClient does not have the CA certificate, or sslrootcert path is wrongCopy ca.crt to the client and reference it with sslrootcertFATAL: certificate authentication failed for userCertificate common name does not match the PostgreSQL role nameRegenerate the client certificate with CN matching the exact role nameSSL error: sslv3 alert handshake failureClient and server share no compatible cipher or protocol versionCheck ssl_min_protocol_version and ssl_ciphers, update client OpenSSL if outdatedconnection works with sslmode=require but fails with verify-fullServer certificate CN does not match the hostname used to connectReissue server certificate with CN matching the DNS name clients useserver won't start after enabling sslMissing or misnamed certificate files in postgresql.conf pathsVerify file paths with sudo -u postgres psql -c "SHOW ssl_cert_file;" and check the PostgreSQL log
Next steps
Configure PostgreSQL 17 connection pooling with PgBouncer for high availability
Configure PostgreSQL 17 streaming replication for high availability with automatic failover
Configure PostgreSQL 17 SSL encryption and advanced security hardening
Set up Vault as a PKI certificate authority with SSL automation and intermediate CA
Monitor PostgreSQL performance with Prometheus and Grafana dashboards
Running this in production?
Want this handled for you? Running this at scale adds a second layer of work: certificate rotation across fleets, monitoring expiry dates, and auditing cipher policy drift after every PostgreSQL minor upgrade. See how we run infrastructure like this for European teams.
---
### Set up HAProxy SSL termination with Let's Encrypt certificates
URL: https://binadit.com/tutorials/setup-haproxy-ssl-termination-with-lets-encrypt
Category: networking
Difficulty: intermediate
Time: ~35 minutes
Author: Binadit Tech Team
> Learn how to terminate SSL/TLS at HAProxy using Let's Encrypt certificates, redirect HTTP to HTTPS, automate renewal with deploy hooks, and harden your cipher suites for production load balancing.
What this solves
HAProxy is a fast, reliable load balancer, but it does not fetch or manage TLS certificates on its own. This tutorial shows you how to obtain free certificates from Let's Encrypt with Certbot, combine them into the PEM bundle HAProxy expects, and wire up automatic renewal so certificates never expire silently.
You will also configure HTTP to HTTPS redirection, backend health checks, load balancing, and modern cipher suites so your termination point is both functional and hardened.
Step-by-step configuration
Install HAProxy
HAProxy handles the TLS termination and forwards decrypted traffic to your backend servers.
sudo apt update
sudo apt install -y haproxy
sudo dnf install -y haproxy
Install Certbot
Certbot requests and renews certificates from Let's Encrypt. Install the standalone plugin so it can bind to port 80 temporarily during issuance.
sudo apt install -y certbot
sudo dnf install -y certbot
Stop HAProxy before issuing certificates
The standalone challenge needs port 80 free. Stop HAProxy temporarily, or skip this step if you plan to use the DNS challenge instead.
sudo systemctl stop haproxy
Obtain a certificate with the standalone challenge
This works when port 80 is reachable from the internet and no other process is bound to it. Replace example.com with your actual domain.
sudo certbot certonly --standalone -d example.com -d www.example.com --agree-tos -m admin@example.com --non-interactive
Note: If HAProxy must stay online, use the DNS challenge instead so port 80 is never touched.
Alternative: obtain a certificate with the DNS challenge
The DNS challenge lets you issue certificates without opening port 80 or stopping HAProxy, and is required for wildcard certificates. This example uses the manual DNS plugin; swap in your DNS provider's Certbot plugin for full automation.
sudo certbot certonly --manual --preferred-challenges dns -d example.com -d '*.example.com' --agree-tos -m admin@example.com
Certbot will print a TXT record to add to your DNS zone. Add it, wait for propagation, then continue the prompt to complete validation.
Build the combined PEM bundle
HAProxy expects a single file containing the certificate, intermediate chain, and private key, in that order. Create a directory to store these bundles and a script to rebuild them after every renewal.
sudo mkdir -p /etc/haproxy/certs
sudo chmod 700 /etc/haproxy/certs
sudo bash -c 'cat /etc/letsencrypt/live/example.com/fullchain.pem /etc/letsencrypt/live/example.com/privkey.pem > /etc/haproxy/certs/example.com.pem'
sudo chmod 600 /etc/haproxy/certs/example.com.pem
sudo chown root:root /etc/haproxy/certs/example.com.pem
Never use chmod 777. The PEM bundle contains your private key. Keep it owned by root with 600 permissions so only the HAProxy process, running as root or via setcap, can read it. Anything more permissive exposes your key to every local user.
Configure HAProxy global and defaults sections
These sections set process-wide behavior and sane connection defaults before you define frontends and backends.
global
log /dev/log local0
log /dev/log local1 notice
chroot /var/lib/haproxy
stats socket /run/haproxy/admin.sock mode 660 level admin
stats timeout 30s
user haproxy
group haproxy
daemon
maxconn 4096
defaults
log global
mode http
option httplog
option dontlognull
timeout connect 5s
timeout client 30s
timeout server 30s
errorfile 400 /etc/haproxy/errors/400.http
errorfile 403 /etc/haproxy/errors/403.http
errorfile 500 /etc/haproxy/errors/500.http
errorfile 502 /etc/haproxy/errors/502.http
errorfile 503 /etc/haproxy/errors/503.http
errorfile 504 /etc/haproxy/errors/504.http
Configure the HTTPS frontend with SSL termination
This frontend binds to port 443, loads the PEM bundle, and forwards traffic to a backend pool. It also sets headers so backend applications know the original request used HTTPS.
frontend https_front
bind *:443 ssl crt /etc/haproxy/certs/example.com.pem
mode http
option forwardfor
http-request set-header X-Forwarded-Proto https
default_backend web_backend
Set up HTTP to HTTPS redirection
All plain HTTP traffic should be redirected to HTTPS rather than served unencrypted. Add a dedicated frontend on port 80 that issues a permanent redirect.
frontend http_front
bind *:80
mode http
http-request redirect scheme https code 301 unless { ssl_fc }
default_backend web_backend
Note: Certbot's standalone challenge also needs port 80. If you keep this redirect frontend running permanently, use the DNS challenge or the HTTP-01 webroot method with an ACL exception for /.well-known/acme-challenge/ instead of stopping HAProxy for renewals.
Allow ACME HTTP-01 renewals without downtime
Add an ACL that lets Certbot's renewal requests through the redirect frontend, so you never need to stop HAProxy again after the initial certificate issuance.
frontend http_front
bind *:80
mode http
acl is_acme_challenge path_beg /.well-known/acme-challenge/
use_backend acme_backend if is_acme_challenge
http-request redirect scheme https code 301 unless { ssl_fc } !is_acme_challenge
default_backend web_backend
backend acme_backend
server certbot 127.0.0.1:8888
Run Certbot's standalone challenge on an internal port so this backend can reach it during renewal, or switch entirely to the DNS challenge for a simpler setup.
Configure the backend with health checks and load balancing
The backend pool defines your application servers, the load balancing algorithm, and active health checks that remove unhealthy servers automatically.
backend web_backend
mode http
balance roundrobin
option httpchk GET /health
http-check expect status 200
default-server inter 5s fall 3 rise 2
server web1 203.0.113.10:8080 check
server web2 203.0.113.11:8080 check
server web3 203.0.113.12:8080 check backup
The fall 3 and rise 2 settings mean a server is marked down after 3 consecutive failed checks and back up after 2 consecutive successes, avoiding flapping on transient errors.
Harden SSL/TLS settings and cipher suites
Restrict HAProxy to TLS 1.2 and 1.3, disable weak ciphers, and enable HSTS so browsers refuse to downgrade to plain HTTP.
global
ssl-default-bind-ciphers ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384
ssl-default-bind-ciphersuites TLS_AES_128_GCM_SHA256:TLS_AES_256_GCM_SHA384:TLS_CHACHA20_POLY1305_SHA256
ssl-default-bind-options ssl-min-ver TLSv1.2 no-tls-tickets
frontend https_front
bind *:443 ssl crt /etc/haproxy/certs/example.com.pem
mode http
http-response set-header Strict-Transport-Security "max-age=31536000; includeSubDomains"
http-request set-header X-Forwarded-Proto https
default_backend web_backend
For a deeper dive into ACL-based traffic rules and security headers, see configure HAProxy SSL termination and security headers.
Validate and reload HAProxy
Always validate the configuration syntax before reloading, since a bad config can drop your listeners.
sudo haproxy -c -f /etc/haproxy/haproxy.cfg
sudo systemctl restart haproxy
sudo systemctl enable haproxy
Automate renewal with a deploy hook
Certbot can run a script automatically whenever it renews a certificate. Use this to rebuild the PEM bundle and reload HAProxy without manual steps.
sudo mkdir -p /etc/letsencrypt/renewal-hooks/deploy
#!/bin/bash
set -e
DOMAIN="example.com"
cat /etc/letsencrypt/live/${DOMAIN}/fullchain.pem /etc/letsencrypt/live/${DOMAIN}/privkey.pem > /etc/haproxy/certs/${DOMAIN}.pem
chmod 600 /etc/haproxy/certs/${DOMAIN}.pem
chown root:root /etc/haproxy/certs/${DOMAIN}.pem
systemctl reload haproxy
sudo chmod 755 /etc/letsencrypt/renewal-hooks/deploy/haproxy-reload.sh
sudo chown root:root /etc/letsencrypt/renewal-hooks/deploy/haproxy-reload.sh
The script needs to be executable by root only, since Certbot's renewal timer runs as root. There is no reason to make it writable by other users.
Test the renewal process
Certbot ships a dry run mode that simulates renewal without touching your actual certificates, letting you confirm the hook fires correctly.
sudo certbot renew --dry-run
Check that the systemd timer for automatic renewal is active, since Certbot installs this by default on most distributions.
systemctl list-timers | grep certbot
Verify your setup
curl -I http://example.com
curl -Iv https://example.com
openssl s_client -connect example.com:443 -servername example.com < /dev/null | grep -A2 "Protocol\|Cipher"
Confirm HAProxy sees your backend servers as healthy using the runtime API or stats page.
echo "show servers state" | sudo socat stdio /run/haproxy/admin.sock
Run an SSL Labs style check locally with testssl.sh, or verify cipher restrictions directly.
openssl s_client -connect example.com:443 -tls1_1
This last command should fail to connect, confirming TLS 1.1 is disabled.
Common issues
SymptomCauseFixHAProxy fails to start after adding bind sslPEM bundle missing or malformedVerify order: certificate, then chain, then key in one file, check with openssl x509 -in /etc/haproxy/certs/example.com.pem -noout -textCertbot renewal fails with port 80 in useHAProxy is bound to port 80 during standalone challengeSwitch to DNS challenge or add the ACME ACL exception shown aboveBrowser shows certificate warningPEM bundle not reloaded after renewalConfirm the deploy hook ran: check /var/log/letsencrypt/letsencrypt.log and reload HAProxy manuallyBackend marked down unexpectedlyHealth check endpoint returns non-200 or times outTest the check URL directly with curl -i http://203.0.113.10:8080/health and adjust inter/fall/riseWeak cipher still negotiatedOld config cached or client forcing legacy TLSRun haproxy -c -f /etc/haproxy/haproxy.cfg to confirm the active config, then reloadPermission denied reading PEM fileFile owned by wrong user or overly restrictive modeSet ownership to root and mode 600, and confirm HAProxy's systemd unit runs with sufficient privilege to read it, do not use 777
Next steps
Configure HAProxy load balancing with multiple backend servers
Set up HAProxy high availability with keepalived clustering
Configure HAProxy with Consul for dynamic service discovery
Configure HAProxy advanced routing with ACLs and maps
Implement HAProxy rate limiting and DDoS protection
Monitor HAProxy with Prometheus and Grafana dashboards
Running this in production?
Want this handled for you? Setting this up once is straightforward. Keeping certificates renewed, cipher suites current, and backend health monitored across environments is the harder part. See how we run infrastructure like this for European teams.
---
### Configure HAProxy with Consul for dynamic service discovery
URL: https://binadit.com/tutorials/configure-haproxy-consul-service-discovery
Category: networking
Difficulty: advanced
Time: ~50 minutes
Author: Binadit Tech Team
> Learn how to combine HAProxy, Consul and consul-template to build a self-updating load balancer that discovers backends automatically, reacts to health checks, and reloads without downtime.
What this solves
Static HAProxy backend lists break down once services scale up, scale down or move between hosts. This tutorial wires HAProxy to Consul's service catalog through consul-template, so backend servers are generated dynamically from live health check data and configuration reloads happen automatically when the topology changes.
By the end you will have a Consul agent registering services with health checks, consul-template rendering HAProxy configuration from those checks, and a reload pipeline that survives scaling events and failures without manual intervention.
Step-by-step installation
Install HAProxy
HAProxy will act as the load balancer that receives dynamically generated backend definitions.
sudo apt update
sudo apt install -y haproxy
sudo dnf install -y haproxy
Install the Consul agent
Each HAProxy node runs a local Consul agent in client mode, joined to your Consul cluster, to query service health data with low latency.
curl -fsSL https://apt.releases.hashicorp.com/gpg | sudo gpg --dearmor -o /usr/share/keyrings/hashicorp-archive-keyring.gpg
echo "deb [signed-by=/usr/share/keyrings/hashicorp-archive-keyring.gpg] https://apt.releases.hashicorp.com $(lsb_release -cs) main" | sudo tee /etc/apt/sources.list.d/hashicorp.list
sudo apt update
sudo apt install -y consul
sudo dnf install -y dnf-plugins-core
sudo dnf config-manager --add-repo https://rpm.releases.hashicorp.com/RHEL/hashicorp.repo
sudo dnf install -y consul
Configure the Consul client agent
This configuration joins the node to an existing Consul server cluster. Replace 203.0.113.10 with your actual Consul server addresses.
datacenter = "dc1"
data_dir = "/opt/consul"
bind_addr = "{{ GetInterfaceIP \"eth0\" }}"
retry_join = ["203.0.113.10", "203.0.113.11", "203.0.113.12"]
client_addr = "127.0.0.1"
enable_local_script_checks = true
ports {
grpc = 8502
}
sudo mkdir -p /opt/consul
sudo chown -R consul:consul /opt/consul
sudo systemctl enable --now consul
consul members
Note: If you are building a fresh Consul cluster rather than joining an existing one, see install and configure Consul for service discovery with clustering and security first.
Register a service with health checks
Services register themselves in Consul's catalog along with a health check. HAProxy will only route to instances that pass this check.
{
"service": {
"name": "web-app",
"port": 8080,
"tags": ["http"],
"check": {
"http": "http://localhost:8080/healthz",
"interval": "10s",
"timeout": "3s",
"deregister_critical_service_after": "90s"
}
}
}
sudo systemctl reload consul
consul catalog services
consul health checks web-app
Repeat this registration file on every application host, adjusting the port for each instance. Consul tracks each instance independently by node and check status.
Install consul-template
consul-template watches the Consul catalog for changes and renders templates, in this case an HAProxy configuration file, whenever backend state changes.
sudo apt install -y consul-template
sudo dnf install -y consul-template
Note: If your distro's repo does not ship consul-template, download the binary release from HashiCorp and place it in /usr/local/bin with chmod 755.
Create the HAProxy template
This template queries Consul for healthy instances of web-app and generates an HAProxy frontend and backend block. It also defines the stats page and load balancing algorithm.
global
log /dev/log local0
maxconn 20000
user haproxy
group haproxy
defaults
log global
mode http
timeout connect 5s
timeout client 30s
timeout server 30s
option httplog
frontend stats
bind *:8404
stats enable
stats uri /stats
stats refresh 10s
stats auth admin:{{ key "haproxy/stats_password" }}
frontend web_front
bind *:80
default_backend web_back
backend web_back
balance leastconn
option httpchk GET /healthz
{{ range service "web-app" }}
server {{ .Node }}-{{ .Port }} {{ .Address }}:{{ .Port }} check
{{ end }}
Warning: Do not hardcode the stats password in the template. Store it in Consul's KV store with consul kv put haproxy/stats_password 'S3cur3-Stats!Pass' so it never appears in version control.
Configure consul-template to render and reload
This configuration tells consul-template where to write the rendered file and what command to run afterward to validate and reload HAProxy safely.
consul {
address = "127.0.0.1:8500"
}
template {
source = "/etc/consul-template/haproxy.ctmpl"
destination = "/etc/haproxy/haproxy.cfg"
command = "haproxy -c -f /etc/haproxy/haproxy.cfg && systemctl reload haproxy"
command_timeout = "30s"
wait {
min = "2s"
max = "10s"
}
}
The wait block debounces rapid-fire changes during scaling events, so a burst of instance registrations does not trigger dozens of reloads in a few seconds.
Set correct ownership and permissions
consul-template needs to write to the HAProxy config directory and trigger a reload. Rather than opening permissions widely, grant a dedicated service account exactly what it needs.
sudo useradd -r -s /usr/sbin/nologin consul-template
sudo chown -R consul-template:haproxy /etc/haproxy
sudo chmod 750 /etc/haproxy
sudo chmod 640 /etc/haproxy/haproxy.cfg
Never use chmod 777. It gives every user on the system full read, write and execute access to your HAProxy configuration, including the stats credentials. Use group ownership between consul-template and haproxy instead, with 750/640 permissions so only the two accounts that need access have it.
Grant the reload command via sudoers instead of running consul-template as root:
consul-template ALL=(root) NOPASSWD: /bin/systemctl reload haproxy, /usr/sbin/haproxy -c -f /etc/haproxy/haproxy.cfg
Run consul-template as a systemd service
Running consul-template under systemd ensures it restarts on failure and starts on boot alongside Consul and HAProxy.
[Unit]
Description=Consul Template for HAProxy
After=network-online.target consul.service
Wants=network-online.target
[Service]
User=consul-template
Group=haproxy
ExecStart=/usr/bin/consul-template -config=/etc/consul-template/config.hcl
Restart=on-failure
RestartSec=5
[Install]
WantedBy=multi-user.target
sudo systemctl daemon-reload
sudo systemctl enable --now consul-template
sudo systemctl status consul-template
Enable and start HAProxy
Start HAProxy once the first rendered config is in place.
sudo haproxy -c -f /etc/haproxy/haproxy.cfg
sudo systemctl enable --now haproxy
sudo systemctl status haproxy
Verify your setup
Confirm Consul sees the service as healthy, the template rendered correctly, and HAProxy is routing traffic.
consul catalog services
consul health checks web-app
cat /etc/haproxy/haproxy.cfg | grep -A5 backend
curl -I http://127.0.0.1/
curl -u admin:S3cur3-Stats!Pass http://127.0.0.1:8404/stats
Register a second instance of web-app on another port and watch the backend update without restarting HAProxy manually.
sudo journalctl -u consul-template -f
You should see a render event followed by a successful reload log line within a few seconds of the new registration.
Testing failover and scaling scenarios
Simulate a backend failure
Stop the application process on one instance and confirm Consul marks the check critical, and that consul-template removes it from the backend within one interval cycle.
sudo systemctl stop web-app
consul health checks web-app
curl -u admin:S3cur3-Stats!Pass http://127.0.0.1:8404/stats
The stats page should show the server entry as down, and the HAProxy config should no longer list it after the next consul-template render.
Simulate horizontal scale-out
Register three additional instances in quick succession to confirm the debounce window in the wait block coalesces the reload into a single event instead of five separate reloads.
for port in 8081 8082 8083; do
sudo tee /etc/consul.d/web-app-$port.json > /dev/null < Deploy PgBouncer in front of PostgreSQL 17 with session, transaction and statement pooling modes, SCRAM-SHA-256 authentication, keepalived-based HA, and Prometheus monitoring for production workloads.
What this solves
PostgreSQL forks a new backend process for every client connection, which becomes expensive under high concurrency. PgBouncer sits between your application and PostgreSQL 17, multiplexing thousands of client connections onto a small pool of real database connections.
This tutorial covers pooling modes, SCRAM-SHA-256 authentication, running multiple PgBouncer instances behind keepalived and HAProxy for failover, tuning pool sizes for production, and monitoring with SHOW POOLS, SHOW STATS and a Prometheus exporter.
Step-by-step configuration
Install PostgreSQL 17 client libraries and PgBouncer
PgBouncer needs the PostgreSQL client libraries to connect upstream. Install both PgBouncer and the PostgreSQL 17 client tools before configuring anything.
sudo apt update
sudo apt install -y curl ca-certificates gnupg
curl -fsSL https://www.postgresql.org/media/keys/ACCC4CF8.asc | sudo gpg --dearmor -o /usr/share/keyrings/postgresql.gpg
echo "deb [signed-by=/usr/share/keyrings/postgresql.gpg] http://apt.postgresql.org/pub/repos/apt $(lsb_release -cs)-pgdg main" | sudo tee /etc/apt/sources.list.d/pgdg.list
sudo apt update
sudo apt install -y postgresql-client-17 pgbouncer
sudo dnf install -y https://download.postgresql.org/pub/repos/yum/reporpms/EL-9-x86_64/pgdg-redhat-repo-latest.noarch.rpm
sudo dnf -qy module disable postgresql
sudo dnf install -y postgresql17 pgbouncer
Confirm PostgreSQL 17 is reachable
Assume PostgreSQL 17 is already running on a primary server at 203.0.113.10. Verify connectivity before wiring up the pooler, otherwise you will debug PgBouncer for a problem that lives elsewhere.
psql -h 203.0.113.10 -U postgres -d appdb -c "SELECT version();"
If you have not yet configured the primary, see install and configure PostgreSQL 17 with performance tuning and security hardening first.
Choose a pooling mode in pgbouncer.ini
PgBouncer supports three pooling modes. Session pooling assigns one server connection per client for the whole session, the safest but least efficient. Transaction pooling releases the server connection after each transaction, the best default for most web apps. Statement pooling releases it after every statement, but breaks multi-statement transactions and most ORMs, so use it only for stateless read-only workloads.
[databases]
appdb = host=203.0.113.10 port=5432 dbname=appdb
appdb_session = host=203.0.113.10 port=5432 dbname=appdb pool_mode=session
[pgbouncer]
listen_addr = 0.0.0.0
listen_port = 6432
auth_type = scram-sha-256
auth_file = /etc/pgbouncer/userlist.txt
admin_users = pgbouncer_admin
stats_users = pgbouncer_stats
pool_mode = transaction
max_client_conn = 2000
default_pool_size = 25
min_pool_size = 5
reserve_pool_size = 10
reserve_pool_timeout = 3
max_db_connections = 100
max_user_connections = 100
server_idle_timeout = 300
server_lifetime = 3600
query_wait_timeout = 30
client_idle_timeout = 0
client_login_timeout = 60
log_connections = 1
log_disconnections = 1
log_pooler_errors = 1
pidfile = /var/run/pgbouncer/pgbouncer.pid
logfile = /var/log/pgbouncer/pgbouncer.log
Note: the appdb_session database entry lets specific clients (migrations, long transactions, LISTEN/NOTIFY consumers) connect through session mode on the same PgBouncer instance without changing the primary pool's mode.
Generate SCRAM-SHA-256 credentials in userlist.txt
PostgreSQL 17 defaults to scram-sha-256 password hashing. PgBouncer must use the exact same hash format in userlist.txt, plaintext or md5 entries will fail authentication against a SCRAM-only server.
psql -h 203.0.113.10 -U postgres -d appdb -t -A -c "SELECT rolname, rolpassword FROM pg_authid WHERE rolname = 'app_user';"
Copy the output into userlist.txt, keeping the SCRAM-SHA-256 hash exactly as returned. Never write a real password here in plaintext.
"app_user" "SCRAM-SHA-256$4096:base64salt$base64storedkey:base64serverkey"
"pgbouncer_admin" "SCRAM-SHA-256$4096:base64salt$base64storedkey:base64serverkey"
sudo chown pgbouncer:pgbouncer /etc/pgbouncer/userlist.txt /etc/pgbouncer/pgbouncer.ini
sudo chmod 600 /etc/pgbouncer/userlist.txt
Never use chmod 777. userlist.txt contains password hashes for every pooled user. Mode 600 owned by the pgbouncer service user is the minimum needed, anything looser exposes credential hashes to every local account on the box.
Enable and start PgBouncer
Start the service and confirm it is listening on port 6432, the standard PgBouncer port distinct from PostgreSQL's 5432.
sudo mkdir -p /var/log/pgbouncer /var/run/pgbouncer
sudo chown pgbouncer:pgbouncer /var/log/pgbouncer /var/run/pgbouncer
sudo systemctl enable --now pgbouncer
sudo systemctl status pgbouncer
Deploy PgBouncer on a second node for redundancy
A single PgBouncer instance is a single point of failure. Repeat the installation and configuration steps above on a second host (203.0.113.11), pointing to the same PostgreSQL primary and using an identical userlist.txt and pgbouncer.ini.
scp /etc/pgbouncer/userlist.txt root@203.0.113.11:/etc/pgbouncer/userlist.txt
scp /etc/pgbouncer/pgbouncer.ini root@203.0.113.11:/etc/pgbouncer/pgbouncer.ini
Front both PgBouncer nodes with HAProxy
HAProxy load balances client connections across both PgBouncer nodes and removes a failed node from rotation automatically using TCP health checks.
sudo apt install -y haproxy
sudo dnf install -y haproxy
frontend pgbouncer_front
bind 203.0.113.20:6432
mode tcp
default_backend pgbouncer_nodes
backend pgbouncer_nodes
mode tcp
balance roundrobin
option tcp-check
tcp-check connect port 6432
server pgb1 203.0.113.10:6432 check inter 3s fall 3 rise 2
server pgb2 203.0.113.11:6432 check inter 3s fall 3 rise 2
sudo systemctl enable --now haproxy
For a deeper walkthrough of HAProxy tuning and ACL-based routing, see configure HAProxy load balancing with multiple backend servers.
Add keepalived for a floating virtual IP
HAProxy itself needs redundancy. Run keepalived on both HAProxy hosts so a virtual IP fails over automatically if the active node goes down.
sudo apt install -y keepalived
sudo dnf install -y keepalived
vrrp_script chk_haproxy {
script "/usr/bin/killall -0 haproxy"
interval 2
weight 2
}
vrrp_instance VI_PGBOUNCER {
state MASTER
interface eth0
virtual_router_id 51
priority 150
advert_int 1
authentication {
auth_type PASS
auth_pass Ch4ngeThisVrrpSecret
}
virtual_ipaddress {
203.0.113.20/24
}
track_script {
chk_haproxy
}
}
sudo systemctl enable --now keepalived
Set state BACKUP and priority 100 on the second node. For the complete active/passive pattern including firewall rules, see configure keepalived with HAProxy backend health monitoring.
Tune pool sizes and timeouts for production load
default_pool_size controls how many server connections PgBouncer opens per database/user pair in transaction mode. Set it based on PostgreSQL's max_connections divided by the number of PgBouncer instances and databases sharing the primary, leaving headroom for replication and admin connections.
SettingRecommended starting valueWhydefault_pool_size20-40Matches typical CPU core count on the database hostmax_client_conn1000-5000PgBouncer connections are cheap, size for peak app instancesreserve_pool_size10-20% of default_pool_sizeAbsorbs short traffic bursts without queuingserver_idle_timeout300sFrees idle backend connections during low trafficquery_wait_timeout30sFails fast instead of queueing indefinitely under saturation
Verify PostgreSQL's own limit accommodates all pooled connections plus replication slots.
psql -h 203.0.113.10 -U postgres -c "SHOW max_connections;"
Warning: setting default_pool_size too high defeats the purpose of pooling. If PgBouncer's total pool size across all instances approaches PostgreSQL's max_connections, you have removed the buffer that protects the primary during a connection storm.
Monitoring PgBouncer
Query live pool and stats via the admin console
PgBouncer exposes an administrative pseudo-database called pgbouncer. Connect as an admin_users entry to inspect pool state in real time.
psql -h 127.0.0.1 -p 6432 -U pgbouncer_admin pgbouncer -c "SHOW POOLS;"
psql -h 127.0.0.1 -p 6432 -U pgbouncer_admin pgbouncer -c "SHOW STATS;"
psql -h 127.0.0.1 -p 6432 -U pgbouncer_admin pgbouncer -c "SHOW CLIENTS;"
SHOW POOLS reports cl_active, cl_waiting and sv_active columns per database/user pair. A consistently non-zero cl_waiting means default_pool_size is too small for the current load.
Deploy the Prometheus exporter
prometheus-pgbouncer-exporter scrapes SHOW STATS and SHOW POOLS and exposes them as Prometheus metrics for Grafana dashboards and alerting.
sudo useradd --no-create-home --shell /usr/sbin/nologin pgbouncer_exporter
curl -L -o /tmp/pgbouncer_exporter.tar.gz https://github.com/prometheus-community/pgbouncer_exporter/releases/download/v0.10.2/pgbouncer_exporter-0.10.2.linux-amd64.tar.gz
tar -xzf /tmp/pgbouncer_exporter.tar.gz -C /tmp
sudo mv /tmp/pgbouncer_exporter-0.10.2.linux-amd64/pgbouncer_exporter /usr/local/bin/
sudo chown pgbouncer_exporter:pgbouncer_exporter /usr/local/bin/pgbouncer_exporter
[Unit]
Description=PgBouncer Prometheus Exporter
After=network.target
[Service]
User=pgbouncer_exporter
Group=pgbouncer_exporter
Environment=DATA_SOURCE_NAME="postgres://pgbouncer_stats:StrongExporterPass9!@127.0.0.1:6432/pgbouncer?sslmode=disable"
ExecStart=/usr/local/bin/pgbouncer_exporter --web.listen-address=127.0.0.1:9127
Restart=on-failure
[Install]
WantedBy=multi-user.target
sudo systemctl daemon-reload
sudo systemctl enable --now pgbouncer_exporter
curl http://127.0.0.1:9127/metrics | grep pgbouncer_pools
Wire this into an existing Prometheus and Grafana stack as described in set up Prometheus and Grafana monitoring stack with Docker Compose, then build alerts on pgbouncer_pools_cl_waiting and pgbouncer_pools_sv_used.
Failover testing with streaming replication
Simulate a primary failure
If PostgreSQL streaming replication is already in place, test that PgBouncer's target can be repointed to a promoted replica quickly. This is the scenario your HA setup exists for.
sudo systemctl stop postgresql
On the standby, promote it to become the new primary.
sudo -u postgres /usr/lib/postgresql/17/bin/pg_ctl promote -D /var/lib/postgresql/17/main
Repoint PgBouncer to the new primary
Update the host in pgbouncer.ini on both PgBouncer nodes, then reload without restarting, PgBouncer reload keeps existing pooled connections draining gracefully.
[databases]
appdb = host=203.0.113.11 port=5432 dbname=appdb
sudo systemctl reload pgbouncer
For the full replication topology, promotion tooling, and automated failover scripts, follow set up PostgreSQL 17 streaming replication with PgBouncer connection pooling and load balancing.
Verify your setup
sudo systemctl status pgbouncer haproxy keepalived
psql -h 203.0.113.20 -p 6432 -U app_user -d appdb -c "SELECT 1;"
psql -h 127.0.0.1 -p 6432 -U pgbouncer_admin pgbouncer -c "SHOW POOLS;"
curl -s http://127.0.0.1:9127/metrics | grep pgbouncer_up
Common issues
SymptomCauseFixERROR: password authentication faileduserlist.txt has a plaintext or md5 hash but PostgreSQL uses scram-sha-256Regenerate the entry with the exact rolpassword value from pg_authidClients hang waiting for a connectiondefault_pool_size too small for concurrent loadCheck SHOW POOLS for cl_waiting, raise default_pool_size or reserve_pool_sizeApplication sees stale prepared statements after failoverStatement pooling mode used with a stateful ORMSwitch that database entry to pool_mode=session or transactionVirtual IP does not move on failoverFirewall blocks VRRP multicast traffic between HAProxy nodesAllow protocol 112 (VRRP) between node IPs instead of disabling the firewallPrometheus exporter shows no metricspgbouncer_stats user lacks permission or wrong DSNConfirm stats_users includes the exporter's role in pgbouncer.iniPostgreSQL rejects connections after reloadmax_connections exceeded by combined pool sizes across instancesReduce default_pool_size or raise max_connections and restart PostgreSQL
Next steps
Set up PostgreSQL 17 streaming replication with PgBouncer connection pooling and load balancing
Configure PostgreSQL 17 SSL encryption and advanced security hardening
Monitor PostgreSQL performance with Prometheus and Grafana dashboards
Configure PostgreSQL 17 PgBouncer multi-region load balancing
Automate PostgreSQL failover with Patroni and etcd
Running this in production?
Want this handled for you? Setting this up once is straightforward. Keeping it patched, monitored, backed up and performant across environments, including pool sizing as traffic grows, is the harder part. See how we run infrastructure like this for European teams.
---
### Configure Kubernetes secrets management with External Secrets Operator and HashiCorp Vault
URL: https://binadit.com/tutorials/configure-kubernetes-secrets-management-with-external-secrets-operator
Category: devops
Difficulty: advanced
Time: ~75 minutes
Author: Binadit Tech Team
> Learn how to deploy External Secrets Operator on Kubernetes, integrate it with HashiCorp Vault using the Kubernetes auth method, and sync secrets via SecretStore and ClusterSecretStore resources with production-grade RBAC and monitoring.
What this solves
Storing secrets as plain Kubernetes Secret objects means they sit base64-encoded in etcd with no rotation, no audit trail, and no central policy control. External Secrets Operator (ESO) bridges Kubernetes to HashiCorp Vault, pulling secrets on a schedule and materializing them as native Secret objects that your pods consume normally.
This tutorial covers a production-ready setup: Vault as the backend, ESO deployed with Helm, Kubernetes auth method for authentication, namespace-scoped SecretStore and cluster-wide ClusterSecretStore resources, templating, rotation, RBAC isolation, and troubleshooting.
Prerequisites and architecture overview
You need a running Kubernetes cluster (kubeadm, EKS, GKE, or AKS all work) with kubectl configured, Helm 3 installed, and a Vault server reachable from the cluster network. If you have not installed Vault yet, see install and configure Vault for secrets management with high availability first.
The data flow is: ESO controller pods authenticate to Vault using a Kubernetes service account token, Vault validates that token against the Kubernetes API, Vault issues a scoped token tied to a policy, and ESO uses that token to read secrets and write them into Kubernetes Secret objects referenced by your workloads.
Note: This tutorial assumes Vault is already unsealed and reachable at a stable address. Production Vault should run with auto-unseal and TLS, not the dev server mode shown for the local backend example.
Step-by-step configuration
Install the Vault CLI and verify connectivity
The Vault CLI lets you configure policies and auth methods from your workstation or a bastion host.
wget -O- https://apt.releases.hashicorp.com/gpg | sudo gpg --dearmor -o /usr/share/keyrings/hashicorp-archive-keyring.gpg
echo "deb [signed-by=/usr/share/keyrings/hashicorp-archive-keyring.gpg] https://apt.releases.hashicorp.com $(lsb_release -cs) main" | sudo tee /etc/apt/sources.list.d/hashicorp.list
sudo apt update && sudo apt install -y vault
sudo dnf install -y dnf-plugins-core
sudo dnf config-manager --add-repo https://rpm.releases.hashicorp.com/RHEL/hashicorp.repo
sudo dnf install -y vault
export VAULT_ADDR="https://vault.example.com:8200"
vault status
Enable the KV v2 secrets engine
KV version 2 supports versioning, which lets you roll back accidental overwrites of application secrets.
vault login
vault secrets enable -path=kv-eso kv-v2
vault kv put kv-eso/production/api-service DB_PASSWORD="Str0ng-P@ss-2024!" API_KEY="ak_live_9f3c7d2a1b"
Enable the Kubernetes auth method in Vault
This lets Vault verify tokens presented by pods against your cluster's API server, without storing static credentials anywhere.
vault auth enable kubernetes
Retrieve the cluster CA and API endpoint, then configure Vault to talk to your cluster.
kubectl config view --raw --minify --flatten -o jsonpath='{.clusters[].cluster.certificate-authority-data}' | base64 -d > /tmp/k8s-ca.crt
KUBE_HOST=$(kubectl config view --raw --minify --flatten -o jsonpath='{.clusters[].cluster.server}')
vault write auth/kubernetes/config \
kubernetes_host="$KUBE_HOST" \
kubernetes_ca_cert=@/tmp/k8s-ca.crt
Create a least-privilege Vault policy
Scope the policy to only the paths ESO needs to read. Never grant blanket access to the entire KV mount.
path "kv-eso/data/production/*" {
capabilities = ["read"]
}
path "kv-eso/metadata/production/*" {
capabilities = ["list", "read"]
}
vault policy write eso-production-read vault-eso-policy.hcl
Create a Kubernetes auth role bound to a service account
Binding the Vault role to a specific service account and namespace enforces that only ESO pods in that namespace can assume this role, which is the core of namespace isolation.
vault write auth/kubernetes/role/eso-production-role \
bound_service_account_names=external-secrets-sa \
bound_service_account_namespaces=external-secrets \
policies=eso-production-read \
ttl=15m
Deploy External Secrets Operator with Helm
ESO ships an official Helm chart that installs the controller, webhook, and CRDs. Deploy it into a dedicated namespace to keep it isolated from application workloads.
kubectl create namespace external-secrets
helm repo add external-secrets https://charts.external-secrets.io
helm repo update
helm install external-secrets external-secrets/external-secrets \
--namespace external-secrets \
--set installCRDs=true \
--set replicaCount=2 \
--set serviceAccount.name=external-secrets-sa
Two replicas give you a controller failover path if a pod is evicted or the node drains during maintenance.
Verify the operator is running
Confirm the controller, webhook, and cert-controller pods reach Running state before creating any secret store resources.
kubectl get pods -n external-secrets
kubectl get crd | grep external-secrets.io
Create a ClusterSecretStore for cluster-wide access
A ClusterSecretStore is not namespaced and can be referenced by ExternalSecret resources across the cluster, useful for shared credentials like registry pull secrets or platform-wide TLS certs.
apiVersion: external-secrets.io/v1beta1
kind: ClusterSecretStore
metadata:
name: vault-backend
spec:
provider:
vault:
server: "https://vault.example.com:8200"
path: "kv-eso"
version: "v2"
auth:
kubernetes:
mountPath: "kubernetes"
role: "eso-production-role"
serviceAccountRef:
name: "external-secrets-sa"
namespace: "external-secrets"
kubectl apply -f manifests/cluster-secret-store.yaml
kubectl get clustersecretstore vault-backend -o wide
Create a namespace-scoped SecretStore
Use a SecretStore instead of a ClusterSecretStore when a team should only sync secrets within its own namespace, reinforcing the isolation established by the RBAC and Vault role bindings above.
apiVersion: external-secrets.io/v1beta1
kind: SecretStore
metadata:
name: vault-backend-app
namespace: production
spec:
provider:
vault:
server: "https://vault.example.com:8200"
path: "kv-eso"
version: "v2"
auth:
kubernetes:
mountPath: "kubernetes"
role: "eso-production-role"
serviceAccountRef:
name: "external-secrets-sa"
namespace: "production"
kubectl create namespace production
kubectl apply -f manifests/app-secret-store.yaml
Define an ExternalSecret to sync into a Kubernetes Secret
The ExternalSecret is the resource that actually maps Vault paths to keys inside a generated Kubernetes Secret. The refreshInterval controls how often ESO re-reads Vault.
apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
name: api-service-secrets
namespace: production
spec:
refreshInterval: 1h
secretStoreRef:
name: vault-backend-app
kind: SecretStore
target:
name: api-service-secrets
creationPolicy: Owner
data:
- secretKey: DB_PASSWORD
remoteRef:
key: production/api-service
property: DB_PASSWORD
- secretKey: API_KEY
remoteRef:
key: production/api-service
property: API_KEY
kubectl apply -f manifests/api-service-external-secret.yaml
kubectl get externalsecret api-service-secrets -n production
kubectl get secret api-service-secrets -n production -o jsonpath='{.data}'
Use templating to reshape synced secrets
Templating lets you build connection strings or config file formats directly from multiple Vault values instead of requiring the application to assemble them at runtime.
apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
name: api-service-connection
namespace: production
spec:
refreshInterval: 30m
secretStoreRef:
name: vault-backend-app
kind: SecretStore
target:
name: api-service-connection
creationPolicy: Owner
template:
engineVersion: v2
data:
DATABASE_URL: "postgres://app_user:{{ .DB_PASSWORD }}@postgres-primary.production.svc.cluster.local:5432/appdb"
data:
- secretKey: DB_PASSWORD
remoteRef:
key: production/api-service
property: DB_PASSWORD
Configure RBAC to restrict who can read synced Secrets and ExternalSecrets
ESO synchronizing Vault into a namespace does not automatically restrict which cluster users can read that Secret. Apply RBAC in the target namespace so only the application's service account and approved operators can view it.
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
namespace: production
name: secret-reader
rules:
- apiGroups: [""]
resources: ["secrets"]
resourceNames: ["api-service-secrets", "api-service-connection"]
verbs: ["get"]
kubectl apply -f manifests/secret-reader-role.yaml
kubectl create rolebinding api-service-secret-reader \
--role=secret-reader \
--serviceaccount=production:api-service-sa \
-n production
For a deeper walkthrough of scoping cluster roles and service accounts, see configure Kubernetes RBAC with service accounts and cluster roles.
Verify your setup
Confirm ESO successfully authenticated to Vault, synced the secret, and that the resulting Kubernetes Secret has the expected keys.
kubectl describe externalsecret api-service-secrets -n production
kubectl get events -n production --field-selector involvedObject.name=api-service-secrets
kubectl logs -n external-secrets deployment/external-secrets --tail=50
The Status.Conditions field in the ExternalSecret describe output should show Ready: True with the reason SecretSynced. If it shows an error, the message usually names the exact Vault path or permission problem.
Warning: Never store raw Vault root tokens in CI/CD pipelines or ConfigMaps to bootstrap this integration. Use short-lived Kubernetes service account tokens through the auth method configured above, and rotate the Vault policy or role immediately if a token is ever exposed.
Secret rotation and refresh behavior
ESO does not push changes, it polls. When you update a value in Vault with vault kv put, the change appears in the Kubernetes Secret only after the next refreshInterval elapses. Set shorter intervals (5-15m) for credentials that rotate frequently, and longer intervals (1h or more) for stable static config to reduce load on Vault.
Secret typeSuggested refreshIntervalReasonDatabase passwords (static)1hRotated manually or via scheduled job, low churnDynamic database creds15mVault dynamic secrets expire on a TTL, needs frequent renewalTLS certificates6hCerts typically valid for weeks or monthsAPI keys for third-party services30mBalance between freshness and Vault request volume
Applications that read environment variables at startup only will not see rotated secrets until the pod restarts. Pair ESO with a rollout mechanism, such as the Reloader controller or a CI/CD hook, to restart deployments when the underlying Secret changes.
Production hardening and high availability considerations
Run Vault itself in HA mode with Raft integrated storage or Consul as the storage backend, and enable auto-unseal so Vault comes back online automatically after a node restart without manual intervention. See configure Vault auto-unseal with AWS KMS for a working setup.
Run at least two ESO controller replicas across separate nodes using pod anti-affinity, and set resource requests and limits so the controller is not evicted under memory pressure. Enable the ESO Prometheus metrics endpoint and scrape externalsecret_sync_calls_total and externalsecret_sync_calls_error to alert on sync failures before applications notice missing secrets.
kubectl get --raw /api/v1/namespaces/external-secrets/services/external-secrets-metrics:http-metrics/proxy/metrics | grep externalsecret_sync
Enforce network policies so only the ESO namespace can reach Vault's port over the network, reducing the blast radius if a workload pod is compromised. If you are also running Argo CD, review integrate ArgoCD with External Secrets Operator to keep SecretStore and ExternalSecret manifests under GitOps control without committing plaintext values.
Common issues
SymptomCauseFixExternalSecret stuck in SecretSyncedErrorVault role not bound to the correct service account or namespaceCheck vault read auth/kubernetes/role/eso-production-role matches the SecretStore's serviceAccountRefpermission denied reading secretVault policy path does not match KV mount path exactlyRemember KV v2 requires kv-eso/data/ in the policy, not just kv-eso/Secret never updates after Vault changerefreshInterval has not elapsed yetLower refreshInterval or manually force reconcile with kubectl annotate externalsecret api-service-secrets force-sync=$(date +%s) -n production --overwritex509 certificate signed by unknown authorityVault server uses a certificate not trusted by cluster nodesMount the CA bundle into the ESO pod via caBundle in the SecretStore provider configClusterSecretStore works but SecretStore fails in one namespacebound_service_account_namespaces in Vault role restricts to a different namespaceCreate a separate Vault role per namespace or widen the bound namespaces list deliberately
Next steps
Configure Vault dynamic secrets for databases with PostgreSQL and MySQL integration
Set up Vault as a PKI certificate authority with SSL automation and intermediate CA
Implement Kubernetes secrets management with HashiCorp Vault integration
Configure Kubernetes secrets management with Sealed Secrets for secure Helm values
Monitor Kubernetes network policies with Prometheus and Grafana for enhanced cluster security
Running this in production?
Want this handled for you? Running this at scale adds a second layer of work: Vault unseal drills, ESO controller upgrades, policy audits, and alerting when a sync silently fails. Talk to an engineer if you'd rather hand the ops side off.
---
### Integrate Jaeger with Istio service mesh for distributed tracing
URL: https://binadit.com/tutorials/integrate-jaeger-with-istio-service-mesh-tracing
Category: devops
Difficulty: advanced
Time: ~60 minutes
Author: Binadit Tech Team
> Learn how to integrate Jaeger with Istio service mesh to get end-to-end distributed tracing across Kubernetes microservices, including sidecar injection, trace sampling, and ingress gateway access to the Jaeger UI.
What this solves
When requests flow through dozens of microservices behind an Istio service mesh, pinpointing latency or failures without distributed tracing is nearly impossible. This tutorial integrates Jaeger with Istio so every Envoy sidecar automatically emits spans, giving you a full request timeline across services without touching application code.
You will install Istio with tracing enabled, deploy Jaeger as the tracing backend, configure sampling and telemetry, and verify trace propagation end to end through the ingress gateway.
Prerequisites and Kubernetes cluster preparation
Confirm cluster access and resources
You need a running Kubernetes cluster (1.27+) with at least 3 worker nodes, 4 vCPU and 8GB RAM free for Istio control plane, Jaeger, and demo workloads. Verify kubectl access before continuing.
kubectl version --short
kubectl get nodes
kubectl cluster-info
Install prerequisite tools
You need curl, istioctl, and Helm to install Istio and Jaeger components.
sudo apt update && sudo apt install -y curl tar unzip
sudo dnf install -y curl tar unzip
Download and install istioctl
The istioctl binary manages the Istio installation profile and lets you enable tracing at install time.
curl -L https://istio.io/downloadIstio | ISTIO_VERSION=1.23.2 sh -
cd istio-1.23.2
export PATH=$PWD/bin:$PATH
istioctl version --remote=false
Install Helm
Helm is required to deploy Jaeger with the official chart in a repeatable way.
curl https://raw.githubusercontent.com/helm/helm/main/scripts/get-helm-3 | bash
curl https://raw.githubusercontent.com/helm/helm/main/scripts/get-helm-3 | bash
Installing Istio service mesh with tracing enabled
Create the istio-system namespace
All Istio control plane components and observability tools live in this dedicated namespace.
kubectl create namespace istio-system
Install Istio with tracing configuration
The demo profile enables the ingress gateway and sets a default trace sampling percentage. Production deployments should use a custom IstioOperator manifest instead.
apiVersion: install.istio.io/v1alpha1
kind: IstioOperator
spec:
profile: default
meshConfig:
enableTracing: true
defaultConfig:
tracing:
sampling: 10.0
extensionProviders:
- name: jaeger
opentelemetry:
service: jaeger-collector.istio-system.svc.cluster.local
port: 4317
values:
global:
proxy:
tracer: "openelemetry"
istioctl install -f istio-config.yaml -y
Enable automatic sidecar injection
Label the namespace where your microservices run so Istio automatically injects the Envoy sidecar into every pod.
kubectl create namespace demo-app
kubectl label namespace demo-app istio-injection=enabled
Note: If you already manage RBAC for this cluster, review configuring Kubernetes RBAC with service accounts and cluster roles before granting broader access to the istio-system namespace.
Deploying Jaeger and configuring the tracing backend
Add the Jaeger Helm repository
The official Jaegertracing chart deploys the collector, query service, and UI as separate components, which scales better than the all-in-one image.
helm repo add jaegertracing https://jaegertracing.github.io/helm-charts
helm repo update
Deploy Jaeger with production values
This example uses in-memory storage for a quick start. For a durable backend with retention and encryption, see the Elasticsearch backend guide linked below.
provisionDataStore:
cassandra: false
elasticsearch: false
storage:
type: memory
collector:
service:
otlp:
grpc:
name: otlp-grpc
port: 4317
http:
name: otlp-http
port: 4318
query:
service:
type: ClusterIP
helm install jaeger jaegertracing/jaeger \
-n istio-system \
-f jaeger-values.yaml
Confirm Jaeger components are running
You should see collector, query, and agent pods (if enabled) in a Running state.
kubectl get pods -n istio-system -l app.kubernetes.io/name=jaeger
Warning: In-memory storage loses all trace data on pod restart. Do not use this in production. Use Elasticsearch or Cassandra as documented in configuring Jaeger with Elasticsearch backend security and encryption.
Configuring Istio telemetry and trace sampling rates
Create a Telemetry resource
Istio's Telemetry API controls sampling rate and trace propagation headers mesh-wide or per namespace. A 100 percent sample rate is useful for testing but too expensive for production traffic volumes.
apiVersion: telemetry.istio.io/v1alpha1
kind: Telemetry
metadata:
name: mesh-tracing
namespace: istio-system
spec:
tracing:
- providers:
- name: jaeger
randomSamplingPercentage: 10.00
kubectl apply -f telemetry.yaml
Override sampling for a specific namespace
You can apply a higher sampling rate temporarily in a staging or debugging namespace without affecting production traffic elsewhere in the mesh.
apiVersion: telemetry.istio.io/v1alpha1
kind: Telemetry
metadata:
name: debug-tracing
namespace: demo-app
spec:
tracing:
- providers:
- name: jaeger
randomSamplingPercentage: 100.00
kubectl apply -f telemetry-debug.yaml
For high-volume production environments, tune sampling with head-based and tail-based strategies as covered in setting up Jaeger sampling strategies for high-volume production tracing.
Enabling automatic sidecar trace propagation
Deploy a sample multi-service application
Istio's Envoy sidecars generate spans automatically, but your application code must forward the trace context headers on outbound calls, otherwise spans from different services will not link together.
kubectl apply -n demo-app -f https://raw.githubusercontent.com/istio/istio/release-1.23/samples/bookinfo/platform/kube/bookinfo.yaml
Verify header propagation in application code
Each service in the call chain must propagate these headers from incoming requests to any outgoing requests it makes: x-request-id, x-b3-traceid, x-b3-spanid, x-b3-parentspanid, x-b3-sampled, x-b3-flags, and b3 or traceparent for W3C trace context.
kubectl logs -n demo-app deploy/productpage-v1 -c istio-proxy | grep -i trace
Expose the sample app through the ingress gateway
This lets you generate real traffic through the mesh entry point to validate end-to-end tracing.
apiVersion: networking.istio.io/v1alpha3
kind: Gateway
metadata:
name: bookinfo-gateway
namespace: demo-app
spec:
selector:
istio: ingressgateway
servers:
- port:
number: 80
name: http
protocol: HTTP
hosts:
- "*"
---
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
name: bookinfo
namespace: demo-app
spec:
hosts:
- "*"
gateways:
- bookinfo-gateway
http:
- match:
- uri:
exact: /productpage
route:
- destination:
host: productpage
port:
number: 9080
kubectl apply -f bookinfo-gateway.yaml
Verifying distributed traces across microservices
Generate traffic
Send repeated requests through the ingress gateway to generate a trace chain across productpage, details, reviews, and ratings services.
export INGRESS_HOST=$(kubectl -n istio-system get service istio-ingressgateway -o jsonpath='{.status.loadBalancer.ingress[0].ip}')
for i in $(seq 1 20); do curl -s -o /dev/null http://$INGRESS_HOST/productpage; done
Port-forward the Jaeger query UI
Access Jaeger locally to confirm traces are arriving with all expected spans.
kubectl port-forward -n istio-system svc/jaeger-query 16686:16686
Open http://localhost:16686, select the productpage.demo-app service, and search for recent traces. Each trace should show connected spans from productpage through details, reviews, and ratings.
Integrating Jaeger UI with Istio ingress gateway
Create a dedicated gateway for Jaeger
Exposing Jaeger through the Istio ingress gateway avoids running a separate load balancer just for observability tools.
apiVersion: networking.istio.io/v1alpha3
kind: Gateway
metadata:
name: jaeger-gateway
namespace: istio-system
spec:
selector:
istio: ingressgateway
servers:
- port:
number: 80
name: http-jaeger
protocol: HTTP
hosts:
- "jaeger.example.com"
---
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
name: jaeger-vs
namespace: istio-system
spec:
hosts:
- "jaeger.example.com"
gateways:
- jaeger-gateway
http:
- route:
- destination:
host: jaeger-query
port:
number: 16686
kubectl apply -f jaeger-gateway.yaml
Point DNS and test access
Add an A record for jaeger.example.com pointing to the ingress gateway's external IP, then confirm the UI loads.
curl -s -o /dev/null -w "%{http_code}\n" -H "Host: jaeger.example.com" http://203.0.113.10/
Warning: Do not expose the raw Jaeger UI without authentication on a public IP. Add OAuth2 and RBAC as described in configuring Jaeger authentication with OAuth2 and RBAC for enterprise security before exposing it beyond your internal network.
Verify your setup
kubectl get pods -n istio-system
kubectl get telemetry -A
istioctl proxy-config log deploy/productpage-v1.demo-app
kubectl exec -n demo-app deploy/productpage-v1 -c istio-proxy -- curl -s localhost:15000/stats | grep tracing
A healthy setup shows all Istio and Jaeger pods Running, at least one Telemetry resource applied, and non-zero tracing counters on the sidecar's admin stats endpoint.
Common issues
SymptomCauseFixTraces show single spans with no parent-child linksApplication not propagating trace headers on outbound callsForward incoming x-request-id, x-b3-*, and traceparent headers on every outbound HTTP client callNo traces appear in Jaeger UI at allSampling percentage set too low or Telemetry resource not applied to the right namespaceCheck kubectl get telemetry -A and temporarily raise randomSamplingPercentage to 100 for testingJaeger collector pod crashloopingOTLP gRPC port misconfigured or storage backend unreachableCheck kubectl logs -n istio-system deploy/jaeger-collector and confirm storage.type matches your backendSidecar not injected into podsNamespace missing the istio-injection labelRun kubectl label namespace demo-app istio-injection=enabled then restart the deploymentJaeger UI returns 502 through ingress gatewayVirtualService pointing to wrong service name or portConfirm the Jaeger query service name with kubectl get svc -n istio-system and match the port in the VirtualServiceHigh trace volume overwhelming storage backendSampling rate too high for production trafficLower randomSamplingPercentage and use tail-based sampling for error and slow-request capture
Next steps
Implement Istio observability with Jaeger tracing and Kiali dashboard for Kubernetes service mesh
Configure Jaeger distributed tracing on Kubernetes cluster with Helm charts and Elasticsearch backend
Configure Istio distributed tracing with Jaeger and Zipkin for comprehensive microservices observability
Configure advanced Jaeger sampling strategies for high-traffic environments
Set up Istio multi-cluster service mesh with cross-cluster communication
Running this in production?
Want this handled for you? Running this at scale adds a second layer of work: capacity planning for the tracing pipeline, storage retention, sampling tuning as traffic grows, and on-call coverage when the collector falls behind. See how we run infrastructure like this for European teams.
---
### Configure Consul Connect service mesh monitoring with distributed tracing
URL: https://binadit.com/tutorials/configure-consul-connect-service-mesh-monitoring
Category: monitoring
Difficulty: advanced
Time: ~45 minutes
Author: Binadit Tech Team
> Set up comprehensive monitoring for Consul Connect service mesh with Prometheus metrics, Grafana dashboards, Jaeger distributed tracing, and Envoy proxy observability for production-grade service mesh operations.
What this solves
Consul Connect service mesh provides secure service-to-service communication, but operating it reliably requires deep observability into service health, proxy performance, and request flows. This tutorial configures comprehensive monitoring with Prometheus metrics collection, Grafana dashboards for service mesh visualization, and distributed tracing with Jaeger and OpenTelemetry to track requests across your entire service topology.
Prerequisites
You need a running Consul cluster with Connect enabled and at least two services configured to communicate through the service mesh. This tutorial builds on existing Consul Connect infrastructure to add monitoring capabilities.
Update system packages
Start by updating your package manager and installing required dependencies for monitoring components.
sudo apt update && sudo apt upgrade -y
sudo apt install -y wget curl unzip jq
sudo dnf update -y
sudo dnf install -y wget curl unzip jq
Configure Consul metrics collection
Enable Consul telemetry
Configure Consul to export metrics in Prometheus format and enable detailed service mesh telemetry.
telemetry {
prometheus_retention_time = "24h"
disable_hostname = true
metrics_prefix = "consul"
}
connect {
enabled = true
}
ports {
grpc = 8502
http = 8500
}
Configure Connect proxy metrics
Enable detailed metrics collection for Envoy proxies managed by Consul Connect.
connect {
enabled = true
proxy_defaults {
config {
envoy_prometheus_bind_addr = "0.0.0.0:9102"
envoy_stats_bind_addr = "0.0.0.0:9103"
}
}
}
Restart Consul services
Apply the new configuration by restarting Consul on all cluster nodes.
sudo systemctl restart consul
sudo systemctl status consul
curl -s http://localhost:8500/v1/agent/metrics?format=prometheus | head -20
Install and configure Prometheus
Install Prometheus
Download and install the latest version of Prometheus for metrics collection and storage.
wget https://github.com/prometheus/prometheus/releases/download/v2.45.0/prometheus-2.45.0.linux-amd64.tar.gz
tar xvfz prometheus-2.45.0.linux-amd64.tar.gz
sudo mv prometheus-2.45.0.linux-amd64/prometheus /usr/local/bin/
sudo mv prometheus-2.45.0.linux-amd64/promtool /usr/local/bin/
sudo useradd --no-create-home --shell /bin/false prometheus
sudo mkdir -p /etc/prometheus /var/lib/prometheus
sudo chown prometheus:prometheus /etc/prometheus /var/lib/prometheus
Configure Prometheus for Consul metrics
Set up Prometheus to discover and scrape metrics from Consul servers and Connect proxies automatically.
global:
scrape_interval: 15s
evaluation_interval: 15s
scrape_configs:
- job_name: 'consul'
static_configs:
- targets: ['localhost:8500']
metrics_path: /v1/agent/metrics
params:
format: ['prometheus']
scrape_interval: 5s
- job_name: 'consul-connect-proxies'
consul_sd_configs:
- server: 'localhost:8500'
services: []
relabel_configs:
- source_labels: [__meta_consul_service_metadata_proxy_type]
regex: connect-proxy
action: keep
- source_labels: [__meta_consul_service_port]
target_label: __address__
regex: (.*)
replacement: ${1}:9102
- source_labels: [__meta_consul_service]
target_label: service
- source_labels: [__meta_consul_node]
target_label: node
metrics_path: /metrics
scrape_interval: 5s
- job_name: 'envoy-admin'
consul_sd_configs:
- server: 'localhost:8500'
services: []
relabel_configs:
- source_labels: [__meta_consul_service_metadata_proxy_type]
regex: connect-proxy
action: keep
- source_labels: [__meta_consul_service_port]
target_label: __address__
regex: (.*)
replacement: ${1}:9103
- source_labels: [__meta_consul_service]
target_label: service
- source_labels: [__meta_consul_node]
target_label: node
metrics_path: /stats/prometheus
scrape_interval: 10s
Create Prometheus systemd service
Configure Prometheus to run as a system service with proper permissions and resource limits.
[Unit]
Description=Prometheus
Wants=network-online.target
After=network-online.target
[Service]
User=prometheus
Group=prometheus
Type=simple
ExecStart=/usr/local/bin/prometheus \
--config.file /etc/prometheus/prometheus.yml \
--storage.tsdb.path /var/lib/prometheus/ \
--web.console.templates=/etc/prometheus/consoles \
--web.console.libraries=/etc/prometheus/console_libraries \
--web.listen-address=0.0.0.0:9090 \
--web.enable-lifecycle \
--storage.tsdb.retention.time=30d
[Install]
WantedBy=multi-user.target
Start Prometheus
Enable and start the Prometheus service, then verify it can scrape Consul metrics.
sudo chown -R prometheus:prometheus /etc/prometheus/
sudo systemctl daemon-reload
sudo systemctl enable --now prometheus
sudo systemctl status prometheus
curl -s http://localhost:9090/api/v1/targets | jq '.data.activeTargets[] | {job: .labels.job, health: .health}'
Install and configure Grafana dashboards
Install Grafana
Install Grafana for creating comprehensive service mesh monitoring dashboards.
sudo apt install -y apt-transport-https software-properties-common wget
wget -q -O - https://packages.grafana.com/gpg.key | sudo apt-key add -
echo "deb https://packages.grafana.com/oss/deb stable main" | sudo tee -a /etc/apt/sources.list.d/grafana.list
sudo apt update
sudo apt install -y grafana
sudo tee /etc/yum.repos.d/grafana.repo<
Configure Grafana data source
Add Prometheus as a data source for Grafana to visualize Consul Connect metrics.
apiVersion: 1
datasources:
- name: Prometheus
type: prometheus
access: proxy
url: http://localhost:9090
isDefault: true
editable: true
Create Consul Connect dashboard
Deploy a comprehensive dashboard for monitoring Consul Connect service mesh metrics.
{
"dashboard": {
"id": null,
"title": "Consul Connect Service Mesh",
"tags": ["consul", "connect", "service-mesh"],
"timezone": "browser",
"panels": [
{
"title": "Service Health",
"type": "stat",
"targets": [
{
"expr": "consul_health_service_query_tag{status=\"passing\"}",
"legendFormat": "Healthy Services"
}
],
"gridPos": {"h": 8, "w": 6, "x": 0, "y": 0}
},
{
"title": "Proxy Connections",
"type": "graph",
"targets": [
{
"expr": "rate(envoy_cluster_upstream_cx_connect_total[5m])",
"legendFormat": "{{service}} - {{cluster_name}}"
}
],
"gridPos": {"h": 8, "w": 12, "x": 6, "y": 0}
},
{
"title": "Request Rate",
"type": "graph",
"targets": [
{
"expr": "rate(envoy_http_inbound_0_0_0_0_20000_http_requests_total[5m])",
"legendFormat": "{{service}} - Requests/sec"
}
],
"gridPos": {"h": 8, "w": 12, "x": 0, "y": 8}
},
{
"title": "Response Times",
"type": "graph",
"targets": [
{
"expr": "histogram_quantile(0.95, rate(envoy_http_inbound_0_0_0_0_20000_http_request_duration_milliseconds_bucket[5m]))",
"legendFormat": "{{service}} - 95th percentile"
}
],
"gridPos": {"h": 8, "w": 12, "x": 12, "y": 8}
}
],
"time": {
"from": "now-1h",
"to": "now"
},
"refresh": "10s"
}
}
Start Grafana
Enable and start Grafana, then access the dashboard to verify service mesh metrics visualization.
sudo chown -R grafana:grafana /var/lib/grafana/
sudo systemctl enable --now grafana-server
sudo systemctl status grafana-server
echo "Grafana available at http://localhost:3000 (admin/admin)"
Configure distributed tracing with Jaeger
Install Jaeger
Install Jaeger for distributed tracing across your Consul Connect service mesh.
wget https://github.com/jaegertracing/jaeger/releases/download/v1.47.0/jaeger-1.47.0-linux-amd64.tar.gz
tar -xzf jaeger-1.47.0-linux-amd64.tar.gz
sudo mv jaeger-1.47.0-linux-amd64/jaeger-all-in-one /usr/local/bin/
sudo useradd --no-create-home --shell /bin/false jaeger
sudo mkdir -p /var/lib/jaeger
sudo chown jaeger:jaeger /var/lib/jaeger
Configure Jaeger service
Set up Jaeger to collect traces from Envoy proxies in your service mesh with proper storage configuration.
[Unit]
Description=Jaeger Tracing
After=network.target
[Service]
User=jaeger
Group=jaeger
Type=simple
ExecStart=/usr/local/bin/jaeger-all-in-one \
--collector.grpc-server.host-port=:14250 \
--collector.http-server.host-port=:14268 \
--query.host-port=:16686 \
--memory.max-traces=50000 \
--log-level=info
Restart=always
[Install]
WantedBy=multi-user.target
Configure Envoy tracing
Enable distributed tracing in Consul Connect by configuring Envoy proxies to send trace data to Jaeger.
connect {
enabled = true
proxy_defaults {
config {
envoy_tracing_json = jsonencode({
http = {
name = "envoy.tracers.zipkin"
typed_config = {
"@type" = "type.googleapis.com/envoy.extensions.tracers.zipkin.v3.ZipkinConfig"
collector_cluster = "jaeger_collector"
collector_endpoint_version = "HTTP_JSON"
collector_endpoint = "/api/v2/spans"
shared_span_context = false
}
}
})
envoy_extra_static_clusters_json = jsonencode({
jaeger_collector = {
name = "jaeger_collector"
connect_timeout = "1s"
type = "STRICT_DNS"
lb_policy = "ROUND_ROBIN"
load_assignment = {
cluster_name = "jaeger_collector"
endpoints = [{
lb_endpoints = [{
endpoint = {
address = {
socket_address = {
address = "127.0.0.1"
port_value = 14268
}
}
}
}]
}]
}
}
})
}
}
}
Start Jaeger and restart Connect proxies
Start the Jaeger service and restart your Connect proxies to enable tracing.
sudo systemctl daemon-reload
sudo systemctl enable --now jaeger
sudo systemctl status jaeger
# Restart Consul to pick up tracing configuration
sudo systemctl restart consul
# Restart any existing Connect proxies
consul connect proxy -sidecar-for web-service &
echo "Jaeger UI available at http://localhost:16686"
Configure OpenTelemetry integration
Install OpenTelemetry Collector
Deploy the OpenTelemetry Collector to provide advanced telemetry processing and export capabilities.
wget https://github.com/open-telemetry/opentelemetry-collector-releases/releases/download/v0.80.0/otelcol_0.80.0_linux_amd64.tar.gz
tar -xzf otelcol_0.80.0_linux_amd64.tar.gz
sudo mv otelcol /usr/local/bin/
sudo useradd --no-create-home --shell /bin/false otelcol
sudo mkdir -p /etc/otelcol
sudo chown otelcol:otelcol /etc/otelcol
Configure OpenTelemetry for service mesh
Set up the collector to receive traces from Envoy and export them to Jaeger and metrics to Prometheus.
receivers:
otlp:
protocols:
grpc:
endpoint: 0.0.0.0:4317
http:
endpoint: 0.0.0.0:4318
zipkin:
endpoint: 0.0.0.0:9411
prometheus:
config:
scrape_configs:
- job_name: 'envoy-metrics'
static_configs:
- targets: ['localhost:9102']
processors:
batch:
timeout: 1s
send_batch_size: 1024
attributes:
actions:
- key: service.name
action: upsert
from_attribute: service_name
- key: service.version
action: upsert
from_attribute: service_version
exporters:
jaeger:
endpoint: http://localhost:14250
tls:
insecure: true
prometheus:
endpoint: "0.0.0.0:8889"
logging:
loglevel: debug
service:
pipelines:
traces:
receivers: [otlp, zipkin]
processors: [batch, attributes]
exporters: [jaeger, logging]
metrics:
receivers: [prometheus]
processors: [batch]
exporters: [prometheus, logging]
Start OpenTelemetry Collector
Create a systemd service for the OpenTelemetry Collector and start it.
[Unit]
Description=OpenTelemetry Collector
After=network.target
[Service]
User=otelcol
Group=otelcol
Type=simple
ExecStart=/usr/local/bin/otelcol --config=/etc/otelcol/config.yaml
Restart=always
[Install]
WantedBy=multi-user.target
sudo chown -R otelcol:otelcol /etc/otelcol/
sudo systemctl daemon-reload
sudo systemctl enable --now otelcol
sudo systemctl status otelcol
Monitor Envoy proxy metrics
Configure enhanced Envoy metrics
Enable comprehensive Envoy metrics collection including circuit breaker status and connection pool metrics.
connect {
enabled = true
proxy_defaults {
config {
envoy_prometheus_bind_addr = "0.0.0.0:9102"
envoy_stats_bind_addr = "0.0.0.0:9103"
# Enable additional Envoy stats
envoy_stats_config_json = jsonencode({
stats_config = {
histogram_bucket_settings = [
{
match = {
prefix = "http.inbound"
}
buckets = [0.5, 1, 5, 10, 25, 50, 100, 250, 500, 1000, 2500, 5000, 10000]
}
]
}
})
# Enable circuit breaker and outlier detection stats
envoy_extra_static_clusters_json = jsonencode({
circuit_breakers = {
thresholds = [
{
priority = "DEFAULT"
max_connections = 1024
max_pending_requests = 256
max_requests = 1024
max_retries = 3
}
]
}
outlier_detection = {
consecutive_5xx = 3
interval = "30s"
base_ejection_time = "30s"
max_ejection_percent = 50
}
})
}
}
}
Create Envoy proxy dashboard
Deploy a specialized dashboard for monitoring Envoy proxy performance and health metrics.
{
"dashboard": {
"id": null,
"title": "Envoy Proxy Metrics",
"tags": ["envoy", "proxy", "consul-connect"],
"panels": [
{
"title": "Connection Pool Status",
"type": "graph",
"targets": [
{
"expr": "envoy_cluster_upstream_cx_active",
"legendFormat": "{{cluster_name}} - Active Connections"
},
{
"expr": "envoy_cluster_upstream_cx_overflow",
"legendFormat": "{{cluster_name}} - Overflow"
}
]
},
{
"title": "Circuit Breaker Status",
"type": "stat",
"targets": [
{
"expr": "envoy_cluster_circuit_breakers_default_cx_open",
"legendFormat": "{{cluster_name}} - Circuit Open"
}
]
},
{
"title": "Request Success Rate",
"type": "graph",
"targets": [
{
"expr": "rate(envoy_http_inbound_0_0_0_0_20000_http_requests_total{response_code!~\"5..\"}[5m]) / rate(envoy_http_inbound_0_0_0_0_20000_http_requests_total[5m]) * 100",
"legendFormat": "{{service}} - Success Rate %"
}
]
},
{
"title": "Outlier Detection Events",
"type": "graph",
"targets": [
{
"expr": "rate(envoy_cluster_outlier_detection_ejections_active[5m])",
"legendFormat": "{{cluster_name}} - Ejections"
}
]
}
]
}
}
Restart services for enhanced metrics
Apply the enhanced Envoy configuration by restarting Consul and any running proxies.
sudo systemctl restart consul
sudo systemctl restart grafana-server
# Verify metrics endpoints are responding
curl -s http://localhost:9102/metrics | grep envoy_cluster | head -5
curl -s http://localhost:9103/stats | grep circuit_breakers | head -5
Verify your setup
Test the complete monitoring stack
---
### Configure Kubernetes RBAC with service accounts and cluster roles for secure access control
URL: https://binadit.com/tutorials/configure-kubernetes-rbac-with-service-accounts-and-cluster-roles
Category: devops
Difficulty: intermediate
Time: ~25 minutes
Author: Binadit Tech Team
> Learn to implement Kubernetes Role-Based Access Control (RBAC) with service accounts, cluster roles, and role bindings for granular permissions and secure cluster access management.
What this solves
Kubernetes RBAC provides fine-grained access control for your cluster resources by defining who can perform specific actions on which resources. This tutorial shows you how to create service accounts with appropriate permissions using cluster roles and role bindings, ensuring secure access control while maintaining operational flexibility.
Understanding RBAC components
Kubernetes RBAC consists of four main components that work together to control access. Service accounts represent identities for pods and external systems. Roles and ClusterRoles define permissions for specific actions. RoleBindings and ClusterRoleBindings associate subjects (users, groups, or service accounts) with roles.
Note: ClusterRoles apply cluster-wide while Roles are namespace-specific. ClusterRoleBindings grant cluster-wide permissions while RoleBindings grant namespace-specific permissions.
Step-by-step configuration
Verify RBAC is enabled
Check that RBAC is enabled in your Kubernetes cluster by examining the API server configuration.
kubectl auth can-i list pods --as=system:anonymous
kubectl cluster-info dump | grep -i authorization-mode
Create a dedicated namespace
Create a namespace for testing RBAC configurations to isolate your setup.
kubectl create namespace rbac-demo
Create service accounts
Create service accounts with specific metadata and labels for better organization.
apiVersion: v1
kind: ServiceAccount
metadata:
name: pod-reader
namespace: rbac-demo
labels:
app: rbac-demo
role: reader
annotations:
description: "Service account for reading pod information"
---
apiVersion: v1
kind: ServiceAccount
metadata:
name: deployment-manager
namespace: rbac-demo
labels:
app: rbac-demo
role: manager
annotations:
description: "Service account for managing deployments"
---
apiVersion: v1
kind: ServiceAccount
metadata:
name: cluster-admin-sa
namespace: rbac-demo
labels:
app: rbac-demo
role: admin
annotations:
description: "Service account with cluster-wide admin privileges"
kubectl apply -f serviceaccounts.yaml
Create cluster roles with specific permissions
Define cluster roles with granular permissions for different access levels.
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
name: pod-reader-role
labels:
app: rbac-demo
rules:
- apiGroups: [""]
resources: ["pods"]
verbs: ["get", "list", "watch"]
- apiGroups: [""]
resources: ["pods/log"]
verbs: ["get", "list"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
name: deployment-manager-role
labels:
app: rbac-demo
rules:
- apiGroups: ["apps"]
resources: ["deployments"]
verbs: ["get", "list", "watch", "create", "update", "patch", "delete"]
- apiGroups: [""]
resources: ["pods"]
verbs: ["get", "list", "watch"]
- apiGroups: [""]
resources: ["events"]
verbs: ["get", "list", "watch"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
name: namespace-admin-role
labels:
app: rbac-demo
rules:
- apiGroups: [""]
resources: ["*"]
verbs: ["*"]
- apiGroups: ["apps"]
resources: ["*"]
verbs: ["*"]
- apiGroups: ["extensions"]
resources: ["*"]
verbs: ["*"]
kubectl apply -f clusterroles.yaml
Create namespace-specific roles
Create roles that are scoped to specific namespaces for more granular control.
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
namespace: rbac-demo
name: secret-manager
labels:
app: rbac-demo
rules:
- apiGroups: [""]
resources: ["secrets"]
verbs: ["get", "list", "watch", "create", "update", "patch", "delete"]
- apiGroups: [""]
resources: ["configmaps"]
verbs: ["get", "list", "watch", "create", "update", "patch"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
namespace: rbac-demo
name: service-manager
labels:
app: rbac-demo
rules:
- apiGroups: [""]
resources: ["services"]
verbs: ["get", "list", "watch", "create", "update", "patch", "delete"]
- apiGroups: [""]
resources: ["endpoints"]
verbs: ["get", "list", "watch"]
kubectl apply -f roles.yaml
Create cluster role bindings
Bind service accounts to cluster roles to grant cluster-wide permissions.
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
name: pod-reader-binding
labels:
app: rbac-demo
subjects:
- kind: ServiceAccount
name: pod-reader
namespace: rbac-demo
roleRef:
kind: ClusterRole
name: pod-reader-role
apiGroup: rbac.authorization.k8s.io
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
name: deployment-manager-binding
labels:
app: rbac-demo
subjects:
- kind: ServiceAccount
name: deployment-manager
namespace: rbac-demo
roleRef:
kind: ClusterRole
name: deployment-manager-role
apiGroup: rbac.authorization.k8s.io
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
name: cluster-admin-binding
labels:
app: rbac-demo
subjects:
- kind: ServiceAccount
name: cluster-admin-sa
namespace: rbac-demo
roleRef:
kind: ClusterRole
name: cluster-admin
apiGroup: rbac.authorization.k8s.io
kubectl apply -f clusterrolebindings.yaml
Create namespace-specific role bindings
Bind service accounts to namespace roles for scoped permissions.
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
name: secret-manager-binding
namespace: rbac-demo
labels:
app: rbac-demo
subjects:
- kind: ServiceAccount
name: deployment-manager
namespace: rbac-demo
roleRef:
kind: Role
name: secret-manager
apiGroup: rbac.authorization.k8s.io
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
name: service-manager-binding
namespace: rbac-demo
labels:
app: rbac-demo
subjects:
- kind: ServiceAccount
name: deployment-manager
namespace: rbac-demo
roleRef:
kind: Role
name: service-manager
apiGroup: rbac.authorization.k8s.io
kubectl apply -f rolebindings.yaml
Configure service account tokens
Create long-lived tokens for service accounts that need persistent access.
apiVersion: v1
kind: Secret
metadata:
name: pod-reader-token
namespace: rbac-demo
annotations:
kubernetes.io/service-account.name: pod-reader
type: kubernetes.io/service-account-token
---
apiVersion: v1
kind: Secret
metadata:
name: deployment-manager-token
namespace: rbac-demo
annotations:
kubernetes.io/service-account.name: deployment-manager
type: kubernetes.io/service-account-token
kubectl apply -f serviceaccount-tokens.yaml
Create test pods with service accounts
Deploy pods that use the configured service accounts to test RBAC permissions.
apiVersion: v1
kind: Pod
metadata:
name: pod-reader-test
namespace: rbac-demo
labels:
app: rbac-demo
test: pod-reader
spec:
serviceAccountName: pod-reader
containers:
- name: kubectl
image: bitnami/kubectl:latest
command: ['sleep', '3600']
restartPolicy: Never
---
apiVersion: v1
kind: Pod
metadata:
name: deployment-manager-test
namespace: rbac-demo
labels:
app: rbac-demo
test: deployment-manager
spec:
serviceAccountName: deployment-manager
containers:
- name: kubectl
image: bitnami/kubectl:latest
command: ['sleep', '3600']
restartPolicy: Never
kubectl apply -f test-pods.yaml
Implementing advanced RBAC policies
Create resource-specific permissions
Configure roles with permissions for specific resources and resource names.
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
namespace: rbac-demo
name: specific-resource-manager
labels:
app: rbac-demo
rules:
- apiGroups: [""]
resources: ["secrets"]
resourceNames: ["app-secret", "db-secret"]
verbs: ["get", "update"]
- apiGroups: ["apps"]
resources: ["deployments"]
resourceNames: ["app-deployment"]
verbs: ["get", "update", "patch"]
- apiGroups: [""]
resources: ["configmaps"]
verbs: ["get", "list"]
resourceNames: ["app-config"]
kubectl apply -f resource-specific-role.yaml
Implement attribute-based access control
Create roles that use label selectors and field selectors for fine-grained access.
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
name: label-based-access
labels:
app: rbac-demo
rules:
- apiGroups: [""]
resources: ["pods"]
verbs: ["get", "list", "watch"]
- apiGroups: [""]
resources: ["pods/exec"]
verbs: ["create"]
- apiGroups: ["apps"]
resources: ["deployments"]
verbs: ["get", "list", "watch"]
resourceNames: []
- apiGroups: [""]
resources: ["events"]
verbs: ["get", "list"]
- apiGroups: ["metrics.k8s.io"]
resources: ["pods", "nodes"]
verbs: ["get", "list"]
kubectl apply -f attribute-based-role.yaml
Configure admission control integration
Create roles that work with admission controllers for policy enforcement. This example works well with OPA Gatekeeper configurations.
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
name: admission-controller-reviewer
labels:
app: rbac-demo
rules:
- apiGroups: ["admissionregistration.k8s.io"]
resources: ["validatingadmissionwebhooks", "mutatingadmissionwebhooks"]
verbs: ["get", "list"]
- apiGroups: [""]
resources: ["events"]
verbs: ["create", "patch"]
- apiGroups: ["authorization.k8s.io"]
resources: ["subjectaccessreviews"]
verbs: ["create"]
- apiGroups: ["authentication.k8s.io"]
resources: ["tokenreviews"]
verbs: ["create"]
kubectl apply -f admission-control-role.yaml
Verify your RBAC setup
Test service account permissions
Verify that each service account has the expected permissions and restrictions.
# Test pod reader permissions
kubectl auth can-i list pods --as=system:serviceaccount:rbac-demo:pod-reader
kubectl auth can-i create deployments --as=system:serviceaccount:rbac-demo:pod-reader
# Test deployment manager permissions
kubectl auth can-i create deployments --as=system:serviceaccount:rbac-demo:deployment-manager
kubectl auth can-i delete secrets --as=system:serviceaccount:rbac-demo:deployment-manager -n rbac-demo
# Test cluster admin permissions
kubectl auth can-i "*" "*" --as=system:serviceaccount:rbac-demo:cluster-admin-sa
Verify permissions from within pods
Test actual API access from the pods using the service accounts.
# Test from pod-reader pod
kubectl exec -n rbac-demo pod-reader-test -- kubectl get pods --all-namespaces
kubectl exec -n rbac-demo pod-reader-test -- kubectl get deployments
# Test from deployment-manager pod
kubectl exec -n rbac-demo deployment-manager-test -- kubectl get deployments
kubectl exec -n rbac-demo deployment-manager-test -- kubectl create deployment test-deploy --image=nginx
Audit RBAC configuration
Review the complete RBAC setup and identify potential security gaps.
# List all service accounts
kubectl get serviceaccounts -n rbac-demo -o wide
# List all role bindings
kubectl get rolebindings,clusterrolebindings -n rbac-demo -o wide
# Check role definitions
kubectl describe clusterrole pod-reader-role
kubectl describe role secret-manager -n rbac-demo
# Verify token secrets
kubectl get secrets -n rbac-demo | grep token
Security best practices
Security Warning: Never use the cluster-admin role for regular applications. Always follow the principle of least privilege when assigning permissions.
Implement these security practices for production RBAC deployments. Use specific resource names when possible instead of wildcard permissions. Regularly audit and rotate service account tokens. Enable audit logging to track RBAC decisions and access patterns.
Create separate service accounts for each application or service component. Group permissions logically and use descriptive names for roles and bindings. Document the purpose and scope of each service account for your team.
For comprehensive cluster security, combine RBAC with network policies and pod security standards to create defense-in-depth protection.
Common issues
SymptomCauseFix
Service account can't access resourcesMissing role bindingCreate appropriate RoleBinding or ClusterRoleBinding
"Forbidden" errors in pod logsInsufficient permissions in roleAdd required verbs and resources to the role definition
Service account token not foundToken secret not createdCreate Secret with kubernetes.io/service-account-token type
Cross-namespace access deniedUsing Role instead of ClusterRoleUse ClusterRole and ClusterRoleBinding for cross-namespace access
Application can't read own metadataMissing self-inspection permissionsAdd permissions for pods/self and configmaps in the same namespace
Service discovery not workingMissing endpoints and services permissionsAdd get/list verbs for services and endpoints resources
Next steps
Implement network policies for pod-to-pod security
Integrate HashiCorp Vault for advanced secrets management
Configure ingress controllers with automated SSL certificate management
Set up Pod Security Standards with admission controllers
Configure Calico CNI for advanced network microsegmentation
Running this in production?
Want this handled for you? Setting up RBAC once is straightforward. Keeping it patched, monitored, backed up and tuned across environments is the harder part. See how we run infrastructure like this for European SaaS and e-commerce teams.
---
### Implement Deno microservices architecture with service discovery and load balancing
URL: https://binadit.com/tutorials/implement-deno-microservices-architecture
Category: devops
Difficulty: advanced
Time: ~45 minutes
Author: Binadit Tech Team
> Build a production-ready Deno microservices architecture with Consul service discovery, HAProxy load balancing, and comprehensive monitoring using Prometheus. This tutorial covers container orchestration, health checks, and automated failover for scalable applications.
What this solves
Modern applications need to scale horizontally by breaking into smaller services, but managing multiple Deno microservices becomes complex without proper service discovery and load balancing. This tutorial builds a production-ready architecture where Deno services automatically register themselves with Consul, HAProxy distributes traffic based on health checks, and Prometheus monitors the entire stack.
Step-by-step installation
Update system packages
Start by updating your package manager to ensure you get the latest versions of all components.
sudo apt update && sudo apt upgrade -y
sudo apt install -y curl wget unzip software-properties-common
sudo dnf update -y
sudo dnf install -y curl wget unzip
Install Deno runtime
Install Deno using the official installation script, then verify the installation.
curl -fsSL https://deno.land/install.sh | sh
echo 'export DENO_INSTALL="$HOME/.deno"' >> ~/.bashrc
echo 'export PATH="$DENO_INSTALL/bin:$PATH"' >> ~/.bashrc
source ~/.bashrc
deno --version
Install Consul for service discovery
Download and install HashiCorp Consul for service registration and health checking.
CONSUL_VERSION="1.17.0"
wget https://releases.hashicorp.com/consul/${CONSUL_VERSION}/consul_${CONSUL_VERSION}_linux_amd64.zip
unzip consul_${CONSUL_VERSION}_linux_amd64.zip
sudo mv consul /usr/local/bin/
sudo chmod +x /usr/local/bin/consul
consul version
Configure Consul server
Create Consul configuration directory and setup the main configuration file.
sudo mkdir -p /etc/consul.d /opt/consul
sudo useradd --system --home /etc/consul.d --shell /bin/false consul
sudo chown -R consul:consul /etc/consul.d /opt/consul
datacenter = "dc1"
data_dir = "/opt/consul"
log_level = "INFO"
server = true
bootstrap_expect = 1
bind_addr = "0.0.0.0"
client_addr = "0.0.0.0"
ui_config {
enabled = true
}
connect {
enabled = true
}
ports {
grpc = 8502
}
acl = {
enabled = false
default_policy = "allow"
}
Create Consul systemd service
Setup Consul to run as a systemd service with automatic restarts.
[Unit]
Description=Consul
Requires=network-online.target
After=network-online.target
ConditionFileNotEmpty=/etc/consul.d/consul.hcl
[Service]
Type=notify
User=consul
Group=consul
ExecStart=/usr/local/bin/consul agent -config-dir=/etc/consul.d/
ExecReload=/bin/kill -HUP $MAINPID
KillMode=process
Restart=on-failure
LimitNOFILE=65536
[Install]
WantedBy=multi-user.target
sudo systemctl daemon-reload
sudo systemctl enable consul
sudo systemctl start consul
sudo systemctl status consul
Install HAProxy load balancer
Install HAProxy for distributing traffic across Deno microservices with health checks.
sudo apt install -y haproxy
sudo dnf install -y haproxy
Configure HAProxy with Consul integration
Setup HAProxy configuration with service discovery integration and health checks.
global
daemon
maxconn 4096
log stdout local0
stats socket /var/run/haproxy.sock mode 660 level admin
stats timeout 30s
defaults
mode http
timeout connect 5s
timeout client 30s
timeout server 30s
option httplog
option dontlognull
option redispatch
retries 3
frontend api_gateway
bind *:80
bind *:443 ssl crt /etc/ssl/certs/haproxy.pem
redirect scheme https if !{ ssl_fc }
# Route based on URL path
acl is_users_service path_beg /api/users
acl is_orders_service path_beg /api/orders
acl is_health path /health
use_backend users_service if is_users_service
use_backend orders_service if is_orders_service
use_backend health_check if is_health
default_backend api_default
backend users_service
balance roundrobin
option httpchk GET /health
http-check expect status 200
# Dynamic backend discovery via Consul
server-template users 3 _users._tcp.service.consul:8080 check resolvers consul
backend orders_service
balance roundrobin
option httpchk GET /health
http-check expect status 200
server-template orders 3 _orders._tcp.service.consul:8081 check resolvers consul
backend api_default
balance roundrobin
server default 127.0.0.1:8000 check
backend health_check
http-request return status 200 content-type text/plain string "HAProxy healthy"
resolvers consul
nameserver consul 127.0.0.1:8600
accepted_payload_size 8192
hold valid 5s
listen stats
bind *:8404
stats enable
stats uri /
stats refresh 5s
stats admin if TRUE
Install Prometheus for monitoring
Download and install Prometheus to monitor your microservices architecture.
PROMETHEUS_VERSION="2.48.0"
wget https://github.com/prometheus/prometheus/releases/download/v${PROMETHEUS_VERSION}/prometheus-${PROMETHEUS_VERSION}.linux-amd64.tar.gz
tar xvf prometheus-${PROMETHEUS_VERSION}.linux-amd64.tar.gz
sudo mv prometheus-${PROMETHEUS_VERSION}.linux-amd64/prometheus /usr/local/bin/
sudo mv prometheus-${PROMETHEUS_VERSION}.linux-amd64/promtool /usr/local/bin/
sudo mkdir -p /etc/prometheus /var/lib/prometheus
sudo useradd --system --home /var/lib/prometheus --shell /bin/false prometheus
sudo chown prometheus:prometheus /etc/prometheus /var/lib/prometheus
Configure Prometheus with service discovery
Setup Prometheus to automatically discover services registered in Consul.
global:
scrape_interval: 15s
evaluation_interval: 15s
rule_files:
- "/etc/prometheus/rules/*.yml"
alerting:
alertmanagers:
- static_configs:
- targets:
- localhost:9093
scrape_configs:
- job_name: 'prometheus'
static_configs:
- targets: ['localhost:9090']
- job_name: 'consul'
static_configs:
- targets: ['localhost:8500']
metrics_path: /v1/agent/metrics
params:
format: ['prometheus']
- job_name: 'haproxy'
static_configs:
- targets: ['localhost:8404']
metrics_path: /stats/prometheus
- job_name: 'consul-services'
consul_sd_configs:
- server: 'localhost:8500'
services: ['users', 'orders']
relabel_configs:
- source_labels: [__meta_consul_service]
target_label: job
- source_labels: [__meta_consul_node]
target_label: instance
- source_labels: [__meta_consul_service_address]
target_label: __address__
- source_labels: [__meta_consul_service_port]
target_label: __address__
regex: '(.*)'
replacement: '${1}:${__meta_consul_service_port}'
- job_name: 'deno-services'
consul_sd_configs:
- server: 'localhost:8500'
tags: ['deno', 'microservice']
relabel_configs:
- source_labels: [__meta_consul_service]
target_label: service
- source_labels: [__address__]
target_label: __address__
regex: '([^:]+):(\d+)'
replacement: '${1}:${2}'
metrics_path: '/metrics'
Create Prometheus systemd service
Setup Prometheus to run as a systemd service with proper permissions.
[Unit]
Description=Prometheus
Wants=network-online.target
After=network-online.target
[Service]
User=prometheus
Group=prometheus
Type=simple
ExecStart=/usr/local/bin/prometheus \
--config.file /etc/prometheus/prometheus.yml \
--storage.tsdb.path /var/lib/prometheus/ \
--web.console.templates=/etc/prometheus/consoles \
--web.console.libraries=/etc/prometheus/console_libraries \
--web.listen-address=0.0.0.0:9090 \
--web.enable-lifecycle
Restart=always
[Install]
WantedBy=multi-user.target
sudo systemctl daemon-reload
sudo systemctl enable prometheus
sudo systemctl start prometheus
Create Deno microservice template
Create a reusable template for Deno microservices with service registration and metrics.
mkdir -p ~/deno-microservices
cd ~/deno-microservices
import { serve } from "https://deno.land/std@0.208.0/http/server.ts";
export interface ServiceConfig {
name: string;
port: number;
version: string;
consulUrl?: string;
}
export class MicroService {
private config: ServiceConfig;
private routes: Map
Create users microservice
Build the first microservice for user management with Consul registration.
import { MicroService } from "./service-base.ts";
const service = new MicroService({
name: "users",
port: 8080,
version: "1.0.0",
});
// Mock user data
const users = new Map([
["1", { id: "1", name: "Alice Johnson", email: "alice@example.com" }],
["2", { id: "2", name: "Bob Smith", email: "bob@example.com" }],
["3", { id: "3", name: "Carol Wilson", email: "carol@example.com" }],
]);
// GET /api/users
service.addRoute("/api/users", async (req: Request) => {
if (req.method !== "GET") {
return new Response("Method Not Allowed", { status: 405 });
}
return new Response(JSON.stringify(Array.from(users.values())), {
status: 200,
headers: { "Content-Type": "application/json" },
});
});
// GET /api/users/:id
service.addRoute("/api/users/", async (req: Request) => {
if (req.method !== "GET") {
return new Response("Method Not Allowed", { status: 405 });
}
const url = new URL(req.url);
const id = url.pathname.split("/").pop();
if (!id || !users.has(id)) {
return new Response("User not found", { status: 404 });
}
return new Response(JSON.stringify(users.get(id)), {
status: 200,
headers: { "Content-Type": "application/json" },
});
});
if (import.meta.main) {
service.start();
}
Create orders microservice
Build the second microservice for order management with service discovery.
import { MicroService } from "./service-base.ts";
const service = new MicroService({
name: "orders",
port: 8081,
version: "1.0.0",
});
// Mock order data
const orders = new Map([
["1", { id: "1", userId: "1", items: ["laptop", "mouse"], total: 1299.99, status: "shipped" }],
["2", { id: "2", userId: "2", items: ["phone"], total: 899.99, status: "processing" }],
["3", { id: "3", userId: "1", items: ["keyboard"], total: 129.99, status: "delivered" }],
]);
// GET /api/orders
service.addRoute("/api/orders", async (req: Request) => {
if (req.method !== "GET") {
return new Response("Method Not Allowed", { status: 405 });
}
const url = new URL(req.url);
const userId = url.searchParams.get("userId");
let result = Array.from(orders.values());
if (userId) {
result = result.filter(order => order.userId === userId);
}
return new Response(JSON.stringify(result), {
status: 200,
headers: { "Content-Type": "application/json" },
});
});
// GET /api/orders/:id
service.addRoute("/api/orders/", async (req: Request) => {
if (req.method !== "GET") {
return new Response("Method Not Allowed", { status: 405 });
}
const url = new URL(req.url);
const id = url.pathname.split("/").pop();
if (!id || !orders.has(id)) {
return new Response("Order not found", { status: 404 });
}
return new Response(JSON.stringify(orders.get(id)), {
status: 200,
headers: { "Content-Type": "application/json" },
});
});
if (import.meta.main) {
service.start();
}
Create systemd services for Deno microservices
Setup systemd services to manage the Deno microservices lifecycle with automatic restarts.
[Unit]
Description=Deno Users Microservice
After=network.target consul.service
Requires=consul.service
[Service]
Type=simple
User=deno
Group=deno
WorkingDirectory=/home/deno/microservices
ExecStart=/home/deno/.deno/bin/deno run --allow-net --allow-env users-service.ts
Restart=always
RestartSec=5
Environment=SERVICE_ADDRESS=localhost
StandardOutput=journal
StandardError=journal
[Install]
WantedBy=multi-user.target
[Unit]
Description=Deno Orders Microservice
After=network.target consul.service
Requires=consul.service
[Service]
Type=simple
User=deno
Group=deno
WorkingDirectory=/home/deno/microservices
ExecStart=/home/deno/.deno/bin/deno run --allow-net --allow-env orders-service.ts
Restart=always
RestartSec=5
Environment=SERVICE_ADDRESS=localhost
StandardOutput=journal
StandardError=journal
[Install]
WantedBy=multi-user.target
Create dedicated user and setup permissions
Create a dedicated user for running Deno services securely with minimal permissions.
sudo useradd --system --home /home/deno --create-home --shell /bin/bash deno
sudo mkdir -p /home/deno/microservices
sudo cp ~/deno-microservices/* /home/deno/microservices/
sudo chown -R deno:deno /home/deno
sudo chmod 755 /home/deno/microservices
sudo chmod 644 /home/deno/microservices/*.ts
Never use chmod 777. It gives every user on the system full access to your files. Instead, fix ownership with chown and use minimal permissions like 755 for directories and 644 for files.
Start all services
Enable and start all the services in the correct order with dependency checking.
# Start Consul first
sudo systemctl status consul
# Start HAProxy
sudo
---
### Implement Kubernetes security scanning with Falco and OPA Gatekeeper for runtime protection
URL: https://binadit.com/tutorials/implement-kubernetes-security-scanning-with-falco-and-opa
Category: security
Difficulty: advanced
Time: ~45 minutes
Author: Binadit Tech Team
> Set up comprehensive Kubernetes security with Falco for runtime threat detection and OPA Gatekeeper for admission control policy enforcement. This tutorial covers installation, configuration, and custom security policies.
What this solves
Kubernetes clusters need layered security to detect runtime threats and enforce admission policies. Falco monitors system calls and container behavior for suspicious activity, while OPA Gatekeeper validates resources against security policies before deployment. Together they provide comprehensive protection against malicious workloads, privilege escalation, and policy violations.
Step-by-step installation
Install Helm for package management
Both Falco and Gatekeeper use Helm charts for installation. Install Helm first to manage the deployments.
curl https://get.helm.sh/helm-v3.12.0-linux-amd64.tar.gz -o helm.tar.gz
tar -zxvf helm.tar.gz
sudo mv linux-amd64/helm /usr/local/bin/helm
helm version
Add Helm repositories
Add the official repositories for Falco and Gatekeeper charts.
helm repo add falcosecurity https://falcosecurity.github.io/charts
helm repo add gatekeeper https://open-policy-agent.github.io/gatekeeper/charts
helm repo update
Install Falco for runtime security monitoring
Deploy Falco as a DaemonSet to monitor all nodes for suspicious activity. The default configuration includes rules for common threats.
kubectl create namespace falco-system
helm install falco falcosecurity/falco \
--namespace falco-system \
--set driver.kind=ebpf \
--set falco.grpc.enabled=true \
--set falco.grpcOutput.enabled=true
Install OPA Gatekeeper for admission control
Deploy Gatekeeper to validate all resource requests against defined policies before they reach the API server.
kubectl create namespace gatekeeper-system
helm install gatekeeper gatekeeper/gatekeeper \
--namespace gatekeeper-system \
--set replicas=3 \
--set auditInterval=60
Verify installations
Check that both systems are running correctly across all nodes.
kubectl get pods -n falco-system
kubectl get pods -n gatekeeper-system
kubectl get validatingadmissionwebhooks
Configure security policies and rules
Create custom Falco rules
Add detection rules for your specific environment. This example detects unauthorized network connections.
apiVersion: v1
kind: ConfigMap
metadata:
name: falco-custom-rules
namespace: falco-system
data:
custom_rules.yaml: |
- rule: Unexpected outbound connection
desc: Detect unexpected outbound network connections
condition: >
outbound and not fd.typechar=4 and not fd.typechar=6 and
not proc.name in (curl, wget, apt, yum, dnf) and
not container.image.repository in (docker.io/library/alpine, gcr.io/distroless)
output: >
Unexpected outbound connection (command=%proc.cmdline connection=%fd.name
user=%user.name container=%container.name image=%container.image.repository)
priority: WARNING
tags: [network, outbound]
- rule: Privileged container spawned
desc: Detect containers running with privileged access
condition: >
spawned_process and container and proc.vpid=1 and
container.privileged=true
output: >
Privileged container spawned (command=%proc.cmdline user=%user.name
container=%container.name image=%container.image.repository)
priority: CRITICAL
tags: [container, privilege]
kubectl apply -f falco-custom-rules.yaml
Create Gatekeeper constraint templates
Define reusable policy templates that can be applied to different resource types.
apiVersion: templates.gatekeeper.sh/v1beta1
kind: ConstraintTemplate
metadata:
name: requiresecuritycontext
spec:
crd:
spec:
names:
kind: RequireSecurityContext
validation:
properties:
runAsNonRoot:
type: boolean
readOnlyRootFilesystem:
type: boolean
allowPrivilegeEscalation:
type: boolean
targets:
- target: admission.k8s.gatekeeper.sh
rego: |
package requiresecuritycontext
violation[{"msg": msg}] {
container := input.review.object.spec.containers[_]
not container.securityContext.runAsNonRoot
msg := "Container must run as non-root user"
}
violation[{"msg": msg}] {
container := input.review.object.spec.containers[_]
not container.securityContext.readOnlyRootFilesystem
msg := "Container must use read-only root filesystem"
}
violation[{"msg": msg}] {
container := input.review.object.spec.containers[_]
container.securityContext.allowPrivilegeEscalation != false
msg := "Container must not allow privilege escalation"
}
kubectl apply -f require-security-context-template.yaml
Apply security context constraints
Use the template to enforce security contexts on all pods in production namespaces.
apiVersion: constraints.gatekeeper.sh/v1beta1
kind: RequireSecurityContext
metadata:
name: must-have-security-context
spec:
match:
kinds:
- apiGroups: [""]
kinds: ["Pod"]
namespaces: ["production", "staging"]
parameters:
runAsNonRoot: true
readOnlyRootFilesystem: true
allowPrivilegeEscalation: false
kubectl apply -f security-context-constraint.yaml
Create resource limit template
Prevent resource exhaustion attacks by requiring CPU and memory limits on all containers.
apiVersion: templates.gatekeeper.sh/v1beta1
kind: ConstraintTemplate
metadata:
name: requireresources
spec:
crd:
spec:
names:
kind: RequireResources
validation:
properties:
limits:
type: array
items:
type: string
requests:
type: array
items:
type: string
targets:
- target: admission.k8s.gatekeeper.sh
rego: |
package requireresources
violation[{"msg": msg}] {
container := input.review.object.spec.containers[_]
required := input.parameters.limits
provided := container.resources.limits
missing := required[_]
not provided[missing]
msg := sprintf("Container missing required resource limit: %v", [missing])
}
violation[{"msg": msg}] {
container := input.review.object.spec.containers[_]
required := input.parameters.requests
provided := container.resources.requests
missing := required[_]
not provided[missing]
msg := sprintf("Container missing required resource request: %v", [missing])
}
kubectl apply -f require-resources-template.yaml
Apply resource limit constraints
Enforce CPU and memory limits on all production workloads.
apiVersion: constraints.gatekeeper.sh/v1beta1
kind: RequireResources
metadata:
name: must-have-resource-limits
spec:
match:
kinds:
- apiGroups: [""]
kinds: ["Pod"]
namespaces: ["production", "staging"]
parameters:
limits: ["cpu", "memory"]
requests: ["cpu", "memory"]
kubectl apply -f resource-limits-constraint.yaml
Set up monitoring and alerting
Configure Falco alerts
Set up webhook notifications to send security alerts to your monitoring system.
apiVersion: v1
kind: ConfigMap
metadata:
name: falco-config-override
namespace: falco-system
data:
falco.yaml: |
rules_file:
- /etc/falco/falco_rules.yaml
- /etc/falco/falco_rules.local.yaml
- /etc/falco/k8s_audit_rules.yaml
- /etc/falco/rules.d
json_output: true
json_include_output_property: true
http_output:
enabled: true
url: "http://webhook-service.monitoring.svc.cluster.local:8080/falco"
user_agent: "falco/0.35.0"
priority: WARNING
syscall_event_drops:
actions:
- log
- alert
rate: 0.1
max_burst: 1000
kubectl apply -f falco-config-override.yaml
kubectl rollout restart daemonset/falco -n falco-system
Create alerting webhook service
Deploy a simple webhook receiver to process Falco alerts and forward them to your notification system.
apiVersion: apps/v1
kind: Deployment
metadata:
name: security-webhook
namespace: monitoring
spec:
replicas: 2
selector:
matchLabels:
app: security-webhook
template:
metadata:
labels:
app: security-webhook
spec:
containers:
- name: webhook
image: falcosecurity/falcosidekick:2.25.0
ports:
- containerPort: 2801
env:
- name: WEBHOOK_URL
value: "https://hooks.slack.com/services/YOUR/SLACK/WEBHOOK"
- name: DEBUG
value: "true"
resources:
requests:
cpu: 100m
memory: 128Mi
limits:
cpu: 200m
memory: 256Mi
---
apiVersion: v1
kind: Service
metadata:
name: security-webhook
namespace: monitoring
spec:
selector:
app: security-webhook
ports:
- port: 8080
targetPort: 2801
type: ClusterIP
kubectl create namespace monitoring
kubectl apply -f webhook-deployment.yaml
Monitor Gatekeeper violations
Create a monitoring dashboard to track policy violations and system health.
apiVersion: v1
kind: ServiceMonitor
metadata:
name: gatekeeper-metrics
namespace: gatekeeper-system
spec:
selector:
matchLabels:
app: gatekeeper
endpoints:
- port: metrics
interval: 30s
path: /metrics
---
apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
name: gatekeeper-alerts
namespace: gatekeeper-system
spec:
groups:
- name: gatekeeper.rules
rules:
- alert: GatekeeperViolations
expr: increase(gatekeeper_violations_total[5m]) > 10
for: 2m
labels:
severity: warning
annotations:
summary: "High number of Gatekeeper policy violations"
description: "{{ $value }} policy violations detected in the last 5 minutes"
- alert: GatekeeperDown
expr: up{job="gatekeeper"} == 0
for: 5m
labels:
severity: critical
annotations:
summary: "Gatekeeper is down"
description: "Gatekeeper admission controller is not responding"
kubectl apply -f gatekeeper-monitoring.yaml
Test security policies
Test Gatekeeper admission control
Verify that policies block non-compliant workloads by trying to deploy a pod without required security context.
apiVersion: v1
kind: Pod
metadata:
name: insecure-test-pod
namespace: production
spec:
containers:
- name: test
image: nginx:latest
ports:
- containerPort: 80
kubectl apply -f test-insecure-pod.yaml
This should fail with a message about missing security context and resource limits.
Test compliant pod deployment
Deploy a pod that meets all security requirements.
apiVersion: v1
kind: Pod
metadata:
name: secure-test-pod
namespace: production
spec:
containers:
- name: test
image: nginx:latest
ports:
- containerPort: 80
securityContext:
runAsNonRoot: true
runAsUser: 1000
readOnlyRootFilesystem: true
allowPrivilegeEscalation: false
resources:
requests:
cpu: 100m
memory: 128Mi
limits:
cpu: 200m
memory: 256Mi
volumeMounts:
- name: tmp
mountPath: /tmp
- name: var-cache
mountPath: /var/cache/nginx
volumes:
- name: tmp
emptyDir: {}
- name: var-cache
emptyDir: {}
kubectl apply -f test-secure-pod.yaml
Generate test security events
Trigger Falco rules to verify runtime monitoring works correctly.
# Create a test pod that will trigger Falco rules
kubectl run falco-test --image=alpine --rm -it --restart=Never -- sh
# Inside the container, run commands that should trigger alerts:
ps aux
netstat -an
find /etc -name "*passwd*"
wget google.com
exit
Verify your setup
# Check Falco is detecting events
kubectl logs -n falco-system -l app.kubernetes.io/name=falco --tail=50
# Verify Gatekeeper policies are active
kubectl get constraints
kubectl get constrainttemplates
# Check for policy violations
kubectl describe RequireSecurityContext must-have-security-context
kubectl describe RequireResources must-have-resource-limits
# Test webhook connectivity
kubectl logs -n monitoring -l app=security-webhook
Important: Falco generates many events in a typical cluster. Configure appropriate filters and alert thresholds to avoid notification fatigue while ensuring critical security events are not missed.
Common issues
SymptomCauseFix
Falco pods failing to starteBPF driver not supportedUse --set driver.kind=module instead of ebpf
Gatekeeper blocking all podsConstraint too restrictiveAdd namespace exclusions or adjust match criteria
High CPU usage from FalcoToo many syscall eventsTune rules and add filters for noisy processes
Webhook not receiving alertsNetwork policy blocking trafficAllow egress from Falco namespace to webhook service
Policies not applyingGatekeeper not readyWait for all admission webhooks to be ready
Advanced configuration
Performance tuning: In high-traffic clusters, consider using Falco's gRPC output with a separate collector service to reduce resource usage and improve alert processing efficiency.
For production deployments, also consider integrating with Kubernetes RBAC for comprehensive access control and OpenTelemetry monitoring for complete observability across your security stack.
Next steps
Implement Kubernetes RBAC with service accounts for access control
Set up Kubernetes container image security scanning with Trivy
Implement Kubernetes network policies for pod-to-pod security
Configure Kubernetes secrets management with Vault integration
Set up Kubernetes monitoring with Prometheus Operator
Running this in production?
Need this managed? Running this at scale adds a second layer of work: capacity planning, failover drills, cost control, and on-call. See how we run infrastructure like this for European teams.
---
### Configure OpenTelemetry custom metrics for application monitoring with Prometheus and Grafana
URL: https://binadit.com/tutorials/configure-opentelemetry-custom-metrics-for-application-monitoring
Category: monitoring
Difficulty: intermediate
Time: ~45 minutes
Author: Binadit Tech Team
> Set up OpenTelemetry SDK to collect custom application metrics, export them to Prometheus for storage, and visualize performance data in Grafana dashboards with automated alerting.
What this solves
OpenTelemetry custom metrics give you detailed insights into your application's performance beyond basic system metrics. You can track business-specific metrics like user sign-ups, order completion rates, or API response times. This tutorial shows you how to instrument applications with OpenTelemetry, send metrics to Prometheus, and build Grafana dashboards for monitoring and alerting.
Step-by-step installation
Install OpenTelemetry Collector
The OpenTelemetry Collector receives metrics from your applications and forwards them to Prometheus. Download and install the latest collector binary.
wget https://github.com/open-telemetry/opentelemetry-collector-releases/releases/download/v0.91.0/otelcol_0.91.0_linux_amd64.tar.gz
tar -xzf otelcol_0.91.0_linux_amd64.tar.gz
sudo mv otelcol /usr/local/bin/
sudo chmod +x /usr/local/bin/otelcol
Create collector configuration
Configure the collector to receive OTLP metrics and export them to Prometheus format. This config enables metric collection on port 4318 and serves Prometheus metrics on port 8889.
receivers:
otlp:
protocols:
grpc:
endpoint: 0.0.0.0:4317
http:
endpoint: 0.0.0.0:4318
processors:
batch:
timeout: 1s
send_batch_size: 1024
memory_limiter:
limit_mib: 512
exporters:
prometheus:
endpoint: "0.0.0.0:8889"
namespace: "app"
const_labels:
environment: "production"
service:
pipelines:
metrics:
receivers: [otlp]
processors: [memory_limiter, batch]
exporters: [prometheus]
telemetry:
logs:
level: info
Create systemd service for collector
Set up the collector as a systemd service for automatic startup and management.
[Unit]
Description=OpenTelemetry Collector
After=network.target
[Service]
Type=simple
User=nobody
Group=nogroup
ExecStart=/usr/local/bin/otelcol --config=/etc/otelcol-config.yaml
Restart=on-failure
RestartSec=5
StandardOutput=journal
StandardError=journal
[Install]
WantedBy=multi-user.target
Start OpenTelemetry Collector
Enable and start the collector service to begin accepting metrics from your applications.
sudo systemctl daemon-reload
sudo systemctl enable --now otelcol
sudo systemctl status otelcol
Install Prometheus
Install Prometheus to scrape metrics from the OpenTelemetry Collector and store them for querying.
sudo apt update
sudo apt install -y prometheus
sudo dnf install -y epel-release
sudo dnf install -y golang-github-prometheus
Configure Prometheus to scrape OpenTelemetry metrics
Add the OpenTelemetry Collector as a scrape target in Prometheus configuration. This tells Prometheus to collect metrics from the collector's Prometheus endpoint.
global:
scrape_interval: 15s
evaluation_interval: 15s
rule_files:
- "/etc/prometheus/rules/*.yml"
alerting:
alertmanagers:
- static_configs:
- targets:
- localhost:9093
scrape_configs:
- job_name: 'prometheus'
static_configs:
- targets: ['localhost:9090']
- job_name: 'otel-collector'
static_configs:
- targets: ['localhost:8889']
scrape_interval: 10s
metrics_path: /metrics
- job_name: 'node-exporter'
static_configs:
- targets: ['localhost:9100']
Create Prometheus alerting rules
Set up alerting rules for custom metrics to notify you when application performance degrades.
sudo mkdir -p /etc/prometheus/rules
groups:
- name: application_metrics
rules:
- alert: HighErrorRate
expr: rate(app_http_requests_total{status=~"5.."}[5m]) > 0.1
for: 2m
labels:
severity: warning
annotations:
summary: "High error rate detected"
description: "Error rate is {{ $value }} errors per second"
- alert: SlowResponseTime
expr: histogram_quantile(0.95, rate(app_http_request_duration_seconds_bucket[5m])) > 1.0
for: 5m
labels:
severity: critical
annotations:
summary: "Slow response times detected"
description: "95th percentile response time is {{ $value }} seconds"
- alert: LowThroughput
expr: rate(app_http_requests_total[5m]) < 1.0
for: 10m
labels:
severity: warning
annotations:
summary: "Low request throughput"
description: "Request rate is {{ $value }} requests per second"
Start Prometheus
Enable and start Prometheus to begin collecting metrics from the OpenTelemetry Collector.
sudo systemctl enable --now prometheus
sudo systemctl status prometheus
Install Grafana
Install Grafana to create dashboards and visualizations for your OpenTelemetry metrics.
sudo apt install -y software-properties-common
sudo add-apt-repository "deb https://packages.grafana.com/oss/deb stable main"
wget -q -O - https://packages.grafana.com/gpg.key | sudo apt-key add -
sudo apt update
sudo apt install -y grafana
sudo tee /etc/yum.repos.d/grafana.repo << 'EOF'
[grafana]
name=grafana
baseurl=https://packages.grafana.com/oss/rpm
repo_gpgcheck=1
enabled=1
gpgcheck=1
gpgkey=https://packages.grafana.com/gpg.key
EOF
sudo dnf install -y grafana
Configure Grafana data source
Add Prometheus as a data source in Grafana to query your OpenTelemetry metrics.
sudo systemctl enable --now grafana-server
sudo systemctl status grafana-server
Access Grafana at http://your-server:3000 with username admin and password admin. Navigate to Configuration > Data Sources and add Prometheus with URL http://localhost:9090.
Install OpenTelemetry SDK in your application
Add OpenTelemetry instrumentation to your application. This example shows Python implementation with custom metrics.
pip install opentelemetry-api opentelemetry-sdk opentelemetry-exporter-otlp
from opentelemetry import metrics
from opentelemetry.sdk.metrics import MeterProvider
from opentelemetry.sdk.metrics.export import PeriodicExportingMetricReader
from opentelemetry.exporter.otlp.proto.http.metric_exporter import OTLPMetricExporter
import time
import random
# Configure OpenTelemetry
metric_exporter = OTLPMetricExporter(
endpoint="http://localhost:4318/v1/metrics",
headers={}
)
metric_reader = PeriodicExportingMetricReader(
exporter=metric_exporter,
export_interval_millis=5000
)
metrics.set_meter_provider(MeterProvider(metric_readers=[metric_reader]))
meter = metrics.get_meter("app_metrics", "1.0.0")
# Create custom metrics
request_counter = meter.create_counter(
name="http_requests_total",
description="Total number of HTTP requests",
unit="1"
)
response_time_histogram = meter.create_histogram(
name="http_request_duration_seconds",
description="HTTP request duration in seconds",
unit="s"
)
active_connections_gauge = meter.create_up_down_counter(
name="active_connections",
description="Number of active connections",
unit="1"
)
# Example usage
def handle_request(endpoint, status_code):
start_time = time.time()
# Simulate request processing
processing_time = random.uniform(0.1, 2.0)
time.sleep(processing_time)
# Record metrics
request_counter.add(1, {"endpoint": endpoint, "status": str(status_code)})
response_time_histogram.record(processing_time, {"endpoint": endpoint})
return f"Processed {endpoint} in {processing_time:.2f}s"
# Simulate application traffic
if __name__ == "__main__":
endpoints = ["/api/users", "/api/orders", "/api/products"]
for i in range(100):
endpoint = random.choice(endpoints)
status = random.choices([200, 404, 500], weights=[85, 10, 5])[0]
active_connections_gauge.add(1)
result = handle_request(endpoint, status)
active_connections_gauge.add(-1)
print(f"Request {i+1}: {result}")
time.sleep(0.1)
Create Grafana dashboard
Import a custom dashboard configuration to visualize your OpenTelemetry metrics with panels for request rates, response times, and error rates.
{
"dashboard": {
"id": null,
"title": "OpenTelemetry Application Metrics",
"tags": ["opentelemetry", "monitoring"],
"timezone": "browser",
"panels": [
{
"id": 1,
"title": "Request Rate",
"type": "stat",
"targets": [
{
"expr": "rate(app_http_requests_total[5m])",
"refId": "A"
}
],
"fieldConfig": {
"defaults": {
"unit": "reqps",
"min": 0
}
},
"gridPos": {"h": 8, "w": 6, "x": 0, "y": 0}
},
{
"id": 2,
"title": "Error Rate",
"type": "stat",
"targets": [
{
"expr": "rate(app_http_requests_total{status=~\"5..\"}[5m]) / rate(app_http_requests_total[5m])",
"refId": "A"
}
],
"fieldConfig": {
"defaults": {
"unit": "percentunit",
"min": 0,
"max": 1
}
},
"gridPos": {"h": 8, "w": 6, "x": 6, "y": 0}
},
{
"id": 3,
"title": "Response Time (95th percentile)",
"type": "stat",
"targets": [
{
"expr": "histogram_quantile(0.95, rate(app_http_request_duration_seconds_bucket[5m]))",
"refId": "A"
}
],
"fieldConfig": {
"defaults": {
"unit": "s",
"min": 0
}
},
"gridPos": {"h": 8, "w": 6, "x": 12, "y": 0}
},
{
"id": 4,
"title": "Active Connections",
"type": "stat",
"targets": [
{
"expr": "app_active_connections",
"refId": "A"
}
],
"fieldConfig": {
"defaults": {
"unit": "short",
"min": 0
}
},
"gridPos": {"h": 8, "w": 6, "x": 18, "y": 0}
}
],
"time": {
"from": "now-1h",
"to": "now"
},
"refresh": "10s"
}
}
Configure firewall rules
Open the necessary ports for OpenTelemetry Collector, Prometheus, and Grafana to communicate properly.
sudo ufw allow 3000/tcp
sudo ufw allow 9090/tcp
sudo ufw allow 4317/tcp
sudo ufw allow 4318/tcp
sudo ufw allow 8889/tcp
sudo firewall-cmd --permanent --add-port=3000/tcp
sudo firewall-cmd --permanent --add-port=9090/tcp
sudo firewall-cmd --permanent --add-port=4317/tcp
sudo firewall-cmd --permanent --add-port=4318/tcp
sudo firewall-cmd --permanent --add-port=8889/tcp
sudo firewall-cmd --reload
Configure custom metrics collection
Add business metrics to your application
Extend your application with business-specific metrics like user signups, revenue, or feature usage. These metrics provide insights into application performance from a business perspective.
# Additional business metrics
user_signups = meter.create_counter(
name="user_signups_total",
description="Total number of user signups",
unit="1"
)
order_value_histogram = meter.create_histogram(
name="order_value_dollars",
description="Order value in dollars",
unit="USD"
)
feature_usage = meter.create_counter(
name="feature_usage_total",
description="Feature usage by type",
unit="1"
)
# Example business event tracking
def track_user_signup(user_type, source):
user_signups.add(1, {
"user_type": user_type,
"source": source
})
def track_order(order_value, product_category):
order_value_histogram.record(order_value, {
"category": product_category
})
def track_feature_use(feature_name, user_tier):
feature_usage.add(1, {
"feature": feature_name,
"tier": user_tier
})
Set up metric sampling and filtering
Configure the collector to sample high-volume metrics and filter irrelevant data to reduce storage costs and improve query performance.
receivers:
otlp:
protocols:
grpc:
endpoint: 0.0.0.0:4317
http:
endpoint: 0.0.0.0:4318
processors:
batch:
timeout: 1s
send_batch_size: 1024
memory_limiter:
limit_mib: 512
filter/drop_debug:
metrics:
exclude:
match_type: regexp
metric_names:
- ".*_debug.*"
- ".*_test.*"
probabilistic_sampler:
sampling_percentage: 10
exporters:
prometheus:
endpoint: "0.0.0.0:8889"
namespace: "app"
const_labels:
environment: "production"
version: "1.0.0"
metric_expiration: 180s
enable_open_metrics: true
service:
pipelines:
metrics:
receivers: [otlp]
processors: [memory_limiter, filter/drop_debug, batch]
exporters: [prometheus]
telemetry:
logs:
level: info
metrics:
address: 0.0.0.0:8888
Set up Grafana dashboards and alerting
Create alerting rules in Grafana
Set up Grafana alerts that trigger notifications when metrics exceed thresholds. This provides proactive monitoring for your application performance.
curl -X POST http://admin:admin@localhost:3000/api/alert-rules \
-H "Content-Type: application/json" \
-d '{
"title": "High Error Rate Alert",
"condition": "B",
"data": [
{
"refId": "A",
"queryType": "",
"relativeTimeRange": {
"from": 300,
"to": 0
},
"model": {
"expr": "rate(app_http_requests_total{status=~\"5..\"}[5m])",
"refId": "A"
}
},
{
"refId": "B",
"queryType": "",
"model": {
"conditions": [
{
"evaluator": {
"params": [0.1],
"type": "gt"
},
"operator": {
"type": "and"
},
"query": {
"params": ["A"]
},
"reducer": {
"params": [],
"type": "avg"
},
"type": "query"
}
],
"refId": "B"
}
}
],
"intervalSeconds": 60,
"noDataState": "NoData",
"execErrState": "Alerting",
"for": "2m"
}'
Configure notification channels
Set up Slack or email notifications for alerts. This ensures your team gets notified when application issues occur.
curl -X POST http://admin:admin@localhost:3000/api/alert-notifications \
-H "Content-Type: application/json" \
-d '{
"name": "slack-alerts",
"type": "slack",
"settings": {
"url": "https://hooks.slack.com/services/YOUR/WEBHOOK/URL",
"username": "Grafana",
"channel": "#alerts",
"title": "Application Alert",
"text": "{{ range .Alerts }}{{ .Annotations.summary }}\n{{ .Annotations.description }}{{ end }}"
}
}'
Import pre-built dashboard
Load a comprehensive dashboard template that includes panels for all common OpenTelemetry metrics and alerts.
curl -X POST http://admin:admin@localhost:3000/api/dashboards/db \
-H "Content-Type: application/json" \
-d '@otel_dashboard.json'
Verify your setup
# Check OpenTelemetry Collector status
sudo systemctl status otelcol
# Verify collector is receiving metrics
curl http://localhost:8889/metrics | grep app_
# Check Prometheus targets
curl http://localhost:9090/api/v1/targets | jq '.data.activeTargets[] | select(.job=="otel-collector")'
# Test metric ingestion
python3 app_metrics.py
# Query metrics in Prometheus
curl -G http://localhost:9090/api/v1/query --data-urlencode 'query=app_http_requests_total'
# Check Grafana data source
curl http://admin:admin@localhost:3000/api/datasources
Common issues
<
SymptomCauseFix
---
### Configure Jaeger with Elasticsearch backend security and encryption
URL: https://binadit.com/tutorials/configure-jaeger-elasticsearch-backend-security
Category: monitoring
Difficulty: advanced
Time: ~45 minutes
Author: Binadit Tech Team
> Set up secure communication between Jaeger and Elasticsearch using TLS encryption, authentication, and production-grade security hardening for distributed tracing infrastructure.
What this solves
Jaeger with Elasticsearch backend requires proper security configuration for production environments. This tutorial configures TLS encryption between Jaeger components and Elasticsearch, sets up authentication mechanisms, and implements security hardening measures. You'll establish secure communication channels, configure X-Pack security features, and implement role-based access control for your distributed tracing infrastructure.
Prerequisites
You need a running Elasticsearch cluster and basic familiarity with Jaeger components. This builds on basic Jaeger installation and requires understanding of TLS certificate management. The configuration assumes you have administrative access to both Jaeger and Elasticsearch instances.
Step-by-step configuration
Install required packages
Install Elasticsearch, Jaeger components, and certificate management tools needed for secure configuration.
sudo apt update
sudo apt install -y elasticsearch jaeger-collector jaeger-query jaeger-agent openssl curl
sudo dnf update -y
sudo dnf install -y elasticsearch jaeger-collector jaeger-query jaeger-agent openssl curl
Configure Elasticsearch X-Pack security
Enable X-Pack security features in Elasticsearch to provide authentication and authorization capabilities for Jaeger access.
cluster.name: jaeger-cluster
node.name: jaeger-node-1
network.host: 0.0.0.0
http.port: 9200
transport.port: 9300
# Enable X-Pack security
xpack.security.enabled: true
xpack.security.enrollment.enabled: true
# TLS configuration
xpack.security.http.ssl.enabled: true
xpack.security.http.ssl.keystore.path: certs/http.p12
xpack.security.transport.ssl.enabled: true
xpack.security.transport.ssl.verification_mode: certificate
xpack.security.transport.ssl.client_authentication: required
xpack.security.transport.ssl.keystore.path: certs/transport.p12
xpack.security.transport.ssl.truststore.path: certs/transport.p12
# Audit logging for security events
xpack.security.audit.enabled: true
Generate TLS certificates
Create certificate authority and TLS certificates for secure communication between Jaeger and Elasticsearch components.
sudo mkdir -p /etc/elasticsearch/certs
sudo chown elasticsearch:elasticsearch /etc/elasticsearch/certs
sudo chmod 750 /etc/elasticsearch/certs
# Generate CA and certificates
sudo /usr/share/elasticsearch/bin/elasticsearch-certutil ca --out /etc/elasticsearch/certs/elastic-ca.p12 --pass ""
sudo /usr/share/elasticsearch/bin/elasticsearch-certutil cert --ca /etc/elasticsearch/certs/elastic-ca.p12 --ca-pass "" --out /etc/elasticsearch/certs/http.p12 --pass ""
sudo /usr/share/elasticsearch/bin/elasticsearch-certutil cert --ca /etc/elasticsearch/certs/elastic-ca.p12 --ca-pass "" --out /etc/elasticsearch/certs/transport.p12 --pass ""
# Set appropriate permissions
sudo chown elasticsearch:elasticsearch /etc/elasticsearch/certs/*
sudo chmod 660 /etc/elasticsearch/certs/*
Create Jaeger service account
Set up dedicated Elasticsearch user account for Jaeger with minimal required permissions for tracing data access.
sudo systemctl start elasticsearch
sudo systemctl enable elasticsearch
# Wait for Elasticsearch to start
sleep 30
# Reset built-in user passwords
sudo /usr/share/elasticsearch/bin/elasticsearch-setup-passwords auto > /tmp/es-passwords.txt
# Extract elastic password for admin operations
ELASTIC_PASSWORD=$(grep "PASSWORD elastic" /tmp/es-passwords.txt | awk '{print $4}')
echo "Elastic password: $ELASTIC_PASSWORD"
Create Jaeger user and roles
Configure role-based access control with custom roles for Jaeger tracing operations and read-only access patterns.
# Create jaeger_writer role for collector
curl -k -u elastic:$ELASTIC_PASSWORD -X POST "https://localhost:9200/_security/role/jaeger_writer" -H "Content-Type: application/json" -d '
{
"indices": [
{
"names": ["jaeger-*"],
"privileges": ["create", "index", "write", "delete", "manage"]
}
]
}'
# Create jaeger_reader role for query service
curl -k -u elastic:$ELASTIC_PASSWORD -X POST "https://localhost:9200/_security/role/jaeger_reader" -H "Content-Type: application/json" -d '
{
"indices": [
{
"names": ["jaeger-*"],
"privileges": ["read"]
}
]
}'
# Create jaeger_admin role for index management
curl -k -u elastic:$ELASTIC_PASSWORD -X POST "https://localhost:9200/_security/role/jaeger_admin" -H "Content-Type: application/json" -d '
{
"indices": [
{
"names": ["jaeger-*"],
"privileges": ["all"]
}
]
}'
Create Jaeger service users
Set up individual user accounts for Jaeger collector, query service, and administrative operations with appropriate role assignments.
# Create user for Jaeger collector
curl -k -u elastic:$ELASTIC_PASSWORD -X POST "https://localhost:9200/_security/user/jaeger_collector" -H "Content-Type: application/json" -d '
{
"password": "JaegerCollector2024!",
"roles": ["jaeger_writer"],
"full_name": "Jaeger Collector Service"
}'
# Create user for Jaeger query
curl -k -u elastic:$ELASTIC_PASSWORD -X POST "https://localhost:9200/_security/user/jaeger_query" -H "Content-Type: application/json" -d '
{
"password": "JaegerQuery2024!",
"roles": ["jaeger_reader"],
"full_name": "Jaeger Query Service"
}'
# Create user for Jaeger admin operations
curl -k -u elastic:$ELASTIC_PASSWORD -X POST "https://localhost:9200/_security/user/jaeger_admin" -H "Content-Type: application/json" -d '
{
"password": "JaegerAdmin2024!",
"roles": ["jaeger_admin"],
"full_name": "Jaeger Administrator"
}'
Extract certificates for Jaeger
Convert Elasticsearch certificates to PEM format for use by Jaeger components and configure certificate access permissions.
sudo mkdir -p /etc/jaeger/certs
# Extract CA certificate
sudo openssl pkcs12 -in /etc/elasticsearch/certs/elastic-ca.p12 -cacerts -nokeys -out /etc/jaeger/certs/ca.crt -passin pass:""
# Extract client certificate and key
sudo openssl pkcs12 -in /etc/elasticsearch/certs/http.p12 -clcerts -nokeys -out /etc/jaeger/certs/client.crt -passin pass:""
sudo openssl pkcs12 -in /etc/elasticsearch/certs/http.p12 -nocerts -out /etc/jaeger/certs/client.key -passin pass:"" -passout pass:""
# Set appropriate permissions
sudo chown -R jaeger:jaeger /etc/jaeger/certs
sudo chmod 750 /etc/jaeger/certs
sudo chmod 644 /etc/jaeger/certs/ca.crt
sudo chmod 644 /etc/jaeger/certs/client.crt
sudo chmod 600 /etc/jaeger/certs/client.key
Configure Jaeger collector with security
Set up Jaeger collector with TLS encryption, authentication credentials, and secure connection parameters for Elasticsearch backend.
es:
server-urls: https://localhost:9200
username: jaeger_collector
password: JaegerCollector2024!
tls:
enabled: true
ca: /etc/jaeger/certs/ca.crt
cert: /etc/jaeger/certs/client.crt
key: /etc/jaeger/certs/client.key
server-name: localhost
insecure-skip-verify: false
num-shards: 3
num-replicas: 1
index-prefix: jaeger
create-index-templates: true
timeout: 30s
max-span-age: 72h
collector:
grpc:
host-port: 0.0.0.0:14250
tls:
enabled: true
cert: /etc/jaeger/certs/client.crt
key: /etc/jaeger/certs/client.key
http:
host-port: 0.0.0.0:14268
tls:
enabled: true
cert: /etc/jaeger/certs/client.crt
key: /etc/jaeger/certs/client.key
zipkin:
host-port: 0.0.0.0:9411
Configure Jaeger query service
Set up Jaeger query service with read-only Elasticsearch access, TLS encryption, and UI security configuration for web interface access.
es:
server-urls: https://localhost:9200
username: jaeger_query
password: JaegerQuery2024!
tls:
enabled: true
ca: /etc/jaeger/certs/ca.crt
cert: /etc/jaeger/certs/client.crt
key: /etc/jaeger/certs/client.key
server-name: localhost
insecure-skip-verify: false
index-prefix: jaeger
timeout: 30s
max-lookback: 168h
query:
http:
host-port: 0.0.0.0:16686
tls:
enabled: true
cert: /etc/jaeger/certs/client.crt
key: /etc/jaeger/certs/client.key
grpc:
host-port: 0.0.0.0:16685
tls:
enabled: true
cert: /etc/jaeger/certs/client.crt
key: /etc/jaeger/certs/client.key
ui:
config:
archive:
enabled: false
dependencies:
menuEnabled: true
tracking:
gaID: ""
menu:
- label: "About Jaeger"
url: "https://jaegertracing.io"
Configure systemd services
Create systemd service files for Jaeger components with proper security context and resource limits for production deployment.
[Unit]
Description=Jaeger Collector
Documentation=https://jaegertracing.io
After=network.target elasticsearch.service
Requires=elasticsearch.service
[Service]
Type=simple
User=jaeger
Group=jaeger
ExecStart=/usr/bin/jaeger-collector --config-file=/etc/jaeger/collector.yaml
Restart=always
RestartSec=10
KillMode=mixed
KillSignal=SIGTERM
TimeoutStopSec=30
# Security settings
NoNewPrivileges=true
ProtectSystem=strict
ProtectHome=true
ReadWritePaths=/var/log/jaeger
PrivateTmp=true
PrivateDevices=true
ProtectKernelTunables=true
ProtectControlGroups=true
RestrictSUIDSGID=true
# Resource limits
LimitNOFILE=65536
LimitNPROC=4096
[Install]
WantedBy=multi-user.target
Create Jaeger query service
Set up systemd service for Jaeger query with security hardening and proper dependency management on Elasticsearch availability.
[Unit]
Description=Jaeger Query Service
Documentation=https://jaegertracing.io
After=network.target elasticsearch.service
Requires=elasticsearch.service
[Service]
Type=simple
User=jaeger
Group=jaeger
ExecStart=/usr/bin/jaeger-query --config-file=/etc/jaeger/query.yaml
Restart=always
RestartSec=10
KillMode=mixed
KillSignal=SIGTERM
TimeoutStopSec=30
# Security settings
NoNewPrivileges=true
ProtectSystem=strict
ProtectHome=true
ReadWritePaths=/var/log/jaeger
PrivateTmp=true
PrivateDevices=true
ProtectKernelTunables=true
ProtectControlGroups=true
RestrictSUIDSGID=true
# Resource limits
LimitNOFILE=65536
LimitNPROC=4096
[Install]
WantedBy=multi-user.target
Configure log directories and permissions
Set up proper logging directory structure with appropriate permissions for Jaeger service user and log rotation configuration.
sudo mkdir -p /var/log/jaeger
sudo chown jaeger:jaeger /var/log/jaeger
sudo chmod 750 /var/log/jaeger
# Create jaeger user if not exists
sudo useradd -r -s /bin/false jaeger 2>/dev/null || true
# Ensure all certificate permissions are correct
sudo chown -R jaeger:jaeger /etc/jaeger
sudo chmod -R o-rwx /etc/jaeger/certs
Start and enable services
Start Jaeger services with systemd and verify they connect securely to Elasticsearch with proper authentication and encryption.
sudo systemctl daemon-reload
sudo systemctl enable jaeger-collector
sudo systemctl enable jaeger-query
sudo systemctl start jaeger-collector
sudo systemctl start jaeger-query
# Check service status
sudo systemctl status jaeger-collector
sudo systemctl status jaeger-query
Configure firewall rules
Set up specific firewall rules for Jaeger ports with TLS encryption and restrict access to authorized networks only.
sudo ufw allow from 203.0.113.0/24 to any port 14250 comment 'Jaeger gRPC collector'
sudo ufw allow from 203.0.113.0/24 to any port 14268 comment 'Jaeger HTTP collector'
sudo ufw allow from 203.0.113.0/24 to any port 16686 comment 'Jaeger UI'
sudo ufw allow from 203.0.113.0/24 to any port 16685 comment 'Jaeger query gRPC'
sudo ufw reload
sudo firewall-cmd --permanent --add-rich-rule='rule family="ipv4" source address="203.0.113.0/24" port protocol="tcp" port="14250" accept'
sudo firewall-cmd --permanent --add-rich-rule='rule family="ipv4" source address="203.0.113.0/24" port protocol="tcp" port="14268" accept'
sudo firewall-cmd --permanent --add-rich-rule='rule family="ipv4" source address="203.0.113.0/24" port protocol="tcp" port="16686" accept'
sudo firewall-cmd --permanent --add-rich-rule='rule family="ipv4" source address="203.0.113.0/24" port protocol="tcp" port="16685" accept'
sudo firewall-cmd --reload
Verify your setup
Test the secure connection between Jaeger and Elasticsearch and verify all authentication mechanisms are working properly.
# Check Elasticsearch cluster health with authentication
curl -k -u elastic:$ELASTIC_PASSWORD "https://localhost:9200/_cluster/health?pretty"
# Verify Jaeger can connect to Elasticsearch
curl -k "https://localhost:16686/api/services"
# Check TLS certificate validity
echo | openssl s_client -connect localhost:16686 -servername localhost 2>/dev/null | openssl x509 -noout -dates
# Verify collector is receiving data
curl -k "https://localhost:14268/api/traces" -X POST -H "Content-Type: application/json" -d '{
"data": [
{
"traceID": "test-trace-id",
"spans": [
{
"traceID": "test-trace-id",
"spanID": "test-span-id",
"operationName": "test-operation",
"startTime": 1640995200000000,
"duration": 1000000
}
]
}
]
}'
# Check Jaeger indices in Elasticsearch
curl -k -u jaeger_query:JaegerQuery2024! "https://localhost:9200/_cat/indices/jaeger-*?v"
Production security hardening
Configure index lifecycle management
Set up automated index lifecycle policies for secure data retention and automated cleanup of old tracing data.
# Create ILM policy for Jaeger indices
curl -k -u elastic:$ELASTIC_PASSWORD -X PUT "https://localhost:9200/_ilm/policy/jaeger_policy" -H "Content-Type: application/json" -d '
{
"policy": {
"phases": {
"hot": {
"actions": {
"rollover": {
"max_size": "50gb",
"max_age": "7d"
}
}
},
"warm": {
"min_age": "7d",
"actions": {
"allocate": {
"number_of_replicas": 0
}
}
},
"delete": {
"min_age": "30d"
}
}
}
}'
# Apply policy to Jaeger index templates
curl -k -u elastic:$ELASTIC_PASSWORD -X PUT "https://localhost:9200/_index_template/jaeger_template" -H "Content-Type: application/json" -d '
{
"index_patterns": ["jaeger-*"],
"template": {
"settings": {
"index.lifecycle.name": "jaeger_policy",
"index.lifecycle.rollover_alias": "jaeger-write"
}
}
}'
Enable audit logging
Configure comprehensive audit logging for security monitoring and compliance tracking of Jaeger access patterns.
# Add to existing elasticsearch.yml
xpack.security.audit.enabled: true
xpack.security.audit.outputs: [index, logfile]
xpack.security.audit.logfile.events.include: [
"access_denied",
"access_granted",
"anonymous_access_denied",
"authentication_failed",
"connection_denied",
"tampered_request",
"run_as_denied",
"run_as_granted"
]
xpack.security.audit.logfile.events.exclude: ["access_granted"]
xpack.security.audit.index.rollover: "daily"
xpack.security.audit.index.events.include: ["authentication_failed", "access_denied", "tampered_request"]
Set up monitoring alerts
Configure security monitoring and alerting for authentication failures, unauthorized access attempts, and certificate expiration tracking.
# Create watcher for failed authentications
curl -k -u elastic:$ELASTIC_PASSWORD -X PUT "https://localhost:9200/_watcher/watch/jaeger_auth_failures" -H "Content-Type: application/json" -d '
{
"trigger": {
"schedule": {
"interval": "5m"
}
},
"input": {
"search": {
"request": {
"search_type": "query_then_fetch",
"indices": [".security-audit-*"],
"body": {
"query": {
"bool": {
"must": [
{
"term": {
"event_type": "authentication_failed"
}
},
{
"range": {
"@timestamp": {
"gte": "now-5m"
}
}
}
]
}
}
}
}
}
},
"condition": {
"compare": {
"ctx.payload.hits.total.value": {
"gt": 5
}
}
},
"actions": {
"log_alert": {
"logging&a
---
### Setup ScyllaDB backup validation and automated restore testing
URL: https://binadit.com/tutorials/setup-scylladb-backup-validation-and-testing
Category: databases
Difficulty: advanced
Time: ~45 minutes
Author: Binadit Tech Team
> Configure automated ScyllaDB backups with nodetool snapshots, implement validation scripts to verify backup integrity, and set up systemd timers for automated restore testing with Prometheus monitoring.
What this solves
ScyllaDB backup validation ensures your snapshots are recoverable when disasters strike. This tutorial sets up automated backup creation with nodetool, validates backup integrity through checksums and metadata verification, and implements automated restore testing to catch corruption before you need the backups. You'll also configure Prometheus monitoring to track backup health and receive alerts when validation fails.
Step-by-step configuration
Install backup validation dependencies
Install required tools for backup validation, compression, and monitoring.
sudo apt update
sudo apt install -y python3 python3-pip jq pigz parallel curl
sudo dnf install -y python3 python3-pip jq pigz parallel curl
sudo dnf install -y epel-release
Create backup directory structure
Set up organized directories for backups, validation logs, and restore testing.
sudo mkdir -p /opt/scylladb-backup/{snapshots,validation,restore-test,scripts,logs}
sudo chown -R scylla:scylla /opt/scylladb-backup
sudo chmod -R 755 /opt/scylladb-backup
Configure backup automation script
Create the main backup script that handles snapshot creation, validation, and cleanup.
#!/bin/bash
# ScyllaDB Backup and Validation Script
set -euo pipefail
# Configuration
BACKUP_DIR="/opt/scylladb-backup"
SNAPSHOT_DIR="$BACKUP_DIR/snapshots"
VALIDATION_DIR="$BACKUP_DIR/validation"
LOG_DIR="$BACKUP_DIR/logs"
KEYSPACES="${SCYLLA_KEYSPACES:-system_schema}"
RETENTION_DAYS=${BACKUP_RETENTION_DAYS:-7}
TIMESTAMP=$(date +%Y%m%d_%H%M%S)
SNAPSHOT_TAG="backup_$TIMESTAMP"
LOG_FILE="$LOG_DIR/backup_$TIMESTAMP.log"
# Prometheus metrics file
METRICS_FILE="/var/lib/node_exporter/textfile_collector/scylladb_backup.prom"
# Logging function
log() {
echo "$(date '+%Y-%m-%d %H:%M:%S') $1" | tee -a "$LOG_FILE"
}
# Create snapshot
create_snapshot() {
log "Creating snapshot: $SNAPSHOT_TAG"
# Clear any existing snapshots
nodetool clearsnapshot 2>/dev/null || true
# Create new snapshot
if [ "$KEYSPACES" = "all" ]; then
nodetool snapshot -t "$SNAPSHOT_TAG"
else
for ks in $KEYSPACES; do
nodetool snapshot -t "$SNAPSHOT_TAG" "$ks"
done
fi
log "Snapshot created successfully"
}
# Copy and compress snapshot data
copy_snapshot() {
log "Copying snapshot data to backup directory"
local snapshot_backup_dir="$SNAPSHOT_DIR/$TIMESTAMP"
mkdir -p "$snapshot_backup_dir"
# Find and copy snapshot files
find /var/lib/scylla/data -name "$SNAPSHOT_TAG" -type d | while read -r snap_dir; do
# Extract keyspace and table from path
local rel_path=$(echo "$snap_dir" | sed "s|/var/lib/scylla/data/||")
local dest_dir="$snapshot_backup_dir/$rel_path"
mkdir -p "$(dirname "$dest_dir")"
# Copy with compression
log "Copying $snap_dir to $dest_dir"
tar -I pigz -cf "${dest_dir}.tar.gz" -C "$(dirname "$snap_dir")" "$(basename "$snap_dir")"
done
# Create metadata file
create_metadata "$snapshot_backup_dir"
log "Snapshot copy completed"
}
# Create backup metadata
create_metadata() {
local backup_dir="$1"
local metadata_file="$backup_dir/metadata.json"
log "Creating backup metadata"
# Get cluster information
local cluster_name=$(nodetool describecluster | grep "Name:" | awk '{print $2}')
local node_id=$(nodetool info | grep "ID" | awk '{print $2}')
local datacenter=$(nodetool status | grep "$(hostname -I | awk '{print $1}')" | awk '{print $2}')
# Calculate checksums
local checksums_file="$backup_dir/checksums.sha256"
find "$backup_dir" -name "*.tar.gz" -exec sha256sum {} + > "$checksums_file"
# Create JSON metadata
cat > "$metadata_file" << EOF
{
"timestamp": "$TIMESTAMP",
"snapshot_tag": "$SNAPSHOT_TAG",
"cluster_name": "$cluster_name",
"node_id": "$node_id",
"datacenter": "$datacenter",
"keyspaces": "$KEYSPACES",
"backup_size_bytes": $(du -sb "$backup_dir" | awk '{print $1}'),
"file_count": $(find "$backup_dir" -name "*.tar.gz" | wc -l),
"checksums_file": "checksums.sha256",
"node_hostname": "$(hostname)",
"scylla_version": "$(scylla --version | head -1)"
}
EOF
log "Metadata created: $metadata_file"
}
# Validate backup integrity
validate_backup() {
local backup_dir="$SNAPSHOT_DIR/$TIMESTAMP"
local validation_log="$VALIDATION_DIR/validation_$TIMESTAMP.log"
local validation_status=0
log "Validating backup integrity"
# Check metadata exists
if [ ! -f "$backup_dir/metadata.json" ]; then
echo "FAIL: metadata.json missing" >> "$validation_log"
validation_status=1
else
echo "PASS: metadata.json exists" >> "$validation_log"
fi
# Verify checksums
if [ -f "$backup_dir/checksums.sha256" ]; then
cd "$backup_dir"
if sha256sum -c checksums.sha256 >> "$validation_log" 2>&1; then
echo "PASS: All checksums verified" >> "$validation_log"
else
echo "FAIL: Checksum verification failed" >> "$validation_log"
validation_status=1
fi
else
echo "FAIL: checksums.sha256 missing" >> "$validation_log"
validation_status=1
fi
# Test archive extraction
local test_extract_dir="$VALIDATION_DIR/extract_test_$TIMESTAMP"
mkdir -p "$test_extract_dir"
find "$backup_dir" -name "*.tar.gz" | head -3 | while read -r archive; do
if tar -tzf "$archive" > /dev/null 2>&1; then
echo "PASS: Archive readable - $(basename "$archive")" >> "$validation_log"
else
echo "FAIL: Archive corrupted - $(basename "$archive")" >> "$validation_log"
validation_status=1
fi
done
rm -rf "$test_extract_dir"
# Update metrics
update_prometheus_metrics "$validation_status"
if [ $validation_status -eq 0 ]; then
log "Backup validation PASSED"
else
log "Backup validation FAILED - check $validation_log"
exit 1
fi
}
# Update Prometheus metrics
update_prometheus_metrics() {
local validation_status="$1"
local backup_dir="$SNAPSHOT_DIR/$TIMESTAMP"
if [ -f "$backup_dir/metadata.json" ]; then
local backup_size=$(jq -r '.backup_size_bytes' "$backup_dir/metadata.json")
local file_count=$(jq -r '.file_count' "$backup_dir/metadata.json")
else
local backup_size=0
local file_count=0
fi
cat > "$METRICS_FILE" << EOF
# HELP scylladb_backup_last_success_timestamp Last successful backup timestamp
# TYPE scylladb_backup_last_success_timestamp gauge
scylladb_backup_last_success_timestamp $(date +%s)
# HELP scylladb_backup_size_bytes Size of last backup in bytes
# TYPE scylladb_backup_size_bytes gauge
scylladb_backup_size_bytes $backup_size
# HELP scylladb_backup_file_count Number of files in last backup
# TYPE scylladb_backup_file_count gauge
scylladb_backup_file_count $file_count
# HELP scylladb_backup_validation_status Last backup validation status (0=success, 1=failure)
# TYPE scylladb_backup_validation_status gauge
scylladb_backup_validation_status $validation_status
EOF
}
# Clean old backups
cleanup_old_backups() {
log "Cleaning up backups older than $RETENTION_DAYS days"
find "$SNAPSHOT_DIR" -type d -name "[0-9]*" -mtime +$RETENTION_DAYS -exec rm -rf {} + 2>/dev/null || true
find "$VALIDATION_DIR" -name "*.log" -mtime +$RETENTION_DAYS -delete 2>/dev/null || true
find "$LOG_DIR" -name "*.log" -mtime +$RETENTION_DAYS -delete 2>/dev/null || true
log "Cleanup completed"
}
# Main execution
main() {
log "Starting ScyllaDB backup process"
create_snapshot
copy_snapshot
validate_backup
cleanup_old_backups
# Clear snapshot from ScyllaDB
nodetool clearsnapshot "$SNAPSHOT_TAG" 2>/dev/null || true
log "Backup process completed successfully"
}
main "$@"
sudo chmod +x /opt/scylladb-backup/scripts/backup.sh
sudo chown scylla:scylla /opt/scylladb-backup/scripts/backup.sh
Create restore testing script
Build an automated restore test script that validates backup recoverability.
#!/bin/bash
# ScyllaDB Restore Testing Script
set -euo pipefail
# Configuration
BACKUP_DIR="/opt/scylladb-backup"
SNAPSHOT_DIR="$BACKUP_DIR/snapshots"
RESTORE_TEST_DIR="$BACKUP_DIR/restore-test"
LOG_DIR="$BACKUP_DIR/logs"
TEST_KEYSPACE="backup_test_ks"
TEST_TABLE="backup_test_table"
TIMESTAMP=$(date +%Y%m%d_%H%M%S)
LOG_FILE="$LOG_DIR/restore_test_$TIMESTAMP.log"
# Prometheus metrics
METRICS_FILE="/var/lib/node_exporter/textfile_collector/scylladb_restore_test.prom"
# Logging function
log() {
echo "$(date '+%Y-%m-%d %H:%M:%S') $1" | tee -a "$LOG_FILE"
}
# Find latest backup
find_latest_backup() {
local latest_backup=$(find "$SNAPSHOT_DIR" -type d -name "[0-9]*" | sort -r | head -1)
if [ -z "$latest_backup" ]; then
log "ERROR: No backups found in $SNAPSHOT_DIR"
exit 1
fi
echo "$latest_backup"
}
# Create test keyspace and data
create_test_data() {
log "Creating test keyspace and data"
# Drop existing test keyspace if exists
cqlsh -e "DROP KEYSPACE IF EXISTS $TEST_KEYSPACE;" 2>/dev/null || true
# Create test keyspace
cqlsh -e "CREATE KEYSPACE $TEST_KEYSPACE WITH replication = {'class': 'SimpleStrategy', 'replication_factor': 1};"
# Create test table with sample data
cqlsh -k "$TEST_KEYSPACE" -e "
CREATE TABLE $TEST_TABLE (
id UUID PRIMARY KEY,
name TEXT,
created_at TIMESTAMP,
data BLOB
);
INSERT INTO $TEST_TABLE (id, name, created_at, data)
VALUES (uuid(), 'test_record_1', toTimestamp(now()), 0x123456789abcdef);
INSERT INTO $TEST_TABLE (id, name, created_at, data)
VALUES (uuid(), 'test_record_2', toTimestamp(now()), 0xfedcba987654321);
INSERT INTO $TEST_TABLE (id, name, created_at, data)
VALUES (uuid(), 'test_record_3', toTimestamp(now()), 0x1a2b3c4d5e6f);
"
log "Test data created successfully"
}
# Get test data checksum
get_test_data_checksum() {
local checksum=$(cqlsh -k "$TEST_KEYSPACE" -e "SELECT * FROM $TEST_TABLE;" | md5sum | awk '{print $1}')
echo "$checksum"
}
# Perform snapshot of test data
snapshot_test_data() {
local snapshot_tag="restore_test_$TIMESTAMP"
log "Creating snapshot of test data: $snapshot_tag"
nodetool snapshot -t "$snapshot_tag" "$TEST_KEYSPACE"
echo "$snapshot_tag"
}
# Extract backup for testing
extract_backup() {
local backup_dir="$1"
local extract_dir="$RESTORE_TEST_DIR/extract_$TIMESTAMP"
log "Extracting backup to $extract_dir"
mkdir -p "$extract_dir"
# Extract all archives from the backup
find "$backup_dir" -name "*.tar.gz" | while read -r archive; do
local rel_path=$(echo "$archive" | sed "s|$backup_dir/||" | sed 's|\.tar\.gz$||')
local dest_dir="$extract_dir/$rel_path"
mkdir -p "$(dirname "$dest_dir")"
tar -I pigz -xf "$archive" -C "$(dirname "$dest_dir")"
done
echo "$extract_dir"
}
# Restore keyspace from backup
restore_from_backup() {
local extract_dir="$1"
local target_keyspace="${TEST_KEYSPACE}_restored"
log "Restoring keyspace as $target_keyspace"
# Create target keyspace
cqlsh -e "DROP KEYSPACE IF EXISTS $target_keyspace;" 2>/dev/null || true
cqlsh -e "CREATE KEYSPACE $target_keyspace WITH replication = {'class': 'SimpleStrategy', 'replication_factor': 1};"
# Find snapshot data for our test keyspace
local snapshot_path=$(find "$extract_dir" -path "*/$TEST_KEYSPACE/*" -name "backup_*" | head -1)
if [ -z "$snapshot_path" ]; then
log "ERROR: No snapshot data found for $TEST_KEYSPACE"
return 1
fi
# Copy snapshot files to target keyspace data directory
local target_data_dir="/var/lib/scylla/data/$target_keyspace"
sudo systemctl stop scylla-server
# Create target directory structure
find "$snapshot_path" -name "*.db" | while read -r db_file; do
local table_dir=$(dirname "$db_file" | sed "s|.*/$TEST_KEYSPACE/|$target_data_dir/|" | sed 's|/backup_[0-9_]*$||')
sudo mkdir -p "$table_dir"
sudo cp "$db_file" "$table_dir/"
sudo chown scylla:scylla "$table_dir"/*.db
done
sudo systemctl start scylla-server
# Wait for ScyllaDB to start
local retries=30
while ! nodetool status >/dev/null 2>&1 && [ $retries -gt 0 ]; do
log "Waiting for ScyllaDB to start... ($retries retries left)"
sleep 10
retries=$((retries - 1))
done
if [ $retries -eq 0 ]; then
log "ERROR: ScyllaDB failed to start after restore"
return 1
fi
log "Restore completed for keyspace $target_keyspace"
}
# Verify restored data
verify_restored_data() {
local target_keyspace="${TEST_KEYSPACE}_restored"
local original_checksum="$1"
log "Verifying restored data integrity"
# Get restored data checksum
local restored_checksum=$(cqlsh -k "$target_keyspace" -e "SELECT * FROM $TEST_TABLE;" | md5sum | awk '{print $1}')
if [ "$original_checksum" = "$restored_checksum" ]; then
log "SUCCESS: Restored data matches original (checksum: $restored_checksum)"
return 0
else
log "ERROR: Restored data does not match original"
log "Original checksum: $original_checksum"
log "Restored checksum: $restored_checksum"
return 1
fi
}
# Update Prometheus metrics
update_restore_metrics() {
local test_status="$1"
local test_duration="$2"
cat > "$METRICS_FILE" << EOF
# HELP scylladb_restore_test_last_run_timestamp Last restore test timestamp
# TYPE scylladb_restore_test_last_run_timestamp gauge
scylladb_restore_test_last_run_timestamp $(date +%s)
# HELP scylladb_restore_test_status Last restore test status (0=success, 1=failure)
# TYPE scylladb_restore_test_status gauge
scylladb_restore_test_status $test_status
# HELP scylladb_restore_test_duration_seconds Duration of last restore test
# TYPE scylladb_restore_test_duration_seconds gauge
scylladb_restore_test_duration_seconds $test_duration
EOF
}
# Cleanup test data
cleanup_test_data() {
log "Cleaning up test data"
# Drop test keyspaces
cqlsh -e "DROP KEYSPACE IF EXISTS $TEST_KEYSPACE;" 2>/dev/null || true
cqlsh -e "DROP KEYSPACE IF EXISTS ${TEST_KEYSPACE}_restored;" 2>/dev/null || true
# Clean snapshot
nodetool clearsnapshot 2>/dev/null || true
# Remove extract directory
rm -rf "$RESTORE_TEST_DIR/extract_$TIMESTAMP" 2>/dev/null || true
log "Cleanup completed"
}
# Main execution
main() {
local start_time=$(date +%s)
local test_status=0
log "Starting ScyllaDB restore test"
# Find latest backup
local backup_dir=$(find_latest_backup)
log "Using backup: $backup_dir"
# Create test data and get checksum
create_test_data
local original_checksum=$(get_test_data_checksum)
log "Original data checksum: $original_checksum"
# Snapshot test data (for comparison)
local test_snapshot=$(snapshot_test_data)
# Extract backup
local extract_dir=$(extract_backup "$backup_dir")
# Restore and verify
if restore_from_backup "$extract_dir" && verify_restored_data "$original_checksum"; then
log "Restore test PASSED"
test_status=0
else
log "Restore test FAILED"
test_status=1
fi
# Calculate duration and update metrics
local end_time=$(date +%s)
local duration=$((end_time - start_time))
update_restore_metrics "$test_status" "$duration"
# Cleanup
cleanup_test_data
log "Restore test completed in ${duration}s with status $test_status"
exit $test_status
}
main "$@"
sudo chmod +x /opt/scylladb-backup/scripts/restore-test.sh
sudo chown scylla:scylla /opt/scylladb-backup/scripts/restore-test.sh
Configure systemd timers for automation
Set up systemd services and timers for automated backup and restore testing.
[Unit]
Description=ScyllaDB Backup Service
After=scylla-server.service
Requires=scylla-server.service
[Service]
Type=oneshot
User=scylla
Group=scylla
ExecStart=/opt/scylladb-backup/scripts/backup.sh
Environment=SCYLLA_KEYSPACES=all
Environment=BACKUP_RETENTION_DAYS=7
St
---
### Implement enterprise network QoS with Cisco integration using FRRouting and traffic shaping
URL: https://binadit.com/tutorials/implement-enterprise-network-qos-with-cisco-integration
Category: networking
Difficulty: advanced
Time: ~45 minutes
Author: Binadit Tech Team
> Configure enterprise-grade Quality of Service policies with DSCP marking, traffic shaping using tc and HTB, and seamless integration with Cisco equipment through FRRouting BGP and OSPF routing protocols for comprehensive network bandwidth management.
What this solves
Enterprise networks require sophisticated Quality of Service (QoS) policies to prioritize critical traffic and maintain performance across diverse applications. This tutorial configures advanced QoS with DSCP marking, hierarchical traffic shaping, and seamless integration with Cisco infrastructure through FRRouting's BGP and OSPF implementations.
Prerequisites
You need root access to your Linux servers, basic understanding of networking concepts, and existing network infrastructure with Cisco equipment. Your servers should have multiple network interfaces for testing traffic separation.
Step-by-step configuration
Install FRRouting and traffic control tools
Install FRRouting for Cisco protocol compatibility and Linux traffic control utilities for QoS implementation.
curl -s https://deb.frrouting.org/frr/keys.asc | sudo apt-key add -
echo 'deb https://deb.frrouting.org/frr jammy frr-stable' | sudo tee /etc/apt/sources.list.d/frr.list
sudo apt update
sudo apt install -y frr frr-pythontools iproute2 tc iptables-persistent
sudo dnf install -y epel-release
sudo dnf install -y https://rpm.frrouting.org/repo/rpm-repo-0-*.noarch.rpm
sudo dnf install -y frr iproute tc iptables-services
sudo systemctl enable iptables
Enable FRRouting daemons
Configure FRRouting to enable BGP and OSPF daemons for Cisco integration.
bgpd=yes
ospfd=yes
zebra=yes
vtysh_enable=yes
zebra_options=" -A 127.0.0.1 -s 90000000"
bgpd_options=" -A 127.0.0.1"
ospfd_options=" -A 127.0.0.1"
Configure FRRouting BGP for Cisco integration
Set up BGP peering with Cisco equipment and implement route policies for QoS integration.
frr version 8.4
frr defaults traditional
hostname frrouting-qos
log syslog informational
ipv6 forwarding
!
interface eth0
description WAN-Interface
ip address 203.0.113.10/24
!
interface eth1
description LAN-Interface
ip address 192.168.1.1/24
!
router bgp 65001
bgp router-id 203.0.113.10
neighbor 203.0.113.1 remote-as 65000
neighbor 203.0.113.1 description Cisco-BGP-Peer
!
address-family ipv4 unicast
neighbor 203.0.113.1 activate
neighbor 203.0.113.1 route-map QOS-OUT out
neighbor 203.0.113.1 route-map QOS-IN in
exit-address-family
!
router ospf
ospf router-id 192.168.1.1
network 192.168.1.0/24 area 0
network 203.0.113.0/24 area 0
!
route-map QOS-OUT permit 10
set community 65001:100
set ip next-hop unchanged
!
route-map QOS-IN permit 10
match community PRIORITY-TRAFFIC
set local-preference 200
!
ip community-list standard PRIORITY-TRAFFIC permit 65000:100
!
line vty
!
Start and enable FRRouting services
Enable FRRouting services and verify BGP neighbor establishment.
sudo systemctl enable --now frr
sudo systemctl status frr
sudo vtysh -c "show ip bgp summary"
sudo vtysh -c "show ip ospf neighbor"
Configure DSCP marking with iptables
Implement DSCP marking for traffic classification and QoS policy enforcement.
# DSCP marking for enterprise QoS
*mangle
:PREROUTING ACCEPT [0:0]
:INPUT ACCEPT [0:0]
:FORWARD ACCEPT [0:0]
:OUTPUT ACCEPT [0:0]
:POSTROUTING ACCEPT [0:0]
# Voice traffic - EF (Expedited Forwarding) DSCP 46
-A POSTROUTING -p udp --dport 5060:5090 -j DSCP --set-dscp 46
-A POSTROUTING -p tcp --dport 5060:5090 -j DSCP --set-dscp 46
-A POSTROUTING -p udp --dport 16384:32767 -j DSCP --set-dscp 46
# Video conferencing - AF41 DSCP 34
-A POSTROUTING -p tcp --dport 443 -m string --string "zoom" --algo bm -j DSCP --set-dscp 34
-A POSTROUTING -p udp --dport 8801:8810 -j DSCP --set-dscp 34
-A POSTROUTING -p tcp --dport 1935 -j DSCP --set-dscp 34
# Database traffic - AF31 DSCP 26
-A POSTROUTING -p tcp --dport 3306 -j DSCP --set-dscp 26
-A POSTROUTING -p tcp --dport 5432 -j DSCP --set-dscp 26
-A POSTROUTING -p tcp --dport 1433 -j DSCP --set-dscp 26
# Management traffic - CS6 DSCP 48
-A POSTROUTING -p tcp --dport 22 -j DSCP --set-dscp 48
-A POSTROUTING -p tcp --dport 161:162 -j DSCP --set-dscp 48
-A POSTROUTING -p tcp --dport 179 -j DSCP --set-dscp 48
# Web traffic - AF21 DSCP 18
-A POSTROUTING -p tcp --dport 80 -j DSCP --set-dscp 18
-A POSTROUTING -p tcp --dport 443 -j DSCP --set-dscp 18
# Best effort - default DSCP 0
-A POSTROUTING -j DSCP --set-dscp 0
COMMIT
*filter
:INPUT ACCEPT [0:0]
:FORWARD ACCEPT [0:0]
:OUTPUT ACCEPT [0:0]
COMMIT
Apply iptables rules
Load the DSCP marking rules and make them persistent across reboots.
sudo iptables-restore < /etc/iptables/rules.v4
sudo systemctl enable netfilter-persistent
sudo netfilter-persistent save
sudo iptables-restore < /etc/iptables/rules.v4
sudo service iptables save
sudo systemctl enable iptables
Configure HTB traffic shaping
Implement Hierarchical Token Bucket (HTB) for bandwidth management and traffic prioritization.
#!/bin/bash
# Interface configuration
INTERFACE="eth0"
BANDWIDTH="100mbit"
# Clear existing qdisc
tc qdisc del dev $INTERFACE root 2>/dev/null
# Create root HTB qdisc
tc qdisc add dev $INTERFACE root handle 1: htb default 60
# Create root class with total bandwidth
tc class add dev $INTERFACE parent 1: classid 1:1 htb rate $BANDWIDTH
# Voice traffic - 20% guaranteed, 50% max
tc class add dev $INTERFACE parent 1:1 classid 1:10 htb rate 20mbit ceil 50mbit prio 1
tc filter add dev $INTERFACE parent 1: protocol ip prio 1 u32 match ip tos 0xb8 0xfc flowid 1:10
# Video traffic - 30% guaranteed, 60% max
tc class add dev $INTERFACE parent 1:1 classid 1:20 htb rate 30mbit ceil 60mbit prio 2
tc filter add dev $INTERFACE parent 1: protocol ip prio 2 u32 match ip tos 0x88 0xfc flowid 1:20
# Database traffic - 15% guaranteed, 40% max
tc class add dev $INTERFACE parent 1:1 classid 1:30 htb rate 15mbit ceil 40mbit prio 3
tc filter add dev $INTERFACE parent 1: protocol ip prio 3 u32 match ip tos 0x68 0xfc flowid 1:30
# Management traffic - 10% guaranteed, 30% max
tc class add dev $INTERFACE parent 1:1 classid 1:40 htb rate 10mbit ceil 30mbit prio 4
tc filter add dev $INTERFACE parent 1: protocol ip prio 4 u32 match ip tos 0xc0 0xfc flowid 1:40
# Web traffic - 15% guaranteed, 50% max
tc class add dev $INTERFACE parent 1:1 classid 1:50 htb rate 15mbit ceil 50mbit prio 5
tc filter add dev $INTERFACE parent 1: protocol ip prio 5 u32 match ip tos 0x48 0xfc flowid 1:50
# Best effort - 10% guaranteed, remaining available
tc class add dev $INTERFACE parent 1:1 classid 1:60 htb rate 10mbit ceil $BANDWIDTH prio 6
# Add fair queuing to each class for better distribution
tc qdisc add dev $INTERFACE parent 1:10 handle 10: sfq perturb 10
tc qdisc add dev $INTERFACE parent 1:20 handle 20: sfq perturb 10
tc qdisc add dev $INTERFACE parent 1:30 handle 30: sfq perturb 10
tc qdisc add dev $INTERFACE parent 1:40 handle 40: sfq perturb 10
tc qdisc add dev $INTERFACE parent 1:50 handle 50: sfq perturb 10
tc qdisc add dev $INTERFACE parent 1:60 handle 60: sfq perturb 10
echo "QoS configuration applied successfully"
tc class show dev $INTERFACE
Make QoS script executable and apply configuration
Set proper permissions and apply the traffic shaping configuration.
sudo chmod +x /usr/local/bin/setup-qos.sh
sudo /usr/local/bin/setup-qos.sh
Create systemd service for persistent QoS
Ensure QoS configuration persists across system reboots with a systemd service.
[Unit]
Description=Network QoS Configuration
After=network-online.target
Wants=network-online.target
[Service]
Type=oneshot
ExecStart=/usr/local/bin/setup-qos.sh
RemainAfterExit=yes
[Install]
WantedBy=multi-user.target
Enable QoS service and configure bandwidth monitoring
Enable the QoS service and set up monitoring for traffic analysis.
sudo systemctl daemon-reload
sudo systemctl enable --now network-qos
sudo systemctl status network-qos
Configure advanced OSPF for Cisco integration
Set up OSPF areas and route redistribution for enterprise network integration. For complex QoS routing, you might also want to explore OSPF multi-area design with FRRouting for larger deployments.
sudo vtysh
configure terminal
router ospf
area 0 authentication message-digest
area 1 stub
network 10.0.0.0/8 area 1
redistribute connected metric 20 metric-type 2
redistribute static metric 10 metric-type 1
passive-interface default
no passive-interface eth0
no passive-interface eth1
timers throttle spf 200 1000 5000
max-metric router-lsa on-startup 60
!
interface eth0
ip ospf message-digest-key 1 md5 cisco123
ip ospf priority 100
ip ospf cost 10
!
interface eth1
ip ospf message-digest-key 1 md5 cisco123
ip ospf priority 200
ip ospf cost 5
!
write memory
exit
Configure traffic monitoring and QoS statistics
Set up monitoring scripts to track QoS effectiveness and bandwidth utilization.
#!/bin/bash
INTERFACE="eth0"
LOGFILE="/var/log/qos-stats.log"
echo "$(date): QoS Statistics for $INTERFACE" >> $LOGFILE
echo "===========================================" >> $LOGFILE
# Display class statistics
tc -s class show dev $INTERFACE >> $LOGFILE
# Display current DSCP markings
echo "\nDSCP Statistics:" >> $LOGFILE
iptables -t mangle -L POSTROUTING -v -n | grep DSCP >> $LOGFILE
# BGP neighbor status
echo "\nBGP Neighbor Status:" >> $LOGFILE
vtysh -c "show ip bgp summary" >> $LOGFILE
# OSPF neighbor status
echo "\nOSPF Neighbor Status:" >> $LOGFILE
vtysh -c "show ip ospf neighbor" >> $LOGFILE
echo "" >> $LOGFILE
Set up automated QoS monitoring
Create a cron job for regular QoS monitoring and performance tracking.
sudo chmod +x /usr/local/bin/qos-monitor.sh
(crontab -l 2>/dev/null; echo "*/5 * * * * /usr/local/bin/qos-monitor.sh") | sudo crontab -
Configure Cisco integration verification
Verify BGP and OSPF peering with Cisco equipment and QoS policy synchronization.
# Verify BGP neighbor establishment
sudo vtysh -c "show ip bgp neighbor 203.0.113.1"
# Check OSPF database
sudo vtysh -c "show ip ospf database"
# Verify route redistribution
sudo vtysh -c "show ip route ospf"
sudo vtysh -c "show ip route bgp"
Verify your setup
Test the QoS configuration and verify Cisco integration with comprehensive checks.
# Check QoS class configuration
tc class show dev eth0
# Verify DSCP marking rules
iptables -t mangle -L POSTROUTING -v -n
# Test BGP connectivity
sudo vtysh -c "show ip bgp summary"
sudo vtysh -c "show ip bgp neighbors"
# Check OSPF neighbors
sudo vtysh -c "show ip ospf neighbor"
sudo vtysh -c "show ip ospf interface"
# Monitor traffic classification
watch -n 2 'tc -s class show dev eth0'
# Test DSCP marking with ping
ping -Q 184 203.0.113.1 # Test AF21 marking
ping -Q 136 203.0.113.1 # Test AF31 marking
# Check routing table
ip route show
sudo vtysh -c "show ip route"
Performance tuning
Optimize QoS performance for high-throughput enterprise environments.
ParameterDefaultOptimizedPurpose
HTB quantum15008000Better packet scheduling
SFQ perturb105More frequent hash regeneration
OSPF SPF throttle50001000Faster convergence
BGP keepalive6030Faster failure detection
Apply performance optimizations:
# Optimize network buffers
echo 'net.core.rmem_max = 16777216' | sudo tee -a /etc/sysctl.conf
echo 'net.core.wmem_max = 16777216' | sudo tee -a /etc/sysctl.conf
echo 'net.ipv4.tcp_rmem = 4096 65536 16777216' | sudo tee -a /etc/sysctl.conf
echo 'net.ipv4.tcp_wmem = 4096 65536 16777216' | sudo tee -a /etc/sysctl.conf
# Apply changes
sudo sysctl -p
# Optimize BGP timers
sudo vtysh -c "configure terminal"
sudo vtysh -c "router bgp 65001"
sudo vtysh -c "neighbor 203.0.113.1 timers 30 90"
sudo vtysh -c "write memory"
Integration with advanced iptables QoS
For more sophisticated packet marking and filtering rules, consider implementing advanced iptables QoS with fwmark to complement your FRRouting setup.
Common issues
SymptomCauseFix
BGP neighbor won't establishFirewall blocking port 179sudo iptables -A INPUT -p tcp --dport 179 -j ACCEPT
OSPF adjacency failsArea ID mismatchVerify area configuration with show ip ospf interface
Traffic not shaped correctlyWrong interface specifiedCheck with ip link show and update script
DSCP marking not workingiptables rules not appliedsudo iptables-save | grep DSCP to verify
QoS classes emptyFilter rules not matchingUse tcpdump -i eth0 -v to check DSCP values
Route redistribution failsMetric conflictsAdjust redistribution metrics in OSPF config
Security considerations
Security note: QoS configurations can be exploited for DoS attacks. Always implement rate limiting and monitor for unusual traffic patterns.
Implement security hardening for QoS infrastructure:
# Protect against QoS abuse
iptables -A INPUT -p tcp --dport 179 -m limit --limit 10/minute -j ACCEPT
iptables -A INPUT -p tcp --dport 179 -j DROP
# Monitor for DSCP abuse
echo '*/10 * * * * /usr/bin/iptables -t mangle -Z' | sudo crontab -
# BGP authentication
sudo vtysh -c "configure terminal"
sudo vtysh -c "router bgp 65001"
sudo vtysh -c "neighbor 203.0.113.1 password CiscoQoSAuth2024"
sudo vtysh -c "write memory"
Next steps
Implement OSPF multi-area design with FRRouting and advanced routing policies
Configure advanced iptables QoS with fwmark and multiple interfaces
Configure enterprise BGP route filtering with FRRouting communities
Implement network automation with FRRouting SNMP monitoring
Setup enterprise network redundancy with VRRP failover
Running this in production?
Need this managed? Running enterprise QoS at scale adds complexity: capacity planning, policy optimization, multi-vendor integration, and 24/7 monitoring. Our managed platform covers monitoring, optimization and incident response for European enterprise networks.
---
### Implement Kubernetes RBAC with service accounts and role-based access control
URL: https://binadit.com/tutorials/implement-kubernetes-rbac-with-service-accounts
Category: devops
Difficulty: intermediate
Time: ~45 minutes
Author: Binadit Tech Team
> Configure Kubernetes role-based access control (RBAC) with service accounts, roles, and role bindings to enforce secure access policies and namespace isolation in your cluster.
What this solves
Kubernetes RBAC (Role-Based Access Control) controls who can access what resources in your cluster. By default, Kubernetes operates on a principle of least privilege, but without proper RBAC configuration, you may grant excessive permissions or block legitimate access. This tutorial shows you how to create service accounts, define roles with specific permissions, and bind them together for secure, granular access control.
Understanding Kubernetes RBAC fundamentals
RBAC in Kubernetes consists of four main components that work together. Service accounts provide identity for pods and applications. Roles define what actions can be performed on which resources. ClusterRoles work like Roles but apply cluster-wide. RoleBindings and ClusterRoleBindings connect subjects (users, groups, service accounts) to roles.
RBAC operates on the principle of explicit allow - everything is denied by default unless explicitly permitted. Permissions are additive, meaning multiple roles can grant different permissions to the same subject. Namespace-level roles only apply within their specific namespace, while cluster roles can access resources across all namespaces.
Step-by-step configuration
Verify RBAC is enabled
Check that RBAC authorization is enabled in your Kubernetes cluster before proceeding with configuration.
kubectl api-versions | grep rbac
kubectl auth can-i create roles --as=system:serviceaccount:default:default
Create a dedicated namespace
Create a test namespace to demonstrate RBAC policies without affecting existing workloads.
kubectl create namespace rbac-demo
kubectl get namespaces rbac-demo
Create a service account
Service accounts provide an identity for processes that run in pods. Create a service account that will be used by applications needing specific permissions.
apiVersion: v1
kind: ServiceAccount
metadata:
name: app-reader
namespace: rbac-demo
automountServiceAccountToken: true
kubectl apply -f rbac-serviceaccount.yaml
kubectl get serviceaccount -n rbac-demo
Create a namespace-level role
Define a role that grants read-only permissions to pods and services within the rbac-demo namespace. This follows the principle of least privilege by only granting necessary permissions.
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
namespace: rbac-demo
name: pod-reader
rules:
- apiGroups: [""]
resources: ["pods", "services"]
verbs: ["get", "watch", "list"]
- apiGroups: [""]
resources: ["pods/log"]
verbs: ["get"]
kubectl apply -f rbac-role.yaml
kubectl get role -n rbac-demo
Create a role binding
Bind the service account to the role, granting the app-reader service account the permissions defined in the pod-reader role.
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
name: read-pods
namespace: rbac-demo
subjects:
- kind: ServiceAccount
name: app-reader
namespace: rbac-demo
roleRef:
kind: Role
name: pod-reader
apiGroup: rbac.authorization.k8s.io
kubectl apply -f rbac-rolebinding.yaml
kubectl get rolebinding -n rbac-demo
Create a cluster-level role
Define a ClusterRole for permissions that span multiple namespaces or cluster-wide resources like nodes. This example grants read access to nodes and persistent volumes.
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
name: node-reader
rules:
- apiGroups: [""]
resources: ["nodes", "persistentvolumes"]
verbs: ["get", "watch", "list"]
- apiGroups: ["metrics.k8s.io"]
resources: ["nodes", "pods"]
verbs: ["get", "list"]
kubectl apply -f rbac-clusterrole.yaml
kubectl get clusterrole node-reader
Create a cluster role binding
Bind the service account to the cluster role, giving it cluster-wide read permissions for nodes and metrics.
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
name: read-nodes
subjects:
- kind: ServiceAccount
name: app-reader
namespace: rbac-demo
roleRef:
kind: ClusterRole
name: node-reader
apiGroup: rbac.authorization.k8s.io
kubectl apply -f rbac-clusterrolebinding.yaml
kubectl get clusterrolebinding read-nodes
Create a role for write operations
Create a separate role with write permissions for deployment management. This demonstrates separation of concerns between read and write access.
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
namespace: rbac-demo
name: deployment-manager
rules:
- apiGroups: ["apps"]
resources: ["deployments", "replicasets"]
verbs: ["get", "list", "watch", "create", "update", "patch", "delete"]
- apiGroups: [""]
resources: ["pods"]
verbs: ["get", "list", "watch"]
kubectl apply -f rbac-deployment-role.yaml
Create a service account for deployment operations
Create a separate service account for deployment operations to demonstrate role separation.
apiVersion: v1
kind: ServiceAccount
metadata:
name: app-deployer
namespace: rbac-demo
kubectl apply -f rbac-deployer-sa.yaml
Bind the deployer role
Create a role binding that grants the app-deployer service account deployment management permissions.
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
name: manage-deployments
namespace: rbac-demo
subjects:
- kind: ServiceAccount
name: app-deployer
namespace: rbac-demo
roleRef:
kind: Role
name: deployment-manager
apiGroup: rbac.authorization.k8s.io
kubectl apply -f rbac-deployer-binding.yaml
Testing RBAC policies
Test service account permissions
Use kubectl auth can-i to test what each service account can do. This helps verify your RBAC configuration works as expected.
kubectl auth can-i get pods --as=system:serviceaccount:rbac-demo:app-reader -n rbac-demo
kubectl auth can-i create deployments --as=system:serviceaccount:rbac-demo:app-reader -n rbac-demo
kubectl auth can-i create deployments --as=system:serviceaccount:rbac-demo:app-deployer -n rbac-demo
Create a test pod with service account
Deploy a pod that uses one of your service accounts to verify the RBAC configuration works in practice.
apiVersion: v1
kind: Pod
metadata:
name: rbac-test
namespace: rbac-demo
spec:
serviceAccountName: app-reader
containers:
- name: kubectl-test
image: bitnami/kubectl:latest
command: ['sleep', '3600']
restartPolicy: Never
kubectl apply -f rbac-test-pod.yaml
kubectl get pod rbac-test -n rbac-demo
Test permissions from inside the pod
Execute commands inside the test pod to verify that the service account can only perform allowed operations.
kubectl exec -it rbac-test -n rbac-demo -- kubectl get pods -n rbac-demo
kubectl exec -it rbac-test -n rbac-demo -- kubectl get nodes
kubectl exec -it rbac-test -n rbac-demo -- kubectl create deployment test --image=nginx -n rbac-demo
Implementing advanced RBAC patterns
Create resource-specific permissions
Configure RBAC to allow access to specific resources by name, not just resource types. This provides fine-grained control over individual resources.
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
namespace: rbac-demo
name: specific-config-reader
rules:
- apiGroups: [""]
resources: ["configmaps"]
resourceNames: ["app-config", "database-config"]
verbs: ["get"]
- apiGroups: [""]
resources: ["secrets"]
resourceNames: ["app-secret"]
verbs: ["get"]
kubectl apply -f rbac-specific-resource.yaml
Create role aggregation
Use ClusterRole aggregation to combine multiple roles automatically. This is useful for creating composite roles from smaller, reusable components.
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
name: monitoring-reader
labels:
rbac.example.com/aggregate-to-monitoring: "true"
rules:
- apiGroups: [""]
resources: ["pods", "services", "endpoints"]
verbs: ["get", "list", "watch"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
name: aggregate-monitoring
aggregationRule:
clusterRoleSelectors:
- matchLabels:
rbac.example.com/aggregate-to-monitoring: "true"
rules: []
kubectl apply -f rbac-aggregated-role.yaml
Verify your setup
kubectl get serviceaccounts -n rbac-demo
kubectl get roles -n rbac-demo
kubectl get rolebindings -n rbac-demo
kubectl get clusterroles | grep -E "node-reader|aggregate-monitoring"
kubectl get clusterrolebindings | grep read-nodes
kubectl auth can-i --list --as=system:serviceaccount:rbac-demo:app-reader -n rbac-demo
Troubleshooting access issues
Debug RBAC permissions
Use these commands to troubleshoot when applications can't access required resources.
kubectl auth can-i create pods --as=system:serviceaccount:rbac-demo:app-reader -n rbac-demo
kubectl describe rolebinding -n rbac-demo
kubectl describe clusterrolebinding read-nodes
Check service account tokens
Verify that service accounts have valid tokens and can authenticate to the API server.
kubectl get serviceaccount app-reader -n rbac-demo -o yaml
kubectl get secrets -n rbac-demo | grep app-reader
kubectl describe secret $(kubectl get serviceaccount app-reader -n rbac-demo -o jsonpath='{.secrets[0].name}') -n rbac-demo
Common issues
SymptomCauseFix
Pod gets "forbidden" errorsService account lacks permissionsCreate role and rolebinding with required verbs and resources
RoleBinding doesn't workWrong namespace or subject referenceVerify namespace matches and subject kind/name are correct
Can't access cluster resourcesUsing Role instead of ClusterRoleCreate ClusterRole and ClusterRoleBinding for cluster-scoped resources
Service account not foundautomountServiceAccountToken disabledSet automountServiceAccountToken: true in ServiceAccount
Permissions too broadUsing wildcards in rulesSpecify exact resources and verbs instead of "*"
Multiple bindings conflictOverlapping role definitionsRBAC permissions are additive - review all bindings for the subject
Next steps
Implement Kubernetes network policies for pod-to-pod security and traffic isolation
Configure Kubernetes secrets management with Vault integration for secure container orchestration
Set up Kubernetes monitoring with Prometheus Operator and custom metrics
Configure advanced Kubernetes Pod Security Standards with admission controllers
Implement Kubernetes policy enforcement with Open Policy Agent Gatekeeper
Running this in production?
Want this handled for you? Setting this up once is straightforward. Keeping it patched, monitored, backed up and performant across environments is the harder part. See how we run infrastructure like this for European teams.
---
### Configure ScyllaDB cluster monitoring with Prometheus and Grafana dashboards
URL: https://binadit.com/tutorials/configure-scylladb-cluster-monitoring-with-prometheus
Category: monitoring
Difficulty: intermediate
Time: ~45 minutes
Author: Binadit Tech Team
> Set up comprehensive monitoring for ScyllaDB clusters using Prometheus metrics collection and Grafana visualization dashboards. Configure alerting rules for performance monitoring and health checks.
What this solves
ScyllaDB provides extensive metrics through its built-in monitoring endpoints, but collecting and visualizing these metrics requires proper setup. This tutorial shows you how to configure Prometheus to scrape ScyllaDB metrics, set up Grafana dashboards for cluster visualization, and implement alerting rules for proactive monitoring of your NoSQL database cluster.
Step-by-step configuration
Install Prometheus
First, install Prometheus to collect metrics from your ScyllaDB cluster.
sudo apt update
wget https://github.com/prometheus/prometheus/releases/download/v2.48.0/prometheus-2.48.0.linux-amd64.tar.gz
tar xzf prometheus-2.48.0.linux-amd64.tar.gz
sudo mv prometheus-2.48.0.linux-amd64 /opt/prometheus
sudo useradd --system --shell /bin/false prometheus
sudo chown -R prometheus:prometheus /opt/prometheus
sudo dnf update -y
wget https://github.com/prometheus/prometheus/releases/download/v2.48.0/prometheus-2.48.0.linux-amd64.tar.gz
tar xzf prometheus-2.48.0.linux-amd64.tar.gz
sudo mv prometheus-2.48.0.linux-amd64 /opt/prometheus
sudo useradd --system --shell /bin/false prometheus
sudo chown -R prometheus:prometheus /opt/prometheus
Configure Prometheus for ScyllaDB
Create the Prometheus configuration file with ScyllaDB scrape targets. ScyllaDB exposes metrics on port 9180 by default.
global:
scrape_interval: 15s
evaluation_interval: 15s
rule_files:
- "scylla_rules.yml"
alerting:
alertmanagers:
- static_configs:
- targets:
- localhost:9093
scrape_configs:
- job_name: 'prometheus'
static_configs:
- targets: ['localhost:9090']
- job_name: 'scylla'
static_configs:
- targets:
- '203.0.113.10:9180'
- '203.0.113.11:9180'
- '203.0.113.12:9180'
scrape_interval: 10s
metrics_path: /metrics
params:
format: [prometheus]
- job_name: 'scylla-manager'
static_configs:
- targets:
- '203.0.113.10:56090'
scrape_interval: 30s
Create ScyllaDB alerting rules
Define alerting rules specific to ScyllaDB performance and health monitoring.
groups:
- name: scylla.rules
rules:
- alert: ScyllaNodeDown
expr: up{job="scylla"} == 0
for: 1m
labels:
severity: critical
annotations:
summary: "ScyllaDB node is down"
description: "ScyllaDB node {{ $labels.instance }} has been down for more than 1 minute."
- alert: ScyllaHighCPU
expr: scylla_reactor_utilization > 0.8
for: 5m
labels:
severity: warning
annotations:
summary: "High CPU utilization on ScyllaDB node"
description: "CPU utilization is {{ $value }} on {{ $labels.instance }}"
- alert: ScyllaHighLatency
expr: scylla_storage_proxy_coordinator_read_latency{quantile="0.99"} > 100000
for: 2m
labels:
severity: warning
annotations:
summary: "High read latency detected"
description: "99th percentile read latency is {{ $value }}us on {{ $labels.instance }}"
- alert: ScyllaLowDiskSpace
expr: scylla_node_filesystem_avail_bytes / scylla_node_filesystem_size_bytes < 0.1
for: 1m
labels:
severity: critical
annotations:
summary: "Low disk space on ScyllaDB node"
description: "Available disk space is below 10% on {{ $labels.instance }}"
- alert: ScyllaCompactionBacklog
expr: scylla_compaction_manager_pending_tasks > 100
for: 10m
labels:
severity: warning
annotations:
summary: "High compaction backlog"
description: "Compaction backlog has {{ $value }} pending tasks on {{ $labels.instance }}"
- alert: ScyllaHighMemoryUsage
expr: scylla_memory_allocated_bytes / scylla_memory_total_bytes > 0.9
for: 5m
labels:
severity: critical
annotations:
summary: "High memory usage on ScyllaDB node"
description: "Memory usage is {{ $value | humanizePercentage }} on {{ $labels.instance }}"
- alert: ScyllaTimeouts
expr: rate(scylla_storage_proxy_coordinator_read_timeouts_total[5m]) > 1
for: 2m
labels:
severity: warning
annotations:
summary: "High timeout rate detected"
description: "Read timeout rate is {{ $value }}/sec on {{ $labels.instance }}"
- alert: ScyllaErrorRate
expr: rate(scylla_storage_proxy_coordinator_read_errors_total[5m]) > 0.1
for: 1m
labels:
severity: critical
annotations:
summary: "High error rate detected"
description: "Read error rate is {{ $value }}/sec on {{ $labels.instance }}"
- alert: ScyllaStreamingErrors
expr: rate(scylla_streaming_total_incoming_bytes[5m]) == 0 and scylla_node_operation_mode{mode="NORMAL"} == 1
for: 15m
labels:
severity: warning
annotations:
summary: "No streaming activity detected"
description: "No incoming streaming detected on {{ $labels.instance }} during repair/bootstrap"
- alert: ScyllaLargePartitions
expr: scylla_large_partition_exceeding_threshold_total > 0
for: 1m
labels:
severity: warning
annotations:
summary: "Large partitions detected"
description: "{{ $value }} large partitions found on {{ $labels.instance }}"
- alert: ScyllaGCPressure
expr: rate(scylla_memory_free_bytes[5m]) < 0
for: 5m
labels:
severity: warning
annotations:
summary: "Memory pressure detected"
description: "Decreasing free memory trend on {{ $labels.instance }}"
- alert: ScyllaConnectionErrors
expr: rate(scylla_cql_connections_rejected_total[5m]) > 1
for: 2m
labels:
severity: critical
annotations:
summary: "High connection rejection rate"
description: "CQL connection rejection rate is {{ $value }}/sec on {{ $labels.instance }}"
- alert: ScyllaRepairProgress
expr: scylla_repair_segment_total == 0 and on(instance) scylla_node_operation_mode{mode="NORMAL"} == 1
for: 24h
labels:
severity: warning
annotations:
summary: "No repair activity in 24 hours"
description: "Node {{ $labels.instance }} has not run repair in over 24 hours"
Create Prometheus systemd service
Set up Prometheus as a systemd service for automatic startup and management.
[Unit]
Description=Prometheus
Wants=network-online.target
After=network-online.target
[Service]
User=prometheus
Group=prometheus
Type=simple
ExecStart=/opt/prometheus/prometheus \
--config.file=/opt/prometheus/prometheus.yml \
--storage.tsdb.path=/opt/prometheus/data \
--web.console.templates=/opt/prometheus/consoles \
--web.console.libraries=/opt/prometheus/console_libraries \
--web.listen-address=0.0.0.0:9090 \
--web.enable-lifecycle
[Install]
WantedBy=multi-user.target
Install and configure Grafana
Install Grafana for creating dashboards and visualizations of ScyllaDB metrics.
sudo apt install -y software-properties-common
wget -q -O - https://packages.grafana.com/gpg.key | sudo apt-key add -
echo "deb https://packages.grafana.com/oss/deb stable main" | sudo tee /etc/apt/sources.list.d/grafana.list
sudo apt update
sudo apt install -y grafana
sudo tee /etc/yum.repos.d/grafana.repo <
Start monitoring services
Enable and start both Prometheus and Grafana services.
sudo mkdir -p /opt/prometheus/data
sudo chown prometheus:prometheus /opt/prometheus/data
sudo systemctl daemon-reload
sudo systemctl enable --now prometheus
sudo systemctl enable --now grafana-server
Configure Grafana data source
Add Prometheus as a data source in Grafana and import ScyllaDB dashboards.
curl -X POST http://admin:admin@localhost:3000/api/datasources \
-H "Content-Type: application/json" \
-d '{
"name": "Prometheus",
"type": "prometheus",
"url": "http://localhost:9090",
"access": "proxy",
"basicAuth": false
}'
Import ScyllaDB dashboard
Create a comprehensive dashboard for ScyllaDB cluster monitoring with key performance indicators.
curl -X POST http://admin:admin@localhost:3000/api/dashboards/db \
-H "Content-Type: application/json" \
-d @- <<'EOF'
{
"dashboard": {
"id": null,
"title": "ScyllaDB Cluster Overview",
"tags": ["scylla", "database"],
"timezone": "browser",
"panels": [
{
"id": 1,
"title": "Node Status",
"type": "stat",
"targets": [
{
"expr": "up{job=\"scylla\"}",
"legendFormat": "{{instance}}"
}
],
"gridPos": {"h": 4, "w": 6, "x": 0, "y": 0}
},
{
"id": 2,
"title": "CPU Utilization",
"type": "graph",
"targets": [
{
"expr": "scylla_reactor_utilization",
"legendFormat": "{{instance}}"
}
],
"gridPos": {"h": 8, "w": 12, "x": 0, "y": 4}
},
{
"id": 3,
"title": "Read Latency (99th percentile)",
"type": "graph",
"targets": [
{
"expr": "scylla_storage_proxy_coordinator_read_latency{quantile=\"0.99\"}",
"legendFormat": "{{instance}}"
}
],
"gridPos": {"h": 8, "w": 12, "x": 12, "y": 4}
},
{
"id": 4,
"title": "Write Latency (99th percentile)",
"type": "graph",
"targets": [
{
"expr": "scylla_storage_proxy_coordinator_write_latency{quantile=\"0.99\"}",
"legendFormat": "{{instance}}"
}
],
"gridPos": {"h": 8, "w": 12, "x": 0, "y": 12}
},
{
"id": 5,
"title": "Operations per Second",
"type": "graph",
"targets": [
{
"expr": "rate(scylla_cql_reads_total[5m])",
"legendFormat": "Reads - {{instance}}"
},
{
"expr": "rate(scylla_cql_inserts_total[5m])",
"legendFormat": "Writes - {{instance}}"
}
],
"gridPos": {"h": 8, "w": 12, "x": 12, "y": 12}
}
],
"time": {
"from": "now-1h",
"to": "now"
},
"refresh": "5s"
}
}
EOF
Install Alertmanager
Set up Alertmanager to handle alerts generated by Prometheus rules.
wget https://github.com/prometheus/alertmanager/releases/download/v0.26.0/alertmanager-0.26.0.linux-amd64.tar.gz
tar xzf alertmanager-0.26.0.linux-amd64.tar.gz
sudo mv alertmanager-0.26.0.linux-amd64 /opt/alertmanager
sudo useradd --system --shell /bin/false alertmanager
sudo chown -R alertmanager:alertmanager /opt/alertmanager
wget https://github.com/prometheus/alertmanager/releases/download/v0.26.0/alertmanager-0.26.0.linux-amd64.tar.gz
tar xzf alertmanager-0.26.0.linux-amd64.tar.gz
sudo mv alertmanager-0.26.0.linux-amd64 /opt/alertmanager
sudo useradd --system --shell /bin/false alertmanager
sudo chown -R alertmanager:alertmanager /opt/alertmanager
Configure Alertmanager
Set up email notifications for ScyllaDB alerts.
global:
smtp_smarthost: 'localhost:587'
smtp_from: 'alertmanager@example.com'
smtp_auth_username: 'alertmanager@example.com'
smtp_auth_password: 'your-email-password'
route:
group_by: ['alertname']
group_wait: 10s
group_interval: 10s
repeat_interval: 1h
receiver: 'web.hook'
routes:
- match:
severity: critical
receiver: 'critical-email'
- match:
severity: warning
receiver: 'warning-email'
receivers:
- name: 'web.hook'
webhook_configs:
- url: 'http://127.0.0.1:5001/'
- name: 'critical-email'
email_configs:
- to: 'admin@example.com'
subject: 'CRITICAL: ScyllaDB Alert - {{ .GroupLabels.alertname }}'
body: |
{{ range .Alerts }}
Alert: {{ .Annotations.summary }}
Description: {{ .Annotations.description }}
Instance: {{ .Labels.instance }}
Severity: {{ .Labels.severity }}
{{ end }}
- name: 'warning-email'
email_configs:
- to: 'monitoring@example.com'
subject: 'WARNING: ScyllaDB Alert - {{ .GroupLabels.alertname }}'
body: |
{{ range .Alerts }}
Alert: {{ .Annotations.summary }}
Description: {{ .Annotations.description }}
Instance: {{ .Labels.instance }}
{{ end }}
Create Alertmanager service
Set up Alertmanager as a systemd service.
[Unit]
Description=Alertmanager
Wants=network-online.target
After=network-online.target
[Service]
User=alertmanager
Group=alertmanager
Type=simple
ExecStart=/opt/alertmanager/alertmanager \
--config.file=/opt/alertmanager/alertmanager.yml \
--storage.path=/opt/alertmanager/data
[Install]
WantedBy=multi-user.target
Start Alertmanager
Enable and start the Alertmanager service.
sudo mkdir -p /opt/alertmanager/data
sudo chown alertmanager:alertmanager /opt/alertmanager/data
sudo systemctl daemon-reload
sudo systemctl enable --now alertmanager
Configure ScyllaDB monitoring agent
Install and configure the ScyllaDB monitoring agent for enhanced metrics collection.
wget https://github.com/scylladb/scylla-monitoring/archive/refs/tags/scylla-monitoring-4.5.0.tar.gz
tar xzf scylla-monitoring-4.5.0.tar.gz
sudo mv scylla-monitoring-4.5.0 /opt/scylla-monitoring
sudo chown -R prometheus:prometheus /opt/scylla-monitoring
Import advanced ScyllaDB dashboards
Import official ScyllaDB Grafana dashboards for comprehensive monitoring.
cd /opt/scylla-monitoring
sudo -u prometheus ./start-grafana.sh -s prometheus_servers.yml -n node_exporter_servers.yml -G
Verify your setup
Check that all monitoring components are running and collecting metrics properly.
sudo systemctl status prometheus
sudo systemctl status grafana-server
sudo systemctl status alertmanager
curl http://localhost:9090/api/v1/targets
curl http://localhost:3000/api/health
curl http://localhost:9093/api/v1/status
Access Grafana at http://your-server:3000 (admin/admin) and verify that ScyllaDB metrics are being collected. Check the dashboard shows current cluster status and performance metrics.
Common issues
SymptomCauseFix
No metrics from ScyllaDBWrong port or endpointVerify ScyllaDB metrics endpoint: curl http://node:9180/metrics
Prometheus can't scrape targetsFirewall blocking accessOpen port 9180: sudo ufw allow 9180
Grafana shows no dataData source not configuredCheck Prometheus data source URL in Grafana settings
Alerts not firingAlertmanager not connectedVerify Alertmanager target in Prometheus: http://localhost:9090/alerts
Dashboard import failsJSON format errorUse Grafana UI to import dashboard ID 9614 for ScyllaDB
High memory usageToo many metrics retainedAdjust Prometheus retention: --storage.tsdb.retention.time=30d
Next steps
Configure ScyllaDB SSL encryption and authentication for production security
Configure Prometheus Alertmanager with Slack integration for team notifications
Setup ScyllaDB backup automation with S3 integration
Implement ScyllaDB performance tuning and optimization
Configure advanced Grafana dashboards and alerting with custom metrics
Running this in production?
Need help with the operational load? Setting this up once is straightforward. Keeping it patched, monitored, backed up and performant across environments is the harder part. See how we run infrastructure like this for European SaaS and e-commerce teams.
---
## Contact
Binadit B.V., Seinhuiswachter 2, 3034 KH Rotterdam, Netherlands
Email: contact@binadit.com · Phone: +31 10 477 5362
KvK 80923216 · VAT NL861852990B01
Content may be cited by AI assistants with canonical URL attribution.