I disagree with that. Apache's defaults on most distros isn't secure. Even 'enterprise' level distributions often fall into the following traps when packaging Apache:
1-> auto indexing enabled (should be disabled)
2-> user directories enabled (should be disabled)
3-> server signatures 'on' (should be 'off')
4-> server tokens set to 'full' (should be 'prod')
5-> hidden (dot prefixed) files not always blacklisted as unauthorised files in Apache config
6-> same as above for editor specific back up files (eg file~). But this is only an issue for people who bulk upload or edit files on the live server (very naughty).
7-> having less optimal SSL configurations (Apache's default SSL set up isn't PCI compliant).
(IIRC there's a couple of other Apache tweaks, but that's just off the top of my head)
Then you have issues with PHP (eg logging to STDOUT so web users can view PHP errors).
And that's just covering the webserver configuration. There's still holes that need plugging if you actually want to run a secure web server; one of my favourite tools for that is using fail2ban which will auto-blacklist IPs in iptables (Linux firewall) based on if certain attacks are detected. For example it can prevent brute force attacks on SSH brute force attacks (which negates the need to run denyhosts as recommended in the article), HTTP auth, FTP, mail servers, etc.
And touching on SSH, many data centres will enable root log ins on their OS pre-installs by default. Your first job on any such box will be creating a user account and then disabling root log ins (PermitRootLogin no -> /etc/ssh/sshd_config).
Then there's a whole stack of optimisations you can run on iptables to adaptively prevent port scanning, forged TCP/IP packets and such like.
Also many distributions ship FTP, which is insecure by default. Anyone sys admin requiring FTP hosting would be better off using a chrooted SFTP environment (so the security benefits of SSH plus the sandboxed (chroot) environment that FTP often benefits).
Let's also not forget the plethora of internal services (databases, MTAs (mail servers), etc). If you're reading this article then you're probably not a trained systems administrator or not working for a large enough company where each server is running in it's own environment. Thus your MySQL / Postgre / whatever databases are running on your webservers. So you need to ensure that your database is only listening on localhost (127.0.0.1). Same applied for your MTA, unless you are intentionally providing a mail services (it is advisable to have an MTA installed even if you're not providing mail services because you can then set up daemons like fail2ban to automatically e-mail you when attempted break ins happen; which will in turn allow you to spot any determined hackers and permanently ban their IP from your box).
And lastly, if you're really paranoid, run your webserver inside a Linux/UNIX container (FreeBSD Jail, Solaris Zone, Linux OpenVZ) instead of inside a virtual machine as there have been hacks where attackers can break out of the VM and gain access to the host system (as of yet, I've not heard such attacks from containers, but if anyone has any evidence of that then I'd love to know). Running inside a container will give your webserver the sandboxed environment to work with plus an easy route for backups / snapshotting meaning minimal downtime and data loss should the worst happen and your server does become compromised.
(source: http://www.youtube.com/watch?v=hCPFlwSCmvU - it's a plugged vulnerability, but it will give you an idea about the kind of issues VMs face when compared to containers)
In short; distribution's don't ship secure defaults. They ship a happy medium between usability and security. However if you have an internet facing box (particularly one not hidden behind a hardware firewall, as many budget set ups are not), then the happy medium isn't secure enough.
I'd say that most of your points are debatable. They can be divided into two groups:
1) Changes that give you relatively little gain from a security point of view at the cost of massive usability. I'd be happy to run production servers without these changes, and so I claim that these are debatable.
2) Changes that stop inexperienced admins accidentally compromising their servers, at the cost of usability/complexity. Debatable because they're less about security and more about not shooting yourself in the foot. Eg. fail2ban. I use ssh keys only. Brute forcing my ssh won't work, and I'd prefer to not allow outsiders to cause firewall rule changes on my servers based on a script that parses log strings with regular expressions. I trust ssh's security more than I trust fail2ban in not having a vulnerability.
> And lastly, if you're really paranoid, run your webserver inside a Linux/UNIX container
This is pretty much what AppArmor hardening does. LXC (Linux containers) are implmented via AppArmor. A compromised httpd daemon restricted in this way won't be able to much anyway (eg. open outbound ports or go to other areas of the filesystem). And this is what Ubuntu ships by default for many daemons.
> Also many distributions ship FTP, which is insecure by default.
Nowadays this stuff is only installed if you requested it, which is different from "ship"ing it. If you use distribution defaults, you won't have an FTP server installed. If you choose to install one, you'll get the weakness whether you use the distribution, tune it yourself or install from a third party source.
> In short; distribution's don't ship secure defaults. They ship a happy medium between usability and security. However if you have an internet facing box (particularly one not hidden behind a hardware firewall, as many budget set ups are not), then the happy medium isn't secure enough.
Secure enough for production. I have plenty of servers on distribution defaults that haven't been compromised (and I have experience in dealing with others' servers when they have been compromised, so I don't think I'm oblivious). You may prefer adding additional hardening, but it is unnecessary and relies on you knowing what you are doing.
Finally, you've missed my biggest point. For someone new to running servers on the Internet, who can he trust for guidance? You, me, a random guide on the Internet or a distribution vendor?
I don't think you've really read through the points I was raising as the vast majority of them were relating to hiding server based information from unauthorised users and preventing brute force attacks. Neither of those two have any impact on usability what-so-ever (that is, unless your 'user' is an attacker lol).
If you don't mind, I will address your points individually:
"Changes that give you relatively little gain from a security point of view at the cost of massive usability. I'd be happy to run production servers without these changes, and so I claim that these are debatable."
Which items specifically prevents usability? None of them do aside the FTP (but SFTP is supported by nearly every FTP client already) and turning off PHP error logging to STDOUT (really not recommended for production servers!). Everything else doesn't reduce usability for authorised users. And I resent the claim that there's little gain as even on my servers which have a low public profile, I get reports of multiple brute force attacks a week (on average, about separate 5 attacks a week, all of which I manually add to a permanent firewall blacklist when it becomes obvious that they're just retrying attacks everytime fail2ban's autoban expires).
"Changes that stop inexperienced admins accidentally compromising their servers, at the cost of usability/complexity. Debatable because they're less about security and more about not shooting yourself in the foot. Eg. fail2ban. I use ssh keys only. Brute forcing my ssh won't work, and I'd prefer to not allow outsiders to cause firewall rule changes on my servers based on a script that parses log strings with regular expressions. I trust ssh's security more than I trust fail2ban in not having a vulnerability."
You don't understand how fail2ban works if that's your primary concern. Fail2ban isn't public facing, it just monitors logs and then adds an iptables (or any firewall you chose) rules based on the results of the log files. It's iptables that is public facing and thus needs to be protected against vunrabilities; and iptables is proven technology already. What's more, preventing brute force attacks doesn't reduce usability nor prevent you from shooting yourself in the foot; preventing brute force attacks is the !!bareminimum!! you need to do if you have an internet facing server. This is security 101! I do agree with you that SSH keys are preferable to password log ins, but that also adds complexity and can reduce usability if users need access from multiple locations and platforms (there are workarounds, eg memory stick with the key on), and then you still have the issue of brute force attacks on every other log in service.
"This is pretty much what AppArmor hardening does. LXC (Linux containers) are implmented via AppArmor. A compromised httpd daemon restricted in this way won't be able to much anyway (eg. open outbound ports or go to other areas of the filesystem). And this is what Ubuntu ships by default for many daemons."
You missed the point about easy snapshotting though, which is why I raised the point about containers in the 1st place. I will grant you that you can have back up solutions on bare metal systems, but snapshotting is way more usable (which is ironic given that's been your key argument). Plus I did say that advice was for the paranoid (ie not really all that necessary unless you fancy tinkering).
"Nowadays this stuff is only installed if you requested it, which is different from "ship"ing it. If you use distribution defaults, you won't have an FTP server installed. If you choose to install one, you'll get the weakness whether you use the distribution, tune it yourself or install from a third party source."
You're just reiterating what I said though as I didn't say distributions install it by default. However you missed my point there as well, as I was describing how FTP isn't ideal for production systems (FTP sends clear text passwords and doesn't behave itself behind firewalls and/or NATing without adaptive routing (which means that FTPS often doesn't work behind some firewalls).). Thus chrooted SFTP is a much saner solution for production systems.
"Secure enough for production. I have plenty of servers on distribution defaults that haven't been compromised (and I have experience in dealing with others' servers when they have been compromised, so I don't think I'm oblivious). You may prefer adding additional hardening, but it is unnecessary and relies on you knowing what you are doing."
You've been lucky. But what your advocating is little more than 'security by obscurity' rather than pro-actively hardening your box. Having worked on a number of high profile web infrastructures (including farms that take web payments), I wouldn't be doing my job if I ignored my steps; in fact our business wouldn't exist as we would break UK laws.
The Apache configuration I described is ridiculously easy to set up, as is fail2ban. They are the bare minimum any web server should implement. Which is why I placed emphasis on them by placing those suggestions at the top of my post. However if we're going to advocate complacency in order to make our lives easy, then lets also not bother keeping our software up to date, because that's a real pain in the arse at times :p
Getting reports of brute force attacks is useful, but is not an indication of bad security. Hiding the reports under the carpet does not increase security either.
Preventing brute force attacks is not the bare minimum. The bare minimum is to not be vulnerable to brute force attacks in the first place. ssh already has built in protection for this, and it slows brute force attacks to the point where such an attack is impossible in practice, unless you have weak passwords. Mandate large enough keys only, ban passwords, and you're done. Allow password authentication is your real security vulnerability here. Patching over it with fail2ban just hides the issue.
I do understand how fail2ban works. "Just" monitoring logs isn't good enough. The data that appears in logs is not generally considered to be a security sensitive channel. It's string data with poorly defined delineation. It should not be trusted for automatic use, since every channel that dumps data into log files is not vetted for security.
> You've been lucky.
I disagree. You claim I'm advocating security by obscurity, but you're the one who seems to think that hiding version strings gains in security. That's security by obscurity, since you can determine the version of software by observing its behaviour (or just not caring and trying your attack anyway).
I'm not advocating complacency. We just disagree on what complacency is. I claim that if you keep your software up to date (easiest if you do follow distribution defaults), then you are sufficiently secure. The overwhelming majority of security compromises happen because people fail to run updates. The second largest cause is because of vulnerable misconfigurations that people have introduced. Only a tiny fraction of compromises come through a default distribution installation that your hardening would catch, and these holes are rapidly patched by vendors and a simple update will close them. I think that it is much more likely that you'll open the second cause (an introduced misconfiguration) if you try hardening and you don't know what you're doing. Which is why I recommend sticking to distribution defaults.
"Getting reports of brute force attacks is useful, but is not an indication of bad security. Hiding the reports under the carpet does not increase security either."
I didn't say reports increase security, and Fail2ban does more than just reporting, it actively blocks brute force attacks. It's a bit difficult us discussing the merits of certain security measures when you keep focusing on the irrelevant as if those were my security suggestions - it's almost as if you're trying to 'death by a thousand paper cuts' my whole post simply to win an internet argument <_<
"Preventing brute force attacks is not the bare minimum. The bare minimum is to not be vulnerable to brute force attacks in the first place. ssh already has built in protection for this, and it slows brute force attacks to the point where such an attack is impossible in practice, unless you have weak passwords. Mandate large enough keys only, ban passwords, and you're done. Allow password authentication is your real security vulnerability here. Patching over it with fail2ban just hides the issue."
For SSH, you'd be right. But as I've repeatedly said, not all services offer key based log ins and sometimes there's a business requirement for password log ins on systems that could be managed with keys. You're original arguments were about usability yet the arguments you're making now are the lest flexible suggestions raised thus far!
"I do understand how fail2ban works. "Just" monitoring logs isn't good enough. The data that appears in logs is not generally considered to be a security sensitive channel. It's string data with poorly defined delineation. It should not be trusted for automatic use, since every channel that dumps data into log files is not vetted for security."
Logs are fine for parsing as you have to be compromise before the logs are comprimised. But which point, it's already too late.
"I disagree. You claim I'm advocating security by obscurity, but you're the one who seems to think that hiding version strings gains in security. That's security by obscurity, since you can determine the version of software by observing its behaviour (or just not caring and trying your attack anyway)."
In theory I'd agree with you, however a great number of compromised systems were attacked by opportunists scanning version numbers looking for boxes to target with known vulnerabilities. Plus, and once again I'm having to repeat myself, those specific changes I advised are actually required to comply with many compliance laws (eg PCI compliance, required if you make finance transactions in the UK).
"I'm not advocating complacency. We just disagree on what complacency is. I claim that if you keep your software up to date (easiest if you do follow distribution defaults), then you are sufficiently secure. The overwhelming majority of security compromises happen because people fail to run updates."
There's no such thing as 'sufficiently secure' as that only depends on the attackers targeting your system. Today you might be 'sufficiently secure' because your box has not been spotted by any keen attackers, tomorrow might be different.
Also, I'd be more inclined to agree with you if all of the examples you've given weren't off topic from the points I raised or just incorrect (eg changing Apache config being dangerous and/or hard, fail2ban reducing usability, etc).
"The second largest cause is because of vulnerable misconfigurations that people have introduced."
I'd go along with that. I've often said 'users are the biggest security risks' :)
"Only a tiny fraction of compromises come through a default distribution installation that your hardening would catch, and these holes are rapidly patched by vendors and a simple update will close them."
None of the configurations I mentioned (bar the list of paranoid ones) fall into that category; non-optimal configuration isn't a hole that gets patched. Plus even if it was, it wouldn't be fixed with software updates as package managers tend to avoid over-writing live config files else they'd risk doing more damage than good.
"I think that it is much more likely that you'll open the second cause (an introduced misconfiguration) if you try hardening and you don't know what you're doing. Which is why I recommend sticking to distribution defaults."
You can't make a system less secure by changing the settings I recommended as the distro defaults are already on the most open defaults. Plus, and once again I'm repeating myself, the configurations I'm recommending are incredibly easy to implement.
I find it odd that we're actually arguing about whether it's worth making the most basic of changes based on the assumption that those people in question are stupid and their box will probably be ok. Surely a better approach would be to suggest optimisations; guiding them through the process if needs be? After all, it's too late to regret using the defaults if and when you get hacked (and in my line of work, I've had to fix quite a number of boxes where the sys admins have been content just running with the default settings).
"I find it odd that we're actually arguing about whether it's worth making the most basic of changes based on the assumption that those people in question are stupid and their box will probably be ok. Surely a better approach would be to suggest optimisations -guiding them through the process if needs be."
Not unless we're people with excellent reputations that "those people" can recognise, or it is possible to determine that people with these excellent reputations endorse our advice. Otherwise we're just adding to the muddle of information on the Internet, some of which is bad, some of which is good, and it is impossible for non-experts to tell the difference.
My argument is that the distribution is such a reputable source, and that a random article upvoted on HN isn't. If a distribution ships insecure defaults, then you should petition them to fix the defaults rather than publishing "fixes" elsewhere. You'll have to fight your corner, of course, against a bunch of people who might differ from you in your opinion about what is and isn't secure :)
comment out Include conf/extra/httpd-autoindex.conf (line 101), and as you're now no longer using it, it might also be worth taking out autoindex_module from your LoadModule's (saves a small bit of memory, but there wouldn't be any noticeable performance benefits. But as you're not using it, there's no point loading it).
For SSL PCI compliance, have the following config as part of your SSL settings (which you've got commented out currently):
SSLHonorCipherOrder On
SSLCipherSuite ECDHE-RSA-AES128-SHA256:AES128-GCM-SHA256:RC4:HIGH:!MD5:!aNULL:!EDH
SSLProtocol -ALL +SSLv3 +TLSv1 +TLSv1.1 +TLSv1.2
This should force Apache not to default to older insecure SSL protocols and disable SSL compression (HTTP compression via mod_deflate still works here) which leaves HTTPS open to attacks like BEAST.
Bare in mind I'm still testing the above code myself (funny enough, that's actually what I'm doing this very minute) as the BEAST vulnerability is still relatively new (or rather, new enough where it wasn't part of PCI compliance until the last month or so). I'll update this thread in the next few hours if that code doesn't work, but I can't see there being a problem as it follows the standards defined in Apache's manual.
Also make sure you have OpenSSL version 1.0.1 installed (required for TLS1.1 & 1.2). You can check this by running: openssl version from the command line. However if your system is built from a package manager and has been kept relatively up to day, then you shouldn't have a problem there.
Well, rlpb's fundamental point is quite valid: Vendors have done tremendous work in making an OS secure out-of-the-box. I remember exposing default installations to the public Internet a decade or more ago and watching them get compromised in minutes, if not seconds. Virtually every platform required hardening, and the only thing protecting most ordinary users was the fact that exposure was intermittent via expensive, but quite public, dialup connections. Vendors began taking security more seriously by providing sensible defaults, but it's an ongoing battle with zero-day exploits being discovered far too frequently.
indeed. Windows has come a long way too. I'm not the biggest fan of Windows (strictly personal preference) but I used to love Windows 2000. It was clean, simple yet had a lot of power behind it. Yet Microsoft left telnet turn on by default (and this was back before most homes had routes and thus hardware firewalls).
It was an amazingly short sighted move, but such things were typical back then. And it's only from learning the hard way that we've managed to get to the stage we're at now.
However I think it's often forgotten that servers need a different set of security profiles depending on the server's role and where it is sat. For example, a webserver sat behind a hardware load balancer wouldn't necessarily need much SSH protection as the webfarm HTTP traffic should be on a different VLAN to the internal systems administration traffic (which in turn, would be another different VLAN to the company's staff VLAN). So it would be almost impossible to get access to an OpenSSH log in, let alone attack it. Where as most consumer VPS solutions put all their customer servers in the DMZ, which means it's up to the customer to provide software preventions to harden against access that would normally be protected with a complex hardware solution in more professional / clustered set ups.
And this is why you can't fully trust default configs; there simply is no "one size fits all" solution so package maintainers instead opt for the best compromises.
1-> auto indexing enabled (should be disabled)
2-> user directories enabled (should be disabled)
3-> server signatures 'on' (should be 'off')
4-> server tokens set to 'full' (should be 'prod')
5-> hidden (dot prefixed) files not always blacklisted as unauthorised files in Apache config
6-> same as above for editor specific back up files (eg file~). But this is only an issue for people who bulk upload or edit files on the live server (very naughty).
7-> having less optimal SSL configurations (Apache's default SSL set up isn't PCI compliant).
(IIRC there's a couple of other Apache tweaks, but that's just off the top of my head)
Then you have issues with PHP (eg logging to STDOUT so web users can view PHP errors).
And that's just covering the webserver configuration. There's still holes that need plugging if you actually want to run a secure web server; one of my favourite tools for that is using fail2ban which will auto-blacklist IPs in iptables (Linux firewall) based on if certain attacks are detected. For example it can prevent brute force attacks on SSH brute force attacks (which negates the need to run denyhosts as recommended in the article), HTTP auth, FTP, mail servers, etc.
And touching on SSH, many data centres will enable root log ins on their OS pre-installs by default. Your first job on any such box will be creating a user account and then disabling root log ins (PermitRootLogin no -> /etc/ssh/sshd_config).
Then there's a whole stack of optimisations you can run on iptables to adaptively prevent port scanning, forged TCP/IP packets and such like.
Also many distributions ship FTP, which is insecure by default. Anyone sys admin requiring FTP hosting would be better off using a chrooted SFTP environment (so the security benefits of SSH plus the sandboxed (chroot) environment that FTP often benefits).
Let's also not forget the plethora of internal services (databases, MTAs (mail servers), etc). If you're reading this article then you're probably not a trained systems administrator or not working for a large enough company where each server is running in it's own environment. Thus your MySQL / Postgre / whatever databases are running on your webservers. So you need to ensure that your database is only listening on localhost (127.0.0.1). Same applied for your MTA, unless you are intentionally providing a mail services (it is advisable to have an MTA installed even if you're not providing mail services because you can then set up daemons like fail2ban to automatically e-mail you when attempted break ins happen; which will in turn allow you to spot any determined hackers and permanently ban their IP from your box).
And lastly, if you're really paranoid, run your webserver inside a Linux/UNIX container (FreeBSD Jail, Solaris Zone, Linux OpenVZ) instead of inside a virtual machine as there have been hacks where attackers can break out of the VM and gain access to the host system (as of yet, I've not heard such attacks from containers, but if anyone has any evidence of that then I'd love to know). Running inside a container will give your webserver the sandboxed environment to work with plus an easy route for backups / snapshotting meaning minimal downtime and data loss should the worst happen and your server does become compromised.
(source: http://www.youtube.com/watch?v=hCPFlwSCmvU - it's a plugged vulnerability, but it will give you an idea about the kind of issues VMs face when compared to containers)
In short; distribution's don't ship secure defaults. They ship a happy medium between usability and security. However if you have an internet facing box (particularly one not hidden behind a hardware firewall, as many budget set ups are not), then the happy medium isn't secure enough.