Yep, we are pretty much alive. I just had a quick look and I personally got the feeling... we are dead. So I decided to write a quick post, just to make sure this is not true.
We had hard times (company wise), hopefully all bad news are in the past. We finally got the time and energy to continue the work on our primary goal - e107.org and brand new e107 version. The result should be visible in weeks and yes, this is true, I never write it otherwise. I hope the future changes and improvements will bring the so long waited positivism in our community and end up the exhausting process of 'doing everything alone' process. I also hope this will be the start of so long waited way up for e107. We, as persons and company, who have put so much energy and effort in just an idea, are eager to see e107 as a competitor of all world leading CMS systems. Actually I believe this will happen.
I also want to thank for the great support of our FS Net and e107.org community - we had a great time with you, and I believe the best is yet to come.
Installatron, a leader in web application automation, today announced the availability of the e107 content management system, through Installatron Applications Installer, its multi-platform web application auto-installer and auto-upgrader software.
Installatron Applications Installer is the leading multi-platform web application auto-installer and auto-upgrader plugin for the cPanel/WHM, DirectAdmin, Plesk Linux, Plesk Windows, and Kloxo/LxAdmin web-hosting control panels, and a multi-user integration API is available for any hosting control panel or graphical user interface. Installatron Applications Installer was launched in 2004.
For more information about Installatron, including a live demo and a free version, please visit: http://www.installatron.com
We all were witnesses of a great panic and a lot of accusations of how weak and non-safe is e107 recently. The reason was a massive attack against most of the e107 based (community and non-community) sites and e107.org itself.
Let's summarize the facts first.
It's true that a number of security holes was recently reported, and some of them were bad - really bad. I'll write in a separate post the sad story behind some of the reports, which caused SO MUCH damage to a lot of e107 based sites and hosting companies (sad, because it was done by PHP security adviser owner of php-security.org - a popular person - who acted as a teenager - understand totally unprofessional). I'm pretty much sure this story triggered the start of the attack against e107 but this is not a subject of this article.
The reaction of the core development team was fast enough - we did quick fixes, we released number of quick patches and security releases. We were already prepared with a notification system which delivered the information about the recent available security patches direct to site owners' administration area. We also published a lot of information (news on e107.org), all the information we got. I don't think the whole could have been done better.
The panic came AFTER the last security patch. It was caused by a number of bot attacks which were trying to go through an ALREADY PATCHED security hole (the so popular recently contact.php). Bots were following (exactly) the INSTRUCTIONS pointed in one of the advisories - php-security.org/2010/05/19/mops-2010-035-e107-bbcode-remote-php-code-execution- vulnerability/index.html (do you remember the sad story I was talking about earlier?)
What exactly happened?
The bad guys used the fact that this vulnerability was first published and spread around the World Wide Web without the knowledge of e107 core development team. They attacked and infected servers (not secured enough - see below). All infected servers was made attackers. They infected more servers, etc.
The number of infected servers was low at the beginning of the attack - at this time patch was already available. The problem is people do not pay enough attention on this even if shown on the top of their site administration panel, so patching required time. Those who were late, lost the game. Unfortunately those who weren't late, didn't win the game. They were attacked by all infected servers. Although the hole was patched, attacks became DDOS. They affected all servers which can't handle such attacks.
What can do e107 CMS about all this?
Near nothing, nor any other CMS/PHP script can do more. e107 core development team could do one thing only - security patch, preventing php code execution in this particular case. Neither development team nor PHP itself are able to help you stop the requests to your site. I hope everyone understand this.
I saw all kind of advises as 'delete contact.php', 'add .htaccess rules', etc. This could work in very short term.
Here are blocked requests from the logs on this server:
As you see, there is only one solution - server security software.
I can point server owners/shared hosting companies to a solution based on our current experience. It involves an installation of a free software package on a Linux web server, best suitable for RedHat or derivative Linux platforms (read more details about different software requirements below). The solution is fully integrated with Cpanel/WHM server management system.
Additionally, the company behind some applications of this package is offering low cost Cpanel configuration services which allows you to order installation and configuration of all required software on a dedicated server(s) for a very fair price. Learn more about the ConfigServer Cpanel services.
I'll try to explain how the whole is working in few words.
Real time protection (active scanning)
Requests to the server are being analyzed (e.g mod_security rules, suhosin) and temporary blocked if recognized as a hacking attempt. Temporary blocked IPs doesn't have access to the server for 300 seconds (default value). Additionally, number of temporary blocks are resulting in permanent block (csf.deny file). Permanent blocks are added to the server's iptables. The default max number of permanent blocked IPs is 100. You could increase the number if required but be careful:
DENY_IP_LIMIT configuration variable comments wrote ...
This can be important as a large number of IP addresses create a large number of iptables rules (4 times the number of IP's) which can cause problems on some systems where either the number of iptables entries has been limited (esp VPS's) or where resources are limited.
You should find the appropriate number based on your server resources and the strength of the (eventually) current attack against your server.
Server monitoring (on-demand scanning)
ConfigServer Firewall comes with Login Failure Daemon (lfd) which effectively blocks IPs based on the number of login failures. You should start number of cron jobs for various checks (e.g. rkhunter, Chkrootkit which is not listed here etc). In the end you should be able to be informed about the state of your server just reading email reports once per day. eXploit Scanner (if installed) will add additional server monitoring (beside its active scanning, it's able to perform scanning of files, directories and user accounts for suspected exploits, viruses and suspicious resources).
Just a side note - I'm not saying people who has the software below will be 100% secure. There is no such a thing when you are plugged-in to Internet. I'm just saying you'll be as much secured as possible, it'll work for you in most if not in all cases.
This article is not intending to explain How to protect yourself (installation/configuration instructions) but what to install to be protected. Links to appropriate resources are provided together with every product listed below.
Atomic ModSecurity & ModSecurity Rules
Overview and installation instructions: http://www.configserver.com/cp/csf.html
Additional help by configuring the rules: http://www.atomicorp.com/wiki/index.php/Mod_security
This is a part of the Atomic Secured Linux (ASL) project.
Mod Security is an open-source web application firewall. Atomic Rules are set of predefined mod_security rules which are updated on a regular basis.
For a Cpanel users, the best option is to install mod_security via the Easyapache module. See below why.
ConfigServer ModSecurity Control (cmc)
Product page: http://www.configserver.com/cp/cmc.html
The product provides you with an interface to the cPanel mod_security implementation from within WHM. It gives you additional GUI control over your ModSecurity installation.
Product page: http://www.hardened-php.net/suhosin/
Suhosin is an advanced protection system for PHP installations. It is designed to protect servers and users from known and unknown flaws in PHP applications and the PHP core.
iptables SPI firewall (ConfigServer Firewall - csf)
Product page: http://www.configserver.com/cp/csf.html
You'll find full feature list and all required installation instructions.
You pretty much need 3-4 shell lines:
rm -fv csf.tgz wget http://www.configserver.com/free/csf.tgz tar -xzf csf.tgz cd csf sh install.sh
Product page: http://www.clamav.net/lang/en/
It's an open source (GPL) anti-virus toolkit for UNIX, nothing more or less.
ConfigServer eXploit Scanner (cxs)
Product page: http://www.configserver.com/cp/cxs.html
This is the only non-free product in this list. You may or may not buy & install it. I suggest you install it.
It does active scanning of files as they are uploaded to the server. It has Cpanel GUI control panel.
Product page: http://www.mailscanner.info/intro.html
MailScanner is an e-mail security and anti-spam package for e-mail gateway systems. It supports Clam Anti Virus.
Product page: http://www.rootkit.nl/projects/rootkit_hunter.html
Rootkit scanner is scanning tool to ensure you for about 99.9%* you're clean of nasty tools. This tool scans for rootkits, backdoors and local exploits by running various tests.
I'll update this section in the future with information about what could break your e107 site installations and how it can be tweaked/re-configured.
Summary - be smart, not paranoid
Panic creates paranoid thoughts. You need to find the balance between security and flawless working web applications on your server(s). People using hosting services want to be secure and not to block the access of half of the world to their sites. I'll try to keep an updated list here in the future of common needed for e107 installation configuration changes needed to make the CMS and its plugins work well (and not to block/ban innocent visitors).
Anyway, the idea is you should be safe enough against both exploit & DOS attacks. I can confirm that the access the contact form on FS is blocked (403 error) very efficient from mod_security currently. There are many blocked daily attempts and no CPU overload on this server. This is the reason I decided to share all this. It may (it should) work for you as well.
Those of you who are using the 'PHP development tools' (PDT ) plugin for Eclipse should know this is the most wanted ever feature from the PDT development team (Zend company). I think I know the reason why it's still not implemented - it's available for the commercial PDT analogue - Zend Studio for Eclipse.
I did a wide research half year ago and didn't find anything even close to useful. The breakthrough came a month ago. I call it pure luck - I don't know how I ended up to a site (free hosted) in totally illegible for me language (Chinese?) except it's title
Eclipse PDT (PHP Development Tools) - PHP Code Formatter Plugin (prototype)I was able to (almost) understand the rest thanks to Google Translate project.
However I wasn't able to find any reference to the person/team behind this page. If someone can give me information in this matter, please do it. I really wanna say "Thank you" to this guy.
To understand better how lucky I was, I'll tell you that some days later I was trying to open this page with no success - it was restricted to US IP's only (and I'm far from US located).
Aptana 2.+ (tested on 2.0.2) OR Eclipse 3.4+
Eclipse Web Tools Platform (WTP) 3.+ - tested on v3.1.1
Eclipse Dynamic Languages Toolkit (DLTK) 1.+ - tested on v1.0.1
PHP Development Tools (PDT) 2.+ - tested on v2.1.2
Project URL: http://atlanto.web.fc2.com/pdt/workshop/formatter_plugin.html
Direct download (free-source.net mirror) PDT 1.3: va000137.pdt.tools.formatter_0.92.4.jar
Direct download (free-source.net mirror) PDT 2.x: va000137.pdt.tools.formatter_0.92.4.v20081027.jar
Copy the downloaded .jar file to your Aptana/Eclipse 'dropin' folder (INSTALL_PATH/dropins), restart Aptana/Eclipse, done. Yep, so easy it is.
Now go to Preferences / PHP / Code Style / Formatter and set your preferred code styling options.
EDIT: Thanks to tgtje who pointed me to PDT Tools project on SourceForge Japan. You can download the most recent version of Code Formatter dropin from there and I can say: "Thank you so much atlanto from sourceforge.jp" :)
Our only wish is to stay close to the community, so we added one more option for 'real time discussions' to our site services - join our newly created channel #fsnet at irc.freenode.net
For those of you not familiar with IRC, here are some nice & popular IRC clients:
eCheck Seciruty is a tiny tool for detecting malicious PHP scripts and code portions on your website. It was originally build to check e107 CMS based sites, but it can be actually used on any kind of PHP based projects.
This tool is licensed under GNU General Public License - http://www.gnu.org/licenses/gpl.txt
Before you start using the tool, I have to warn you - DON'T PANIC when you first see the 'suspicious' results. Be sure you read the 'Analyzing the results' chapter.
Download most recent version of eCheck Seciruty here
Shell script (echeck.php)
Copy echeck.php somewhere on your server. In this example I'm copying it in /home/secretr/
[secretr@SecretR /]$ cd /home/secretr/ [secretr@SecretR ~]$ ./echeck.php -v eCheck 1.0 beta Report issues or get help on http://free-source.net or irc://irc.freenode.org/e107 [secretr@SecretR ~]$
You can always get quick help
[secretr@SecretR ~]$ ./echeck.php -v eCheck 1.0 beta Report issues or get help on http://free-source.net or irc://irc.freenode.org/e107 [secretr@SecretR ~]$ ./echeck.php -h This is a command line PHP script for checking for/cleaning PHP malicious code. Usage:./echeck.php [options] /path/to/wwwroot Options: -v Script version -I Output a list with infected files only -S Output a list with suspected files only -C Clean files (MAKE A BACKUP BEFORE DOING THIS), confirmation is required -r number Directory depth level [secretr@SecretR ~]$
Now, the only thing you need to know is the path to your web root (e107 root for e107 user). In my case this is /home/secretr/public_html and my e107 Installation is located in e107_0.7 folder. There are two alternatives. You could let eCheck know the path to your web root:
$ ./echeck.php -I -S ./public_html/e107_0.7/
or the opposite - navigate to web root and call the script with the proper path:
[secretr@SecretR ~]$ cd public_html/e107_0.7/ [secretr@SecretR e107_0.7]$ /home/secretr/./echeck.php -I -S ./
Here is the output of eCheck scan on fresh e107 v0.7 CVS copy:
[secretr@SecretR ~]$ ./echeck.php -I -S -r 10 ./public_html/e107_0.7/ Directory depth set to 10 ./public_html/e107_0.7/backend.php...SUSPECTED (shell execution) ./public_html/e107_0.7/e107_plugins/pdf/pdf.sc...SUSPECTED (shell execution) ./public_html/e107_0.7/e107_handlers/resize_handler.php...SUSPECTED (shell execution) Files checked: 1040 Files suspected: 3 Files infected: 0 Files cleaned: 0 Clean errors: 0 Clean warnings: 0 NOTE: SUSPECTED DOES N0T MEAN INFECTED! DIFF AGAINST TRUSTED COPY OF SUSPECTED FILES TO BE SURE EVERYTHING IS OK. SUSPECTED FILES ARE NOT CLEANED! [secretr@SecretR ~]$
There is (still experimental) cleanup option you could try if eCheck finds files marked as INFECTED. I recommend to make a backup of your files first. Additionally, you need write permission on all checked files (e.g. run eCheck as root) and your PHP version should be at least 5.0.
I'll put infected and real malicious files inside my local e107 system to show you what happens:
[secretr@SecretR ~]$ ./echeck.php -C -I -S ./public_html/e107_0.7/ Directory depth set to 100 Did you make a backup? Be sure you did it! Type 'yes' to continue:
You need to confirm (type yes and press enter) to continue the operation
[secretr@SecretR ~]$ ./echeck.php -C -I -S ./public_html/e107_0.7/ Directory depth set to 100 Did you make a backup? Be sure you did it! Type 'yes' to continue: yes ./public_html/e107_0.7/echeckwww.php...SUSPECTED (eval/base64_decode found) ./public_html/e107_0.7/backend.php...SUSPECTED (shell execution) ./public_html/e107_0.7/index.php...INFECTED...CLEANED ./public_html/e107_0.7/e107_plugins/pdf/pdf.sc...SUSPECTED (shell execution) ./public_html/e107_0.7/e107_files/public/shell.php...SUSPECTED (eval/base64_decode found) ./public_html/e107_0.7/e107_handlers/resize_handler.php...SUSPECTED (shell execution) Files checked: 1043 Files suspected: 5 Files infected: 1 Files cleaned: 1 Clean errors: 0 Clean warnings: 0 NOTE: SUSPECTED DOES NOT MEAN INFECTED! DIFF AGAINST TRUSTED COPY OF SUSPECTED FILES TO BE SURE EVERYTHING IS OK. SUSPECTED FILES ARE NOT CLEANED! [secretr@SecretR ~]$
Our index.php was infected with known infection, so eCheck was able to clean it. Note we have one new line - './public_html/e107_0.7/e107_files/public/shell.php'. We'll talk about this one later.
One last example - let's execute eCheck as root (your current user should be sudoer), output everything (all checked files) and write the output to a file - log.txt in our case.
[secretr@SecretR ~]$sudo ./echeck.php ./public_html/e107_0.7/ > ./log.txt [secretr@SecretR ~]$cat log.txt | more Directory depth set to 100 ./public_html/e107_0.7/install_.php....CHECKING...OK ./public_html/e107_0.7/user.php....CHECKING...OK ./public_html/e107_0.7/rate.php....CHECKING...OK ./public_html/e107_0.7/search.php....CHECKING...OK ./public_html/e107_0.7/online.php....CHECKING...OK ./public_html/e107_0.7/fpw.php....CHECKING...OK ./public_html/e107_0.7/print.php....CHECKING...OK ./public_html/e107_0.7/upload.php....CHECKING...OK ./public_html/e107_0.7/page.php....CHECKING...OK ./public_html/e107_0.7/links.php....CHECKING...OK ./public_html/e107_0.7/e107_languages/English/lan_notify.php....CHECKING...OK ./public_html/e107_0.7/e107_languages/English/lan_np.php....CHECKING...OK ./public_html/e107_0.7/e107_languages/English/lan_usersettings.php....CHECKING...OK ./public_html/e107_0.7/e107_languages/English/lan_membersonly.php....CHECKING...OK ./public_html/e107_0.7/e107_languages/English/lan_sitelinks.php....CHECKING...OK ./public_html/e107_0.7/e107_languages/English/lan_upload_handler.php....CHECKING...OK ./public_html/e107_0.7/e107_languages/English/lan_fpw.php....CHECKING...OK ./public_html/e107_0.7/e107_languages/English/lan_download.php....CHECKING...OK --More--
Scan via a browser (echeckwww.php)
For those who don't have shell access to their sites (most common case for shared hosting) there is an alternative.
Copy echeckwww.php to your site root (in my case /home/secretr/public_html/e107_0.7/) and just call it in your favorite browser like this:
You should see something like this (click to enlarge)
Keep in mind you don't have any options you can set in this case. Auto-clean is not available as well
Analyzing the results
Scripts are analyzed in two ways:
Suspected doesn't mean files are infected in some way. Most of the phrases (generic php functions) are used in all kind of software. The process of analyzing the results is your responsibility. If you know the structure of your site, and you have generic knowledge of 'what, where happens', it would be easy to identify the problems (if there are any).
I'll use the example above, more precisely this line from our latest shell example:
./public_html/e107_0.7/e107_files/public/shell.php...SUSPECTED (eval/base64_decode found)
Every e107 user should know that /e107_files/public/ folder should not contain any scripts. Experienced admins would know what to do from now on - checking the file last modified date and investigating the Apache logs to find out how was this file uploaded on the server, eventually reporting the problem to e107 core team.
In other hand we see
./public_html/e107_0.7/backend.php...SUSPECTED (shell execution) ./public_html/e107_0.7/e107_plugins/pdf/pdf.sc...SUSPECTED (shell execution) ./public_html/e107_0.7/e107_handlers/resize_handler.php...SUSPECTED (shell execution)
lines are appearing on and on. These are the false positives I'm talking about. You'll have many of them on a live site with a lot of 3rd party code. You just need to investigate all you see - it's pretty easy to distinguish malicious from creative code.
Where can I get help?