Archive for category PHP

Facebook Hacker Cup 2011 Qualification Round: the PHP code

I wrote about how I completed the 3 exercices of Facebook’s Hacker Cup 2011 qualification round in my previous posts (Double Squares, Peg Game and Studious Student) and now I’ll provide the code I wrote. Since the Facebook qualification round ended, I guess it’s safe to post, and will allow people who have kept their input and output to really confirm if they won or not (it seems some people receive mails saying they do, then saying they don’t… looks like the Hacker Cup was hacked together a bit too quickly).

As I said previously in the Double Squares post, the concept was not to make nice code or think for a long time, but complete the exercices as fast as possible to have a correct output before everyone else. Instead of optimizing the code and write nice long comments, I wrote code that would provide the correct output with the minimal amount of coding efforts (which was achieved by being brutal). Because of this the code is almost not commented. If you want to understand what you see, you’d better look at the appropriate posts.

The code is meant to be run from PHP command line, using the php interpretor, or using php’s cgi binary (with flag -q). It should work with PHP 5.1+ (I use PHP 5.3) and will produce the output for the .txt file with the same name.

Instead of reading the number of entries in the file (first line) I ignored it and will just read each file until I reach EOF, ignoring empty lines (like the final line). There might be some ugly hacks, but it wouldn’t be a Hacker Cup without some hacks.

The code, in a 23kB .zip file

Remember that this was written as fast as possible. The goal was not to make nice code, but to make code quickly. It will work and produce correct output given any valid input data.

I’ll be back next week!

Tags: , , ,

PHP can do anything, what about some ssh?

Last time I already tried to prove PHP can do anything when it comes to network protocols by implementing a DNS server. This time I’m doing it again with a server-side implementation of the SSH2 protocol.

You probably know SSH at least by its name. It’s a of secure telnet replacement which also allows many other things such as port forwarding, remote file management (with sftp) and more.

With PHP I could write a fully working SSH server in only 3 days. Of course I didn’t implement every single extension there is to SSH, but I’ve implemented:

  • SSH2 protocol only (no SSH1, anyway who uses that anymore?)
  • Encryption protocols: aes128-cbc,blowfish-cbc,serpent256-cbc,cast128-cbc,3des-cbc (via mcrypt)
  • Message digests: hmac-sha1,hmac-sha1-96,hmac-md5,hmac-md5-96 (via hash)
  • No compression as I cannot easily keep a compression context active (the gzip extension in php is missing a way to create a compression context)
  • Password and public key (ssh-dss and ssh-rsa) identification
  • Ability to program an interactive shell in PHP (there are send and recv functions in a separate class, anyone can have some fun and write something out of that. Should be possible to make a wrapper to communicate with a shell launched with proc_open)
  • Support for multiple channels
  • SFTP subsystem
  • Can be easily extended to add support for custom channels or re-use the ssh protocol for something else

My goal when writing this was to provide a replacement for the FTP protocol for the customers of my hosting service. FTP has many drawbacks, including no encryption (you can with ftps or ftpes) and the way ftp transmits data (another connection has to be opened on a different port, leading most of the time to some problems for people behind a NAT and/or firewalled servers).

With this ssh server supporting sftp, I finally got the replacement I was looking for. Of course it uses more CPU than a C ssh server (about 3 times more) but the difference isn’t that big. Next steps will include fork()’ing to open channels (will allow the SFTP server to chroot) and maybe support for some SSH extensions.

To implement the SSH protocol the following PHP extensions were used:

  • OpenSSL: used to generate secure bits when negociating the key, and used to generate the host signature on connection. I was hoping to use openssl_verify() to verify the key used when logging in, but I couldn’t manage to convert the ssh-rsa key to something openssl would understand, so I re-implemented signature verification with gmp.
  • MCrypt: The ssh protocol is encrypted (usually with something like AES128). mcrypt has the required functions to handle encryption in block mode
  • Hash: each packet transmitted over SSH is optionally signed with a HMAC signature. In order to generate and verify those signatures I used hash_hmac()
  • And finally the most important: GMP. As I was missing some functions to properly handle the initial Diffie-Hellman key exchange (and later to implement publickey authentication) I had to re-implement those in PHP. Of course working with 1024 bits integers is not something we can use the native int type for. GMP (and bc) allows such calculations (and I used them). I was missing the ability in gmp to read from/convert to binary values, so I had to add the use of bin2hex() and pack(‘H*’, …) to be able to work with binary values easily. GMP computations are only used when negociating keys (the ssh rfc recommands doing this once an hour, or every gigabyte of data transmitted) or when using the publickey authentification.

What did I create a ssh server for? The same thing I created a DNS server for fun and for KalyHost. In order to provide services updated in realtime I wrote a database-backed dns server a while ago, and now a ssh server (which can be database-backed too by extending the “Base” class).

The sourcecode can be downloaded from the SVN: http://ookoo.org/svn/pinetd2/trunk/code/classes/Daemon/SSHd/ this depends on pinetd2, a framework I wrote which allows to easily create daemons in PHP, and which I already used to create various things (FTP, Mail server, etc).

People willing to help on pinetd2 (code and/or documentation) are welcome. If you do not mind being called crazy because you make something else than webpages in PHP, you can contact me by mail or on IRC (or by leaving a comment on this post too if you wish to).

Tags: , , , ,

SimpleDNSd: new features & bugfix

Anyone using SimpleDNSd is strongly encouraged to update to latest SVN (you will have to erase the php-5.3.1 directory and recompile php to get it patched correctly). The current SVN version will most likely go release candidate and be released later.

Current SVN release includes many fixes and improvements, including for SimpleDNSd, the DNS daemon written in PHP.

This includes:

  • Support for delegation-only zones: it is now possible to handle TLDs via SimpleDNSd. I did a test by adding “free” domain names to the KalyHost service. Those domains can be ordered for free, and a webinterface is made available to control the domain DNS, allowing you to test SimpleDNSd and see how easily changes are done in realtime.
  • Support for PHP new requested feature (PHP bugreport #51295): queries to the DNS daemon were failing or returning wrong data randomly because of this bug. It took me a while to point this out as it was rather random. Basically current PHP implementation of SQLite3 has no “busy timeout”, meaning requests will fail immediatly if database is busy.
    I had to add a busyTimeout() method in SQLite3 (similar to the one already existing for the old sqlite PHP extension) and use it. This means we’ll have to wait for this patch to be added to a current PHP release before pinetd2 can be released as stable.
  • PZC: “Progressive Zone Change”. This is one feature no other DNS daemon has (or maybe they do, I don’t know). This feature allows to schedule change of a domain to a new zone. When the scheduled time comes closer, the DNSd will send expiration time smaller and smaller to make records expire on the time the zone will change.

A bit more about PZC:
Let’s say we have domain “example.com” pointing to zone A. Calling API method domainPzc(‘example.com’, ‘B’, time()+86400); will make domain example.com pointing to zone B in 24 hours. In the meantime, no returned record will expire after the scheduled time for zone change: any record obtained 15 seconds before zonechange will be marked to expire in 15 seconds.
This features allow a really precise control of “DNS Propagation”: you decide exactly when zone change will happen. Note that if you have a record in your zone expiring in 3 days, you shouldn’t schedule zone changes less than 3 days before effective date, or it might not have the expected behaviour.

As far as I know, no other DNS server support such a feature allowing to switch to a different zone with full control of when it will “propagate”.

(I know some resolvers out there will not follow expiration times given by the authoritative DNS server, however I like to think those are only a minority, and that PZC will give the expected behaviour for almost everyone)

Tags: , , , , , ,

PInetd2: new website, and release coming soon

If you never heared about PInetd, it probably means you never tried to search on google how to make a DNS server in PHP, for example.

PInetd is a generic framework allowing to run a wide range of daemons, mostly answering on TCP or UDP ports.

The main goal is to make generic protocol interfaces that can easily be overloaded to implement new features, new auth mechanism, etc… For example the FTP daemon is extended as FTPd_SQL for SQL auth.

The pinetd2 project is under development since October 4th 2007 (will be 2 years old soon), and has reached an appreciable maturity level. Most bugs were cleared and right now the system is running well. Some expected features (http admin interface, transports, etc) are not completed yet, but should appear within the next weeks, before the first official release with version 2.0.0.

PInetd includes framework tools to easily make a daemon listening on a TCP or UDP port, and also include some pre-made modules: FTP, HTTP, Mails (pop/smtp), DNS, NetBatch…

While official documentation is still mostly missing, previous posts here explain how to configure a DNS daemon, and the wiki contains basic documentation.

The official PInetd website is now open, and it is possible to subscribe to news feed to receive updates either by RSS or by mail.

Tags:

SVN error 200030 – It’s PHP’s fault!

If you updated to SVN 1.6.x and created a new repository in the new SVN format (which uses SQLite), you probably found that you were getting an XML error that says:

<m:human-readable errcode="200030">
Could not open the requested SVN filesystem
</m:human-readable>

After some searches I found someone who had the same problem, and fixed it. Basically the problem is due to the SQLite version embed into PHP 5.2.x. PHP includes SQLite 3.3.7 while SVN will usually depend on 3.6.15 (or whatever you have on your system).

To fix this problem you have various solutions:

  • Disable PHP as did our japanese friend
  • Upgrade SQLite version provided with php, or tell php to use system’s SQLite with –with-sqlite=/usr
  • Create your svn repository with –pre-1.6-compatible

Hope this article will help all those who, like me, got the error 200030 with SVN without understanding much where it comes from.

Tags: , ,

Yahoo going closer to Microsoft, Rasmus’ reaction

We got a press release from Yahoo saying that Yahoo! and Microsoft are going to share search/advertizing. In fact only the first paragraph seems of any interest, the rest of the press release is mostly marketting bullshit.

Yahoo! and Microsoft announced an agreement that will improve the Web search experience for users and advertisers, and deliver sustained innovation to the industry. In simple terms, Microsoft will now power Yahoo! search while Yahoo! will become the exclusive worldwide relationship sales force for both companies’ premium search advertisers.

This sounds rather awful, and reply from Rasmus was quick.

Rasmus announces he wants to leave...

Of course I immediatly tried to confirm this on IRC…

[21:20:51] <MagicalTu> http://twitter.com/rasmus/status/2908526145 <- wtf ?
[21:21:21] <Rasmus> See the Microsoft-Yahoo press release

So now, we’ll see how it evolves, but I guess this press conference will change a lot of things, including in PHP’s world.

Tags: , ,

Your own PHP DNS daemon

After my initial announcement of a PHP DNS Daemon, and some performance tests, and since it has been widely reported over internet (Zend DevZone, and today on Nexen which is deeply involved with French php community), I finally found the strength to write a guide on “how to get started with DNSd”.

Basically, you’ll need subversion, gcc, make and most prerequisites for PHP (ICU, etc). This supposed you already have some knowledge in system administration.

The first step will be to fetch the latest SVN code somewhere in your Linux box (I usually put that in /usr/local).

$ cd /usr/local
$ svn co http://ookoo.org/svn/pinetd2/trunk pinetd2
(checkout lines)
$ cd pinetd2/php
$ ./do_php.sh
(will download & compile PHP 5.3.0, and maybe complain about missing stuff,
 just install whatever is missing, if you need help, post the last error
 lines here or contact me)
$ cd ..
$ ./start.sh
Please edit config.xml and remove the line containing this text

Arrived at this point, a file config.xml has been created and needs to be edited. I added comments inside it to help you. You’ll need to do a few things:

  1. Remove the <RemoveMe> tag, and edit the <Name> tag to include your machine’s name.
  2. Change the storage engine to use SQLite3. PInetd’s MySQL driver is not compatible yet with DNSd. You can copy the line from the example, but remember that if your zone file is in /tmp, it might get erased at startup (depends on your linux distribution).
  3. There is an empty <DNSd> tag, fill it with the second choice in the comment (the <PeersArray> one). Change the Signature, peer name, etc to fit your needs for the Type=”control” line : this is the definition of “who will be able to create/remove records on this DNS server”.
  4. At the end of the file, remove all processes that do not have Daemon=”DNSd”. There sould only be 3 remaining processes.
    If you are not root, set PortOffset to an arbitrary value greater than 1024, like for example 10000.

Once you’ve reached this point, you should be ready to go. Try starting the daemon.

[2009-02-19 10:10:33:30945] DEBUG: pinetd v2.0.0alpha running on...
[2009-02-19 10:10:33:30945] WARN: SUID security level is defined...
[2009-02-19 10:10:33:30945] WARN: Warning: Chroot security level...
[2009-02-19 10:10:33:30945] DEBUG: My name: localhost
[2009-02-19 10:10:33:30947] INFO: Loading Daemon\DNSd\UDP on port 10053, bound to ip 127.0.0.1
[2009-02-19 10:10:33:30948] INFO: Loading Daemon\DNSd\TCP on port 10053, bound to ip 127.0.0.1
[2009-02-19 10:10:33:30949] INFO: Loading process Daemon\DNSd\Process

As you can see, I am not root, so PInetd will complain about the impossibility of chroot()ing or setuid()ing, however it’s just for testing, so we don’t really care.

At this point, if you configured the DNS daemon like me with a PortOffset of 10000, with a peer named “MyPeer” and a secret of “qwerty”, running “dnsd_test.php” in the “test” directory will create an “example.com” domain.

$ php dnsd_test.php
Connected to localhost

Now, you can test it:

$ dig +short -p10053 @localhost example.com
127.0.0.1

Of course the PHP DNS Daemon is not completed yet (fixed a potential denial of service yesterday) and probably still have many bugs, so I strongly advice against using it on any production system yet.
Yes, I am using it for this blog, and for another ~200 domains, to find bugs and make the solution more stable, however if you wish to contribute by testing on a production system too, make sure you are ready to have all your websites becoming down and your server taking fire.

The next step for you is to look in test directory at the “dnsd_test.php” script, and make your own pages using DNSd. You do not need to use PHP 5.3.0 to use this class, so you can basically use it anywhere. PHP 5.3.0 is only required for the DNS daemon itself.

If you have any question, feel free to leave a comment, I’ll try to complete this article as problems are raised by people who use this.

Tags: , , ,

PHP DNS Daemon: performances

Many people seems to have taken interest into my weird idea to write a PHP DNS daemon in PHP, so I decided to run some experiments. The first one was to setup ~200 domains to this dns server (including my blog’s one), the second one is to run queryperf against it, and against bind9 running on the same host, with the same configuration (ie. same domain list, no recursion, etc).

First, the host:

  • CPU: 2x Intel Xeon E5405 (2GHz) ; a total of 8 cores
  • RAM: 8GB RAM (4x 2GB DDR2 @667Mhz)
  • Hard Disk: 2x1TB HDD (RAID 1, 3ware Inc 7xxx/8xxx-series PATA/SATA-RAID) ; total of 1TB usable
  • OS: Linux Gentoo 64bits 2008.0 (multilib) with Linux Kernel 2.6.27-gentoo-r2

The test itself will be a 1 million random queries generated by gen-data-queryperf.py with 40% of random domains.

Some words on results

First, I’d like to say that pinetd2 is still under development, some parts are still not implemented (the DNS server is able to act as a DNS server, that’s the important part for me), and also some optimizations weren’t done yet (for example a query will always cause the same SQL statements to be run, I could prepare those).
The fact I’m running SQLite means the SQL server isn’t able to cache results (the db file might be modified by anyone, anytime, however I don’t know the exact internals of SQLite), and I don’t cache anything either.

When I started writing DNSd, I didn’t especially try to go on performances, features were importants, and realtime was too. Many improvements to speed can still be done (I’m thinking “prepared statments” right now, but also caching domains list, etc) and would help to get those numbers closer to ISC BIND.

The fact DNSd is 1/4 the speed of BIND (2531.89 queries/sec instead of 10071.2 queries/seq, my dns server is runnnin at 25.14% the speed of bind) is impressive. I guess we’ll need more tests, with different backends (MySQL is also supported, in theory) and different hosts, but I was supposing the database overhead would be bigger than that (well, SQLite is fast, but I wasn’t expecting that fast).

To tell you the truth, I am surprised by those results, however these are results on a real host, really running domains (like my blog’s domain), which makes me believe those results are the closest I could get from DNSd performances on a real host.

Now, the raw test results with both bind and PHP DNSd, running from the same host (to avoid network latency, and since I got 8 cores with almost no CPU usage as it’s morning in France, it shouldn’t make a big difference).

Other test results with other hardwares are welcome. I’ll try running the same kind of tests on less powerful hardware too, just to see what I get.

Read the rest of this entry »

Tags: , , ,

PHP DNS Daemon

As the subject suggests, I just wrote an opensource DNS daemon in PHP. I already know what any sane person is thinking right now:

[18:25:06] <Derick> MT`AwAy: you’re mad

Now that it’s said, let’s move on ; if you want to tell me I’m crazy, you can post it in a comment here, it makes me happy. I had some reasons to dislike bind9 which finally made me write my own DNS daemon, and I’ll explain that here.

My need was to have a stable dynamic DNS server working in most environnements, with an easy to configure master/slave relationship (with realtime synchronisation), and a way to change records instantly from PHP…

For those who already played with bind9, there’s a feature called dlz which basically allows to ask bind to get data from an SQL server. I could just configure another MySQL per slave, and put replication there, but it’s not that fast and I had some stability issues (both with MySQL replication, and with bind crashing in some weird cases).

So, instead of trying to fix bind9′s code (which would also include fixing MySQL replication – or trying newer MySQL’s row-based replication) or searching for another solution (there are zillions of dns servers around there, but they all have features I’ll never need), I decided to write my own DNS server (built on top of PInetd, my PHP networking framework for PHP 5.3.0), with only the features I needed.

So here are some of the features of this DNS daemon:

  • Supports RFC 1035, and some others too (IPv6 AAAA records, and DNS OPT, with the goal of supporting DNSSEC at some point).
  • Does NOT support AXFR nor IXFR, I have no need for the standard zone transfer protocol
  • Data can be updated realtime with a provided client class, connecting to the server via a shared secret authentication (client computes checksum of his name, timestamp and secret, server replies with the same kind of data). Once authentication is finished data is sent cleartext, but adding SSL encoding wouldn’t be that hard (just need to create a “STARTTLS” function, that’s on the TODO list).
  • Slave stays connected to master (keepalive packet sent every 15 minutes), and gets updates realtime.
  • On first connection, slave will get all zones/domains/records from the master. If it gets disconnected later and reconnect, it will search for his last update, and will ask only for newer data from the master.
  • Slave synchronisation is done in a separate process, meaning that even when processing a lot of updates, service is available.

As a test, I ran AFNIC’s zonecheck on a virtual zone I created on the server, and it works!

dig @dyndns1.ookoo.org version.dnsd ch txt

If you want to test this, I got a test domain with a record creation page, however I won’t post it here (or it might get abused) so if you want to test, feel free to contact me.

And finally the conclusion is simple: php can do anything you want to do, and even things you never wanted to.

PS: I’m looking for someone to look after PInetd‘s website, like removing this lipsum news, etc… (I don’t have time for that).

Tags: , ,

Using SSL keys for client authentification

As you may have seen on services such as MyOpenId, you can either login on your account with a password, or you can generate a client identification SSL certificate and let your users identify with one of the most secure way (as long as they don’t get their keys stolen).

SSL identification?

The idea is pretty simple. Usually communications between a SSL client and server are done via a public/private key system. While I never checked in depth, I assume it’s pretty much like for mails: the client generates a client certificate, connects to the server, gets the server’s public key, sends its public key to the server, then use its own private key and the server’s public key to exchange data.
SSL identification happens there: instead of generating a key, the client will use a previously generated private key, and send its public key alongside a certificate previously obtained from the server. The server will be able to check the certificate, which proves that the client’s key has been signed by the server CA (another private certificate held by the server, only used to make client certificates).

This way, the server will know that the key used by the client was signed by him and can be trusted.

How did this happen? Who gave its key to the client? How can you do the same on your website with PHP? What? Why are you talking about PHP 5.3.1?

Read the rest of this entry »

Tags: , , , ,