1. Never use a subquery inside a WHERE field IN(...) list

2. How-to: Update values in a table from a calculation that uses the table's own values, in a single query

Let's begin!

1. Never use a subquery inside a WHERE field IN(...) list

   BAD/SLOW:

   SQL copy to clipboard
   1. SELECT * FORM table1 WHERE datefield IN
   2.   (SELECT datefield FROM table2 GROUP BY datetype)

   
   

   SELECT * FORM table1 WHERE datefield IN
     (SELECT datefield FROM table2 GROUP BY datetype)
   

   It's tempting to use a subquery inside an IN(...) list, but your query speed will slow exponentially as more rows are added. Don't be tricked by query caching either, since every new INSERT will erase the cache. The solution is to use an INNER JOIN and subquery instead.

   GOOD/OPTIMIZED:

   SQL copy to clipboard
   1. SELECT *
   2. FROM table1 AS t1
   3. INNER JOIN (
   4.   SELECT datefield FROM table2 GROUP BY datetype
   5. ) AS t2 ON t1.datefield=t2.datefield;

   
   

   SELECT *
   FROM table1 AS t1
   INNER JOIN (
     SELECT datefield FROM table2 GROUP BY datetype
   ) AS t2 ON t1.datefield=t2.datefield;
   

    

2. How-to: Update values within a table using the table's own fields

   The UPDATE command allows you to join other tables. We can use a subquery on the same table to calculate new field values, then join this subquery as a "table."

   For this example, a table contains a date field with values two weeks apart, but we want to change them to one week apart. (A user-defined variable @row is needed to keep track of the row number in order to calculate the new date.)

   SQL copy to clipboard
   1. # Change dates from bi-monthly to weekly
   2. SET @row = -1;
   3. UPDATE u_tabledates AS t1
   4. INNER JOIN
   5.   (
   6.   SELECT *, DATE_SUB(u_date, INTERVAL (@row := @row + 1) week) AS new_date
   7.   FROM u_tabledates ORDER BY u_date ASC
   8.   ) AS t2 ON (t2.u_id=t1.u_id)
   9. SET t1.u_date = t2.new_date

   
   

   # Change dates from bi-monthly to weekly
   SET @row = -1;
   UPDATE u_tabledates AS t1
   INNER JOIN
     (
     SELECT *, DATE_SUB(u_date, INTERVAL (@row := @row + 1) week) AS new_date
     FROM u_tabledates ORDER BY u_date ASC
     ) AS t2 ON (t2.u_id=t1.u_id)
   SET t1.u_date = t2.new_date
   

The keyword for performance is "caching." By taking the load off the processor and database, a website can easily handle hundreds of thousands of visitors. Many frameworks offers some sort of caching method that stores the final rendered HTML. This is easy to implement for anonymous users, but what about logged-in users?

I recently released an open-source Drupal module called Authenticated User Page Caching (Authcache) (and a demo site) for logged-in users. Drupal is a great PHP/MySQL CMS/framework for building community-based websites, though it suffers from a "one-more-query" syndrome--a single page request can have hundreds of SQL queries! The reason why authenticated caching is difficult is because most of the time, when you're logged into a website, the content changes according to your account (like a "You're logged in as [username]" link), so it's pointless to save the final HTML to the server's cache. Instead the entire page has to be rendered by PHP on each request, requiring precious CPU cycles & database hits. Ajax/JavaScript can be used for customizing the cached HTML, however. Of course, this technique can't be used with every website--some sites are just too dynamic--but it can work if the content of pages is similar across users.

Caching Flowchart

Every millisecond counts when it comes to performing page requests--not only are cached pages served faster to the user, but it frees up services/memory faster on the server, giving more bang for your buck. This is important not only on high-volume websites, but when you're on a hosting plan that doesn't give you gig upon gigs of memory or high-performance CPU processing power.

The modular nature of Drupal and its excellent API makes it rather straightforward to implement this type of caching. Other frameworks may prove more difficult. The Authcache module can use a number of different caching engines, such as APC, memcached, or even static files. I've found that APC offers the fastest retrieval times, allowing pages to be served in under 1 millisecond!

Implementing Page Caching
The tricky part with page caching is knowing when to retrieve from the cache and when to delete outdated cache. If there is a POST request, such as when a comment is made, then Drupal/PHP will execute the page code normally and invalidate the cache. When we are logged in as an admin, we don't display or save cache pages (because we want our admin links and don't want users to see them). We may also expire our cache after a certain amount of time to always display the most up-to-date content, which is useful on a front page and search result page. Yes, these little details can be a hassle, but the end result is well worth it if your goal is a faster site.

What About Dynamic Content?
This is where the beauty of Ajax/JavaScript comes in. We don't need to re-render an entire page, just perform a separate confined action. For example, instead of having PHP render "Hello, [username]" text for each page for logged in users along with everything, store the username in a cookie and have JavaScript render the username. Then create a separate cache for pages for logged in users and one for anonymous users. If certain pages absolutely must be dynamic, then they can be excluded from the cache. Another example: I like to keep a view tally on my articles in this Wordpress blog. All my pages are cached, however, so to update the count my pages ping a small PHP script after loading, which checks the user's cookie to see if the article has been viewed, and if not, then it quickly connects to the database, sends an update query, then exits, all independently of the what matters the most: serving the page to the user.

Caching Tricks
There are also several browser caching techniques that can be used when cached HTML:

• Sending "HTTP 304 Not Modified" headers – Save a timestamp of when the cached page was last modified and send it with a "Last Modified" header. Then we can examine the user's "If-Modified-Since" request headers and send a 304 response to any pages that the browser has already saved in their cache. (header("HTTP/1.1 304 Not Modified"); A 304 response does not send any HTML.) This saves on bandwidth and allows the browser to display the page directly from memory without having to redownload and parse the HTML again.

• Using "max-age" headers – The "max-age" response header informs browsers not to download the URL again for "max-age" seconds (header("Cache-Control: max-age=3600, must-revalidate");). We can this header to cache Ajax responses. This is useful for blocks of user-specific content. For example, if a user votes on a poll, we would want to display the poll results and not the poll vote form. But we don't want to retrieve the poll results on each page view, so we can tell the browser to just cache the block for 10 minutes. A cookie with a timestamp can be used for cache invalidation (e.g., if a user changes their vote, a "vote" cookie containing the unix timestamp is saved, then the Ajax request uses this cookie value when generating the request URL).

Installing Cygwin
(You can skip this if you're backing up on a Linux machine.)
The first task is installing Cygwin on your Windows box, which provides a Linux-like environment for running our backup tools (rsync+ssh and cron). When running setup.exe, you should select as many packages as you feel comfortable installing by clicking . Be sure to install all of "Admin," "Base," "Devel," and "Net." You can always run setup.exe later to update your existing Cygwin installation.

Creating SSH Public/Private Keys
Since we want to copy our files securely over SSH without being prompted for a password each time we login, we need to generate a public key for our backup machine. At the shell prompt, enter:

ssh-keygen -t dsa

Ignore the questions by pressing "Enter" at any prompts, such as the file location or passphrase. Now open ~/.ssh/id_dsa.pub (e.g., c:/cgwin/home/(Windows Username)/.ssh/id_dsa.pub) file and copy the contents. Log in to your remote/Linux machine and paste this info into the file ~/.ssh/authorized_keys. If the file doesn't exist, just create a new one. You must also modify your SSH daemon to use allow Pubkey authentications. Edit /etc/ssh/sshd_config and uncomment:

RSAAuthentication yes
PubkeyAuthentication yes

On your backup machine, test your pubkey by typing (replacing username and remotehost):

ssh username@remotehost

You should now be able to login into your Linux server without typing in a password!

The rsync Backup Script

In other words, rsync is one of the best backup tools available.

Create your backup directory:

mkdir /rsync

And create your backup script inside it, named backup.sh, making sure to replace username@localhost:

#!/bin/bash
rsync -avz -e ssh username@remote:/var/www/ /rsync/www/ --log-file=/rsync/rsync.log

/var/www/ is the full path on the remote machine. /rsync/www/ is where to save on your backup machine. If you use a non-standard SSH port, you can use "ssh -p portNum" in quotes.

You can also backup your MySQL database files, log files, code repositories, or whatever else you desire. If you don't have access to the MySQL files directory, you can setup a mysqldump script on the remote machine to save to a file.

You may also want to create "snapshots" of your backups, so these files don't get overwritten by unwanted changes. Here's a snapshot script (snapshot.sh) that gzips MySQL database files:

#!/bin/bash
archivedate=$(date +%Y-%m-%d)
tar cvf - /rsync/mysql | gzip > /rsync/archive/mysql_$archivedate.tar.gz

Cygwin: Installing the "cron" service
Crontabs are used for automating our backup scripts. On Cygwin, cron must be setup as a Windows service in order to run in the background:

chmod +x /bin/cygwin1.dll
mkdir /etc/cron.d
cygrunsrv --install cron -p /usr/sbin/cron -a -D
cygrunsrv --start cron

To see if the cron service is running, type:

ps -a | grep cron

Adding the Crontab:
On your backup server, type:

crontab -e

Press "i" and add:

0 0 * * * /rsync/backup.sh

Press "ESC" then "w" and enter to save.

This will run the backup script at midnight every day. You can also add a line for the snapshot.sh script... replace the first "0" with "30" to run the script 30 minutes later.

That's it! Now just make sure your backup server has your hard drives in a RAID 1 (mirroring) or better configuration for additional data redundancy!