Increasing computer speed can often mean hardware changes. Even an older computer's performance can be improved through upgrades that are often simple to perform. Here is a list of possible changes you can make, and criteria to base your decision on.
RAM: Dollar for dollar, you'll see more improvement in your computer's performance by adding RAM, than any other upgrade. Before you buy, open up your PC's case and check the sticker on its memory modules to see exactly what kind it is. If you're running an old OS, like Win98SE, remember that 512 MB is the limit. With XP and Vista, your only limitations are your pocketbook, and the number of slots on your motherboard.
CPU: You may be surprised that a faster CPU is second on the list. Your computation speed often is not a problem, the way having log jammed memory is. But if your motherboard can accept a faster processor, gains can be made. The gains can be thought of as incremental, rather that the dramatic sort you'll have by increasing RAM from say 256 MB to 1 GB.
GPU: People want to play their games at the highest settings, but many times can not because their graphics card is just too puny. Even if you're not a gamer, and you have a system with video on the motherboard, getting a GPU would free up RAM that has to be shared with video. Ideally, you'll have a motherboard with a PCI-E slot, but do not fret if you've got an AGP, or even all PCI slots. There are a lot of good, fast cards out there for all of them.
SATA: By all means, if your motherboard has SATA ports, get a SATA hard drive. But if it does not, you can still get some of the speed gains by installing an expansion card. It will plug in to a PCI slot, giving you 1 or 2 interior ports, and one external. (You can get a PCI-E version, but if your motherboard has PCI-E it's almost sure to have SATA ports. In this case, the only reason for a card would be if all the ports were in use, and you wanted to do something like connect to an external drive.)
Motherboard: I'm putting motherboards last on the list, not because of performance improvements, but because of the difficulties involved. Indeed, a board with a faster bus can do wonders. The trouble is that a new motherboard may require you to get a new CPU and RAM just to match socket types. This may be more expensive than you're willing to take on. Also, a lot of stuff will have to come out of your case to swap motherboards. If you've never done it before, you'll want plenty of time so you will not be rushed.