"Memory" means short-term memory for computers. In contrast, hard drives provide long-term memory.
Everything the computer is works with has to be in active memory. Things it isn't using at the moment can be stored on the hard drive. A large part of Windows is "loaded" into memory at startup. When you start a program, the CPU loads many of it's files into memory. When you press a key on the keyboard the CPU loads the keyboard input into memory. When you "save" a document the CPU "writes" it to the hard drive, but it remains in memory too. The CPU is obviously the component that does the "thinking", but just like you, it can't do much unless it gathers all the details into short term memory.
If the CPU needs more memory than is installed in the computer, it shifts content it isn't using at the moment to "virtual memory". Virtual memory is located in the "swap file" on the hard drive. It's called swap because Windows "swaps" memory contents between physical memory and this virtual memory.
For example, if you have two large programs open and switch from one to the other, Windows may need to swap the one you were using out of physical memory into virtual memory. Then it brings the one you want to use now out of virtual memory and puts it in physical memory. All this takes some time because virtual memory is much slower than physical memory. More memory is good because it means there's less swapping going on.
Don't get confused by the "Free Memory Myth". Yes there are programs that can "free up memory", but that's just the opposite of the best way for Windows to manage memory.
You'll improve the performance (speed) of your computer more by increasing memory, say to 512 MB, than by getting a faster processor.
If you're buying a new computer look for at least 256 MBytes of memory. If you want higher performance, particularly with Windows XP, be sure you get 512 MB of 266 MHz or faster DDR SDRAM.
The keyword here is DDR (Double Data Rate). 512 MB is the size. 266 MHz is the data rate -- twice that of ordinary SDRAM (Synchronous Dynamic Random Access Memory.
Thinking about adding memory to upgrade your computer? Don't buy cheap memory -- it's often the source of mysterious PC problems. Try Crucial Technology to get top tier memory at a good price. They're the online arm of Micron -- a well-known producer. Their website makes it easy to find the right memory for your computer, or other devices like digital cameras. If you're worried about installing memory, check out these great instructions at Crucial.
The "Memory Configurator" at Shopper.com is another way to locate the memory you need. There are links on buying and installing memory there too. Or try Kingston, who also sells good memory. Samsung also makes quality memory. Get a good price for Samsung memory at Newegg.com.
The hardest thing about installing memory is getting it seated properly. In my computer, just getting at it is a problem. In the case of memory that pivots into place, it's not easy to see exactly how to do it. Memory that plugs in straight is often hard to seat, and if you don't get it started right, you can break the socket or the motherboard trying to force it in. Check out these great instructions from Crucial Technology for installing memory. If you've never done it before you may want to get someone to help, or get it done at a shop.
Before you start to install new memory, before you even open your new memory package, read the ESD precautions on the upgrade page.
See Rob Williams' Computer Memory Upgrade guide. This is a very complete reference.