Thread is the most essential of fly tying materials (assuming you don't count the hook as a material). Fly tying thread's role is pretty simple, it covers the hook and lashes material to that hook. And it sounds pretty simple but, of course, fly tyers make it a lot more complex. To simplify, there are two basic characteristics of threads - material and denier or size - that together give a thread most of its properties and determines the uses of that thread. There are certainly other characteristics that are important and I will write more about those later.
Before you ask, "Can't I just use sewing thread?" Probably not. First, most sewing thread is cotton and, other than silk, most tying threads are synthetics. Synthetics are much better at being able to not stretch when wet, at not soaking up water, and most importantly, at getting wet and dry repeatedly without rotting. Second, fly tying thread is spooled on spools that fit into fly tying bobbins. Lastly, thread is relatively cheap and the tiny bit of money you are saving is not worth the hassle, in my opinion.
Using the right thread is important, in part because it makes a better, more durable fly but just as much because it makes tying that fly easier and thus more enjoyable. You could tie a large streamer with a really small diameter thread but it will take you longer, you will use more thread, and it will generally be more of a pain in the ass than it needs to be. Modern fly tying threads are great and we have a huge variety of choices.
Types of Fly Tying Threads
Initially, flies were tied using silk, a natural material, but since then, we have used a number of synthetic materials in place of silk. I doubt that any of these materials or threads were created for fly tying but rather, fly tying companies used readily available materials with the characteristics they were looking for. There are still a few manufacturers of silk thread, although probably the best known one - Pearsalls - is no longer produced and older threads are selling for rather inflated prices. Tyers like silk threads when trying to recreate traditional flies, in part, because wet silk gives the fly a translucency that synthetic threads do not. However, the fly tying 101 version of this paragraph should probably be, silk is traditional but you will probably never want or need to use it.
The the most common modern fly tying threads are nylon, polyester (poly), and gel spun polyethylene (GSP or G.S.P.), each of which differ a bit in their characteristics, their strength per diameter, and their cost. Nylon was created to replace silk - and it largely has. Lesser used thread materials today are monofilament; a nearly invisible nylon similar to fishing line / tippet material; Kevlar, and rayon threads.
Nylon thread has a bit of stretch which helps keep it from breaking and that stretch helps secure materials to the hook. It holds dye well but it has a lower breaking strength compared to polyester or GSP threads. Some common brand names of nylon threads are Uni-thread's Nylon, Danville flat waxed nylon, and Wapsi Ultra tread. For its diameter, polyester thread is a little stronger than nylon, doesn't take dye quite as well, and does not stretch as much. A number of commonly used threads are polyester threads including UNI-thread's standard line (3/0, 6/0, and 8/0 - more on sizes later), Veevus 14/0 and 16/0, and one of my favorite small fly thread, Gordon Giffith's Sheer Threads. GSP threads are the strongest and have no stretch - in fact GSP threads will bend hooks if you're not careful. GSP dyes the poorest, is very slick, and is the most expensive. GSP threads have many applications and you can use them for any fly but the excel in two applications; 1) very small flies where you want a strong thread with minimal build up, and 2) big flies where you need to REALLY secure a material to the shank. Note that some synthetic threads made of gel spun polyethylene (GSP) have commercial names like "nano-silk" or "power silk".
Most tyers use nylon or polyester threads for most trout sized flies and polyester or GSP for larger streamers. For the smallest of flies, tyers are typically using GSP, Nano-silk (a specialized GSP), or polyester. Polyester threads tend to have a bit of roughness which helps dubbing stick and maybe makes materials less likely to slip out of the thread. GSP is particularly slick.
Thread Sizes Demystified
As you have probably gathered from above, there are basically two different thread size systems. To make it more interesting, many companies use both of them, maybe using the "aught" (X/0) system for one line of thread and the denier system for another line of thread. And to make it more confusing, it is not like the aught system is used for one type of material (nylon or polyester) and denier system for another. However, gel spun polyethylene threads - as far as I know - always use the denier system.
Aught System
This was the system that was used when I started tying flies. It is a measure of the diameter of the thread. For fly threads, the larger the number before the "aught", the smaller the diameter of the thread and typically the lower the breaking strength - though that does vary by material type. The largest "aught system" thread you'll find is typically 3/0 and the smallest 18/0 or 24/0 for really small flies.
Some standard thread sizes and their uses are:
3/0 - larger streamers and nymphs
6/0 - most nymphs and dry flies
8/0 - smaller nymphs and dry flies
12/0, 14/0, 16/0, and 17/0 - small flies on hook sizes 18 or smaller
18/0, 20/0, or 24/0 - for the very smallest of flies (these are typically some sort of GSP or other specialized material).
Denier System
The denier system measures the weight of the thread - gram weight of 9,000 meters of thread to be exact. In other words, 9 kilometers (9,000 meters) of 100 denier thread weights 100 grams. The greater the denier number, the heavier the thread and for the same material, the larger its diameter will be.
30 denier - this is about the smallest I have seen, where you would use 18/0 and smaller threads
50 or 70 denier - small or typical "trout-sized" flies, breaking strength depends upon material but use where you would use 8/0 thread
100 denier - many standard / larger sized trout flies, I like 100 denier GSP and flat waxed nylon for a lot of streamers, approximately equivalent 6/0 thread.
150 and 200 denier - larger trout flies and streamers, 180 denier is approximately equal to 3/0 thread.
Danville lists their 6/0 nylon thread as 70 denier nylon, their Flymater+ comes in 140 and 210 deniers, and their flat waxed nylon is 210 denier nylon.
Comparing Aught and Denier Systems
All things being equal, the smaller the thread diameter - or the greater the "aught" number - the lighter the weight (denier). However there are differences in the materials which means that a nylon 140 denier thread will likely have a lower breaking strength than a 75 denier GSP thread. Like with any material, there are tradeoffs and those tradeoffs depend upon the application. For more on that, below are some links - I particularly like Charlie Craven's FlyFisherman magazine article (link).
Thread Shapes
In general terms, threads tend to be round, oval, or flat and at least for nylon, polyester, and GSP threads. Threads are typically either a single strand of material or are made of many strands of material. This matters because single strand threads can not be flattened or split (for things like split thread dubbing) but they are less likely to fray and require less bobbin twisting to get them in the shape you want. Multi-strand threads are often more easily split (though some are twisted tightly). Sometimes you want a round thread, sometimes you want a thread that will lay flatter. For the Fly Tying 101 student, it probably doesn't matter yet.
Choosing the Right Thread
There is no right thread but there are wrong threads for particular purposes. Probably the single most important thing is to pick a thread size that will work for the size fly you are tying. For most trout-sized nymphs and streamers, a 70 or 100 denier thread or a 6/0 or 8/0 thread is about right. For streamers, a 140 or 150 denier or 3/0 thread is probably a better choice.
Thread color has not been mentioned yet except in how different materials take dyes. I do not think thread color matters that much. Most of the time it is completely hidden and when it is not, you can always color it with a marker. I have threads of many, many colors but I now mostly buy white and black along with a few more fluorescent colors (bright chartreuse and pink) on hand. In smaller sizes threads, I like some more natural colors - olive, tan, rust - for flies where there aren't many materials or as much dubbing to hide the thread. No need to get too fancy about it.
More importantly, pick up a few different threads and give them a try. Twist the bobbin and see if they flatten out or cord up. For typical trout flies, I'd buy a nylon thread or two and a polyester thread brand or two. Below are some of the more commonly used threads for "trout-sized" flies.
Uni-Thread (3/0, 6/0, and 8/0): For a long time, this was the standard thread you found everywhere and it is still a great thread. As the name suggests, it is a single (uni) strand, in this case, of polyester. Uni-thead is round and lightly waxed which helps create a slightly tacky thread which helps secure dubbing and other materials.
UTC Ultra Thread (70 and 140 denier): UTC's standard thread is nylon with a light wax applied. It is a great all-around thread for most trout sized flies - choose the 70 denier for smaller flies and the 140 hooks sized 12 or larger. UTC thread generally lay flatter than UNI threads but it can be twisted up to create a rounder thread.
Danville Flymaster (6/0 is about 70 denier): Is either waxed or unwaxed, this is and excellent thread for most "trout sized" flies.
Danville Flymaster+ (140 and 210 deniers): Wraps flat but can be twisted to a rounder shape, a great all around thread, particularly for a little bit larger flies (sizes 12 and larger). Great thread for saltwater flies.
Danville Flat Waxed Nylon (210 denier): I might pick up a spool to see the difference in between this and the other Danville threads. I use this a lot for streamers.
Veevus offers a number of great threads. In the "aught system", these are polyester threads.
Benecchi and Gordon Griffith are good options for tyer that are looking for strong but thin threads.
I'd forgo GSP threads and some of the specialized thread like monofilament nylons until you have mastered working with nylon and polyester threads. There are reasons to use GSP but rarely do beginners need a GSP thread.
Wrapping it Up (no pun intended...)
Threads can be a little confusing, for sure. And manufacturers do not help as much as they could - or should. GSP threads are almost always labeled as such but rarely do you see nylon or polyester threads labeled with the type of material they are made of. I generally find nylon and polyester pretty interchangeable though they do have some differences in their breaking strength, roughness and how they grab dubbing and other materials, and their color and sheen.
I have a ton of different threads but for standard sized trout flies, I find myself using UTC most of the time. For larger flies, I generally am using Danville flat waxed nylon or Flymaster+ or GSP.