Personally, I think what makes literary monsters truly scary is the combination of the real with the unreal.  Yes, make then fantastical, but also make them believable in a strange sort of way.  Perhaps give them personalities and motives just like you would a character. 

Also, make sure your characters have some sort of emotional reaction toward the monsters, whether it be intense fear or mere loathing.  Play on the emotions of the reader to make the mosters truly scary.

And what Michael said is definitely a good point - leave something up to the imagination of the reader.  Always assume that readers can create much scarier monsters in their imaginations than you could ever create on paper.  Leave enough open in the description for a reader's imagination to fill in the blanks.  

Jack, excellent advise to you from the previous posts--pros all (and many of them, note, are editors who see such stories on a daily basis).

Rather than repeating any of what has been said, I will add another thought:

Monsters can be cool, scary, unusual, frightening, disturbing, whatever--BUT if they do not pose a real threat to the heroes, they are just very strange-looking roadbumps.

Make it CLEAR that a monster is not just an irritation or annoyance to the hero; for a monster to be really threatening and memorable there must be a cost. If the reader feels that the odds are as likely that the monster will kill the hero as that the hero will kill the monster, you will generate real suspense. To fear the monster, the reader must respect the monster. Respect its very real threat.

Granted, this can be exceedingly difficult to pull off in a novel or series where the main heroes are clearly going to survive to the end. [So, shhh, this is just between us: create a hero or two who will end up being killed by the monster. Or maybe the HERO hero is somehow scarred--physically or mentally--by his encounter with the monster. Especially effective if he must later face the same monster again (think Moby Dick). Then there is a real cost.]

Plus all the other stuff Paul, Michael, Ty, Lyn, G.W., and Richard said.

Good luck. And remember to tap into one of the best resources for monsters: your own nightmares.

I'm with these fellas

What's a hero? Someone who defies odds to . . . accomplish something. So, dealing with the monster has to be a serious challenge - one that is neither easy nor intuitive. The harder your monsters are to defend against or defeat, the more tension they will build. How does a six- or seven-foot tall barbarian slay a dragon that can loom thirty feet above him? You can't even use a sword against spectral oponents. What makes a werewolf scary? You have to have a special weapon to kill it. Same with vampires. Zombies have numbers. Orcs come in hordes. The kraken comes up from beneath your ship and you have no ground on which to fight it. Fleas are ubiquitous and just darned annoying. Mosquitos may or may not carry West Nile virus. Dogs have sharp teeth and sometimes attack in packs. Snakes come at you from a terribly indefensible angle. Bears and lions are big and loud and have enormous claws and teeth - and are sometimes flea-ridden. So . . . the monster is only as interesting/horrifying as the process for eliminating or navigating its threat. Same as any challenge.

Mike, if we're going to cite some classic fantasy and S&S monsters as paragons of the craft, I'll add some of Tolkien's: The Balrog, Shelob, Smaug. Now those are some memorable monsters!

I can't nod my head emphatically enough here.  When you see the monster, you know exactly what you're afraid of.  When you only catch an occasional glimpse of what could be a monster, the things to be afraid of are limited only by your imagination--which often takes the opportunity to run wild.

Other than that, my personal opinion is that if it's a monster, it should be alien.  Not from another planet, just not a human with claws.  Keep that air of mystery about it.  To use Nicholas's example, the Fellowship didn't know what the hell the Balrog was.  They just knew that it was very old, and that one of the most powerful men in Middle Earth didn't want anything to do with it.  Once the reader understands the monster, it won't be scary anymore.  It can still cause tension, but not fear.