There are basically two methods of making hills and mountains for your model railroad, the Papier-Mâché Method and the Window Screening Method. ("Papier-Mâché" is French, and is often misspelled as "paper mache" in English.)
If your layout is temporary, or you don't want to spend much money, you can use the papier-mâché method. Basically, you just scrunch up some newspapers into the general shape of the hill or mountain, then cover them with sheets of newspaper dipped in papier-mâché paste.
The traditional method of making papier-mâché paste is to add one part water to one part flour. Some people prefer to boil the paste, saying that it makes the consistency smoother. Water to flour ratio is changed to five parts water to one part flour in this case. Other people use a three to one or a one to one ratio of water and polyvinyl acetate wood glue (white glue). Still others say that starch makes a fine paste. Adding a dash of cinnamon to the mixture gives the paste a pleasant smell; adding a dash of salt and sugar reduces the chances of the product developing mould.
You simply dip sheets of newspaper (fold them a couple of times to make them thicker) in the paste and drape them over the scrunched up newspapers. The sheets will be soft and pliable, so mould them into hills and valleys as you like. Let the papier-mâché dry overnight. You will find that it is hard and rigid, like a piñata.
Paint it with a water-based paint in shades of green (for grass) and brown (for dirt) and grey (for rock). You can also use spray paint, but make sure it is matte, not glossy, paint. While the paint is still wet, sprinkle some grass (available at the hobby shop) or green-coloured sawdust. When the paint is dry, add some lichen for bushes and trees (also available at any hobby shop).
Window Screening and Wooden Supports
If your train layout is permanent, a more durable method of making mountains and hills is to use screening and wooden supports. This also has the advantage that you can run tracks on the mountain itself, providing you make the appropriate supports. Basically, you cut wooden support blocks to the appropriate size, then drape window screening (called "fly screen" in Australia) over the blocks, stapling the screen to the blocks. You should fasten the blocks to the table with screws or nails first, so they don't move around.
If you want, you can build a second track bed around or through the mountain, elevated above the table. Simply use support blocks and plywood to make a rigid base for the track.
Once the screening is in place, pinch it in places to make rugged edges, ditches, and valleys. Then mix up some common household plaster (available at any hardware store to fix holes in a plaster wall) and slobber it over the screen. There is no need to be careful, just make sure all the screen is covered. A thicker plaster mix works much better than a thin mix, as the latter allows the screen to show through.
When you get to the rugged vertical bits, use horizontal strokes of a knife to simulate rock cuts. Let the plaster dry overnight and then paint as described above.
Finally, finish it off with lichen bushes and trees as appropriate.
Don't Forget Access
When you are deciding how big to make your mountain, and where to place it, also think about derailments. Murphy's Law guarantees that, at some point, your train is going to derail in the tunnel through your mountain. Make sure the tunnel entrance is large enough so you can reach in and get your train out.
Or consider these alternatives: If you have made a papier-mâché mountain, and not glued it to the table, you can simply lift it off the table.
If you have made a permanent wood block and wire screen mountain, then consider cutting access holes in the table, below the mountain, so you can reach in and right the train or remove it. Don't make the holes too close to the track, because if the train does derail at that spot, you don't want it to fall through the hole onto the floor.
You can get more information on model railroading at