I'm going to give my input on this matter, since there are a bunch of people who obviously are commenting with some flawed knowledge(not pointing any fingers).
First off, opiates are one of the most dangerous kinds of drugs, as they are very addictive. While people are prescribed them all the time, this does not mean they are safe. They are powerful, and their physically addictive nature leads to further, and more dangerous use. The effect they have on you lessens and lessens with use, so people often take more and more(they are possibly the easiest to overdose on), or resort to insufflation(snorting) to increase the potency of the drugs.
Ecstasy is not physically addictive, however, I do know many people who become addicted to E. It's a very powerful drug, and a good one, if used responsibly. I've read that far more people die from drinking TOO much water than dehydration, etc, so be careful with advice, especially from individuals who haven't had experiences with a drug themselves. The main problem here lies in the comedown, which is very real, and lasts a fairly long time(often very noticable for a couple days, and tapering off over the course of a week, sometimes more). After a night on E, some people can't deal with the next week of the exact opposite, and take E again, to bring them back up. This is both much less effective, and far more harmful than individual use. Used occasionally(usually at least 2 weeks between doses, the more the better), ecstasy isn't very harmful, but more often, and the effects tend to mount up. The high doesn't get any better, but the comedown continues to get worse, and can easily ruin lives, personal relationships suffer, depression ensues, etc. The physical side effects, like the potential short term memory loss, are far less to worry about than the mental effects from the comedown. A powerful substance, which, if used properly, I don't see any problems with.
Psychedelics are a bit more complex. Mushrooms, LSD, etc, cause next to no harmful physical side effects, but they are very very powerful drugs, and very unpredicatable. Used properly, in the right environment, they can be great fun and rarely cause serious problems, but their power can be scary. Psychological problems don't develop from nothing, but can be revealed by psychedelics. Psychedelics also have the power to change the way you look at things, including your life. This can be good, or bad, depending on how you look at things. For each hippy that drops acid and decides that wandering around following a pink floyd coverband for the next decade is a good plan, there's another person who goes back to their normal daily life a happier person. Also, in this respect, Steve Jobs and Bill Gates both dropped acid, and Steve Jobs considers it one of the most important things he's ever done, so take that for what it's worth. Also, the necessity for a good setting cannot be stressed enough. You are NOT in control on these drugs, you cannot think straight, and you will be incredibly irrational, so a safe comfortable environment is necessary. Bad trips are a fear, although made out to be far worse and more prevalent than some would have you believe, and it's possible you could harm yourself through a number of different ways while affected by psychedelics.
Marijuana, more specifically, THC, is the safest drug I can think of. Tolerance is slow to build, and not a physical phenomenon. Smoking weed can be harmful to the lungs, but there are numerous ways to counteract this. Also of note, cannabis has never been proven to cause cancer. This does not mean it doesn't, but it does mean that it's very hard to prove that it does. It has tar, and other harmful byproducts, but this is by no means the same as smoking cigarettes. You could find dozens upon dozens of runners who smoke daily, and continue to improve their running times. It's very rare to find someone with any sort of lung problems from only smoking weed. Furthermore, there are numerous ways to combat even the minimal effects the inhalation of smoke causes you. Vaporizing and cooking with cannabutter both completely remove the harmful sideeffects of marijuana. Even using water filtration(yes, smoking a bong) counteracts the majority of the harmful effects by removing nearly all the tar. Added to this, the fact that THC is impossible to overdose from, and has never been proven to cause any harmful sideeffects on its own, and you have a very safe drug.
Finally, someone mentioned Ritalin, Adderall, etc. I had a roommate in college who, at the age of 20, had a heart attack. He's been on heavy ADHD medication for years, and there's not a doubt in my mind that these drugs contributed to this. The Pharmaceutical industry, as well as the FDA, are incredibly corrupt and biased. They have a system in place to maintain control of a hugely profitable industry, and ensure that people continue to be dependent on their aid. The fact that something is a "legal" drug prescribed to people has NO effect on its safety, especially in comparison to other drugs.
Hopefully I've cleared up some confusion. I may be mistaken on some of these points, but I'm fairly sure of what I've said. I strongly suggest
www.erowid.org
, it's a great website with a huge amount of information, far more than anyone here, including myself, can provide. Drugs aren't all fun and games, and can be incredibly dangerous, but everyone should know the truth about these substances, and erowid helps to spread this knowledge in a fairly unbiased manner.