I initially did not like my undergrad experience. I was poor and felt excluded at one of the wealthiest and so-called prestige schools in the country. It was free, though, through athletic scholarship. I was very immature and knew it at some level, so there was lots of rationalizations which didn't help me.
My a-ha moment occurred while sitting on the campus bus on a beautiful day. What a lucky moment. I felt shame (not in a bad way) at feeling so entitled and decided to make what went on in the four corners of the classroom the main priority. I was way luckier than the Teamsters union guys I worked with on my summer job, who were taking on second jobs to get their kids through the local state schools. Those guys, who did not have a formal education would have kicked my rear if I didn't take advantage of the opportunity I had, and frankly, I owe them a lot for caring so much. Their sons and daughters were incredibly fortunate, too. The point is that hard work is just something to be appreciated anywhere. I begged my way into an honors program, and despite not being the caliber of student as the other 9 participants in the program, every month I got better and better and at the end was at the top of the program. I finished my final thesis in the middle of the night on an IBM typewriter in a hotel near Franklin Field at the Penn Relays. The great thing about it was that my fellow participants were my biggest supporters - they knew I was a bit behind academically at the start and by and large people appreciate real effort. Of course, I gave support back in return. I ended up liking my peers tremendously. As tlm1959 suggests, you can find your home, and people matter much more than places.
If the school is University of Illinois, you can find as much challenge as you want. I am from Illinois and did not consider the school, and probably should have (getting home would have been much easier). I was dissuaded by a training run on a visit to campus, where I saw the heat shimmering off the endless flat cornfields, and in my immaturity, said this wasn't for me. Beauty is what you make of it. Lots of great U of I runners likely cherish those runs on the flat terrain of Savoy, and I can see why, especially in the winter where the flat terrain permits seeing things with a clarity that intense running can bring about it. In any event, U of I is a great school, with far more options in terms of fields of study than a smaller school (my school had virtually no vocationally useful majors save for a small (then) engineering school). Like almost every one else at my school, I ended up paying for grad school, and although it worked out very well, the option of working right away in a good job in a field you like is a nice option to have. My stepsister went to U of I and changed majors three times, with a career as a comptroller for a large manufacturer. Illinois did everything right by her.
Transferring is not a bad thing, but the school is what you make of it. Be appropriately skeptical of yourself, and make sure the alternative is truly better for you.