I coached a girl years ago that got a partial scholarship to run at a NAIA school. Her freshman year, she saw some good improvement and ended up being an All-American in cross country. She liked her coach and had chopped about 30 seconds off her best high school 5k time. However, she felt like she was missing out on the fun social stuff from school (she was very much the social extrovert) because their practices were held in the late afternoon/early evening.
The last couple weeks of the semester, after XC season was done, she had some down time and really enjoyed getting to go eat dinner with her non-XC friends and other social activities. When she came back from Christmas break, she told her coach that she was quitting and giving up her scholarship.
Since her parents weren't rich, she had to work to make up for the loss of the scholarship money. She ended up having to work as many hours as she was putting in running before to earn enough money. Ironically, one of the jobs she worked was in the cafeteria, so she couldn't have dinner with her friends anyway.
Basically, what ended up happening was she traded the time she spent running (she actually liked to run and still does run recreationally to this day) for time spent working in the cafeteria and other campus jobs she didn't really enjoy. She did get a little more social time since she wasn't traveling to meets on the weekends, but overall, she wishes she had stuck with the running for multiple reasons.
OP, get through your freshman year (as a previous poster said, the first year transition is tough) and re-evaluate after that time. As another poster said, you have your whole life to party and "have fun". I can't tell you how many kids I coached who didn't come out for track as seniors so they could "have fun" ended up regretting their decisions.