I went into this only agreeing because he was moving across the country etc. and it felt like the right thing to do to say a quick good bye. I also said I preferred to do lunch because I had some things to do in the afternoon/wanted to keep it casual and he asked if "this was the only time in a month I was free so I was trying to cram everyone in."
The lunch thing seems to be an issue, as he said "I wish I could see you for more than just lunch though."
I really thought this was just going to be a quick thing, hence why I agreed, and yes it does seem like he's looking to hang out more. I was hoping to just make a clean situation of this and get out. His comments seem to insinuate things like "are you really that busy to only be able to do this time" and obviously I don't want to flat out say that there are other things I do that have higher priority. I think I'm just going to have to stick to the original plan..