There are a huge number of clubs around London, some of which have very competitive members, and most of which put on track sessions on at least a weekly basis.
Some suggestions for competitive clubs would be Highgate Harriers (based near Hampstead Heath in North London); Serpentine (based all over the place, but do sessions in Hampstead Heath, Paddington and Battersea); Hercules Wimbledon (South West London). But there are dozens, you really won't struggle to find a good club as the scene is fantastic.
As to where to run, it depends where you live, seeing as London is huge. Living anywhere central will be prohibitively expensive, so I would recommend either North London or South West London. Anywhere else and you will struggle to find more than smallish parks, and tarmac paths.
I live in the South West, and within 10 minutes jogging, I can get onto Wimbledon Common, which is attached to Richmond Park, and on which you could do miles of running purely on soft surfaces; with some amazing views. Or I can get down to the Thames, running West that is also mainly on trails, and going east you have some incredible views of the City.
North London has Hampstead Heath, which is big and hilly, and is also a great place to run.