You "live in" a country, a state, province or prefecture, a city, or a neighborhood:

I used to live in Canada.

She lives in Beverly Hills.

Sometimes you can use "live on" for explaining which side of a city you live in:

I live on the South side of Chicago.

You live at a specific address:

I live at 112 West Crescent Street.