You "live in" a country, a state, province or prefecture, a city, or a neighborhood:
I used to live in Canada.
She lives in Beverly Hills.
Sometimes you can use "live on" for explaining which side of a city you live in:
I live on the South side of Chicago.
You live at a specific address:
I live at 112 West Crescent Street.