Carlton is a city in Madison County, Georgia, United States. The population was 263 at the 2020 census. An early variant name of Car…Carlton is a city in Madison County, Georgia, United States. The population was 263 at the 2020 census. An early variant name of Carlton was "Berkeley", to avoid confusion with Carrollton, Georgia. The Georgia General Assembly incorporated Carlton in 1892.