We develop an economic geography model where mobile skilled workers choose to either work in a production sector or to become part of an unproductive elite. The elite sets income tax rates to maximize its own welfare by extracting rents, thereby influencing the spatial structure of the economy and changing the available range of consumption goods. We show that either unskilled labor mobility, or rent-seeking behavior, or both, are likely to favor the occurence of agglomeration and of urban primacy. In equilibrium, the elite may tax the unskilled workers but does not tax the skilled workers, and there are rural-urban transfers towards the agglomeration. The size of the elite and the magnitude of the tax burden that falls on the unskilled decrease with product differentiation and with the expenditure share for manufacturing goods. All these results are broadly in line with observed patterns of urban primacy and economic development in sub-Saharan African countries.