In this paper we model systemic risk by making use of the conditional quantile regression to identify the most systemically important and vulnerable banks in the South Africa (SA) banking sector. We measure the marginal contributions of each bank to systemic risk by computing the delta Conditional Value at Risk which measures the difference between system risk of individual banks when they are in a normal state and when they are in distress state. Using daily stock market closing prices of six South African banking banks from 19 June 2007 to 11 April 2016; our back tested systemic risk measures suggest that the contribution of South African banks to systemic risk tends to significantly increase during periods of financial crises. The two largest banks namely First Rand Bank and Standard Bank are found to be the highest contributors to systemic risk while the smallest bank namely African Bank is found to be the least contributor to the overall systemic risk in South African banking sector. Based on the delta Conditional Value at Risk; we show that there is a need to go beyond micro prudential regulation in order to sustain stability in the South African banking sector.