Statistics from the Wikipedia list of U.S. cities by population, with numbers for 2010 and 2014 for cities 100,000 and over. My analysis excludes Kent, WA and Macon, GA which had non-organic annexations.
Cities that gained population had, on average, 297,130 people. Cities that lost had, on average, 223,384 people. The smaller you are, the smaller you get. A scatter plot (log or absolute population) was not very informative, other than to show that the shrinking cities were all smaller. The largest city that lost population was (hold your breath) Detroit, which lost 33,522, or 4.7% over 4 years - that annualizes to 1.2% lost per year.
Cities of 100,000 or more added 4,099,428, a 5.02% growth rate for growing cities for 2010-2014 (annualized to 1.23%).1111
In the same period, cities of 100,000 or more lost 65,007, a -1.46% loss rate for shrinking cities for 2010-2014 (annualized to -0.366%).
This means on net, cities over 100,000 added a total of 4,034,421 people, for an overall growth rate of 4.68% (annualized to 1.15%).
Compare this, for the same period, to a US population growth of 3.28% (0.809% per year). Cities above 100,000 grew faster than the country as a whole.
The center of growth was at 36.232 N, 99.661 W (western Oklahoma).
The center of shrinking, on average, were at 38.145, 84.188 (suburbs of Lexington, KY). Not surprising that there's more growth to the west (and somewhat to the south).
Weighting it by % gain or loss, it doesn't move very much. The center of shrinking moves about 40 miles, and the center of growth moves less than that.