Coach Nigel Carolan hailed the work being done in Connacht as the Eagles capped a fine day for the province.
As Pat Lam's side were beating Zebre in the Heineken Cup, the second string snatched a late win in the B&I Cup at the Sportsground.
"It is a sign of the progress in the province that we've two teams in international competitions and both getting wins," said Carolan.
Full-back Darragh Leader was the hero when he landed a late penalty to snatch victory.
Connacht trailed 12-3 at the break, the visitors grabbing tries from Jack Preece and Charlie Mulchrone. Captain Jack Carty hit a penalty for Connacht, who made a flying start to the second half as centre Rory Parata touched down and Carty converted.
Rotherham hooker Tom Cruse notched their third try as Lloyd Hayes added the extras.
But tries from Leader and Eoghan Masterson, both converted by Carty, put Connacht 24-22 up. Hayes kicked a penalty for Rotherham but Leader had the last say.
Connacht Eagles– D Leader; F Carr, R Parata, B Murphy, S Macauley; J Carty (c), F Murphy; D Buckley, D Heffernan, F Bealham (S Meuargia 60), D Nolan, M Kearney, C Kindregran (E Masterson 48), J Connolly (U Dillane 60), D Qualter (J Rael 60).
Rotherham Titans – M Keating; J Broadley (S Scanlon 70), J Gill, J Roberts (L Hayes 46), C Wilson; D Murphy, C Mulchrone (D White 54); M Dias (B Sowery 46) (Dias 72), T Cruse, C Quigley (M Tampin 60), B Maddison (c), B Thomas (D Sanderson 54), A Rieder, J Preece, E Williamson (A Birch 62).
REF – A Jackson (RFU)