Dan Biggar kicked two penalties and a drop goal to give the hosts a 9-6 half-time lead, with a further penalty from the outhalf and an excellent finish from man of the match Eli Walker sealing a famous win.
Toulouse's only response came from a penalty and drop goal from Jean-Marc Doussain. Their failure to at least pick up a losing bonus point throws Pool 2 wide open.
The Ospreys will need to beat Leicester and Treviso and hope other results go their way to advance, but yesterday's result will lift the spirits of Wales as a rugby nation.
Biggar had booted them into a third-minute lead before a loose pass from Justin Tipuric halted a promising attacking situation.
Biggar doubled the lead with a well-taken drop goal at the end of the opening quarter after another pinpoint up and under from the outhalf had caused panic in the visiting ranks.
Doussain got Toulouse on the board in the 29th minute, but the Ospreys twice turned down shots at goal in an effort to make their numerical advantage tell with a try but, after being repelled by some superb Toulouse defence, Biggar was soon asked to step up to make it 9-3.
Doussain hit back with a drop goal right on half-time.
Toulouse skipper Jean Bouilhou was sin-binned for a high tackle on wing Walker on the hour, with Biggar slotting the resulting penalty.
Walker, who had looked a constant threat, then escaped some weak tackling to finish off excellent Ospreys build-up play. Biggar could not add the extras but the Welsh side held on for a deserved victory.
Scorers – Ospreys (E Walker try; D Biggar 3 pens, dg) Toulouse (J-M Doussain 2 pens)
Ospreys: R Fussell, R Jones, A Bishop, A Beck, E Walker, D Biggar, K Fotuali'i; R Bevington (DJ Jones 58), R Hibbard, C Johnstone (A Jones 44), I Gough (L Peers 44), J King, R Jones, J Tipuric, J Bearman (C Johnstone 75).
Toulouse: M Medard, V Clerc, Y David (Y Jauzion 67), G Fickou, Y Huget, J-M Doussain, L Burgess; V Kakovin (N Kabasele 49), G Botha (C Tolofua 49), C Johnston (Y Montes 63), Y Maestri (R Millo-Chluski 60), P Albacete, J Bouilhou (G Lamboley 69), T Dusatoir (J B Poux 49), L Picamoles.
Referee: G Clancy (IRFU)