You are correct but its due to scale.
The energy conversion of a big power plant is higher, better,than a smaller conversion. So the energy conversion of fossil fuel in a big, efficient, power station, is better than inside a small engine in your vehicle.
Of course if those big efficient power statons were solar, wind, nuclear, then the fossil fuel issue is diminished but the manufacturing of those power stations, they contain consumption of fossil fuel, so everything you do at the point of consumption (drive a car, say) is necessarily harmful to the environment somewhere to some degree.
The most green thing you could do is never exist, or have no children, the next level up is be vegetarian, higher levels of less-green is ride a bike or use mass transit (efficiency) systems. So to a some degree your existence now is poisoning the future. Of course if everyone was a scientist so existence led to a future which was sustainable and technological then you're permitted to exist, for now. 
So our robots in space crunching up meteorites and nano-scale bots make up new things we then use, and we just spread out across the galaxy, we'd likely survive a few trillion years, rather then few thousand years with the current geo-political reality of we carved up the planet into race, religion and countries who waste time disagreeing.
