There is a data-collection plugin for MRTG called mrtg-ping-probe that returns the min/max latencies (and jitter if you want); if you are using Routers2 with MRTG then it has a special mode to handle this which graphs it as a floating bar. You can get the plugin from many places; I think it is in the MRTG contrib folder, and the Routers2 extras folder.
Having said that, Smokeping does the job far better, though it is a different package. If you have alot of these you want to monitor, it may be a better choice. If you have a singlelatency to monitor and many other MRTG metrics already in place, then maybe use MRTG for the latency graph.
Steve