Who would like to access a website which takes time for loading? I don't have a lot of patience
According to my knowledge you don't need to worry about the effect on ranking, because website speed time will play a very minor role in ranking of your site. However, if your website loads too slow, then there are lot of chances that you may loose your potential visitors quickly.
If you are still concerned about this issue, then I would recommend you to follow matt cutts blog, where he has described the relationship between website speed and its effect on ranking.