There was a post recently on another blog (I can't remember for the life of me which blog it was) about when was the best time of day to take pictures of a garden. Overall, when shooting on a sunny day, either sunrise or sunset are the best time to shoot. The side lighting produces interesting shadows and the warmer light produces richer, more saturated colors, much like the image below of some geranium 'Rozanne', which, during the day look very cool blue. At dusk, though, the warming effects of the light tint the flowers more of a mauve.
These times also give opportunities for interesting backlighting effects (below). Backlight has the interesting effect of highlight any little hairs and things on plants, making them seem to glow.
Honestly, though, I think that the best time to shoot gardens is during an overcast day. For an illustration of this, see below.
This first image is shot right around noon on a bright, sunny day. Notice how flat and harsh the image is. The shadows are hard-edged, and when paired with the glare off the leaves, create a very "busy' image. Colors are washed out in some places and too "hot" in others.
Now, compare to this image. Colors are rich and saturated and the overall tone is even and balanced. The light is soft and calming and there is very little glare to distract the eye. Overcast days act like a natural diffuser and soften light and lessen the presence of hard shadows and glare. If you find yourself faced with a foggy day, rejoice! This is my favorite time to take garden pictures. You can create a really amazing sense of mystery and depth with fog, making smaller gardens seem larger. You can also do yourself a favor and get a polarizer, which can help by cutting glare even more.