Uphill on the downhill?

So remember my "Trail Ride in the Dark" post? I commented on how during the downhill portion of the ride it was dark and I rode slower because of limited view and by the time I turned around it was lighter out so I was able to pick up speed a little. Well I'm wanting to rethink that based on yesterday's ride.
I rode the same trail only in the afternoon. The down hill was disappointing to me because I couldn't get much over 10mph. The trail consists of silty, sandy material and I kept justifying my slow speed to the probability that it was drier yesterday than before thus making it softer. Then I turned around to go back uphill and my average speed jumped to 12mph.
This makes me think that it is more uphill (we're talking gradual maybe 2% if we're lucky) down to the river. I do drop a bit before I get to the river's wildlife preserve area... maybe it drops too much and has to come back up? I mean water flows towards the river not away from it - it has to be downhill. Its messing with my brain. I'm going to have to go back with a GPS and see what the elevations are.
It could have been the breeze, though I didn't notice it until I was back on black top headed home, but I was riding perpendicular to the downhill portion into the wind... I don't know, but I do know that I was doing a brick and regardless of the mind bending experience I still kicked butt on the bike, but my body had its revenge during the run. I find that if I chant like Dora from Finding Nemo that it really helps. Just keep running, just keep running...