After implementing the sort by date feature we had a lot of downtime due to the low amount of ram on the main server. We moved the main site to a server that has 8GB of RAM and I hope that will solve most of the problems cause by the crawlers when we updated the index with new videos. From now on we will try to update the site faster (at least hourly).