Update (7/3 12:30 pm PDT; 7:30 pm GMT): The problem has been fully resolved. Logs should now be functioning normally.
Update (7/2 9:30 am PDT; 4:30 pm GMT): Further good news on the logs. We’re now only about 24 hour behind and barring any new problems, engineering expects us to be fully caught up by tomorrow.
Update: There is some positive news on the logs front. The backlog is now at 50% of its peak level, which puts us on a pace to be fully caught up by this weekend (currently, activity through the 27th should be captured in the logs). However, one point to be aware of: the time stamp on the logs is showing as the time that the logs were processed in the queue, not as the time the activity took place. Which means that until this backlog is processed and we’re caught up, the time stamps on logs will not correspond to the time the activity took place. Admittedly this is not ideal (we recognize the time stamps should reflect the time the activity actually took place), but we’re not going to try fixing this until after we have everything working normally again.
Here are updates on three problems affecting significant numbers of groups or users.
GROUP ACTIVITY LOGS AND RECENT ACTIVITY NOT UPDATING
As many moderators have probably noticed, the “Group Activity” logs in the Management section have not been updating. The team has been working closely with Tibco, the company behind a key piece of software that Groups relies on to process our incredible volume of these logs, to address the issue, but the problem is not yet solved. And while the work will continue over the weekend (it’s 7:30 pm on Friday in California as I write this), I wouldn’t expect this issue to be resolved before some time next week.
Unfortunately, the queuing system that processes these logs is also the system that updates the “recent activity” module (on group home pages, the Groups Front Page, and My Groups), so this module will be out of date as well until the logs problem is resolved.
MESSAGE SEARCH UPDATE
We’re painfully aware that this is the bug that will not die. And while there is some good news, it’s balanced by some not as good news (caused mainly by the logs problem):
- The Good News: Approximately 8000 groups have been migrated to the new message search system and for these groups, message search is working correctly again (with a couple caveats described in the Bad News below).
- The Bad News I: The new message system also relies on the same queuing system as the logs. So messages posted since the logs began to backlog (for most groups on the 19th) won’t appear in search, even for groups on the new system.
- The Bad News II: No additional groups can be migrated until the queuing system is fixed.
- The Bad News III: A couple (of the nine) servers the new message search system is running on are behaving flakily. So sometimes it may be necessary to repeat a search a couple of the time to get the right results (though we hope to address that problem soon).
VERIZON AND COMCAST DELIVERY ISSUES
We began receiving reports today that some Verizon and Comcast users are again having trouble receiving messages from Yahoo! Groups. Unfortunately, there is nothing we can do to address this issues. We are sending the messages to Verizon and Comcast users, but the ISPs are either bouncing them, delaying them, or dropping them entirely. And this despite the fact that we have filled out (multiple times) the information that Verizon and Comcast ask for in these situations.
At this point, the only thing that is likely to help the problem is for enough Groups users to complain to Verizon and Comcast so that they feel compelled to take action. And when you do complain, there are three things you can do to make it more likely your complaint will be acted upon:
1. If there is information about your latest bounced message on your “bounce history page”, send it along with your complaint.
http://groups.yahoo.com/member/preferences/communication/bounce_history?email=YOUREMAILADDRESS (is the direct link to your bounce history)
2. If the problem is messages being delayed (rather than not delivered), send them the full headers of some of your delayed messages.
3. Point them to this group, which is designed for ISP Admins:
We’re sorry for the hassle and inconvenience these problems cause.
While we read comments on the blog, we do NOT take action on individual complaints. The ONLY way to get individual help is to contact Customer Care.