Update (7/3 12:30 pm PDT; 7:30 pm GMT): The problem has been fully resolved. Logs should now be functioning normally.

Update (7/2 9:30 am PDT; 4:30 pm GMT): Further good news on the logs. We’re now only about 24 hour behind and barring any new problems, engineering expects us to be fully caught up by tomorrow.

Update: There is some positive news on the logs front. The backlog is now at 50% of its peak level, which puts us on a pace to be fully caught up by this weekend (currently, activity through the 27th should be captured in the logs). However, one point to be aware of: the time stamp on the logs is showing as the time that the logs were processed in the queue, not as the time the activity took place. Which means that until this backlog is processed and we’re caught up, the time stamps on logs will not correspond to the time the activity took place. Admittedly this is not ideal (we recognize the time stamps should reflect the time the activity actually took place), but we’re not going to try fixing this until after we have everything working normally again.

Here are updates on three problems affecting significant numbers of groups or users.

GROUP ACTIVITY LOGS AND RECENT ACTIVITY NOT UPDATING
As many moderators have probably noticed, the “Group Activity” logs in the Management section have not been updating. The team has been working closely with Tibco, the company behind a key piece of software that Groups relies on to process our incredible volume of these logs, to address the issue, but the problem is not yet solved. And while the work will continue over the weekend (it’s 7:30 pm on Friday in California as I write this), I wouldn’t expect this issue to be resolved before some time next week.

Unfortunately, the queuing system that processes these logs is also the system that updates the “recent activity” module (on group home pages, the Groups Front Page, and My Groups), so this module will be out of date as well until the logs problem is resolved.

MESSAGE SEARCH UPDATE
We’re painfully aware that this is the bug that will not die. And while there is some good news, it’s balanced by some not as good news (caused mainly by the logs problem):
- The Good News: Approximately 8000 groups have been migrated to the new message search system and for these groups, message search is working correctly again (with a couple caveats described in the Bad News below).

- The Bad News I: The new message system also relies on the same queuing system as the logs. So messages posted since the logs began to backlog (for most groups on the 19th) won’t appear in search, even for groups on the new system.
- The Bad News II: No additional groups can be migrated until the queuing system is fixed.
- The Bad News III: A couple (of the nine) servers the new message search system is running on are behaving flakily. So sometimes it may be necessary to repeat a search a couple of the time to get the right results (though we hope to address that problem soon).

VERIZON AND COMCAST DELIVERY ISSUES
We began receiving reports today that some Verizon and Comcast users are again having trouble receiving messages from Yahoo! Groups. Unfortunately, there is nothing we can do to address this issues. We are sending the messages to Verizon and Comcast users, but the ISPs are either bouncing them, delaying them, or dropping them entirely. And this despite the fact that we have filled out (multiple times) the information that Verizon and Comcast ask for in these situations.

At this point, the only thing that is likely to help the problem is for enough Groups users to complain to Verizon and Comcast so that they feel compelled to take action. And when you do complain, there are three things you can do to make it more likely your complaint will be acted upon:

1. If there is information about your latest bounced message on your “bounce history page”, send it along with your complaint.
http://groups.yahoo.com/member/preferences/communication/bounce_history?email=YOUREMAILADDRESS (is the direct link to your bounce history)
2. If the problem is messages being delayed (rather than not delivered), send them the full headers of some of your delayed messages.
3. Point them to this group, which is designed for ISP Admins:
http://tech.groups.yahoo.com/group/ygmailadmin/

We’re sorry for the hassle and inconvenience these problems cause.

- Gordon
Yahoo! Groups

P.S. As always, if you need help from beyond Yahoo! beyond the information in this post, your only option is to contact Customer Care:
http://help.yahoo.com/l/us/yahoo/groups/original/ownmod/forms/forms_index.html

While we read comments on the blog, we do NOT take action on individual complaints. The ONLY way to get individual help is to contact Customer Care.

Did you know that there are more than two thousand groups on Yahoo! Groups dedicated solely to the men and women in the Armed Forces? These include groups dedicated to the men and women in all four branches of the US Armed Forces, groups for Blue Star Moms, and much, much, more.

We know that keeping our troops safe is something many of you find near and dear, which is why Yahoo! recently teamed up with Comedy Central’s Stephen Colbert of the Colbert Report and the philanthropic organization, DonorsChoose.org to lend a hand to the children and families of those who support our troops.

Stephen Colbert, DonorsChoose.org and Yahoo! are working together to give the children of military parents a hand in their education and in their communities.

As part of the “Support Our Troops” campaign, launched in partnership with Colbert, DonorsChoose.org is running a friendly competition among the Armed Forces — Army, Air Force, Marines, Navy and Coast Guard – to see which branch’s supporters can have the biggest impact on classrooms. As of today, we have raised more than $130,000, and the Army is barely hanging on to the lead (check out the leaderboard as the tension builds in real-time).

Literally hundreds of projects have been submitted by teachers at military-serving schools across the country in an effort to show students (and parents!) how proud we all are.

If you want to lend a hand to the DonorsChoose.org campaign click here to donate. Can’t? Show your solidarity by dressing up your Avatar on Yahoo! like Stephen Colbert, or, accessorize your own avatar with things like an eagle, bear, Colbert Nation hat, Che Colbert t-shirt, fireplace, and a desk, and help us spread the word about the Support Our Troops campaign.

Also, be sure to check out video clips from the Colbert Report, including one where Colbert shaves his head in support of the troops!

Thanks for your support!

DonorsChoose.org & Yahoo!

UPDATE: The maintenance was complete at 4:15 pm PDT (12:15 am GMT).  If you are still experiencing problems they are unrelated to this maintenance. Please contact Customer Care so that we can look into the problem:
http://help.yahoo.com/l/us/yahoo/groups/original/members/forms/forms_index.html

On June 11th, 2009 Yahoo! Groups will be undergoing maintenance. During this time 50% of groups will be unavailable for up to an hour. The outage is scheduled to begin some time between 2 and 3 pm PT (10-11 pm GMT).

While the maintenance is going on, all features of affected group, including messages, photos, attachments, etc. will be completely unavailable. However, once the outage concludes, any messages that were posted to these groups via email during this downtime will be distributed.

In an effort to better communicate to all of you, a “blast message” will be posted to all groups later today linking to this blog post. This message will be visible on all groups to all users.

Thank you for your patience—during the outage, this blog will remain up (and comments will remain open), and we will update you all as information becomes available.

- Gordon
Yahoo! Groups

Yesterday around 7pm, message posts to groups were not being delivered and went into a holding state.  The problem has been resolved and the posts since then are continuing to be sent out.  There will be a delay as the system works through the backlog of messages.  The delay will last a few hours but all messages should be posted by the end of the day and all new posts should resume back to normal.

Thanks for your patience,
- Raj
Yahoo Groups