Update 8/6/09 7:00 pm PDT; 2:00 am GMT (8/7): The RepDB problem has been fully resolved and the system is again functioning normally for all groups on all four database clusters. We appreciate the patience people showed as we worked through this issue.

(Note: If you are still seeing problems in your groups, they have another cause. Please contact Customer Care so that we can investigate.)

This blog post was originally titled “New members not showing up on the member list”, but the problem is actually broader than that and I wanted to take a moment to fully explain.

As background, for groups with 350 or more members, Yahoo! Groups uses a “replicated database system“, which is a key component of the backend systems that have allowed Yahoo! Groups to scale to the size it has (100+ million users across more than 12 million groups, with some groups having more than a million members)

Unfortunately, we have a bug at the moment that is causing our “RepDB system” to back up and process incredibly slowly (most affected groups have not had a RepDB update since late on the 7/17). The impact on affected groups includes the following:

• The member list will be out of date with new members not yet appearing on the list (though they will appear in the management logs, which are fully up to date), making it impossible for moderators to change the settings (or remove) these members.
• Members who are not showing up on the member list will also not receive messages via email (although they can still read messages on the web or post via email).
• Members who have recently unsubscribed (but who were set to receive individual emails or digests) will continue to receive messages from the group until the RepDB system is up to date.
• Changes to email settings will not take effect.
• Members who have been reactivated out of bouncing status will still not get email.
• Exports from the member list will not be up to date (although “downloads” will be).

Apologies for the hassles we know this is causing. Solving this problem is the team’s top priority.

- Gordon Strause
Yahoo! Groups

UPDATE (7/16: 4:55 pm PDT; 11:55 pm GMT): The outage is complete.  If you are still having problems accessing your group (or other issues), please contact Customer Care. All comments to the blog are read by the team, but only Customer Care can actually provide support.

UPDATE: The outage scheduled for Wednesday, July 15, will not take place. Instead, on Thursday, July 16th, there will be a site-wide outage impacting all Yahoo! Groups.

During this time, you will not be able to access any of your groups, and, mail will be delayed. Once the site is back up, mail delivery will resume. This outage will start at about 3pm PT (10pm UTC/GMT-to determine what time the outage will take place in your neighborhood, please visit this link: http://www.timeanddate.com/worldclock/converter.html) and should last roughly 2 hours.

This week (July 14, 15, and 16), we’ll be performing maintenance on the remaining 75% of Yahoo! Groups that were not affected by last Wednesday’s outage.

This maintenance will take place over a three-day TWO DAY period (July 14, 15, and 16), as we’ll be working on 25% of Yahoo! Groups at a time. This means you may be able to access some of your groups, but not all, during the outages. For the affected groups, mail will be delayed, but will return to normal once the outage is complete.

All of the outages will take place at 3pm PT (10pm UTC/GMT-to determine what time the outage will take place in your neighborhood, please visit this link: http://www.timeanddate.com/worldclock/converter.html) on the 14th, 15th, and 16th.

While we know that outages sometimes seem to come at exactly the wrong time, we again hope to have these next three done within 2 hours.

Once the maintenance has been completed, we’ll update this post.

Thanks for your patience!

Melissa Daniels
Yahoo! Community Manager

Update (7/9 4:30 pm PDT;  11:30 pm GMT): Pending notifications and email commands should now be going working normally again (although unlike messages these notifications and commands are not queued, so ones that did not go out during that period will not do so; the implication is that pending messages and members during that time will have to be approved on the web and those trying to subscribe/unsubscribe/changes settings via email will have to resend their requests).

Update (7/9 10:15 am PDT; 7/9 5:15 pm GMT): We’re looking into reports of  email notification of pending members/messages/photos (as well as the confirmation messages for email commands like subscribe and unsubscribe) not being delivered.

Update (7/8 10:30 pm PDT; 7/9 5:30 am GMT): The logs problem has again been resolved and the backlog has been cleared.

In the last six weeks we have posted on the blog about a number of bugs. Here is a quick summary of the status of each of these issues:

BUGS STILL OPEN:

- Delays in Processing Activity Logs: This problem was resolved at the end of last week but returned (though on a much smaller scale) yesterday morning. The good news is that the queues are now headed in the right direction (in other words, the backlog is getting smaller), but we are currently still running about 10 hours behind.

- Message Search: Over the last two weeks, progress on the message search bug has been gated by the logs issue, both because the new message search system is dependent on “the queue” working correctly and because the same team is responsible for both systems. So as soon as the logs issue is fully resolved, this team will begin to migrate more groups on to the new message search system.

- Groups Being Recategorized as Adult: Despite a great deal of effort, we have not yet discovered the cause of the bug that is causing groups to be wrongly misclassified as adult, so we are still working on this problem. In the mean time, we have halted all advertising in adult groups until the issue is resolved.

- Text Disappearing in Some Messages Replies: This problem is being caused by the way some versions of Outlook (as well as Juno Mail) are formatting replies to messages (they are inserting the text of the reply in the wrong place, which is causing the script in Groups which deletes olds headers and footers to delete this text as well).. We have developed a potential fix that we are currently testing in QA, but users can also address the problem themselves by changing their subscription from “fully featured” to “traditional”.

NOTE: You can find information, including suggested work arounds, for other known issues here:
http://help.yahoo.com/l/us/yahoo/groups/original/issues/

RESOLVED BUGS

- Program Initialization/Temporarily Unavailable Errors: This problem has been resolved.

- Problems with Photos Uploads (50% Failure Rates): This problem has been resolved.

- Moderator Unable to Change Home Page Photos: This problem has been resolved.

- Yahoo! Mail UK Delivery Issues: This problem has been resolved.

- Messages Not Wrapping on the Preview Page: This problem has been resolved.

- New Attachments Feature Disabled (or settings changing): This problem has been resolved.

- Verizon & Comcast Delivery Issues: Not actually a Yahoo! Groups bug (the delivery issues were on their end), but based on the fact that we are no longer getting frequent reports of delivery problems to these ISPs, we believe the problem has been resolved.

UPDATE (7/8 3:50 pm PDT, 10:50 GMT): Maintenance is complete.  All groups should again be accessible. Thank you to the affected groups for your patience!

Today, Wednesday, July 8th we will be performing scheduled maintenance on the Yahoo! Groups databases. During this time, some of you may not be able to access your groups, and group mail will be delayed.

Maintenance should take roughly two hours, and will begin at 1pm PT (8pm UTC/GMT- to determine what time the outage will take place in your neighborhood, please visit this link: http://www.timeanddate.com/worldclock/converter.html).

Once the maintenance has been completed, we will update this post.

Thanks for your patience!

Melissa Daniels
Yahoo! Community Manager

UPDATE (7/02 3:30 pm PDT; 10:30 pm GMT):  The problem has now been resolved and all Mail UK users should now be receiving messages from Yahoo! Groups normally, including the backlog of messages that had not been delivered earlier (if you are still having delivery issues, please contact Mail Customer Care).  Our apologies for the inconvience.

We’re aware of the reports from Yahoo! Mail UK users that they are not receiving messages from their Yahoo! Groups and both the Groups and Mail engineering teams are investigating the problem.  Will update this blog post with more info as we have it.

Update (7/3 12:30 pm PDT; 7:30 pm GMT): The problem has been fully resolved. Logs should now be functioning normally.

Update (7/2 9:30 am PDT; 4:30 pm GMT): Further good news on the logs. We’re now only about 24 hour behind and barring any new problems, engineering expects us to be fully caught up by tomorrow.

Update: There is some positive news on the logs front. The backlog is now at 50% of its peak level, which puts us on a pace to be fully caught up by this weekend (currently, activity through the 27th should be captured in the logs). However, one point to be aware of: the time stamp on the logs is showing as the time that the logs were processed in the queue, not as the time the activity took place. Which means that until this backlog is processed and we’re caught up, the time stamps on logs will not correspond to the time the activity took place. Admittedly this is not ideal (we recognize the time stamps should reflect the time the activity actually took place), but we’re not going to try fixing this until after we have everything working normally again.

Here are updates on three problems affecting significant numbers of groups or users.

GROUP ACTIVITY LOGS AND RECENT ACTIVITY NOT UPDATING
As many moderators have probably noticed, the “Group Activity” logs in the Management section have not been updating. The team has been working closely with Tibco, the company behind a key piece of software that Groups relies on to process our incredible volume of these logs, to address the issue, but the problem is not yet solved. And while the work will continue over the weekend (it’s 7:30 pm on Friday in California as I write this), I wouldn’t expect this issue to be resolved before some time next week.

Unfortunately, the queuing system that processes these logs is also the system that updates the “recent activity” module (on group home pages, the Groups Front Page, and My Groups), so this module will be out of date as well until the logs problem is resolved.

MESSAGE SEARCH UPDATE
We’re painfully aware that this is the bug that will not die. And while there is some good news, it’s balanced by some not as good news (caused mainly by the logs problem):
- The Good News: Approximately 8000 groups have been migrated to the new message search system and for these groups, message search is working correctly again (with a couple caveats described in the Bad News below).

- The Bad News I: The new message system also relies on the same queuing system as the logs. So messages posted since the logs began to backlog (for most groups on the 19th) won’t appear in search, even for groups on the new system.
- The Bad News II: No additional groups can be migrated until the queuing system is fixed.
- The Bad News III: A couple (of the nine) servers the new message search system is running on are behaving flakily. So sometimes it may be necessary to repeat a search a couple of the time to get the right results (though we hope to address that problem soon).

VERIZON AND COMCAST DELIVERY ISSUES
We began receiving reports today that some Verizon and Comcast users are again having trouble receiving messages from Yahoo! Groups. Unfortunately, there is nothing we can do to address this issues. We are sending the messages to Verizon and Comcast users, but the ISPs are either bouncing them, delaying them, or dropping them entirely. And this despite the fact that we have filled out (multiple times) the information that Verizon and Comcast ask for in these situations.

At this point, the only thing that is likely to help the problem is for enough Groups users to complain to Verizon and Comcast so that they feel compelled to take action. And when you do complain, there are three things you can do to make it more likely your complaint will be acted upon:

1. If there is information about your latest bounced message on your “bounce history page”, send it along with your complaint.
http://groups.yahoo.com/member/preferences/communication/bounce_history?email=YOUREMAILADDRESS (is the direct link to your bounce history)
2. If the problem is messages being delayed (rather than not delivered), send them the full headers of some of your delayed messages.
3. Point them to this group, which is designed for ISP Admins:
http://tech.groups.yahoo.com/group/ygmailadmin/

We’re sorry for the hassle and inconvenience these problems cause.

- Gordon
Yahoo! Groups

P.S. As always, if you need help from beyond Yahoo! beyond the information in this post, your only option is to contact Customer Care:
http://help.yahoo.com/l/us/yahoo/groups/original/ownmod/forms/forms_index.html

While we read comments on the blog, we do NOT take action on individual complaints. The ONLY way to get individual help is to contact Customer Care.

Did you know that there are more than two thousand groups on Yahoo! Groups dedicated solely to the men and women in the Armed Forces? These include groups dedicated to the men and women in all four branches of the US Armed Forces, groups for Blue Star Moms, and much, much, more.

We know that keeping our troops safe is something many of you find near and dear, which is why Yahoo! recently teamed up with Comedy Central’s Stephen Colbert of the Colbert Report and the philanthropic organization, DonorsChoose.org to lend a hand to the children and families of those who support our troops.

Stephen Colbert, DonorsChoose.org and Yahoo! are working together to give the children of military parents a hand in their education and in their communities.

As part of the “Support Our Troops” campaign, launched in partnership with Colbert, DonorsChoose.org is running a friendly competition among the Armed Forces — Army, Air Force, Marines, Navy and Coast Guard – to see which branch’s supporters can have the biggest impact on classrooms. As of today, we have raised more than $130,000, and the Army is barely hanging on to the lead (check out the leaderboard as the tension builds in real-time).

Literally hundreds of projects have been submitted by teachers at military-serving schools across the country in an effort to show students (and parents!) how proud we all are.

If you want to lend a hand to the DonorsChoose.org campaign click here to donate. Can’t? Show your solidarity by dressing up your Avatar on Yahoo! like Stephen Colbert, or, accessorize your own avatar with things like an eagle, bear, Colbert Nation hat, Che Colbert t-shirt, fireplace, and a desk, and help us spread the word about the Support Our Troops campaign.

Also, be sure to check out video clips from the Colbert Report, including one where Colbert shaves his head in support of the troops!

Thanks for your support!

DonorsChoose.org & Yahoo!

UPDATE: The maintenance was complete at 4:15 pm PDT (12:15 am GMT).  If you are still experiencing problems they are unrelated to this maintenance. Please contact Customer Care so that we can look into the problem:
http://help.yahoo.com/l/us/yahoo/groups/original/members/forms/forms_index.html

On June 11th, 2009 Yahoo! Groups will be undergoing maintenance. During this time 50% of groups will be unavailable for up to an hour. The outage is scheduled to begin some time between 2 and 3 pm PT (10-11 pm GMT).

While the maintenance is going on, all features of affected group, including messages, photos, attachments, etc. will be completely unavailable. However, once the outage concludes, any messages that were posted to these groups via email during this downtime will be distributed.

In an effort to better communicate to all of you, a “blast message” will be posted to all groups later today linking to this blog post. This message will be visible on all groups to all users.

Thank you for your patience—during the outage, this blog will remain up (and comments will remain open), and we will update you all as information becomes available.

- Gordon
Yahoo! Groups

Yesterday around 7pm, message posts to groups were not being delivered and went into a holding state.  The problem has been resolved and the posts since then are continuing to be sent out.  There will be a delay as the system works through the backlog of messages.  The delay will last a few hours but all messages should be posted by the end of the day and all new posts should resume back to normal.

Thanks for your patience,
- Raj
Yahoo Groups

UPDATE ON MESSAGE SEARCH (6/8 3:00 pm PDT/11:00 pm GMT): The good news is that the new message search is working correctly on one of our new production search servers. The bad news is that it’s not yet working correctly on all of them. Once this last issue is resolved, we will migrate groups that have signed up on to the new system. Our apologies for the delay.

Below is an update on our highest priority bugs, all of which our engineering team is actively working to squash.  Thank you for your patience as we work to resolve these issues, and we’re sorry for any hassles these bugs have caused you and your groups.

- Gordon
Yahoo! Groups

P.S. If you would like to report additional issues not mentioned below, please contact Customer Care . While we do read all comments, we need the information that the Customer Care form collects to be able to properly investigate a bug.

TOP PRIORITY BUGS

MESSAGE SEARCH PROBLEMS
This is our number one priority bug since it’s affecting so many groups and has no work around. As folks who read the blog know, we have been working hard to rebuild our message search system for a number of months .

The Bad News: The final phases of the this project have proved trickier than expected, which is why we haven’t met our goal of having groups that have signed up to be migrated on the new system by mid-May.

The Good News: Talked again with the engineer leading this project this morning, and he believes we’re very close to being able to roll out the new system more broadly. In fact, he hopes to solve the last remaining issues and actually begin migrating more groups as early as tomorrow (Wednesday, May 26th),  although I think Thursday (Pacific Time) is more realistic.  And assuming he encounters no unexpected problems, we should still be able to have all groups that have signed up to be migrated on the new system by the end of the month.

PROGRAM INITIALIZATION OR TEMPORARILY UNAVAILABLE ERRORS
We know that many groups are seeing this error when they land on the group home page or pending messages page. Usually, the error will go away if you refresh the page, but if the problem persists despite multiple refreshes, please contact Customer Care .

GROUPS BEING MISTAKENLY CLASSIFIED AS ADULT
There has been an ongoing known issue where groups are mistakenly recategorized behind the "adult wall" (meaning that no one under 18 can access). And while annoying, until recently Customer Care has been able to fix this mistake quickly for affected groups (when it is brought to their attention).

But in the past few weeks, for a handful of groups, the problem has become much more frustrating because they are being reclassified as adult every single day. In other words, the problem is being fixed by Customer Care, but a bug is causing the group is to placed back behind the adult wall again the next day.

We are working hard to identify what is triggering this bug (it has nothing to do with the content of the group), but as of yet have not identified the root cause. Once we do, we will roll out a fix. In the mean time, if your group is being affected by this issue, please let Customer Care know , and they will manually fix your group every day until the problem stops.

NEW TEXT DISAPPEARING IN MESSAGE REPLIES
When replying to group messages, some users are experiencing problems with the text they typed not displaying in their reply message.  Instead only the text of the message they are replying to is displaying.

The problem appears to be mainly affecting users of Outlook (and apparently Juno Web Mail), and it is only affecting replies to certain messages. We know the problem is related to HTML formatting issues, but we are still investigating the precise root cause and how to resolve the issue.

For more information, including suggested work arounds, see our "known issue" page here .

MODERATORS UNABLE TO CHANGE THE GROUP HOME PAGE PHOTO
Some moderators are currently getting errors when they try to update their group’s home page photo. Please contact Customer Care if this problem is affecting you.

We should be able to manually fix the problem for your group right away (though it may take us longer to roll out a more global fix that will prevent the problem from occurring in the first place).

PROBLEMS UPLOADING PHOTOS
Some users are reporting problems uploading photos, with only half the photos they submit being successfully uploaded in any given upload attempt. If this problem is affecting you, please contact Customer Care .