UPDATE (7/16: 4:55 pm PDT; 11:55 pm GMT): The outage is complete.  If you are still having problems accessing your group (or other issues), please contact Customer Care. All comments to the blog are read by the team, but only Customer Care can actually provide support.

UPDATE: The outage scheduled for Wednesday, July 15, will not take place. Instead, on Thursday, July 16th, there will be a site-wide outage impacting all Yahoo! Groups.

During this time, you will not be able to access any of your groups, and, mail will be delayed. Once the site is back up, mail delivery will resume. This outage will start at about 3pm PT (10pm UTC/GMT-to determine what time the outage will take place in your neighborhood, please visit this link: http://www.timeanddate.com/worldclock/converter.html) and should last roughly 2 hours.

This week (July 14, 15, and 16), we’ll be performing maintenance on the remaining 75% of Yahoo! Groups that were not affected by last Wednesday’s outage.

This maintenance will take place over a three-day TWO DAY period (July 14, 15, and 16), as we’ll be working on 25% of Yahoo! Groups at a time. This means you may be able to access some of your groups, but not all, during the outages. For the affected groups, mail will be delayed, but will return to normal once the outage is complete.

All of the outages will take place at 3pm PT (10pm UTC/GMT-to determine what time the outage will take place in your neighborhood, please visit this link: http://www.timeanddate.com/worldclock/converter.html) on the 14th, 15th, and 16th.

While we know that outages sometimes seem to come at exactly the wrong time, we again hope to have these next three done within 2 hours.

Once the maintenance has been completed, we’ll update this post.

Thanks for your patience!

Melissa Daniels
Yahoo! Community Manager

Update (7/9 4:30 pm PDT;  11:30 pm GMT): Pending notifications and email commands should now be going working normally again (although unlike messages these notifications and commands are not queued, so ones that did not go out during that period will not do so; the implication is that pending messages and members during that time will have to be approved on the web and those trying to subscribe/unsubscribe/changes settings via email will have to resend their requests).

Update (7/9 10:15 am PDT; 7/9 5:15 pm GMT): We’re looking into reports of  email notification of pending members/messages/photos (as well as the confirmation messages for email commands like subscribe and unsubscribe) not being delivered.

Update (7/8 10:30 pm PDT; 7/9 5:30 am GMT): The logs problem has again been resolved and the backlog has been cleared.

In the last six weeks we have posted on the blog about a number of bugs. Here is a quick summary of the status of each of these issues:

BUGS STILL OPEN:

- Delays in Processing Activity Logs: This problem was resolved at the end of last week but returned (though on a much smaller scale) yesterday morning. The good news is that the queues are now headed in the right direction (in other words, the backlog is getting smaller), but we are currently still running about 10 hours behind.

- Message Search: Over the last two weeks, progress on the message search bug has been gated by the logs issue, both because the new message search system is dependent on “the queue” working correctly and because the same team is responsible for both systems. So as soon as the logs issue is fully resolved, this team will begin to migrate more groups on to the new message search system.

- Groups Being Recategorized as Adult: Despite a great deal of effort, we have not yet discovered the cause of the bug that is causing groups to be wrongly misclassified as adult, so we are still working on this problem. In the mean time, we have halted all advertising in adult groups until the issue is resolved.

- Text Disappearing in Some Messages Replies: This problem is being caused by the way some versions of Outlook (as well as Juno Mail) are formatting replies to messages (they are inserting the text of the reply in the wrong place, which is causing the script in Groups which deletes olds headers and footers to delete this text as well).. We have developed a potential fix that we are currently testing in QA, but users can also address the problem themselves by changing their subscription from “fully featured” to “traditional”.

NOTE: You can find information, including suggested work arounds, for other known issues here:
http://help.yahoo.com/l/us/yahoo/groups/original/issues/

RESOLVED BUGS

- Program Initialization/Temporarily Unavailable Errors: This problem has been resolved.

- Problems with Photos Uploads (50% Failure Rates): This problem has been resolved.

- Moderator Unable to Change Home Page Photos: This problem has been resolved.

- Yahoo! Mail UK Delivery Issues: This problem has been resolved.

- Messages Not Wrapping on the Preview Page: This problem has been resolved.

- New Attachments Feature Disabled (or settings changing): This problem has been resolved.

- Verizon & Comcast Delivery Issues: Not actually a Yahoo! Groups bug (the delivery issues were on their end), but based on the fact that we are no longer getting frequent reports of delivery problems to these ISPs, we believe the problem has been resolved.

UPDATE (7/8 3:50 pm PDT, 10:50 GMT): Maintenance is complete.  All groups should again be accessible. Thank you to the affected groups for your patience!

Today, Wednesday, July 8th we will be performing scheduled maintenance on the Yahoo! Groups databases. During this time, some of you may not be able to access your groups, and group mail will be delayed.

Maintenance should take roughly two hours, and will begin at 1pm PT (8pm UTC/GMT- to determine what time the outage will take place in your neighborhood, please visit this link: http://www.timeanddate.com/worldclock/converter.html).

Once the maintenance has been completed, we will update this post.

Thanks for your patience!

Melissa Daniels
Yahoo! Community Manager

UPDATE (7/02 3:30 pm PDT; 10:30 pm GMT):  The problem has now been resolved and all Mail UK users should now be receiving messages from Yahoo! Groups normally, including the backlog of messages that had not been delivered earlier (if you are still having delivery issues, please contact Mail Customer Care).  Our apologies for the inconvience.

We’re aware of the reports from Yahoo! Mail UK users that they are not receiving messages from their Yahoo! Groups and both the Groups and Mail engineering teams are investigating the problem.  Will update this blog post with more info as we have it.

Update (7/3 12:30 pm PDT; 7:30 pm GMT): The problem has been fully resolved. Logs should now be functioning normally.

Update (7/2 9:30 am PDT; 4:30 pm GMT): Further good news on the logs. We’re now only about 24 hour behind and barring any new problems, engineering expects us to be fully caught up by tomorrow.

Update: There is some positive news on the logs front. The backlog is now at 50% of its peak level, which puts us on a pace to be fully caught up by this weekend (currently, activity through the 27th should be captured in the logs). However, one point to be aware of: the time stamp on the logs is showing as the time that the logs were processed in the queue, not as the time the activity took place. Which means that until this backlog is processed and we’re caught up, the time stamps on logs will not correspond to the time the activity took place. Admittedly this is not ideal (we recognize the time stamps should reflect the time the activity actually took place), but we’re not going to try fixing this until after we have everything working normally again.

Here are updates on three problems affecting significant numbers of groups or users.

GROUP ACTIVITY LOGS AND RECENT ACTIVITY NOT UPDATING
As many moderators have probably noticed, the “Group Activity” logs in the Management section have not been updating. The team has been working closely with Tibco, the company behind a key piece of software that Groups relies on to process our incredible volume of these logs, to address the issue, but the problem is not yet solved. And while the work will continue over the weekend (it’s 7:30 pm on Friday in California as I write this), I wouldn’t expect this issue to be resolved before some time next week.

Unfortunately, the queuing system that processes these logs is also the system that updates the “recent activity” module (on group home pages, the Groups Front Page, and My Groups), so this module will be out of date as well until the logs problem is resolved.

MESSAGE SEARCH UPDATE
We’re painfully aware that this is the bug that will not die. And while there is some good news, it’s balanced by some not as good news (caused mainly by the logs problem):
- The Good News: Approximately 8000 groups have been migrated to the new message search system and for these groups, message search is working correctly again (with a couple caveats described in the Bad News below).

- The Bad News I: The new message system also relies on the same queuing system as the logs. So messages posted since the logs began to backlog (for most groups on the 19th) won’t appear in search, even for groups on the new system.
- The Bad News II: No additional groups can be migrated until the queuing system is fixed.
- The Bad News III: A couple (of the nine) servers the new message search system is running on are behaving flakily. So sometimes it may be necessary to repeat a search a couple of the time to get the right results (though we hope to address that problem soon).

VERIZON AND COMCAST DELIVERY ISSUES
We began receiving reports today that some Verizon and Comcast users are again having trouble receiving messages from Yahoo! Groups. Unfortunately, there is nothing we can do to address this issues. We are sending the messages to Verizon and Comcast users, but the ISPs are either bouncing them, delaying them, or dropping them entirely. And this despite the fact that we have filled out (multiple times) the information that Verizon and Comcast ask for in these situations.

At this point, the only thing that is likely to help the problem is for enough Groups users to complain to Verizon and Comcast so that they feel compelled to take action. And when you do complain, there are three things you can do to make it more likely your complaint will be acted upon:

1. If there is information about your latest bounced message on your “bounce history page”, send it along with your complaint.
http://groups.yahoo.com/member/preferences/communication/bounce_history?email=YOUREMAILADDRESS (is the direct link to your bounce history)
2. If the problem is messages being delayed (rather than not delivered), send them the full headers of some of your delayed messages.
3. Point them to this group, which is designed for ISP Admins:
http://tech.groups.yahoo.com/group/ygmailadmin/

We’re sorry for the hassle and inconvenience these problems cause.

- Gordon
Yahoo! Groups

P.S. As always, if you need help from beyond Yahoo! beyond the information in this post, your only option is to contact Customer Care:
http://help.yahoo.com/l/us/yahoo/groups/original/ownmod/forms/forms_index.html

While we read comments on the blog, we do NOT take action on individual complaints. The ONLY way to get individual help is to contact Customer Care.

Did you know that there are more than two thousand groups on Yahoo! Groups dedicated solely to the men and women in the Armed Forces? These include groups dedicated to the men and women in all four branches of the US Armed Forces, groups for Blue Star Moms, and much, much, more.

We know that keeping our troops safe is something many of you find near and dear, which is why Yahoo! recently teamed up with Comedy Central’s Stephen Colbert of the Colbert Report and the philanthropic organization, DonorsChoose.org to lend a hand to the children and families of those who support our troops.

Stephen Colbert, DonorsChoose.org and Yahoo! are working together to give the children of military parents a hand in their education and in their communities.

As part of the “Support Our Troops” campaign, launched in partnership with Colbert, DonorsChoose.org is running a friendly competition among the Armed Forces — Army, Air Force, Marines, Navy and Coast Guard – to see which branch’s supporters can have the biggest impact on classrooms. As of today, we have raised more than $130,000, and the Army is barely hanging on to the lead (check out the leaderboard as the tension builds in real-time).

Literally hundreds of projects have been submitted by teachers at military-serving schools across the country in an effort to show students (and parents!) how proud we all are.

If you want to lend a hand to the DonorsChoose.org campaign click here to donate. Can’t? Show your solidarity by dressing up your Avatar on Yahoo! like Stephen Colbert, or, accessorize your own avatar with things like an eagle, bear, Colbert Nation hat, Che Colbert t-shirt, fireplace, and a desk, and help us spread the word about the Support Our Troops campaign.

Also, be sure to check out video clips from the Colbert Report, including one where Colbert shaves his head in support of the troops!

Thanks for your support!

DonorsChoose.org & Yahoo!

UPDATE: The maintenance was complete at 4:15 pm PDT (12:15 am GMT).  If you are still experiencing problems they are unrelated to this maintenance. Please contact Customer Care so that we can look into the problem:
http://help.yahoo.com/l/us/yahoo/groups/original/members/forms/forms_index.html

On June 11th, 2009 Yahoo! Groups will be undergoing maintenance. During this time 50% of groups will be unavailable for up to an hour. The outage is scheduled to begin some time between 2 and 3 pm PT (10-11 pm GMT).

While the maintenance is going on, all features of affected group, including messages, photos, attachments, etc. will be completely unavailable. However, once the outage concludes, any messages that were posted to these groups via email during this downtime will be distributed.

In an effort to better communicate to all of you, a “blast message” will be posted to all groups later today linking to this blog post. This message will be visible on all groups to all users.

Thank you for your patience—during the outage, this blog will remain up (and comments will remain open), and we will update you all as information becomes available.

- Gordon
Yahoo! Groups

Yesterday around 7pm, message posts to groups were not being delivered and went into a holding state.  The problem has been resolved and the posts since then are continuing to be sent out.  There will be a delay as the system works through the backlog of messages.  The delay will last a few hours but all messages should be posted by the end of the day and all new posts should resume back to normal.

Thanks for your patience,
- Raj
Yahoo Groups

UPDATE ON MESSAGE SEARCH (6/8 3:00 pm PDT/11:00 pm GMT): The good news is that the new message search is working correctly on one of our new production search servers. The bad news is that it’s not yet working correctly on all of them. Once this last issue is resolved, we will migrate groups that have signed up on to the new system. Our apologies for the delay.

Below is an update on our highest priority bugs, all of which our engineering team is actively working to squash.  Thank you for your patience as we work to resolve these issues, and we’re sorry for any hassles these bugs have caused you and your groups.

- Gordon
Yahoo! Groups

P.S. If you would like to report additional issues not mentioned below, please contact Customer Care . While we do read all comments, we need the information that the Customer Care form collects to be able to properly investigate a bug.

TOP PRIORITY BUGS

MESSAGE SEARCH PROBLEMS
This is our number one priority bug since it’s affecting so many groups and has no work around. As folks who read the blog know, we have been working hard to rebuild our message search system for a number of months .

The Bad News: The final phases of the this project have proved trickier than expected, which is why we haven’t met our goal of having groups that have signed up to be migrated on the new system by mid-May.

The Good News: Talked again with the engineer leading this project this morning, and he believes we’re very close to being able to roll out the new system more broadly. In fact, he hopes to solve the last remaining issues and actually begin migrating more groups as early as tomorrow (Wednesday, May 26th),  although I think Thursday (Pacific Time) is more realistic.  And assuming he encounters no unexpected problems, we should still be able to have all groups that have signed up to be migrated on the new system by the end of the month.

PROGRAM INITIALIZATION OR TEMPORARILY UNAVAILABLE ERRORS
We know that many groups are seeing this error when they land on the group home page or pending messages page. Usually, the error will go away if you refresh the page, but if the problem persists despite multiple refreshes, please contact Customer Care .

GROUPS BEING MISTAKENLY CLASSIFIED AS ADULT
There has been an ongoing known issue where groups are mistakenly recategorized behind the "adult wall" (meaning that no one under 18 can access). And while annoying, until recently Customer Care has been able to fix this mistake quickly for affected groups (when it is brought to their attention).

But in the past few weeks, for a handful of groups, the problem has become much more frustrating because they are being reclassified as adult every single day. In other words, the problem is being fixed by Customer Care, but a bug is causing the group is to placed back behind the adult wall again the next day.

We are working hard to identify what is triggering this bug (it has nothing to do with the content of the group), but as of yet have not identified the root cause. Once we do, we will roll out a fix. In the mean time, if your group is being affected by this issue, please let Customer Care know , and they will manually fix your group every day until the problem stops.

NEW TEXT DISAPPEARING IN MESSAGE REPLIES
When replying to group messages, some users are experiencing problems with the text they typed not displaying in their reply message.  Instead only the text of the message they are replying to is displaying.

The problem appears to be mainly affecting users of Outlook (and apparently Juno Web Mail), and it is only affecting replies to certain messages. We know the problem is related to HTML formatting issues, but we are still investigating the precise root cause and how to resolve the issue.

For more information, including suggested work arounds, see our "known issue" page here .

MODERATORS UNABLE TO CHANGE THE GROUP HOME PAGE PHOTO
Some moderators are currently getting errors when they try to update their group’s home page photo. Please contact Customer Care if this problem is affecting you.

We should be able to manually fix the problem for your group right away (though it may take us longer to roll out a more global fix that will prevent the problem from occurring in the first place).

PROBLEMS UPLOADING PHOTOS
Some users are reporting problems uploading photos, with only half the photos they submit being successfully uploaded in any given upload attempt. If this problem is affecting you, please contact Customer Care .

Over the past few years, we’ve seen a rise in groups dedicated towards improving the lives of others: either through community outreach, recycling (or Freecycling), Do-it-Yourself projects, or doing things naturally and locally.

Just like you, here at Yahoo! we’re always looking for new ways to improve our community, products, and the lives of others. Recently, Yahoo! For Good teamed up with the TV show Everyday Edisons to launch a "Make-It-Green" Program, geared towards improving the world around us by doing right by our planet. To learn more about this program, check out the details below from the "Make-It-Green" team and Yahoo! For Good.

Do you have product ideas that will make our lives a little easier and our planet a little greener?  Yahoo! wants you to submit your world changing ideas to our Make-It-Green program. This could mean your ideas get turned into real products and you’d get $2,500 and a share of the sales.

What is Make-It-Green?
Make-It-Green is turning great ideas that improve people’s lives and help our planet into actual products. Yahoo! is looking for the best green products or do-it-yourself projects and we need your help. Submit your idea, and if it’s chosen, you could see your product on store shelves, get $2,500 and a share of the sales, and possibly appear on the TV show Everyday Edisons .

You can also give the “thumbs up” to your favorites ideas. Your votes will help surface ideas we will consider making into products that will be sold in retail stores.

What is a Green Idea?
A green idea can mean using fewer resources, reusing materials, or finding a better way to get rid of unwanted stuff. To give you an idea of what we are looking for, a recent green product that we love is the Kyoto Box – a solar-powered cooker constructed from two cardboard boxes, aluminum foil and an acrylic cover.  It saves energy, trees, the atmosphere, and even lives—and costs about $6 to make!

Where can I learn more?
Submit your great green idea or pick your favorites at makeitgreen.yahoo.com .

After you’ve checked out the site, come back and tell us what you think. Or, share your Green Group here! We’d love to hear about the different ways you’re giving back to your community, with the help of Yahoo! Groups.

Melissa Daniels & Connie Chan
Yahoo! For Good