Apache Infrastructure Team

Wednesday March 04, 2020

More secure and robust downloads

Explanation of a shift of download backup sites.[Read More]

Tuesday February 25, 2020

Welcome to Roller 6.0!

After some bumpy DNS issues, Roller 6.0 is live!

Sunday January 05, 2020

Another oar in the water

The Infrastructure team (Infra) works behind the scenes to make it possible for Apache's galaxy of committers to do the cool stuff they do, and for the open-source world to get, use, and rely on applications Apache projects produce. Infra supports additions to code repositories, a constant stream of conversation among project committers and contributors, and about 100 terabytes a month of software downloads.

Infra provides not just services, but knowledge. Infra's team and project committers rely on clear and accurate documentation about everything from how to update personal information to how to keep the Apache servers online, secure, and speedy.

Over 20 years of work, however, Infra has built up a substantial quantity of documentation that may be hard for its intended audience to find, out of date, or no longer relevant. So, in December, 2019, the ASF hired Andrew Wetmore as a part-time Technical Writer-Editor. His job is to curate the existing documentation and, in coordination with the rest of the Infra team and the wider ASF community, extend and improve it.

Andrew is a member of the PMC of Apache Royale, and spent fifteen years leading QA and documentation teams for software projects ranging from kitchen-table startups to major corporations. He lives in rural Nova Scotia, on the east coast of Canada, where he is the editor of a small publishing house.

Tuesday September 10, 2019

Subversion-to-Git service (git.apache.org) post mortem, and the path forward

What happened

On August 31st 2019, the machine hosting our subversion-to-git mirrors and synchronization process for GitHub suffered a catastrophic drive error due to a power failure at our data center in Virginia. The power failure was, unfortunately, of such a nature, that recovering the disk data was not possible. Four days into the failure, on September 4th 2019, we received confirmation from the data center that the data redundancy had also failed, meaning we had no measure of restoring to a new disk.

What this means right now

Currently, all GitHub mirrors that originate in subversion, and thus relied on this service, are not being synchronized with their subversion source. As git relies on on-disk subversion meta-data, as opposed to in-repo, we are not able to obtain the meta-data and continue synchronizing unless a full recreation of the mirrors is performed. This means starting from the first revision in any given subversion repository and working towards the most current one, a process that may well take a few days or weeks, depending on the size of the repository (by number of commits) and the number of running jobs at that time.

What we intend to do, going forward

Our most immediate action has been to revisit off-site backup strategies to ensure that our services are as resilient as possible, as well as re-assess and re-categorize various machines with regards to backup strategies.

With backups revisited, and on the more long-term side of things, discussions have been centered around what we want to offer, and how that will shape our design of the system. We want to balance the need for features against robustness and speed at the core of the service, as well as perform some fall cleaning of the service, and as such, the Infrastructure team has decided to restart the service with a blank slate, incorporating features as the needs arise and are discussed. We will also be reaching out to the projects with subversion-to-git mirrors currently on GitHub, and ask for a positive confirmation that they wish to continue with this service, so as to clean up the number of repositories that are no longer in use. We are also redesigning the core service, coupling it tighter with our subversion offerings.

We estimate the git mirror service to be revamped and rebooted in a matter of weeks, as cycles allow (this is occurring in tandem with other service upgrades, which puts the timeline somewhat into the future), and will add mirror repositories on an ad-hoc basis as requests come in.

Notable changes to service offering

As we are starting with a blank slate, please be advised of the following changes to the service as it starts back up:

  • There will no longer be a git.apache.org  URL for git mirrors, to lessen the confusion with gitbox.apache.org.  Projects wishing to point to a git copy of their subversion repository should use their respective GitHub URLs.
  • Repositories are re-created from scratch. As such, it may take days from a recreation is started till the sync process begins to kick in.

Tuesday April 30, 2019

Apache and GitHub - a friendly PSA about awesomeness

With the news of the Apache Software Foundation teaming up more closely with GitHub, we feel it natural to elaborate a bit on what has been going on and what this means for you as a committer and/or user of Apache software.

A little bit of history

The Apache Software Foundation started experimenting with git as a source code repository system in 2008, and ventured into GitHub in 2010, where we were graciously offered whatever resources we needed.

At first, this was merely a mirror of our existing git and subversion repositories, but as time went on, and projects expressed an interest in utilizing the many user-friendly features of GitHub, we started work on enabling projects to make proper use of GitHub some three years ago in the middle of 2016. This project, aptly named `gitbox`, ensured that committers could make full use of the GitHub features, while we kept a place within our own infrastructure for people inclined to continue using our infrastructure for their work. As git is decentralized by its very nature, we were able to use GitHub to augment rather than replace our git workflow, bringing our software development to the millions of users on GitHub in addition to the existing Apache community and committers, on a case-by-case basis.

In 2018, we made the decision to combine the two different git service offerings we had into one service, allowing all Apache projects to use GitHub if they so desired. Before then, we had two distinct git services: gitbox and git-wip-us, the initial git service that had been available since 2010. We coordinated the move from git-wip to gitbox with the various Apache projects, and in early 2019 we had migrated all projects to the new service, enabling GitHub features for all git-based Apache projects.

With Microsoft's acquisition of GitHub in 2018, and their commitment to help strengthen open source development, we have received additional resources to help lower the bar for contributions, and we'd like to thank GitHub for their support of the Apache Software Foundation through all nine years of using their platform.

What this means for you as a committer


As stated above, our GitHub integration is an augmentation of our existing service. It is available to all committers on git-based projects to make use of, should they so wish. All new git repositories will automatically be available on both GitHub and Gitbox.

For those wishing to take full advantage of GitHub's features, one can link their GitHub and Apache accounts through https://gitbox.apache.org/setup/ which will grant their GitHub account write access to the repositories you'd traditionally have access to at Apache.

People that wish to continue using their Apache committer accounts to commit code may continue doing so on gitbox.apache.org with their Apache credentials. Nothing has changed in that respect.

As Apache is a very email-centered organization, all GitHub activity is naturally linked to our mailing lists to ensure the same level of openness in the development of our software.

What this means for you as a user of Apache software


For many projects, the move to GitHub means a lower bar to both contributing as well as troubleshooting and submitting issues to the projects, through the GitHub issue and pull request features.

Our commitment to provenance, quality and open governance remains the same, and with our tight integration with GitHub through our linked account service, we are able to bring what made Apache a mark of quality to the many users and contributors on GitHub.


As always, if you have any questions, comments, remarks or feedback about this, we welcome you to reach out to the Apache Infrastructure Team at: users@infra.apache.org

Sunday January 27, 2019

Rate-limiting on Apache services

Over the past few days we have implemented rate limiting on selected services across the ASF.

As our foundation grows, so do the number of users and robots utilizing our services. In order to accommodate as many as possible with what resources we have, we have opted to implement rate-limiting to ensure that everyone can get their fair share of use of our services across the globe. The first services to have rate-limiting implemented are:

  • JIRA (issues.apache.org)
  • MoinMoin Wiki (wiki.apache.org)
  • BugZilla (bz.apache.org)

If you are a normal user of our services:

This very likely will never affect you, and you can go about your business just like normal :) If you DO experience errors or 429 (rate limited) response codes, please do let us know.

If you are a robot or otherwise automated tool:

There are now limits in place for how much CPU time you can use, varying from service to service. If you get limited, you will receive a HTTP 429 response instead of the normal 200, and a short text blob will explain that you have crossed our resource limits and have been rate-limited. It will also explain why, and when you can expect to be unblocked again (generally within two minutes time). Scrapers, bots etc using our services should check for a 429 response code and act accordingly (or just slow down the discovery pace in general, as that benefits all of us).

A general note about the rate limiting system, now and in the future:

Rate limits are applied across IP blocks to discourage distributed abuse, thus if you have 1.2.3.4 abusing a service, 1.2.3.5 would potentially also be affected by the rate limits till they expire.

Later this year, we will be rolling out rate limits on more services, and we encourage people automating tasks to honor the 429 responses across all ASF services.

We would also like to point out that there are, as before, additional global limits in place regarding the use of our services, which can be found at: https://www.apache.org/dev/infra-ban.html

Thursday January 10, 2019

Roller updated to 5.2.2

We've updated blogs.a.o to the latest version of Roller, 5.2.2!!


Friday December 07, 2018

Relocation of Apache git repositories on git-wip-us.apache.org to gitbox.apache.org

[IF YOUR PROJECT DOES NOT HAVE GIT REPOSITORIES ON GIT-WIP-US PLEASE DISREGARD THIS POST]

Hello Apache projects,

I am writing to you because you may have git repositories on the git-wip-us server, which is slated to be decommissioned in the coming months. All repositories will be moved to the new gitbox service which includes direct write access on github as well as the standard ASF commit access via gitbox.apache.org.

Why this move?
The move comes as a result of retiring the git-wip service, as the hardware it runs on is longing for retirement. In lieu of this, we have decided to consolidate the two services (git-wip and gitbox), to ease the management of our repository systems and future-proof the underlying hardware. The move is fully automated, and ideally, nothing will change in your workflow other than added features and access to GitHub.

Timeframe for relocation
Initially, we are asking that projects voluntarily request to move their repositories to gitbox. The voluntary time frame is between now and January 9th 2019, during which projects are free to either move over to gitbox or stay put on git-wip. After this phase, we will be requiring the remaining projects to move within one month, after which we will move the remaining projects over.

To have your project moved in this initial phase, you will need:

  • Consensus in the project (documented via the mailing list)
  • File a JIRA ticket with INFRA to voluntarily move your project repos over to gitbox (as stated, this is highly automated and will take between a minute and an hour, depending on the size and number of your repositories)

To sum up the preliminary timeline;

  • December 9th 2018 -> January 9th 2019: Voluntary (coordinated) relocation
  • January 9th -> February 6th: Mandated (coordinated) relocation
  • February 7th: All remaining repositories are mass migrated


This timeline may change to accommodate various scenarios.

Using GitHub with ASF repositories
When your project has moved, you are free to use either the ASF repository system (gitbox.apache.org) OR GitHub for your development and code pushes. To be able to use GitHub, please follow the primer at: https://reference.apache.org/committer/github We appreciate your understanding of this issue, and hope that your project can coordinate voluntarily moving your repositories in a timely manner.

All settings, such as commit mail targets, issue linking, PR notification schemes etc will automatically be migrated to gitbox as well.

Monday September 17, 2018

Position Available: Infrastructure Systems Administrator

UPDATE:  We have received enough applicants at this time. Thank you all for your interest. 

The Apache Software Foundation (ASF) seeks to fill an Infrastructure Systems Administrator position. You will be responsible for working with the existing technical infrastructure team.  The ASF manages a world-wide network of open source software which includes more than 1600 software code repositories, a worldwide distribution and mirroring system for software, change management, issue tracking, and software management for 300+ open source initiatives and over 10,000 contributors around the world.

Applicants should have a strong background in Computer and Information Science, and should be familiar with modern DevOps environments. Applicants must demonstrate the ability to work in a remote team environment alongside others working in diverse locations around the world and in different timezones. The successful applicant will work with the existing Infrastructure team to manage the ASF's critical infrastructure and resources. Infrastructure team members are expected to work a weekly on-call rotation with the rest of the team.

Our infrastructure team also supports our broader community by enabling the creation of self-service tooling. The successful candidate will be able to balance the needs of our critical infrastructure and the needs of our community to self-serve. These two demands can often be in conflict and thus an ability to navigate such complex environments is a distinct advantage.

Familiarity with Puppet (or a similar configuration management tool) Linux (Ubuntu-based), Virtual Machines, Subversion/Git, Python, and full development environment stacks are a requirement. Further, the candidate should possess great documentation skills and should be well versed in not only developing and assisting in technical solutions, but in documenting them.

Daily tasks will include handling of alarms, outages, and security concerns on a timely basis; working with our many communities on their needs and issues; managing tickets on a timely basis; rolling out and upgrading services; reducing our large technical debt; and maintaining a professional and collegial atmosphere. The team coordinates primarily through daily chat usage, weekly meetings, and email. Social skills and ability to integrate closely with the team are expected.

Preferred qualifications include a Bachelor's Degree in Computer Science or similar background from an accredited university, though demonstrable and appropriate on-the-job experience is an acceptable substitute for formal qualifications. Familiarity with how open source communities work is a definite positive.

English as a spoken and written language is required in order to facilitate team collaboration.

This is a remote work position, the ASF does not require nor provide office locations. Travel once per year is required, for a team meetup and will typically be coincident with an Apache conference.

Sunday March 26, 2017

Bringing GitPubSub to the Apache Jenkins build server

When it comes to [Jenkins], it has long been known that [polling must die].

While we could go and create post commit hooks in all the ASF hosted Git repositories, that is something that realistically is just creating an added maintenance burden. In any case, we have [GitPubSub].

The question then becomes, how do we integrate [GitPubSub] with [Jenkins]? Thankfully, ASF committer stephenc is also an active committer to the [Jenkins] project and created a [plugin] that connects to [GitPubSub] parses the events and passes them through to the Jenkins [SCM API].

What does this mean?

* You can turn your Git polling down - way way down - to something like once per day. This should significantly reduce the load on both the ASF git servers and builds.apache.org
* Your builds will be triggered in seconds rather than having to wait for the next polling run.
* You can try out using Multi-branch projects much like the [Maven] project has been doing for [Maven core] and [Maven Surefire]

If the reaction to this change proves positive, the next step will be to integrate SvnPubSub with Jenkins and bring the benefits to the Subversion based projects too

See also this blog post by Stephen Connolly:

https://www.cloudbees.com/blog/using-multi-branch-pipelines-apache-maven-project

[polling must die]: http://kohsuke.org/2011/12/01/polling-must-die-triggering-jenkins-builds-from-a-git-hook/
[GitPubSub]: https://www.apache.org/dev/gitpubsub.html
[Jenkins]: https://jenkins.io/
[plugin]: https://github.com/stephenc/asf-gitpubsub-jenkins-plugin
[SCM API]: https://plugins.jenkins.io/scm-api
[Maven]: https://maven.apache.org
[Maven core]: https://builds.apache.org/job/maven-3.x-jenkinsfile/
[Maven Surefire]: https://builds.apache.org/job/maven-surefire-pipeline/

Posted on behalf of Committer Stephen Connolly (stephenc)

Sunday January 01, 2017

blogs.a.o moved, upgraded and improved

Hi All,

blogs.apache.org   - the site you are reading now! has had a bit of an update.

1. We moved it from an aged VM Host to the Cloud (thanks LeaseWeb!)

2. We puppetised the entire service, from install to deploy (see our Github Mirror )

3. We upgraded the Apache Roller software from 5.0.3 to the latest 5.1.2

4. We enabled LDAP for logins. That's right! Every single ASF Committer can now just login! No more creating an INFRA Jira ticket just to get a Roller account on blogs.apache.org

Other stuff remains the same - meaning if you are a Blog Administrator you still need to invite committers into your blog, you still need to choose to make them an Author or Admin etc - Roller doesnt support anything more than login auth for LDAP currently - but I bet the project would love to see the LDAP integration extended and improved if you feel the need!.

Anyhow, our first new year present to our ASF Committers, a shiny updated blog instance,

 Enjoy, and have a great 2017!!


Monday July 25, 2016

Position Available: Infrastructure Systems Administrator/Architect

UPDATE:  We have received enough applicants at this time. Thank you all for your interest. 

The Apache Software Foundation (ASF) seeks to fill an Infrastructure Systems Administrator/Architect position. You will be responsible for working with the existing technical infrastructure team, and VP of Infrastructure at the Apache Software Foundation. The ASF manages a world-wide network of open source software which includes more than 750 software code repositories, a worldwide distribution and mirroring system for software; change management, issue tracking, and software management for 300+ Open Source initiatives and more than 11,000 contributors around the world.


Applicants should have a strong background in Computer and Information Science, and should be familiar with modern Dev/Ops environments. Applicants must demonstrate the ability to work in a remote team environment alongside others working in diverse locations around the world and in different timezones. The successful applicant will work with the existing Infrastructure team and VP, Infrastructure to manage the ASF's critical infrastructure and resources. Infrastructure team members are expected to work an on-call rotation with the rest of the team.

Our infrastructure team also supports our broader community by enabling the creation of self-service tooling. The successful candidate will be able to balance the needs of our critical infrastructure and the needs of our community to self-serve. These two demands can often be in conflict and thus an ability to navigate such complex environments is a distinct advantage.

Familiarity with Puppet (or a similar configuration management tool) Linux (Debian-based), Virtual Machines, Subversion/Git and full development environment stacks are a requirement. Further, the candidate should possess great documentation skills and should be well versed in not only developing and assisting in technical solutions, but in documenting them.

Preferred qualifications include a Bachelor's Degree in Computer Science or similar background from an accredited university, though demonstrable and appropriate on-the-job experience is an acceptable substitute for formal qualifications. Familiarity with how open source communities work is a plus.

English as a spoken and written language is required in order to facilitate team collaboration.

This is a remote work position, the ASF does not require nor provide office locations. Travel required for conferences and general team meetups.

Contact vp-infra@apache.org with your CV.

Thursday June 30, 2016

ASF JIRA Outages and Troubleshooting

As people have noticed, our JIRA instance (arguably the largest public instance in the world) has been suffering from a yet unknown issue as of late. We are reasonably sure that this is related to specific queries being made against the instance (possibly automated queries from scrapers), but have yet to identify the exact cause of the problem.

The failure condition arises when the database connection pool is exhausted, despite being configured and sized appropriately. These connections all appear idle, but when the pool is full, no new connections can be established, and the instance falls over, requiring a restart. 

We are working closely with Atlassian, the creator of JIRA, to remedy the situation. Unfortunately, this requires running diagnostics on the production JIRA instance, which in and of itself causes performance degradation and downtime. Over the past several days, we've identified and implemented some changes to the pool parameters which we hope will help stabilize the instance while we continue our diagnostic work.

We expect that there may still be some moments of downtime and occasional restarts. Any longer duration outages will be announced via Twitter/infrabot and status.apache.org.

Friday February 12, 2016

AppVeyor CI now available for GitHub Mirrors

The ASF Infrastructure team is happy to announce that projects can how have AppVeyor CI setup on their GitHub mirrors.

 The only thing you need to do is create an INFRA ticket at issues.apache.org with the following information:

  • Repo Name
  • Mailing list to send build notifications to (optional)

There are already a few projects using AppVeyor on their GitHub mirror, and we now have an Organization role account for central management (and I have gone through an updated previous tickets with new links to badges).

If you have any questions, you can ask us in Hipchat or you can email infrastructure@apache.org

Monday October 19, 2015

Dear Apache

My name is Daniel Takamori and I'm so happy to be joining the Infra team here at Apache.  I'm from Oregon in the United States and really enjoy the rain.  While at Oregon State University I studied mathematics and physics with a lean towards error correcting codes and mathematical modelling.  Some of my hobbies are playing Go in which I'm ranked 6.9 kyu by the AGA, cooking with eggs and green things, and old school platforming video games.  In a former life I worked on underwater remotely operated vehicles and automated gardening systems.  Traveling is something I liked to do once; living in Hungary was awesome and I hope to visit again. Oregon is a great place to live, with all the trees, rain and burritos but maybe things will change in the future.  My handle Pono is my Hawaiian name, and I'm really proud to use it.

Previously I was at the Oregon State University Open Source Lab and really enjoyed my time there; getting to know the Open Source communities and even work with Apache!  It was a real eye opening experience to the world of what software and DevOps (lol who knows what that even means).  I'm very excited to continue working with the community and even more excited to start this next chapter with such an amazing group.

See you around internets!

Calendar

Search

Hot Blogs (today's hits)

Tag Cloud

Categories

Feeds

Links

Navigation