This is a read-only archive. Find the latest Linux articles, documentation, and answers at the new!

Feature: Open Source

New York Times opens up code

By Tina Gasperson on October 26, 2007 (9:00:00 PM)

Share    Print    Comments   

The New York Times likes open source -- so much so that, as it gradually moves more of its print operations online, it is nurturing a Web development team that has released two of its own open source projects.

XSL Cache is a PHP extension the Times is using to cache stylesheets on its Web site. DBSlayer is a tool the team developed to overcome LAMP scaling limitations that caused database replication processes to overwhelm the DB connection limits.

New York Times senior software architects Jacob Harris and Derek Gottfrid say they've received a mixed reception from the community, because some people just can't understand why a print media company would jump feet first into the open source philosophy. But open source software use isn't new to the Times, says Gottfrid. "I've been here a number of years, and open source has always played an integral part in everything we do."

Recently, the team has experienced growth, according to Gottfrid, in that custom applications developed in-house are "shifting from a proprietary posture. As we were building out and replacing old infrastructure, there were some gaps, so we wrote additional code. And some of those things we're open-sourcing. It's a small, humble effort."

Gottfrid says that upper management has been supportive. "Open source has grown so much in terms of people being aware of it, and our managers have grown up in that environment. Five or 10 years ago we wouldn't have had this reception."

To support their community efforts, Harris and Gottfrid launched a blog at where they and other members of the development team share their thoughts about code, tools, the New York Times site, and sometimes even fun little widgets they've written. Harris, who join the Times only about a year ago, was a big influence on the launch of the open source projects. "We have a lot of internal data that we could potentially share," he says, "and we have applications we're interested in exposing."

Harris says he's received some pushback from members of the open source community. "We went to OSCON, and some people were very excited that we were there, and some were like, what the hell are you doing here. A lot of people feel like we're pretty much irrelevant, and that they know what our business model should be. When you talk to geeks, they say 'just stop printing the newspaper,' but that would pretty much bankrupt us overnight."

Gottfrid says they didn't take it personally. "We've all been involved in the community for a long time. We want to figure out how to have a meaningful engagement with the community, to try to explain some of our problems so we're not just thought of as a monolith. We want to have a discussion with the community we're wading into. So we're figuring out the right way to do outreach; we're trying to find a sympathetic ear and engage some feedback."

Gottfrid says the most frequent comment he hears about the New York Times' foray into open source Web development is "'That's surprising.' They general think of us just as a newspaper, so they don't understand why we would be doing this. Is there a secret angle? But as soon as they read some posts, they start to break it down and understand. Now there's a bunch of people interested in this."

Gottfrid's first release was DBSlayer. "It's a lightweight database connection-pooling tool that we use internally. We were always saying, 'We should open-source this thing.' And that got a positive response from our manager -- they said, 'Go ahead, let's see where you can take this.' It hasn't taken the world by storm, but it is somewhat of a niche thing. It's a new program, so we haven't developed it fully and it is still in the early stages. But people have submitted patches and we've gotten good comments. It's been a very positive response. If we want to open-source other stuff in the future, it makes it a lot easier because we've established that we can."

Harris says one of the things they've talked about doing in the future is "opening up some of our content in the form of widgets or APIs, and see what the general public would do with it."

Gottfrid hints at the ultimate goal of NYT's move toward open source. "The key goes back to the title of the blog: Open. We wanted to have an open dialogue because we value the conversation with the development community. We want to be a part of that, and have a place at the table and listen and give feedback and have that respect. As we open more and more, it gives us a built-in audience that we can seed these things with, and a number of people that we already have a good relationship with."

Gottfrid says there are things you can do to make it easier to jump into the open source community -- among them, "Be humble." He says they don't try to act like they already know where their place is. "We've never presented our solution as a definitive solution for all problems. It worked well for us, and hopefully someone else can make it work for them." And don't forget to be persistent. "Just because you release something today doesn't mean everyone will show up the next day and download it and praise you and shower you with accolades."

Harris and Gottfrid stress the importance of trusting your developers. "This [move toward open source] wasn't a corporate mandate," Gottfrid says. "There were no vice presidents involved in this at all. This was Jake and me monkeying about and saying, 'Wouldn't this be cool?' The fact that they empowered us speaks volumes about the New York Times entrusting non-senior people to have a voice on the outside and represent the company. There's a growing awareness that developers play an integral role in how the medium is evolving."

Harris agrees. "We're no longer just a print company, we're a technology company. We need to express ourselves in technological terms. The best way to do that is to give the developers a voice."

Tina Gasperson writes about business and technology from an open source perspective.

Share    Print    Comments   


on New York Times opens up code

Note: Comments are owned by the poster. We are not responsible for their content.

New York Times opens up code

Posted by: Anonymous [ip:] on October 26, 2007 11:58 PM
El Mundo is the number 1 newspaper in Spain and has a big commitment with free software.

They even keeps some mirrors at to support 'the community'.

If you can read spanish there's a very interesting documents waiting for you about tech specs and system specs for this newspapers online version:


New York Times opens up code

Posted by: Anonymous [ip:] on October 28, 2007 02:19 AM
wow, they're hitting limits on mysql, really surprising considering how enterprise ready that database is.


Re: New York Times opens up code

Posted by: Anonymous [ip:] on October 28, 2007 03:18 AM
Depends on how they architected their application, and how persistent the database connections tend to be (how long they're held open in a session). Remember, this is a BIG newspaper, and they're going to get a lot of hits. It's pretty easy to make what seems like an innocent design decision and have it blow up in your face under real-world loads. I've been there.... (I can saturate an Oracle database back end server without blinking an eye if there is a badly designed app in front of it.) Sounds like they came up with some creative solutions, and rather than patenting it (like a lot of other people we know would), they decided to share it with the community. I don't know how broad the application is, but I still think it's a neat approach.


scotty st.ives alice in wonderland ? I thought mama sang base.

Posted by: Anonymous [ip:] on October 29, 2007 07:30 PM
Where do those comments about a stack of blank pages go. Is there something evil about trying to make software that is perhaps most easily accessable to the public easy to use and better than anything else available by using open standardized formates like the DOM3 for XML. Using a brouser formate that makes the last page of a brouser link a blank page available for word prosesing, mouse wire framed formated text, picture cropping, and video file drag and drop assembly, using simple XML wrapers to make personalized web page construction a no brainer. Dose open source standard software organized and promoted by proprietary software manufacturers suposedly donated to the public interest tresspass patent, copywrite, and national security. Is there some good reason the document object modle is off limits or is someone being censored.


I thought mama sang base and daddy sang tennor?

Posted by: Anonymous [ip:] on October 29, 2007 07:53 PM
Perhaps only with his back to the wall against the fernace with his shirt tail on fire. I was thinking how more features for linux based cell phones mite make them more attractive to phone line service providers. ABS, ignition, suspension programs could be written so a linux based cell phone could connect using a usb to a vehicle to provide monitoring and redundant back up of opperational features if only for recreational, custom or racing vehicles. Some motorcycles need a little help with the battery on cold starts. The usb connector could charge cell phones as well, suggesting cell phone us usb as a universal connector.It then occured to me that a car would need a usb port and then it occured to me that it would have to be either male or female. At this point it seems reasonable that a hardware standard be established that makes a universal asexual connector, either back to back usb or a new connector possessing duality. With enough features and no one to claim the little lamb lost in the woods, a standard audio codec could be used to access coustom features over a basic moble phone service. A usb flash drive, probably allready available could be used as a conversion interface allowing a person to connect video or cammera flash to the usb stick so a PC need only a usb connector rather than a host of dissimilar connections. If the usb stick had wireless capacity then all these periferals could be networked.


Re: I thought mama sang base and daddy sang tennor?

Posted by: Anonymous [ip:] on October 29, 2007 07:58 PM
now you could mount a camcorder to television tray and use a linux based cell phone as a universal remote controller to teach the dinner tray how to find its way to the kitchen.


sorry a first

Posted by: Anonymous [ip:] on October 29, 2007 08:19 PM
I think the web page expires. Then when a comment trys to post it misses the page and never shows up. Or there is a measure of censure afoot.


Open up all php code

Posted by: Anonymous [ip:] on October 30, 2007 07:29 AM
How about to open up the ENTIRE php code that made New York Times' website? Vinh Khoi must be proud then.


free movies

Posted by: Jen on October 31, 2007 01:54 AM
I found has a lot of movies anime music and games!This website is quite great! And it's free for you to streaming and download! Check it out and enjoy it! It will save you much time and money! Don't forgot to thanks me........ha ha


New York Times opens up code

Posted by: Anonymous [ip:] on January 21, 2008 11:47 PM
The NYT's confused and misguided editorial drivel has spilled over into the news articles over the past few years. Too bad NYT isn't worth reading these days, otherwise I might visit their web site.


New York Times opens up code

Posted by: Anonymous [ip:] on February 19, 2008 06:18 PM


This story has been archived. Comments can no longer be posted.

Tableless layout Validate XHTML 1.0 Strict Validate CSS Powered by Xaraya