Mozilla DevCenter
oreilly.comSafari Books Online.Conferences.
advertisement

Sponsored Developer Resources

Atom 1.0 Feed RSS 1.0 Feed RSS 2.0 Feed

Related O'Reilly Books





What Is Firefox What Is Firefox
Brian King provides a brief look at Firefox's origins and evolution, and then dives into its support for web standards like CSS and XML, its debugging and extension capabilities, and some cool new features in the upcoming 1.5 release. If you're considering a switch to Firefox, this article may help make the decision for you.


Mozilla as a Development Platform: An Interview with Axel Hecht  Axel Hecht is a member of Mozilla Europe's board of directors, and a major contributor to the Mozilla project. At O'Reilly's European Open Source Convention (October 17-20), Dr. Hecht will be talking about Mozilla as a development platform. O'Reilly Network interviewed Dr. Hecht to find out if the long-held dream of Mozilla as a development platform was about to come true.   [O'Reilly Network]

A Firefox Glossary  Brian King, with some help from Nigel McFarlane, covers everything from about:config to "zool" in this fun, fact-filled Firefox glossary. It's by no means exhaustive, but you'll find references to specific chapters or hacks throughout the glossary to Nigel's book, Firefox Hacks. When you're ready to dig deeper, check out his book.   [O'Reilly Network]

Important Notice for Mozilla DevCenter Readers About O'Reilly RSS and Atom Feeds  O'Reilly Media, Inc. is rolling out a new syndication mechanism that provides greater control over the content we publish online. Here's information to help you update your existing RSS and Atom feeds to O'Reilly content.  [Mozilla DevCenter]

Hacking Firefox  This excerpt from Firefox Hacks shows you how to use overlays (essentially hunks of UI data) to make something you want to appear in the Firefox default application, perhaps to carry out a particular function of your extension. For example, you might want to add a menu item to the Tools menu to launch your extension. Overlays allow existing Firefox GUIs to be enhanced.   [O'Reilly Network]

Mozile: What You See is What You Edit  Most modern browsers don't allow you to hit "edit" and manipulate content as easily as you view it, WYSIWYG-style. Mozile, which stands for Mozilla Inline Editor, is a new Mozilla plug-in for in-browser editing. This article by Conor Dowling provides an overview of Mozile and what in-browser editing means.
  [ Mozilla DevCenter]

The Future of Mozilla Application Development  Recently, mozilla.org announced a major update to its development roadmap. Some of the changes in the new document represent a fundamental shift in the direction and goals of the Mozilla community. In this article, David Boswell and Brian King analyze the new roadmap, and demonstrate how to convert an existing XPFE-based application into an application that uses the new XUL toolkit. David and Brian are the authors of O'Reilly's Creating Applications with Mozilla.   [Mozilla DevCenter]

Remote Application Development with Mozilla, Part 2  In their first article, Brian King, coauthor of Creating Applications with Mozilla, and Myk Melez looked at the benefits of remote application development using Mozilla technologies such as XUL and web services support. In this article, they present a case study of one such application, the Mozilla Amazon Browser, a tool for searching Amazon's catalogs.   [Mozilla DevCenter]

Remote Application Development with Mozilla  This article explores the uses for remote XUL (loaded from a Web server), contrasts its capabilities with those of local XUL (installed on a user's computer), explains how to deploy remote XUL, and gives examples of existing applications.   [Mozilla DevCenter]

Mozdev.org Made Easy  Now that mozilla.org is about to release Mozilla 1.2 and Netscape has come out with the latest version of their own Mozilla-based browser, Netscape 7, this is a great time to see what other people are building with Mozilla's cross-platform development framework. Here's a little history about, and a roadmap to, mozdev.org.   [Mozilla DevCenter]

XML Transformations with CSS and DOM  Mozilla permits XML to be rendered in the browser with CSS and manipulated with DOM. If you're already familiar with CSS and DOM, you're more than halfway to achieving XML transformations in Mozilla. This article demonstrates how to render XML in the browser with a minimum of CSS and JavaScript.   [Mozilla DevCenter]

Roll Your Own Browser  Here's a look at using the Mozilla toolkit to customize, or even create your own browser.   [Mozilla DevCenter]

Let One Hundred Browsers Bloom  In this article, David Boswell, coauthor of Creating Applications with Mozilla surveys some of the more interesting, and useful, Mozilla-based browsers available now.   [Mozilla DevCenter]

Using the Mozilla SOAP API  With the release of Mozilla 1.0, the world now has a browser that supports SOAP natively. This article shows you how Web applications running in Mozilla can now make SOAP calls directly from the client without requiring a browser refresh or additional calls to the server.   [Web Development DevCenter]





Today's News
August 30, 2014

Chris McAvoy: Open Badges and JSON-LD

The BA standard working group has had adding extensions to the OB assertion specification high on its roadmap this summer. We agreed that before we could add an extension to an assertion or Badge Class, we needed to add machine readable schema definitions for the 1.0 standard.

We experimented with JSON-Schema, then JSON-LD. JSON-LD isn’t a schema validator, it’s much more. It builds linked data semantics on the JSON specification. JSON-LD adds several key features to JSON, most of which you can play around with in the JSON-LD node module.

  1. Add semantic data to a JSON structure, link the serialized object to an object type definition.
  2. Extend the object by linking to multiple object type definitions.
  3. A standard way to flatten and compress the data.
  4. Express the object in RDF.
  5. Treat the objects like a weighted graph.

All of which are features that support the concept behind the Open Badges standard very well. At its core, the OB standard is a way for one party (the issuer) to assert facts about another party (the earner). The assertion (the badge) becomes portable and displayable at the discretion of the owner of the badge.

JSON-LD is also quickly becoming the standard method of including semantic markup on html pages for large indexers like Google. Schema.org now lists JSON-LD examples alongside RDFa and Microdata. Google recommends using JSON-LD for inclusion in their rich snippet listings.

We’ve been talking about JSON-LD on the OB standard working group calls for a while now. It’s starting to feel like consensus is forming around inclusion of JSON-LD markup in the standard. This Tuesday, September 2nd 2014, we’ll meet again to collectively build a list of arguments for and against the move. We’ll also discuss a conditional rollout plan (conditional in that it will only be executed if we get the thumbs up from the community) and identify any gaps we need to cover with commitments from the community.

It’s going to be a great meeting, if you’re at all interested in JSON-LD and Open Badges, please join us!

[Source: Planet Mozilla]

Doug Belshaw: Weeknote 35/2014

This week I’ve been:

  • Surviving while being home alone. The rest of my family flew down to Devon (it’s quicker/easier than driving) to visit the in-laws.
  • Working on lots of stuff around the house. There are no plants left in my garden, for example. We decided that we want to start from ‘ground zero’ so I went on a bit of a mission.
  • Suffering after launching myself into the week too hard. I’d done half of the things I wanted to do all week by Tuesday morning. By Wednesday I was a bit burnt out.
  • Writing blog posts:
  • Accepting an invitation to join Code Club’s education advisory committee.
  • Finding out about an opportunity to work with a well-known university in the US to design a module for their Ed.D. programme. More on that soon, hopefully!
  • Clearing out the Webmaker badges queue (with some assistance from my amazing colleagues)
  • Inviting some people to talk to me about the current Web Literacy Map and how we can go about updating it to a version 2.0.
  • Finishing and sending the rough draft of a video for a badge which will be available on the iDEA award website when it launches properly.
  • Starting to lift weights at the gym. I actually started last week, but I’ve already noticed it help my swimming this week. Improved stamina, and the bottom of my right hamstring doesn’t hurt when I get out of the pool!

Next week I’m at home with a fuller calendar than usual. That’s because I’m talking to lots of people about future directions for the Web Literacy Map. If you’ve started using it, then I’d love to interview you. Sign up for that here.

[Source: Planet Mozilla]

Eric Shepherd: The Sheppy Report: August 29, 2014

This week, my WebRTC research continued; I spent a lot of time watching videos of presentations, pausing every few seconds to take notes, and rewinding often to be sure I got things right. It was interesting but very, very time-consuming!

I got a lot accomplished this week, although not any actual code on samples like I’d planned to. However, the pages on which the smaller samples will go are starting to come together, between bits of actual content on MDN and my extensive notes and outline. So that’s good.

I’m looking forward to this three-day Labor Day holiday here in the States. I’ll be back at it on Tuesday!

What I did this week?

  • Copy-edited the Validator glossary entry.
  • Copy-edited and cleaned up the Learning area page Write a simple page in HTML.
  • Created an initial stub documentation project plan page for updating the HTML element interface reference docs.
  • Turned https://developer.mozilla.org/en-US/docs/Project:About into a redirect to the right place.
  • Read a great deal about WebRTC.
  • Watched many videos about WebRTC, pausing a lot to take copious notes.
  • Built an outline of all the topics I want to be sure to cover. I’m sure this will continue to grow for a while yet.
  • Gathered notes and built agendas for the MDN community meeting and the Web APIs documentation meeting.
  • Updated the WebRTC doc plan with new information based on my initial notes.
  • Offered more input on a bug recommending that we try to add code to prevent people from using the style attribute or any undefined classes.
  • Filed bug 1060395 asking for a way to find the pages describing the individual methods and properties of an interface in the Web API reference
  • Fixed bug 1058814 about hard-to-read buttons by correcting the styles used by a macro.
  • Dealt with expense reports.
  • Started very initial work on WebRTC doc tree construction, preparing to reshuffle and clean up the existing, somewhat old, pages, and to add lots of new stuff.
  • Started work on trying to figure out how to make the SubpageMenuByCategories macro not lose headers; it’s calling through to MakeColumnsForDL, which specifically only works for a straight-up <dl>. Fixing this to work correctly will be my first task on Tuesday.

Meetings attended this week

Monday

  • MDN bug triage meeting
  • #mdndev planning meeting

Tuesday

  • Developer Relations weekly meeting.
  • 1:1 with Teoli. This went on for an hour instead of the usual 30 minutes, due to the enormous amount of Big Stuff we discussed.

Wednesday

  • MDN community meeting

Friday

A pretty good week all in all!

[Source: Planet Mozilla]

Benoit Girard: Visual warning for slow B2G transaction landed

With the landing of bug 1055050, if you turn on the FPS counter on mobile you will now notice a rectangle around the screen edge warning you that a transaction was out of budget.

  • The visual warning will appear if a transaction took over 200ms from start to finish.
  • Yellow indicates the transaction took over 200ms.
  • Orange will indicate the transaction took about 500ms.
  • Red will indicate the transaction is about 1000ms or over.

preview: https://www.dropbox.com/s/so927ty2pn59m3x/2014-08-18%2018.10.16.mp4

What’s a transaction?

It’s running the rendering pipeline and includes (1) Running request animation frame and other refresh observers, (2) Flushing pending style changes, (3) Flushing and reflowing any pending layout changes, (4) Building a display list, (5) Culling, (6) Updating the layer tree, (7) Sending the final result to the compositor, (8) Syncing resources with the GPU. It does NOT include compositing which isn’t part of the main thread transaction. It does not warn for other events like slow running JS events.

Why is this important?

A transaction, just like any other gecko event, blocks the main thread. This means that anything else queued and waiting to be service will be delayed. This means that many things on the page/app will be delayed like: animations, typing, canvas, js callbacks, timers, the next frame.

Why 200ms?

200ms is already very high. If we want anything in the app to run at 60 FPS that doesn’t use a magical async path then any event taking 16ms or more will cause noticeable stutter. However we’re starting with a 200ms threshold to focus on the bigger items first.

How do I fix a visual warning?

The warnings are just provided for instant feedback. Problems should be analyzed using the standard performance workflow. This means using tools like the profiler, display list, layers dump and soon the upcoming DevTools timeline.


[Source: Planet Mozilla]

Priyanka Nag: Maker Party Bhubaneshwar

Last weekend I had a blast in Bhubaneshwar. Over two days, I was there at two different colleges for two Maker parties.

Saturday (23rd August 2014), we were at the Center of IT & Management Education (CIME) where we were asked to address a crowd of 100 participants whom we were supposed to teach webmaking. Trust me, very rarely do we get such crowd in events where we get the opportunity to be less of a teacher and more of a learner. We taught them Webmaking, true, but in return we learnt a lot from them.

Maker Party at Center of IT & Management Education (CIME)

On Sunday, things were even more fabulous at Institute of Technical Education & Research(ITER), Siksha 'O' Anusandhan University college, where we were welcomed by around 400 participants, all filled with energy, enthusiasm and the willingness to learn.

Maker Party at Institute of Technical Education & Research(ITER)

Our agenda for both days were simple....to have loads and loads of fun! We kept the tracks interactive and very open ended. On both days, we did cover the following topics:
  • Introduction to Mozilla
  • Mozilla Products and projects
  • Ways of contributing to Mozilla
  • Intro to Webmaker tools
  • Hands-on session on Thimble, Popcorn and X-ray goggles and Appmaker
Both days, we concluded our sessions by giving away some small tokens of appreciation like e T-shirts, badges, stickers etc, to the people who had been extra awesome among the group. We concluded the awesomeness of the two days by cutting a very delicious cake and fighting over it till its last pieces.
 
Cake.....
Bading goodbye after two days was tough, but after witnessing the enthusiasm of everyone we met during these two events, I am very sure we are going to return soon to Bhubaneshwar for even more awesomeness.
 
A few people who are two be thanked for making these events sucessful and very memorable are:
  1. Sayak Sarkar, the co-organizer for this event.
  2. Sumantro, Umesh and Sukanta from travelling all the way from Kolkata and helping us out with the sessions.
  3. Rish and Prasanna for organizing these events.
  4. Most importantly, the entire team of volunteers from both colleges without whom we wouldn't havebeen able to even move a desk.
 p.s - Not to forget, we did manage to grab media's attention as well. The event was covered by a local newspaper.
The article in the newspaper next morning

[Source: Planet Mozilla]

Pascal Finette: Follow Your Fears

Building trophies in my soul…

[Source: Planet Mozilla]

Daniel Stenberg: Firefox OS Flatfish Bluedroid fix

Hey, when I just built my own Firefox OS (b2g) image for my Firefox OS Tablet (flatfish) I ran into this (known) problem:

Can't find necessary file(s) of Bluedroid in the backup-flatfish folder.
Please update the system image for supporting Bluedroid (Bug-986314),
so that the needed binary files can be extracted from your flatfish device.

So, as I struggled to figure out the exact instructions on how to proceed from this, I figured I should jot down what I did in the hopes that it perhaps will help a fellow hacker at some point:

  1. Download the 3 *.img files from the dropbox site that is referenced from bug 986314.
  2. Download the flash-flatfish.sh script from the same dropbox place
  3. Make sure you have ‘fastboot’ installed (I’m mentioning this here because it turned out I didn’t and yet I have already built and flashed my Flame phone successfully without having it). “apt-get install android-tools-fastboot” solved it for me. Note that if it isn’t installed, the flash-flatfish.sh script will claim that the device is not in fastboot mode and stop with an error message saying so.
  4. Finally: run the script “./flash-flatfish.sh [dir with the 3 .img files]“
  5. Once it had succeeded, the tablet reboots
  6. Remove the backup-flatfish directory in the build dir.
  7. Restart the flatfish build again and now it should get passed that Bluedroid nit

Enjoy!

[Source: Planet Mozilla]

Wladimir Palant: Using a Firefox extension to work around Selenium WebDriver's limitations

My Google search link fix extension had a bunch of regressions lately and I realized that testing its impact on the search pages manually isn’t working — these pages are more complicated than it looks like, and there are lots of configuration options affecting them. So I decided looking into Selenium WebDriver in order to write integration tests that would automate Firefox. All in all, writing the tests is fairly simple once you get used to the rather arcane API. However, the functionality seems to be geared towards very old browsers (think IE6) and some features are nowhere to be found.

One issue: there is no way to focus an element without clicking it. Clicking isn’t always an option, since it might trigger a link for example. That issue turned out to be solved fairly easily:

driver.execute_script("arguments[0].focus()", element)

The ability to pass elements as parameters to WebDriver.execute_script is very useful, so it is surprising that it doesn’t seem to be documented properly anywhere.

But what about working with tabs or middle-clicking links? It seems that tabbed browsing wasn’t invented yet back when that API was designed, so it only has a concept of windows — not very useful. So WebDriver will only let you work with the currently selected tab, inactive tabs are off limits. And WebDriver.execute_script isn’t any help here either, it won’t let you run privileged code.

After briefly considering using send_keys functionality to open Web Console on about:config and typing code into it (yes, it looks like that would actually work), I decided to go with a less crazy solution: install an additional extension to implement the necessary functionality. So if a test wants the element to be middle-clicked it can trigger a custom event:

driver.execute_script('''
  var event = document.createEvent("Events");
  event.initEvent("testhelper_middleclick", true, false);
  arguments[0].dispatchEvent(event);
''', element)

And the extension listens to that event:

window.gBrowser.addEventListener("testhelper_middleclick", function(event)
{
  let utils = event.target.ownerDocument.defaultView
                   .QueryInterface(Ci.nsIInterfaceRequestor)
                   .getInterface(Ci.nsIDOMWindowUtils);
  let rect = event.target.getBoundingClientRect();
  utils.sendMouseEvent("mousedown", rect.left + 1, rect.top + 1, 1, 1, 0);
  utils.sendMouseEvent("mouseup", rect.left + 1, rect.top + 1, 1, 1, 0);
}, false, true);

This works nicely, but what if you want to get data back? For example, I want to know which URLs were requested at the top level — in particular, whether there was a redirect before the final URL. Selenium only allows you to get notified of URL changes that were initiated by Selenium itself (not very helpful) or poll driver.current_url (doesn’t work). The solution is to have the extension register a progress listener and write all URLs seen to the Browser Console:

window.gBrowser.addTabsProgressListener({
  onStateChange: function(browser, webProgress, request, flags, status)
  {
    if (!(flags & Ci.nsIWebProgressListener.STATE_IS_WINDOW))
      return;
    if (!(flags & Ci.nsIWebProgressListener.STATE_START) && !(flags & Ci.nsIWebProgressListener.STATE_REDIRECTING))
      return;
    if (request instanceof Ci.nsIChannel)
      Cu.reportError("[testhelper] Loading: " + request.URI.spec);
  }
});

You can use driver.get_log("browser") to retrieve the full list of console messages. Each message also has a timestamp which allows for example only extracting the URLs seen after the previous check.

Side-note: I first considered using MozMill for this. However, it is geared very much towards Firefox development and much of the Selenium functionality would have to be reimplemented (locating installed Firefox instance, default Firefox preferences for a test profile, dismissing alerts on web pages and so on).

[Source: Planet Mozilla]

Priyanka Nag: Maker Party gets grander in Pune this time

While going through my twitter time-line this evening, I noticed Michelle Thorne's tweet stating that India leads with most Maker Party action this season.
Well, who doubts that! In India, we have a maker parties being organized almost every second day. My facebook wall and twitter timelime is like overloaded with posts, photos and updates from all the Maker Parties happening around me.

Maker Party, Pune


Well, if you are still not aware of this one, we are having the grand daddy of this maker parties in Pune on the 6th of September 2014. The executive director of Mozilla Foundation, Mark Surman, is going to be personally present for this event. Just like all maker parties, this event is an attempt to map and empower a community of educators and creative people who share a passion to innovate, evolve and change the learning landscape.

A few quick updates about this event:
  •  Event date - 6th and 7th September
  •  Event venue - SICSR, Model Colony, Pune
  • Rough agenda for the event is going to be:
    • 6th September 2014 (Day 1) 
      • 10am - 11am : Mozilla introduction
      • 11am - 12 : About Hive initiative
      •  12 - 1pm: Rohit Lalwani - Entrepreneurship talk
      •  1-2pm : Lunch break
      •  2pm - 3pm: Webmaker begins with Appmaker
      •  3pm - 4pm: Webmaker continues with Thimble
      •  4pm - 4.45pm: Webmaker continues with Popcorn
      •  4.45pm - 5.30pm : Webmaker continues with x-ray goggles
      • 5.30pm - 6pm: Prize distribution (against best makes of the day etc). Science fair also ends
      • 6pm - 7pm : Birds of feature
      • 7pm : Dinner (venue - TBD)
Science fair will be from 12 noon to 6pm.
  
    •  7th September 2014 (Day 2) 
      • 1st Half: Community Meetup and Discussions on the future roadmap for Hive India,
        Long term partnership prospect meeting with partners.
      •  2nd Half: Community training sessions on Hive and Train the trainer events.
 
For this event, we are having a variety of different training sessions, workshops and science displays - starting from 3D printing to wood-works, Origami to quad-copter flying and even film making.

If you have still not registered for this event, heres your chance:


Loading...
[Source: Planet Mozilla]

Mike Ratcliffe: View jQuery and jQuery Live events in Firefox DevTools

A little while back we landed a feature that made it possible to interact with events from the markup panel of Firefox developer tools.

A lot of people were excited about this but we soon realized that jQuery wraps event handlers in a proxy making all handlers look like this:

function() {  
  return fn.apply( thisObject || this, arguments );
};

Numerous facepalms later and we now have support for jQuery and jQuery Live events. This support will be available in tonight's nightly:

jQuery and jQuery Live events

Because library support is important we have created an API that allows developers and library authors to create "event parsers" that allow our tools to make sense of their event systems.

This API is priviledged code so to create an event parser you will need to create a simple Firefox extension.

The API looks like this:

var parser = {  
  id: "myLibrary events", // Unique id
  hasListeners: function(node) {
    // Hunt for node's listeners and return true as soon as one is 
    // encountered.
  },
  getListeners: function(node) {
    // Hunt for node's listeners and return an array of objects
    // representing those listeners. Each object should look like this:
    {
      type: "click",
      handler: function clicked() {...},

      // These tags will be displayed as attributes in the events popup.
      tags: "jQuery,Live", 

      // Hide or show fields
      hide: {               
        debugger: false, // Debugger icon
        type: false, // Event type e.g. click
        filename: false,
        capturing: false,
        dom0: false
      },

      override: {
      // The following can be overridden:
        type: "click",
        origin: "http://www.mozilla.com",
        searchString: 'onclick="doSomething()"',
        DOM0: true,
        capturing: true
      }
    }
  }
},
normalizeHandler: function(fnDO) {  
  // Take a handler debug object and use the debugger to walk the scope 
  // chain to discover the function you would like to be displayed.

  // See https://hg.mozilla.org/integration/fx-team/diff/9add1ec0251d/toolkit/devtools/event-parsers.js#l1.98 for an example.
}

gDevTools.registerEventParser(parser);  

We have bugs logged to add support for the following libraries:

You still here? What are you waiting for? Go get hacking!

[Source: Planet Mozilla]

Nicholas Nethercote: Update on reviewing our data practices and Bugzilla development database disclosure

As we indicated in the post titled “MDN Disclosure”, we began several remediation measures, including a review of data practices surrounding user data. We have kicked off a larger project to better our practices around data, including with respect to the various non-Mozilla projects we support. We are implementing immediate fixes for any discovered issues across the organization, and are requiring each business unit to perform a review of their data practices and, if necessary, to implement additional protections based on that review.

As we proceed through our broader remediation program, we discovered an incident that occurred in the Bugzilla community, one of the community projects we support. A member of the Bugzilla community discovered that development database dump files containing email addresses and encrypted passwords were posted on a publicly accessible server. They were alerted to this incident by a security bug filed by a contributor. While it is important to note that the disclosure of this development database does not affect bugzilla.mozilla.org, we continue to believe that the broader community would benefit from our increased focus on data practices and therefore will continue with our plan of including the Bugzilla project as well as other community projects in the data practices initiatives we’ve described above.

We are committed to continuing to improve our data practices to minimize the likelihood of these and other types of incidents.

Sincerely,

Mozilla Security

[Source: Planet Mozilla]

Jared Wein: New in Firefox Nightly: Recommended and easy theme switching through Customize mode

Firefox menuWe shipped the Australis project with Firefox 29, but the Firefox team hasn’t stopped working on making Firefox the easiest browser to personalize. Firefox allows easy customizing through the new Customize mode, and now in Firefox Nightly people will find a quick and easy to way to set the theme of the browser.

After entering Customize mode, a new menu is shown at the footer of the window. Clicking on this menu will show any installed themes as well as a list of five recommended themes.

These recommended themes were picked from the Add-ons for Firefox website by members of the Firefox User Experience team. All of the themes are licensed through Creative Commons. Some are CC-BY and others are CC-BY-SA.

Themes menu

Hovering over a theme in the menu will preview the appearance of the theme. Clicking on one of the themes will change the applied theme.

An applied theme

We haven’t figured out yet what the rotation will be for recommended themes. Any input on how often or how we should go about putting together the next list is greatly appreciated.

Full management of themes and add-ons is still available through the Add-ons Manager. Recommended themes that have not been applied will not show up in the Add-ons Manager. Once a recommended theme is applied, it will appear in the Add-ons Manager and can be uninstalled from there.


Tagged: firefox, planet-mozilla, usability [Source: Planet Mozilla]

Julien Vehent: Postgres multicolumn indexes to save the day

I love relational databases. Well designed, they are the most elegant and efficient way to store data. Which is why MIG uses Postgresql, hosted by Amazon RDS.

It's the first time I use RDS for anything more than a small website. I discover its capabilities along the way.  Over the past few days, I've been investigating performance issues. The database was close to 100% CPU, and the number of DB connections maintained by the Go database package was varying a lot. Something was off.

I have worked as a junior Oracle & Postgres DBA in the past. In my limited experience, database performances are almost always due to bad queries, or bad schemas. When you wrote the queries, however, this is what you blame last, after spending hours looking for a bug in any other components outside of your control.

Eventually, I re-read my queries, and found one that looked bad enough:

// AgentByQueueAndPID returns a single agent that is located at a given queueloc and has a given PID
func (db *DB) AgentByQueueAndPID(queueloc string, pid int) (agent mig.Agent, err error) {
	err = db.c.QueryRow(`SELECT id, name, queueloc, os, version, pid, starttime, heartbeattime,
		status FROM agents WHERE queueloc=$1 AND pid=$2`, queueloc, pid).Scan(
		&agent.ID, &agent.Name, &agent.QueueLoc, &agent.OS, &agent.Version, &agent.PID,
		&agent.StartTime, &agent.HeartBeatTS, &agent.Status)
	if err != nil {
		err = fmt.Errorf("Error while retrieving agent: '%v'", err)
		return
	}
	if err == sql.ErrNoRows {
		return
	}
	return
}
The query locates an agent using its queueloc and pid values. Which is necessary to properly identify an agent, except that neither queueloc nor pid have indexes, resulting in a sequential scan of the table:
mig=> explain SELECT * FROM agents WHERE queueloc='xyz' AND pid=1234;
QUERY PLAN                                                   
--------------------------------------------------------------
 Seq Scan on agents  (cost=0.00..3796.20 rows=1 width=161)
   Filter: (((queueloc)::text = 'xyz'::text) AND (pid = 1234))
(2 rows)

This query is called ~50 times per second, and even with only 45,000 rows in the agents table, that is enough to burn all the CPU cycles on my RDS instance.

Postgres supports multicolumn indexes. The fix is simple enough: create an index on the columns queueloc and pid together.

mig=> create index agents_queueloc_pid_idx on agents(queueloc, pid);
CREATE INDEX

Which results in an immediate, drastic, reduction of the cost of the query, and CPU usage of the instance.


mig=> explain SELECT * FROM agents WHERE queueloc='xyz' AND pid=1234;
QUERY PLAN                                                     
---------------------------------------------------------------------------------------
 Index Scan using agents_queueloc_pid_idx on agents  (cost=0.41..8.43 rows=1 width=161)
   Index Cond: (((queueloc)::text = 'xyz'::text) AND (pid = 1234))
(2 rows)

migdbcpuusage.png

Immediate performance gain for a limited effort. Gotta love Postgres !

[Source: Planet Mozilla]

Pete Moore: Weekly review 2014-08-27

Highlights from this week

1. Play Store - armv6

The main goal of the last week has been to enable fennec builds on esr31 branch. Last week I updated the build process to use a different mechanism to generate the version code in the play store for armv6 apks generated from the esr31 branch. This week has been about enabling these builds and release builders.

This work is tracked in Bug 1040319 – Ensure that Fennec builds from mozilla-esr31 have a buildID to allow for armv6/Android 2.2 users to update to mozilla-esr31 apks.

2. Working with contributors

I’ve been working with kartikgupta0909 this week on IRC - hoping he is going to fix Bug 1020613 - vcs sync should only push tags/heads that have changed since last successful push for us.

I added metadata to bugs, and created a bugzilla search for them to appear in, which I linked to from our contributions wiki page (and I created a sublink to RelEng contributions page from our main Release Engineering page).

3. Other

Regular type support work, which can be seen in bugs below.

Goals for next week:

  • Return to l10n work
  • Prepare for RelEng arch meeting in September

Bugs I created this week:

Other bugs I updated this week:

[Source: Planet Mozilla]

Doug Belshaw: Soliciting feedback on v1.1 of the Web Literacy Map

The Web Literacy Map constitutes the skills and competencies that Mozilla and its community of stakeholders believe to be necessary to read, write and participate effectively on the web.

Sea

The Web Literacy Map currently stands at v1.1 but as I blogged recently, a lot has happened since we launched the first version at MozFest last year! That’s why we’re planning to update it to v2.0 by early January 2015.

I’ll be connecting with key people over the coming weeks to ask for a half-hour (recorded) conversation which will then be shared with the community. In the meantime we’d appreciate your feedback. Here’s what Atul Varma had to say:

So I feel like the weblit map is cool as it is, but as has been discussed previously, there are a number of areas that are important but cross-cut through existing competencies, rather than necessarily constituting their own competencies by themselves… what if we created a set of lenses through which the competencies could be viewed?

There’s a couple of ways you can give your feedback:

Leaving your name means we can follow up with questions if necessary (for clarification, etc.) I look forward to hearing what you have to say! All opinions are welcome. Pull no punches. :-)


Questions? I’m @dajbelshaw on Twitter or you can email me: doug@mozillafoundation.org

[Source: Planet Mozilla]

More News


Sponsored by: