What Caught My Eye - Week 12

Technology & Tech Culture:

  • This video featuring Monty Montgomery from Xiph is a phenomenal and cogent explanation of how analog to digital to analog conversion works, and why the stair-step chart we often see is a horrible lie.
  • Hiring is a bit of voodoo, and often times I hear people talk about Passion, which is a bit of a bugbear of mine. This post talks about how excellent candidates can be overlooked because the signifiers of 'Passion' aren't there.

Arts and Culture

Examining "Reproducibility in Computer Science"

I'm heartened to see Professor Shriram Krishnamurthi spearheading an effort to help evaluate yesterday's reproducibility project.

We are evaluating the results presented by the study at the University of Arizona. Our goal is to allow authors and any other interested parties to review that study’s findings about an individual paper, and attempt to reconstruct their findings. We will summarize the results here.
We are grateful to Collberg, et al. for initiating this discussion and making all their data available. This is a valuable service based on an enormous amount of manual labor. Even if we end up disagreeing with some of their findings, we remain deeply appreciative of their service to the community by highlighting these important issues.
We do wish disagree with the use of the term “reproducibility”, which many people associate with an independent reconstruction of the work. For instance, this paper spells out the difference between repeatability and reproducibility and provides an interesting case study.

I've already submit a dispute, where code isn't available where it was thought that it was. I plan to do a little more looking, time permitting.

Others have successfuly built projects reported as failures. I think that the community is going to help repair the results of the other study.

Combined, they're going to say much more about the state of the field!

Edited to Add: 

Looks like Dr. Krishnamurthi's effort sprung from two discussions on social media, which are worth trawling for insight: 

Source: http://cs.brown.edu/~sk/Memos/Examining-Re...

Reproducibility in CS

So, there’s a project from the University of Arizona that’s trying to investigate CS papers and see how reproducible they are.

Image from the Reproducibility in Computer Science page, retrieved March 19, 2014. 

Image from the Reproducibility in Computer Science page, retrieved March 19, 2014. 

It’s a cool project. Made a pretty picture to be sure! The problem is, their pretty picture is… pretty misleading when it comes to build failures.

There’s many more like this — I took a cursory look through their results, and here’s a couple of categories:

Path setup failures:

Miscellaneous student errors:

I’m a little sad that this is what they decided to put out. Their definition of reasonable effort is

In our experiments we instructed the students to spend no more than 30 minutes on building the systems. In many cases this involved installing additional libraries and compilers, editing makefiles, etc. The students were also instructed to be liberal in their evaluations, and, if in doubt, mark systems as buildable.

Yet, as you can see above, it appears nobody actually looked at what the students said. They were taken at their word, and they were far from liberal!

Moreover, I also disagree that an undergrad given 30 minutes is a good check. Especially considering the quality of the above results.

Shaming researchers through bad methodology like this isn't going to get more people to put out code.

I love the idea of this investigation, but I don’t love the methodology.

Edited to Add:: One thing that I did love from this project was their proposal in the technical report:

Our proposal is therefore much more modest. Rather than forcing authors to make their systems available, we propose instead that every article be required to specify the level of reproducibility a reader or reviewer should expect.

This is a worthwhile baby-step in and of itself, and should probably be integrated into paper classification systems. THis would eventually provide evolutionary pressure towards reproducibility.

What Caught My Eye - Week 11

This one is going to be short and sweet. I've been sucking at keeping track this week, and sufficiently busy and burnt-out that I've not collected as much as usual.

Arts and Culture

  • The Black Hand Files:

    A few months ago, my child came in and calmly announced, "I have a friend named Black Hand. He's here right now. You can't see him."

  • William Shakespeare presents: Terminator The Second

  • Some more on John Roderick's trip to a Miley Cyrus Concert

    John Roderick explains that, contrary to his first impression, he watches the show in utter amazement, slowly understanding that Miley Cyrus is in complete control of every aspect of her career.

    The whole event was completely unironic. There was no cynicism to it. The net result was a total positivity towards the 8-10000 girls all dressed like sex workers. You realize that they are not dressing like that for the male gaze, because there are no men here. They are dressing like that for one another, for their own pleasure, and in homage to Miley, and she's dressing like that for them. And the other amazing thing was that not a single person looked at me like I was a creep. Every single girl that made eye contact with me — which was hundreds and hundreds of them — they all smiled and were like "Hi!" or "Excuse me!" I never for a moment had that feeling of "Omg, What are you doing here?" That was absent from the place.

What Caught My Eye - Week 10

Diversity, Culture, Feminism

  • "When a pipeline leaks, we don't blame the water" is a key theme of this piece by Frances Hocutt, describing how she left her love, chemsistry. This piece, and the next are from a fantastic looking magazine, Model View Culture, which looks to be filled with interesting writing.

    So, someone you know is considering leaving a STEM field and you wish she wouldn't, or you vaguely wish she felt more supported in her current position. You have opinions about the proverbial leaky pipeline. You're sad or angry that you'll be the only woman in the lab once your coworker leaves. You're frustrated that your brilliant, driven mentee quit her job and left the field. You want to get more women in STEM, so you focus your efforts on trying to recruit new women and girls. You're recognized for your outreach efforts -- and your colleague's going-away lunch takes you by surprise.

    So. What can you do?

    You can recognize that our choices to leave are rational decisions that demonstrate self-knowledge and self-respect. We have weighed whether we love the work more than we hate the context we do it in. You can accept our analysis and respect our agency, and not try to convince us that you know better or that we should have worked (even) harder. If you’re part of a majority and we are not, you can acknowledge that we've probably already worked harder than you have to get to the same place.

  • Quantify Everything: A Dream of a Feminist Data Future talks about a historical view of data, and how the definition is changing.

    “Data” has historically been a neglected byproduct of action and interaction, and looking after it has been less a priority than an accident. That data has taken on masculine and technologically essential attributes in recent years is a testament to how quickly and pervasively market semantics can work. For centuries, collecting, caretaking, curating and analyzing data has been the domain of women’s work—look at the histories of librarianship, nursing and programming.

  • TRIGGER WARNING: This is an article about the insidious spread of trigger warnings

    Trigger warnings are presented as a gesture of empathy, but the irony is they lead only to more solipsism, an over-preoccupation with one’s own feelings—much to the detriment of society as a whole. Structuring public life around the most fragile personal sensitivities will only restrict all of our horizons. Engaging with ideas involves risk, and slapping warnings on them only undermines the principle of intellectual exploration. We cannot anticipate every potential trigger—the world, like the Internet, is too large and unwieldy. But even if we could, why would we want to? Bending the world to accommodate our personal frailties does not help us overcome them.

Art and News

Just what it says, guy turns himself into a haybale