On the topic of fuzz testing, Python has an excellent library for property-based testing called Hypothesis [0] [1]. I don't think it does guided testing like AFL or libFuzzer (which OSS-Fuzz uses), but it's very powerful nonetheless.
I developed a fuzzer that generates random values based on ABNF rules, such as often appear in IETF RFCs. So it can be used for testing RFC implementations. It's written in Java but can be called from other tools.
There is a good wrapper for afl, https://pypi.python.org/pypi/python-afl that allows you to relatively easily use afl with your python code. Just be sure to take advantage of looping or you'll be limited by the python runtime start-up overhead.
Seconded. I've used it for testing various functions in recommendation systems, and it's great at finding boundary conditions you didn't think of, or issues with floating point math. And, as it was the first time I used any testing tools like this, it was a fun emotional experience to have an antagonistic test suite.
I wrote a wrapper around CPython for fuzz testing with libFuzzer [1]. It's dog slow because it's designed for anyone to be able to write relatively simple test cases in Python (so every iteration of the fuzzer starts the interpreter, executes the test, tears it down...). But a neat concept IMHO.
Since people are listing fuzzers, there are also the commercial alternatives: Peach Fuzzer (www.peachfuzzer.com/) and Synopsys Defensics (https://goo.gl/wroKI8). While my experience is not that recent, these two seem to dominate the network protocol fuzzing domain, whereas some of the open alternatives listed here appear to have become superior in general purpose 'parser' fuzzing. That said, played with honggfuzz recently and it really shows promise, especially if you have access to the source code (https://github.com/google/honggfuzz).
I'm not against doing the work to create the pull (unless someone else wants to) but looking at https://www.openssh.com/report.html, it's not clear to me if openssh[at]openssh.com or openssh-unix-dev[at]mindrot.org should be used for something like this.
Try firing it at yourself to start with, then ask the openssh community once you've got some useful data?
Edit: Thanks for the downvote! Every time I consider sending in a pile of automated bug-reports to a project where I'm not already part of the community, I look at them myself first. If that's bad, I'm happy to be bad.
> Recent security stories confirm that errors like buffer overflow and use-after-free can have serious, widespread consequences when they occur in critical open source software.
This project is awesome and incredibly valuable but what alternatives are there to making the libraries it checks more secure besides rewriting them in another language? When languages exist where buffer overflows and use-after-free are essentially impossible it's a bit depressing that we have to rely on fuzzing unless fuzzing can find these kinds of bugs with high reliability?
> what alternatives are there to making the libraries it checks more secure besides rewriting them in another language?
I'm not sure what you mean by rewriting a library.
For example, if a library is written in Ruby, fuzzing might not be suitable. you shouldn't rewrite it in C, so that you can fuzz it. (I thinkI misunderstand your point here. Please correct me.)
If we want to discover vulnerabilities in Ruby, we should fuzz the Ruby VM directly. We shouldn't fuzz a Ruby library.
Because it has nothing to do with the story or the rest o the thread, this reads as empty snark. Since someone seems to have the bright idea of publishing this kind of snark on nearly every thread on HN, you can generally expect to see the light grey bottom of any thread you try it on here.
1. it's irrelevant. This is an open source project, google code was a project hosting environment.
2. Google code "bit the dust" after being around 10+ years, and being replaced by something else ....
(and had tools/etc and hand holding for the deprecation).
So, 10 years from now, if someone has made "the github of open source fuzzing", and it's not this project, yeah, i guess it's fair that it might bite the dust.
But honestly, much like Google code, if 10 years from now, someone else has replaced oss fuzz with something that has hundreds of thousands of open source projects, i'd call that success, not failure.
Definitely this. The hard work, from my experience, in hooking up to a fuzzer is writing the wrappers to hook in to the fuzzer, and creating a set of examples from which to begin fuzzing intelligently.
Automated fuzzing is the icing on top of that hard work. It magnifies the utility of doing the work to fuzz by making it run continuously and by throwing more computational resources at it to find more bugs. But, unlike, say, coding to some closed API where you're up a creek if the service goes away, most of the hard work is portable, and you can still run it on your own cluster. AFL and libFuzzer are open source and quite awesome on their own.
OK, thanks for the answers. We had product code on code.google.com and it was definitely a bummer when the site shut down. I'm possibly alone in the galaxy in saying this but I really liked their bug tracker which was lightweight but just functional enough to be really useful.
> In order for a project to be accepted to OSS-Fuzz, it needs to have a large user base and/or be critical to Global IT infrastructure
Let's see - Firefox and/or the Tor browser? I imagine Google wouldn't be too happy about doing free security research for Firefox, but it seems to fit the bill quite well for the goals and mission of the Core Infrastructure Initiative organization.
I think that represents a fundamental misunderstanding of Google's incentives. (disclaimer: I'm typing this from Google, but it's purely my own opinion.) (And wishing my experiment would finish running faster.)
Google's biggest revenue stream is ads. It has many others, but that's the big one. Ads are seen by people who use Google search, and who browse many ad-supported websites. Google also has a lot of users of its services, e.g., gmail, who have accounts. Some also entrust valuable data to Google.
Google therefore has a great deal of incentive to make sure that: (a) Nobody messes with the ability of general users to browse the Internet safely (preserving ad revenue); and (b) It's very hard to compromise users computers and gain access to their Google/gmail/whatever accounts (reducing support costs and keeping users happy), or destroy their data, or exfiltrate it. Even if the compromise was the user's computer, it's still a very bad experience.
With fuzzing itself, as DannyBee alluded to below, one of the leads of a lot of this infrastructure, Kostya Serebryany, is also personally passionate about seeing it be taken up to make the software of the world better. I've seen this in action - he convinced me to use libFuzzer to improve TensorFlow's robustness, and mentioned that he was doing so in part so I'd take that experience back to Carnegie Mellon and spread the word. :) (and it worked - https://github.com/tensorflow/tensorflow/commit/7231d01fcb2c... for example).
One could argue, especially considering some of the disclosure situations surrounding Project Zero that have occurred, that it exists more to shame competitors than improve the world. :) To my knowledge, only like once have I heard the words "Project Zero" in relation to a flaw in Google software.
But as Google definitely does do security testing on competitors' products such as Windows, it definitely goes to show that Google would be more than happy to test for security flaws in Firefox. And regardless of Google's motives, finding security flaws does, in the end, make a more secure Internet for everybody.
That's encouraging -- I had cynically assumed it was just a naive PR play -- but have they screwed over other Google projects by publishing 0-days at the end of their arbitrary time window? For example, do they publish zero-days on widely-deployed versions of Android?
Google actually did and does a whole lot to support firefox security. There's a lot of collaboration in the security space, e.g. Firefox and Chrome security teams heavily collaborate on fixing the TLS/CA ecosystem.
Also the interest is very often shared, because a whole pile of libraries are used by both browsers: freetype, libjpeg-turbo, libpng, giflib, expat, webrtc, opus, ...
Why wouldn't they be happy? Google's a diverse company, it's not like every department/team has an unwritten goal to push Chrome at the expense of everything else.
> "it's not like every department/team has an unwritten goal to push Chrome at the expense of everything else."
It often feels like it, yes. Could it be that you don't see it because you are already using Chrome?
I use Edge on my Windows gaming machine (for minimal maintenance, I install drivers, Steam, and that's pretty much it) and configured Edge to clean everything including cookies on exit, like a permanent stateless Private Mode available for the occasional search.
End result: many Google apps (Search, Maps, YouTube, and probably many others) constantly display huge "Get Chrome" banners, that end up not only painful for the user experience, but contain plain lies: "A more modern, faster browser" says one, whereas Edge/Firefox/Chrome are now objectively in the same ballpark.
This constant soliciting across vast swaths of the web by one dominant actor controlling the end-to-end experience is abusive.
On the other hand, I use Chrome on my Windows machine - the constant dialogs that pop up over the Edge shortcut whenever I load up Chrome telling me how great Edge is are really tiresome.
This is pretty much solely in retaliation for the incredibly annoying "Get Chrome" behavior Google has been engaging in heavily for years. Beyond privacy issues, just avoiding these incessant banners makes it worth avoiding Google websites.
On my work machine I have to use IE for some internal sites that arbitrarily block FF and Chrome. At least once a week IE10 opens a second tab on startup pitching the wonders of IE11 to me and offering to install it immediately.
That's different. MS wants you to upgrade to at least IE11 (or Edge) because IE10 is no longer supported [1], and keeping using it is dangerous. (But maybe you cannot upgrade because of !@#$% corporate policies.)
[1] https://en.wikipedia.org/wiki/Internet_Explorer_10 says "On January 12, 2016, support ended for IE10 on Windows operating systems capable of running Internet Explorer 11, due to new support policies dictating that only the newest version of IE available for a supported version of Windows will be supported. IE10 will only be supported on Windows Server 2012 and Windows Embedded 8 Standard."
I use Firefox and I agree with you, I don't like the "Use Chrome" banners that I see when using IE (i dont get them with ff). However, baring that, is there anything else they've done?
IIRC they still pay Mozilla for making Google the default homepage
> "IIRC they still pay Mozilla for making Google the default homepage"
True.
> "is there anything else they've done?"
Nah, and frankly that's just irrational in-my-head ranting, their behavior is entirely expected rational behavior. Google is a data junkie which critically all-caps NEEDS as much data as possible about us, in order to machine-learn our lives from head to toe so that it can make money helping companies sell us more stuff.
Of course Apple isn't bothering us to use Safari when browsing apple.com, it doesn't care as long as you buy Macs. Of course Microsoft neither, it doesn't care as long as you use their platforms.
But Google is different, it needs your data to operate, and so it will do whatever it takes to get its data fix, including bullying.
</rant>
EDIT+OFFTOPIC: cool article on your blog about AFL, thanks for sharing it :)
>> "IIRC they still pay Mozilla for making Google the default homepage"
> True.
Not true. Mozilla switched to Yahoo a few years back; Google doesn't give them any money. And it's not the default homepage, it's the default search engine (the default homepage always starts out `about:home`).
Eh I guess you're right, as you're mozillian and as I remembered Yahoo too. But can you help me understand this: I was unsure whether today's Mozilla deal was with Yahoo or Google, so I double-checked before asserting True.
And running firefox with a fresh new profile, I confirm all search sources (awesome bar, in-chrome search box, about:home in-content search box) default to querying Google, not Yahoo.
Is it new to be back to Google? Or maybe it's specific to Nightly builds? (53.0a1 2016-12-01) -- EDIT no, same behavior under 50.0.2 stable. And it's not the OS either, I tried Ubuntu and Windows --. Any idea why, then?
Yahoo is only the default for users in North America. Bidau for China and Yandex for Russia. In most other cases Google is the default as their search service produces good results for the user.
