Monday, 31 May 2010

Why is Haskell used so little in the industry?

Bugspy asked an interesting question on Stack Overflow recently:
It is a wonderful, very fast, mature and complete language. It exists for a very long time and has a big set of libraries. Yet, it appears not to be widely used. Why ? I suspect it is because it is pretty rough and unforgiving for beginners, and maybe because its lazy execution makes it even harder
Jeff Atwood, cofounder of Stack Overflow, has since closed the question for being subjective and argumentative. Sadly, Marc Gravell chose to edit my answer and delete half of my comments so the discussion no longer makes sense. However, my response will be of interest to anyone considering gambling their own money on Haskell so I shall repeat it here.
The first point of interest is Norman Ramsey's meta-answer which was written from his point of view as an academic. Norman explains why he believes that major institutes have already made "great use" of Haskell:
"Don't tell that to Credit Suisse or Standard Chartered, two banks which have made great use of Haskell. And don't tell Peerium or Galois or any of the other successful startups that use Haskell. Don't tell any Commercial Users of Functional Programming. And especially don't tell their customers!"
Citing Credit Suisse and Standard Chartered is just name-dropping big banks in the hope that it will deceive people into thinking that these institutes are seriously invested in Haskell. They are not. In reality, the number of full-time Haskell developers at Credit Suisse has long since fallen to zero and one diehard fled to Standard Chartered. This is a tiny fraction of a percent of the total development effort at those institutes and does not constitute "great use of Haskell" by any stretch of the imagination.
Citing CUFP is also misleading. CUFP is primarily a Haskell conference attended almost entirely by academics and students, none of whom have customers.
Norman Ramsey goes on to say:
"Industry rewards getting products out quickly, even if the products are barely adequate. Some ways to get things out quickly include...Throw lots of teams at the problem; first one with a successful approach wins."
This is an absurdly-inaccurate caricature of industry. In reality, it is precisely these kinds of comments from key figures of the Haskell community that drive industrialists away.
In the same thread on Stack Overflow, Don Stewart claimed that "Haskell, if anything, is more widely used than most / all FP languages" and later cited the prevalence of Haskell among CUFP delegates as evidence. This could not be further from the truth, of course. Mathematica has millions of users and dozens of commercial libraries. The F# Hub has 21,228 registered members and F# has several commercial libraries (e.g. WebSharper, CoherentPDF, F# for Visualization and F# for Numerics). In contrast, the Haskell-Cafe has only 3,339 subscribers and Haskell has never had any commercial libraries.
Industry is very diverse and it often rewards taking risks with new technologies because that can give you a competitive advantage. We regularly take risks with fringe technologies like OCaml and F#. If you get it right, it really can pay dividends. If you get it wrong, you'll go the way of Red Nucleus (one of the many deceased Haskell startups). We decided to diversify in 2007 and I was challenged with the task of finding other functional programming languages for us to use. I studied Haskell extensively and made the informed decision not to go anywhere near it for the following reasons:
  • Lack of compelling examples. After much searching I found only one program, Power Serious by Doug McIlroy, that I found compelling but it is just an academic curiosity.
  • Lack of genuine success stories. After a bit of digging, I was horrified to discover that almost all of the "success stories" listed on the Haskell in Industry page were fakes. Anygma may ship a product in the future but do not intend to write it in Haskell. Ansemond had a product and are actually a real software company but (according to their blog) there was "no Haskell" in their product. Antiope didn't list any products at all, let alone ones written in Haskell. Bluespec were advertising their product but their customer testimonials were from "Elvis Lives", "Chicken Man", "Painful Eliminations", "Mr. Bigglesworth" and two people from the founders' University. The prominent member of the Haskell community from Credit Suisse told us that his employer was seriously invested in Haskell with over a hundred developers and that he was personally responsible for much of their Haskell development but this turned out to be another deception: Credit Suisse have over a hundred developers using other languages, he had been a large proportion of a tiny number of Haskell developers and Haskell never caught on at Credit Suisse (they are now merely maintaining legacy Haskell code). Erlang Training and Consulting were placed on the Haskell in Industry page as spam. They had apparently never made any use of Haskell whatsoever. Galois are apparently the only self-sustaining company seriously invested in Haskell in the world but their only visible guy is notable only for having listed a PhD on LinkedIn that he hadn't earned and for generating enormous quantities of contentless propaganda such as the analysis of delegates to a Haskell conference above and his analyses of IRC chat. Nokia was a link to a job advert not directly related to Haskell. Openomy was a link to a blog with no products. Many "startups" including Red Nucleus were listed but never saw the light of day. SeeReason Partners, LLC was another company with no products or services at all, let alone Haskell-related ones. One of the "founders" assured us back in 2007 that they would be putting up a real company page but that never happened. This systematic deception of the public by most of the prominent members of the Haskell community was the last nail in Haskell's coffin from our perspective. Regardless of technical merit, the risk of being associated with such people is far too great.
  • User unfriendly. The compiler alone is a nightmare to install (GHC 6.12 just took me 2 days!) and the libraries are even worse (cabal install breaks on about 30% of packages). Trying to put customers through that pain makes about as much business sense as going into private dentistry without anesthetics.
  • Commerce unfriendly. The entire Haskell infrastructure is built on the assumption that software is distributed in source form. Nobody has ever succeeded in selling closed source Haskell libraries to Haskell programmers. So we could only expect to sell source code (an unnecessary risk not imposed by competing languages) or use it in-house. Almost all Haskell users are academics and many, like Norman Ramsey, are vehemently anti-industry.
  • Ill advised. Even Simon Peyton-Jones advised against us trying to use Haskell in industry because they neither wish nor intend to support real users. Haskell is an on-going experimental research project constantly at risk of major breaking changes.
  • Bad science. The Haskell community generate an unprecedented amount of bad science, setting out to draw the conclusion that Haskell is great and using every trick in the book to make that conclusion seem feasible. In many cases, it requires quite some effort to uncover the truth. For example, Saynte published results for "naively" parallelized ray tracers and concluded that Haskell was the easiest language to parallelize efficiently, requiring only a single line change. This deception was perpetrated by starting with a "serial" version that had already been extensively reworked in order to make it amenable to parallelization and then disabling optimizations only for the competitors (and even adding their compile times in one case!) in order to make Haskell appear competitive. Moreover, that reworking could only have been done on the basis of extensive benchmarking and development. Even the academics publishing Haskell research are up to it. For example, in the recent paper Regular Shape-polymorphic parallel arrays Manuel Chakravarty et al. artificially close the gap between Haskell and C by benchmarking only cache ignorant algorithms (that spend all of their time stalled on unnecessary cache misses), incorrectly states that such algorithms are "widely used", describes the one-line change required to parallelize the C code as "considerable additional effort" despite the fact that both the serial and parallel C solutions are substantially more concise than their Haskell counterparts, refused to provide me with their C code so that I could try to verify their findings (until I published this blog post, their code is now here) and cherry picked Haskell's highest performing results as a function of the number of threads (which peaks at an unpredictable number before seeing performance degradation). When I voiced my concerns, Don Stewart abused his moderator status on the forum by deleting my criticisms. To date, these flaws in the paper have not been addressed.
So I decided that we would diversify into F# instead of Haskell. With the benefit of hindsight, that was absolutely the correct decision. Perhaps other people will choose to risk using Haskell in industry but I would strongly advise them to be very cautious about the "information" that will doubtless be bestowed upon them.

57 comments:

mehdi.asgari said...

"CUFP is primarily a Haskell conference attended almost entirely by academics and students, none of whom have customers."

Are you sure ?
Look at the list of some speakers in
2009:
* Francesco Cesarini, Erlang Training and Consulting, UK
* Kathleen Fisher, AT&T, USA
* Jim Grundy, Intel, USA
* Yaron Minsky, Jane Street Capital, USA
* Ganesh Sittampalam, Credit Suisse, UK
* Ulf Wiger, Ericsson, Sweden

2008:
Lennart Augustsson, Credit Suisse
Adam Granicz, IntelliFactory
Jim Grundy, Intel
Andy Martin, IBM
Yaron Minsky, Jane Street Capital
Ulf Wiger, Ericsson

Flying Frog Consultancy Ltd. said...

@Mehdi: "Are you sure?". Several people have told me that delegates (not speakers) are primarily Haskell researchers and students and many of the CUFP videos substantiate that. For example, the CUFP 2009 discussion: "Who here has used any kind of functional language in a commercial capacity?". Only half of the delegates had even attempted to use any functional language in a commercial capacity (!). Then a survey of delegates by language => mostly Haskell. Then Yaron Minsky explains that what he wants out of CUFP is students to recruit. Then a student delegate stands up and says he is about to finish his studies. Later, an academic starts monologuing about functional programming in the real world. At 16m20s, someone states that many OCaml developers attended the co-located ICFP but did not bother attending CUFP.

augustss said...

@Jon, so in your mind, what constitutes a success story?
BTW, of course there is a large number of academics at CUFP; it's colocated with ICFP and a number of academic workshops.

Flying Frog Consultancy Ltd. said...

@Lennart: To me, a success story is a serious investment in a technology that pays off in the form of growth. The bigger the risk, the more compelling the story.

@Marc: The question asked why industry are not adopting Haskell and I think an important aspect of the answer is the people. People in industry find the behaviour I described abhorrent. I respect what you are doing but you can see why removing references to people from my argument makes no sense in this case.

Ganesh Sittampalam said...
This comment has been removed by the author.
Ganesh Sittampalam said...
This comment has been removed by the author.
mehdi.asgari said...

I have subscribed to receive followup comments on this post; Why do you keep deleting "Ganesh Sittampalam"'s comments ?

Flying Frog Consultancy Ltd. said...

@Mehdi: "Why do you keep deleting "Ganesh Sittampalam"'s comments?". I have no desire to discuss this with him again.

augustss said...

Did you delete Ganesh's comments because he was objecting to your outright lies about him?

Flying Frog Consultancy Ltd. said...

@Lennart: "Did you delete Ganesh's comments because he was objecting to your outright lies about him?". Here is another example of Ganesh falsely claiming that the use of Haskell at Credit Suisse was on the increase just before it was stopped and you jumped ship to Standard Chartered. If you gave an honest description of what really happened you could save other people from making the same mistakes.

augustss said...

@Jon, Yes, I'd say Ganesh's post was about accurate. When I left there were 5 developers using Haskell, not all of 5 full time, but 4 sounds about right. And there were around 10 other people using the framework that we made. Haskell was not declining when I left, and that was not the reason I left. I did think that Credit Suisse might shift towards F#, because it made good sense to do so since they are a Windows only shop. And F# provides excellent interop with Windows and COM (excellent, but slow).
I changed jobs because it was a green field opportunity, and it was a very interesting new job.

BTW, reading your comments about Bluespec in the link you provided made me realize how totally clueless you are about that company.

Ganesh Sittampalam said...

@Jon: It was a perfectly true statement. You can check our job adverts from that period if you don't believe me.

Flying Frog Consultancy Ltd. said...

@Lennart: Thanks for the response. You say "there were around 10 other people using the framework that we made" but Ganesh says "Our code is used by several hundred or perhaps low thousands of users". That is the kind of discrepancy I was referring to.

@Ganesh: "You can check our job adverts from that period if you don't believe me". Replacing employees is not growth.

casey said...

I like many of the ideas in the Haskell language and community. The main reason I don't use Haskell is that the language does not lend itself well to a model-driven approach. I like OO for expressing models as pure data, but the record syntax is too unwieldy. I like functions and immutability for my logic. Why can't Haskell make both of these concise and readable?

Don Stewart said...

Jon, just FYI, I'm actually completing my PhD in a few weeks. I'll send you a copy of the dissertation, if you're interested.

I've been working on it part-time since moving into business, not that its any business of yours.

Anyway, you just had to ask, but instead you made up lies about someone you've never met, and used that in a technical forum. Classy.

Ganesh Sittampalam said...

@Jon: There's no discrepancy, it's two different things - one set of people were actually writing Haskell using one framework, the other set were *using* a compiled component implemented in Haskell.

I never said we were replacing rather than adding employees, that's also your invention. But in any case you don't need to be adding new employees to be doing more stuff; I was just pointing to things you could find publicly that support our continued interest in Haskell at that time.

rl said...

Just for the record, it was mostly me who did what you refer to "cherry picking" in the regular arrays paper. It is, in fact, extremely bad style to single out one author when talking about a paper (and a library) that has been coauthored by 5 people.

As to "cherry picking": yes, we do talk about maximal speedups. We do this to answer a very simple, valid and in my opinion obvious question: what are the maximal speedups we can achieve on these machines for these programs? All other data is also in the paper (see, in particular, Figures 7-9 which together take up about a page and are rather hard to miss). It is quite misleading to refer to this as "cherry picking". Also, I'd be very interested in finding out where you saw a "severe performance degradation" in these graphs.

By the way, you can find the current version of the library on Hackage. Older versions are in the DPH darcs repository. It is unfortunate that the lack of this information has prevented you from verifying our results but for some reason, I can't remember you asking for it - did you?

augustss said...

@Jon, as Ganesh already said, there's no discrepancy. There were about 10 people who did some amount of Haskell development using the framework. There were also other parts of the system that were written in Haskell and distributed as binaries that had hundreds of users (maybe 1000, I'm not sure). I don't think the latter fact was mentioned in our paper.

Flying Frog Consultancy Ltd. said...

@Don: "Jon, just FYI, I'm actually completing my PhD in a few weeks". Good luck!

@Roman Leshchinskiy: "It is unfortunate that the lack of this information has prevented you from verifying our results". You seem to have referenced only the Haskell code that I already have and not the C code that I stated was missing. Where is the C code?

"I'd be very interested in finding out where you saw a "severe performance degradation" in these graphs". You saw only max 2.6× speedup on the Laplace benchmark on 8 cores, probably because your algorithm has no temporal locality at all. I have a naive 18-line F# solution here that is 6.6× faster than your parallel Haskell. Contrary to the statements in your paper, the blame lies with your choice of algorithm and not the machine and these cache ignorant algorithms have not been "widely used" for decades. Harold Prokop described an elegant cache oblivious Laplace solver here way back in 1999. Incredibly, the essence of the cache oblivious algorithms is their reliance upon recursion. Something I'd have expected a functional programmer to leap on!

Ivan said...

Looks like my comment about GHC compile times and the relevance of that chosen sample package was deleted as well... oh well, at least I'm in good company!

Flying Frog Consultancy Ltd. said...

@Ivan: The relevance of what chosen sample package?

Ivan said...

I meant your "compelling example"; there seems to be no relevance on what it is or why it is a compelling example.

I also commented on your GHC compile times; unless you were bootstrapping your way from the last C-only release of GHC (6.2 I think) on a really old machine, there is no way it took you 2 days; on my 2GHz Core 2 Duo laptop it takes about 45 minutes; on my older 1.6 GHz machine it used to take about an hour and a half. Both of these are fair cries from 2 days.

I'd also be interested in how you found that Cabal breaks on 30% of packages; were you deliberately trying to build every package available without checking to see what it was when you came up with that breakage statistic?

Flying Frog Consultancy Ltd. said...

@Ivan: I found that example compelling because none of the tools I usually use could express a solution to that problem as elegantly. To date, it is by far most impressive piece of Haskell code I have ever seen.

Regarding installing GHC, I was not referring to compile time but the installation procedure, i.e. the problems the Haskell Platform is trying to address.

Regarding cabal install, I was trying to install just the software required to run Saynte's ray tracer.

Ivan said...

You haven't said what it's a compelling solution of, just that it is compelling. As such, it's difficult to tell how good a solution it is if you don't say what the problem was.

Regarding installation, the Haskell Platform doesn't take that much longer to compile (let's be generous, maybe an extra hour). Unless your system is that messed up or you are wanting to install really weird packages and have to build the C libs and toolchain yourself, it still won't take 2 days. Hell, I can probably recompile a large chunk of my system in 2 days (and I probably should to take into account the number of improvements in the latest version of gcc).

And if you're referring to the ray tracer found here, the only extra libraries you need are deepseq, primitive, parallel and vector; if 30% of those libraries failed then you had 1.2 packages that failed to build: http://poorlytyped.blogspot.com/2010/01/haskell-ray-tracing-parallel.html

rl said...

"You seem to have referenced only the Haskell code that I already have and not the C code that I stated was missing. Where is the C code?" To answer your question, the C code is indeed in one of the places I referenced, namely in the DPH repository. Look in examples/regularArrays, it's there somewhere.

You didn't state that it was the C code that's missing, though. Rather, you said that Manuel "never published his code (making the work unverifiable)". The C code isn't needed to verify our work. The performance of the C code isn't a contribution of the paper (these are listed in the introduction). It's included as a point of reference to give the reader something to compare the performance of the Haskell code to. The question it answers is: how do the straightforward Repa implementations of the algorithms we present compare to straightforward implementations of these same algorithms in C. We could have omitted this without affecting the contributions of the paper. However, this is usually the first question that people ask so we chose to answer it.

I asked: "I'd be very interested in finding out where you saw a 'severe performance degradation'". You answered: "You saw only max 2.6× speedup on the Laplace benchmark". That's not a performance degradation. I'm still very interested in an answer to my question.

"Contrary to the statements in your paper, the blame lies with your choice of algorithm and not the machine" What statements? We explain why we think these particular algorithms perform the way they do. You are quite right that there are other algorithms that might perform differently. However, these are the algorithms that we picked as examples for explaining how to use the library. I suspect you are of the opinion that the choice wasn't a good one but that's not the argument you are making. This doesn't have anything to do with "bad science", either.

Flying Frog Consultancy Ltd. said...

@Roman Leshchinskiy: "the C code is indeed in one of the places I referenced, namely in the DPH repository. Look in examples/regularArrays, it's there somewhere". That is your serial C code for the Laplace and matmult. In your paper, you state that parallelizing the C code required "considerable additional effort". Where is the parallel C code you referred to in your paper?

"You didn't state that it was the C code that's missing, though.". I originally said "Without the C code being used, this is not reproducible => bad science".

"However, this is usually the first question that people ask so we chose to answer it". Then you desperately need to seek peer review from people with relevant knowledge. The subject of your paper has been extremely heavily studied for decades. The correct reason why your scalability is so poor is widely known (see cache complexity).

"What statements?". I'll write a new blog post detailing each issue in turn.

"We explain why we think these particular algorithms perform the way they do". Your explanations are wrong.

"This doesn't have anything to do with "bad science", either". This has everything to do with bad science. You make statements in your abstract that are contradicted by your own results. You claim algorithms that went out of date decades ago are still "widely used". You call a single line of C code "considerable effort". You describe 2.6× speedup on 8 cores as "good parallel scalability" when algorithms published years ago scale far better (and run over 10× faster in absolute terms).

simonmar said...

Welcome to Jon Harrop's magic world where everything is not as it seems!

1. haskell-cafe has 3339 current subscribers (not 715), ahd that doesn't include those who read it via GMane and other sources (e.g. me).

2. No commercial libraries? Plenty of libraries on Hackage came from industry, and there are many more unreleased libraries being used.

3. Ill-advised: Simon PJ never said any such thing. I think you've just completely misinterpreted Haskell's unofficial "avoid success at all costs" slogan. Clue: it's a joke!

4. User unfriendly. How on earth did it take you 2 days to install GHC? How long does it take to type 'apt-get install ghc', or 'yum install ghc', or download the .exe and double-click on it, depending on what platform you're on? If it really did take you 2 days to install GHC, and it's not your fault (NB.!) then please submit a bug report, let us respond to it, and include the link in your post. Otherwise it's pure FUD.

5. I see you've dropped the claim that I censored your posts on haskell-cafe. FWIW, I didn't, anyway.

fededevita said...

I agree 100% with Jon. As a member of the financial industry, when I had to choose what language(s) to use, I chose F# in order to have exposure to functional structures. It's painless to add to the existing code we have and it owes a lot to Haskell (the GHC is, if I am not mistaken, partly funded by Microsoft, is it not?).

The London Quant Group - a non-profit organization for London based quants, i.e. people working in finance with a quantitative background: mathematicians, physicists, etc. - is organizing a technology conference this month and Don Syme is coming to present F#. That's one way the industry goes: the languages used are those that are made for the industry. Haskell is beautiful, but it was not made for the industry.

Flying Frog Consultancy Ltd. said...

@Simon Marlow: Thanks for the response but:

1. The F# Hub already has over 6× more registered members and its registration process is a higher activation barrier and it is growing much faster (1,000 new members in the last month alone). Mathematica has over an order of magnitude more users again. You cannot seriously believe that Haskell is the world's most widely used FPL.

2. You are describing open source and in-house libraries, respectively. They are not commercial libraries.

3. Simon PJ advised me against trying to use Haskell commercially when I met him over lunch in Cambridge in 2008 and described our needs.

4. Apt got me an old GHC and old cabal install. The libraries I needed required a cabal install upgrade but cabal install broke itself. I managed to fix it by hand with help from people on the mailing list (it was confused about its own install paths, amongst other things). Then I tried to install GHC 6.12 from source because I didn't realise GHC is infamously difficult to build. I had to give up on that, of course. The tale is on the beginners list. A friend told me he once wasted 2 weeks trying to build GHC so I am not alone. Why publicly shun valuable feedback as "FUD"?

5. Ok. Someone, who wishes to remain anonymous, from your circle of friends told me that you had said it was you. I apologise if he was wrong but that's all I have to go on. I can only assume it was Don Stewart again then...

simonmar said...

@Jon

I didn't claim that Haskell was the most widely used FP language - just correcting a figure in your post (it's still wrong, BTW, please fix it).

On the installing GHC point, your post still contains nothing but FUD: you claim that GHC is a "nightmare to install" when for the vast majority of users it is nothing of the kind. You had particular needs and ran into specific problems - by all means describe those, but don't make wild unfounded generalisations.

Simon PJ says he didn't tell you not to use Haskell. I think you've got the wrong end of the stick. We're actively encouraging industrial use of Haskell - look at the recently announced funded project to work with industrial users of parallel Haskell, for instance.

You are one of only 3 (I think, less than 5 anyway) people who has ever had their posts moderated on haskell-cafe, in the 10 years the list has been running. We're an open and inclusive community. The moderation bit is only used as a last resort. It's a big deal to get yourself moderated on haskell-cafe, yet you managed it. Good job! And haskell-cafe is not the only public forum where this kind of thing has happened, I gather.

Flying Frog Consultancy Ltd. said...

@Simon Marlow:

"it's still wrong". You wrote 3339 so I updated it to that figure. How is it wrong?

"your post still contains nothing but FUD". Your attitude is driving potential industrial users away.

"We're actively encouraging industrial use of Haskell". The Industrial Haskell Group has had no new industrial members since it was conceived over a year ago. That is an objective reflection of the fact that something is wrong, i.e. you cannot blame me for that as well. I suggest you stop calling feedback "FUD" and start trying to address real problems like the ones I have highlighted, starting with your own attitude. You have a golden opportunity to learn from my real industrial experience (actually selling products!) and you're wasting it.

"You are one of only 3 (I think, less than 5 anyway) people who has ever had their posts moderated on haskell-cafe..." A circular ad-hominem argument. You don't see that every day.

"We're an open and inclusive community". No you aren't. When I contested Don Stewart's pseudoscience he openly stated that he would do anything and everything to censor me.

"And haskell-cafe is not the only public forum where this kind of thing has happened, I gather". Either you are referring to Don Stewart censoring me elsewhere or you have fallen for one of his fantasies. I'd honestly love to hear what you're referring to here!

rl said...

"In your paper, you state that parallelizing the C code required 'considerable additional effort'." You misread the one half-sentence you are referring to. We say that C code doesn't scale well "without considerable additional effort". This doesn't imply that we actually parallelised any C code for this particular paper. We have parallelised enough C code previously to know that this statement is true. In fact, I have never heard anyone claim that parallelising C is effortless. In any case, the paper isn't about parallelising C. It's about parallelising Haskell.

"I originally said 'Without the C code being used, this is not reproducible => bad science'." You misread your blog post. You said this: "never published his code (making the work unverifiable)". This is not true because we did publish all our code. I would kindly ask you to correct this. Also, as I stated before you don't need any C code to verify the contributions of the paper which are plainly stated in the introduction. C isn't mentioned there.

"Then you desperately need to seek peer review from people with relevant knowledge." You will be happy to hear that research papers are, indeed, peer-review before being accepted for publication.

"The correct reason why your scalability is so poor is widely known (see cache complexity)." I highly doubt that cache inefficiency is the main reason for the parallel behaviour of the Laplace solver on our x86 machine. You are very welcome to prove us wrong, however, now that you have found all our code.

"I'll write a new blog post detailing each issue in turn." To avoid wasting your and everyone else's time, I would strongly suggest that you carefully reread the paper before doing so as you seem to misunderstand it's topic. It is about a Haskell library. It includes a couple of example algorithms to demonstrate how to use the library and to give an indication of its performance. These algorithms have been chosen because they are well known, easy to explain, easy to understand and demonstrate various aspects of the library. The paper is not about those algorithms.

"You make statements in your abstract that are contradicted by your own results." Which statements?

"You claim algorithms that went out of date decades ago are still 'widely used'." Which algorithms? Matrix multiplication, FFT or Laplace solvers based on point Jacobi iteration?

"You call a single line of C code 'considerable effort'." What single line of C code?

"You describe 2.6× speedup on 8 cores as 'good parallel scalability'" You misread the paper. We describe the other 5 benchmarks as "good parallel scalability". We specifically explain why the one particular benchmark you are referring to doesn't scale as well as the others and why this, in our opinion, doesn't indicate that the library itself has scalability problems. Your argument seems to be that the performance of this benchmark is due to its cache behaviour rather than memory bandwidth limitations. If that turns out to be true (which, again, I doubt), it would be a valuable correction, of course, but a minor one. It still wouldn't show any problems with the library itself and wouldn't invalidate any of the paper's contributions.

I'm left with the impression that you are accusing us of "bad science" because you disagree with our choice of example algorithms and with our assessment of the effort required to parallelise C programs. Am I mistaken?

Flying Frog Consultancy Ltd. said...

"This doesn't imply that we actually parallelised any C code for this particular paper". Had you tested your theory by asking an expert to parallelize your C code you would have seen immediately that your speculation was completely wrong in the cases in question. Stating conclusions about an experiment you had not done was bad science.

"We have parallelised enough C code previously to know that this statement is true". Perhaps you can explain the one-line counter example I gave your co-author a month ago?

"You misread your blog post". I correctly quoted my original peer review on Reddit, not this blog post.

"You will be happy to hear that research papers are, indeed, peer-review before being accepted for publication". Which experts on shared-memory parallel programming are reviewing your paper?

"I highly doubt that cache inefficiency is the main reason for the parallel behaviour of the Laplace solver on our x86 machine. You are very welcome to prove us wrong, however, now that you have found all our code". Over a decade of research already proved you wrong, of course. I can add a counter example written in F# to the new blog post, if you like.

"These algorithms have been chosen because they are well known, easy to explain, easy to understand and demonstrate various aspects of the library. The paper is not about those algorithms". The "contributions" section of your paper talks specifically about these algorithms and incorrectly states that they are "widely used".

"Which statements?" See the new blog post.

"Which algorithms? Matrix multiplication, FFT or Laplace solvers based on point Jacobi iteration?" If you do not distinguish between algorithms implementing Laplace solvers then blaming the memory bandwidth of the Xeon in your paper was wrong: cache friendly Laplace solvers get a lot more performance out of that machine because they scale better.

"What single line of C code?" The one I gave your coauthor here over a month ago.

"this, in our opinion, doesn't indicate that the library itself has scalability problems". Can your library express the cache friendly algorithms that people actually use?

"Your argument seems to be that the performance of this benchmark is due to its cache behaviour rather than memory bandwidth limitations. If that turns out to be true (which, again, I doubt), it would be a valuable correction, of course, but a minor one. It still wouldn't show any problems with the library itself and wouldn't invalidate any of the paper's contributions". It contradicts the last of the contributions listed in your paper. FWIW, if you can create a library that makes it easy to construct cache oblivious divide-and-conquer algorithms that will scale well, that would be extremely valuable.

"I'm left with the impression that you are accusing us of "bad science" because you disagree with our choice of example algorithms and with our assessment of the effort required to parallelise C programs. Am I mistaken?" Not quite. You presented theories as fact when there are trivial counter examples and you claimed that your chosen algorithms are widely used when they are not. Given the incorrect conclusions you drew about the poor scalability of your solution, I suspect none of the coauthors or peer reviews has any experience in multicore programming because these are really basic mistakes.

Flying Frog Consultancy Ltd. said...

On the upside, perhaps you can tweak your library to generate cache oblivious code, fix all of your scalability problems and make a vastly more valuable contribution. In the future, I would recommend getting peer review from people involved in state-of-the-art research on multicore programming, e.g. at MIT.

simonmar said...

@Jon, you quoted me as saying "your post still contains nothing but FUD", and then said "Your attitude is driving potential industrial users away."

The phrase you quoted seems to be missing its context. Allow me to repeat the paragraph of which it was a part, perhaps you missed it:

"On the installing GHC point, your post still contains nothing but FUD: you claim that GHC is a "nightmare to install" when for the vast majority of users it is nothing of the kind. You had particular needs and ran into specific problems - by all means describe those, but don't make wild unfounded generalisations."

(this is my last post here; thanks for leaving my comments intact.)

Flying Frog Consultancy Ltd. said...

@Federico: Thanks!

@Simon Marlow: I think your own generalization is based upon a self-selected group of people but we cannot prove either way.

rl said...

"Stating conclusions about an experiment you had not done was bad science." We do not state conclusions about an experiment. We mention, in a half-sentence, what we consider an uncontroversial property of C code, namely that it isn't easy to parallelise. This property has nothing whatsoever to do with the conclusions of the paper.

Nevertheless, you will be happy to learn that I have spent quite a lot of time teaching students specifically about the cache behaviour and parallelisation of matrix multiplication, complete with examples. So this is actually an experiment that I personally have done repeatedly. And indeed, parallelising C code did require considerable effort.

"Perhaps you can explain the one-line counter example I gave your co-author a month ago?" Yes. The one line in your counter-example is not C. The additional effort went into implementing the OpenMP extensions and I'm confident it was quite considerable. Personally, I prefer Cilk which is a much nicer language extension.

That said, I quite agree that comparing our library to parallel C extensions would be worthwhile. This is far beyond the scope of the paper, however.

"Which experts on shared-memory parallel programming are reviewing your paper?" Peer reviews for academic publications are anonymous.

"I correctly quoted my original peer review on Reddit, not this blog post." We are discussing your blog post which contains blatantly wrong statements about our paper. Please amend them.

"Can your library express the cache friendly algorithms that people actually use?" I presume you mean divide-and-conquer algorithms such as the Laplace solver you referenced before. The FFT implementation in the paper is an example of a divide-and-conquer algorithm. In general, Repa can express divide-and-conquer algorithms as long as they are regular.

"FWIW, if you can create a library that makes it easy to construct cache oblivious divide-and-conquer algorithms that will scale well, that would be extremely valuable." See above.

Flying Frog Consultancy Ltd. said...

"We mention, in a half-sentence, what we consider an uncontroversial property of C code, namely that it isn't easy to parallelise". As I have shown, you can parallelize it by adding a single line of code.

"you will be happy to learn that I have spent quite a lot of time teaching students specifically about the cache behaviour". Then why was your interpretation of your Laplace benchmark results completely wrong? Why did you not mention the fact that the scalability you obtained was far worse than that found by others in previous work? Indeed, why did you state exactly the opposite in the last sentence of your abstract when it is blatantly contradicted by your own results?

"The one line in your counter-example is not C". That was not a requirement from the wording of your paper.

"Peer reviews for academic publications are anonymous". Then seek peer review yourself. If nobody at your institute knows this stuff then ask on-line or e-mail a prominent author on the subject.

"We are discussing your blog post which contains blatantly wrong statements about our paper. Please amend them". Please take your own advice.

"The FFT implementation in the paper is an example of a divide-and-conquer algorithm". But the effectiveness of your parallelization of that was not measured because you did the embarrassingly parallel 3D FFT.

augustss said...

@fededevita. It's great that you're using a functional language, and I think people should what they need. I also work in the financial industry and we picked Haskell because it's the only language that suits our needs.

Ivan said...

@Jon you still haven't given any explanation about why it took you so long to install GHC.

From my experience on the Haskell-Cafe mailing list, #haskell IRC channel and helping to tutor a first year class teaching Haskell, the only time that anyone has had any trouble installing Gentoo was when they wanted to install it on a non-standard architecture or operating system (e.g. one student wanted to install it on Haiku Alpha 2, but no-one has bootstrapped GHC for it yet).

So, why don't you put your money where your mouth is and tell us what you did that made it take 2 days to install GHC?

Furthermore, you still haven't said what Power Serious is meant to be a compelling example of.

Finally, it's interesting to note that you mis-spelt the name of the only person who has commented here that agreed with you...

Flying Frog Consultancy Ltd. said...

@Ivan: I already explained here and here and here and several other places that I cannot be bothered to Google for you right now.

"helping to tutor a first year class teaching Haskell". So you're not in industry?

"Furthermore, you still haven't said what Power Serious is meant to be a compelling example of". Haskell, Ivan. Haskell.

Ivan said...

@Jon OK, having a quick read through your email messages indicates to me that you didn't spend 2 days trying to install GHC, you spent 2 days trying to bludgeon your way through using libraries that were possibly incompatible with the version of GHC you were using plus your system wasn't in a consistent state (not having the PATH set properly, etc.).

As for the example, if you meant that as the only example of a Haskell program used in industry, then you should have said so. I'm not sure when you looked though, as there have been frequent messages about tools like Cryptol from Galois that use Haskell in the past few years. That also doesn't count internal usage of Haskell (like what Facebook seems to use).

And no, I am not in industry; I am currently a PhD student. However, I'm not studying language design, functional programming, etc.; I consider myself a mathematician (even though I'm doing my PhD in a computer science department).

However, looking at your web page, you don't seem to be directly involved in industry either: you write books with some consultancy work. So can you really claim authorative knowledge that Haskell isn't widely used in industry?

Oh, and one

Flying Frog Consultancy Ltd. said...

"possibly incompatible with the version of GHC you were using plus your system wasn't in a consistent state". That was from a fresh install so the inconsistencies were the fault of the packages or installer.

"if you meant that as the only example of a Haskell program used in industry". That was not an example of a Haskell program in industry, Ivan. Just a compelling example of Haskell being powerful anywhere. When you're considering gambling on a fringe technology you look for such examples. They do not have to come from industry.

"you don't seem to be directly involved in industry either: you write books with some consultancy work". Industry is the production of goods or services such as our books/software and consultancy. So you cannot really be any more in industry than we are.

"So can you really claim authorative knowledge that Haskell isn't widely used in industry?". You seem to have misunderstood what this post is about. I'm explaining why I chose not to lead my company into using Haskell in industry back in 2007.

benkuin said...

hi
I'm not an haskell fan either, partly for the same reason you describe. But what I wonder is this:

Your blogpost entry refers to a comparison between haskell vs. other prog. langs, but the ending and the discussion below is mainly about haskell vs. f#. Isn't this comparing apples with oranges.

I mean couldn't say the most minus-points here about gnu/linux centric programming language environments?

The problem isn't haskell in the first place. It's the lack of a modern language infrastructure for functional languages (hint hlvm).

Flying Frog Consultancy Ltd. said...

@benkuin: Yes, you are quite right that the factors are not separated. However, I do not believe they are separable. For example, there is no industrial-strength modern infrastructure for functional languages in the open source world precisely because these kinds of languages are not developed in such pragmatic directions (they are vehicles for research).

I should also stress that the technical issues with the Haskell language were offputting but business concerns were higher priority and I saw many of the same problems in Haskell that I had already seen in OCaml.

Simon Marlow's comment about Microsoft actively pushing Haskell in industry is potentially very exciting because they may decide to attack some of these problems. Of course, they already tried Haskell.NET and it was not feasible but perhaps it could be made feasible by building a next-gen CLR around Haskell's ideas (e.g. push purity as far into the core APIs as you can go, so Haskell will not have the same problems interoperating that it does on .NET).

Flying Frog Consultancy Ltd. said...

BTW, there are cross-cutting platforms like the JVM and Mathematica that do run on Linux and offer a commerce-friendly platform for third-party libraries (like ours).

We did try a Mathematica-based product but it was a commercial failure. Perhaps we could have done something with the JVM but I'm not sure what. I investigated Scala at the same time as Haskell and F#, of course, and decided against Scala for similar reasons (very academic mindset and, if there is any interest in targetting industry then it is directed at enterprise Java grunts who have nothing in common with our customers).

Incidentally, I have a lot of respect for Rich Hickey's work on Clojure because he is addressing these problems and trying to build a viable language for industry.

Isaac Gouy said...

@Jon > Then I tried to install GHC 6.12 from source because I didn't realise GHC is infamously difficult to build.

Neither did I, and I haven't found it difficult to build - but I've only built it from source 4 or 5 times, and that was always on Ubuntu.

It probably depends on how soon you notice the hilarious README comment - "NB. you need GHC installed in order to build GHC..."

Paolo Giarrusso said...

You did not find compelling examples for Haskell? I guess
Why functional programming matters
is one of the most common ones (I found it in an introductory course).

Other links on lazy vs strict languages (i.e., mostly, Haskell vs. most other ones) are present in this thread on Lambda the Ultimate.

Paolo Giarrusso said...

Another comment I'd like to make is about Norman Ramsey meta-answer. If you read that in full, he admits that there are just a few users of Haskell. He mentions 2 startups, you admit that Galois is valid and I don't know anything about Peerium.

"Industry rewards getting products out quickly..." seems almost entirely valid. I even know one example of "throwing lots of teams at the problem" - inside Intel, Netburst and Core were developed by different teams, and in the end Netburst was thrown out of the window after producing maybe the worst Intel processors (or at least, helping AMD a lot). But I agree that this particular subpoint is not typical.

Paolo Giarrusso said...

As a final point, it was interesting to compare your discussion to this presentation:
http://donsbot.wordpress.com/2010/07/11/engineering-large-projects-in-a-functional-language/

Judging both things critically, it seems that the Haskell community is aware and trying to address some of your concerns - e.g., the Haskell platform is meant to simplify installing Haskell "with batteries included". But I haven't yet a strong opinion on this kind of issues, I'll have to think more about them. Thanks for your discussion!

Flying Frog Consultancy Ltd. said...

@Paolo: Compelling examples of functional programming are not compelling examples for Haskell. Why would anyone choose Haskell over another functional language? The thread on LtU that you've cited already contains excellent refutations of the alleged benefits of Haskell.

BTW, if you want a compelling example of why you should avoid Haskell look no further than this hilarious thread on Reddit where several Haskell experts tried and failed for days to translate a simple quicksort in F# into Haskell correctly.

The Haskell Platform is certainly a step in the correct direction but it is a very long way from the kind of end user experience you get from a polished industrial release like F# in VS2010. Moreover, Don Stewart actually pretends in that presentation that 20-200kLOC projects are "large" by industrial standards. I'm sure I don't have to tell you just how ridiculous that is.

Ben Lippmeier said...

I've posted the final version of the Repa paper here. In the benchmarking section I've tried to clarify the fact that the algorithms we're using are simple ones primarily intended to help explain the library.

The numbers were taken with the version of Repa in the GHC head. I have yet to update the Hackage package (which will be about 10x slower ATM due to recent inliner changes).

Providing an easy way to write block-based cache efficient algorithms is something we're currently working on.

Another line of work is managing the fact that GHC doesn't do strength reduction on its loops, which makes calculation of array indices fairly expensive. This doesn't matter as much on a super scalar processor when the algorithm is memory bound, but will become more of an issue when we're using more cache efficient algorithms.

AFAIC failure to discuss the limitations in one's own work harms the writer more than the reader. I have no real interest in sitting around feeling special, and if my problems were already solved then I'd be out of a job. So Jon: please continue to point them out, I'm your #1 fan.

Flying Frog Consultancy Ltd. said...

@Ben: Forget about block-based algorithms and strength reduction and focus on divide-and-conquer for cache obliviousness and temporal locality. That is where the orders of magnitude are...

mjg said...

It seems that more valid critics would target lack of competitive implementations beside GHC, and inefficiency of its garbage collection that seems to hardly scale above ~12 processors.
Cloud Haskell is supposed to change that though.


On the other hand, I believe that introducing Haskell98 and then Haskell2010 standards were explicitly meant to provide a stable language platform that seems to work between versions, doesn't it?

Curt Sampson said...

I was responsible for building Tsuru Capital's IT systems from scratch back in 2008, and it was entirely built in Haskell, mostly running on Unix but with some parts of the system written for and running on Windows as well. I described my experience in detail in my 2009 ICFP paper. Tsuru is still using primarily Haskell to this day and they're quite happy with it.

My experience was quite different from yours. I had no problem getting GHC and the libraries I needed up and running, and this was long before the Haskell Platform was available. I started out with pre-built binaries, but we eventually moved to building the compiler from source, and this was as easy as building any compiler has been for me.

I'm familiar with a lot of the people and organizations you talk about, and I find your statements to be highly exaggerated at best.

That said, I would have probably made the same decision as you to go with F# when selling commercial libraries since, for various reasons I won't get in to, there's not a large culture of selling commercial libraries in the Haskell world.

sn said...

@Jon: As a past small-time Haskell programmer now a small-time F# programmer, I've enjoyed reading this blog post. Its been quite educational.

"look no further than this hilarious thread on Reddit where several Haskell experts tried and failed for days to translate a simple quicksort in F# into Haskell correctly"
Are you sure that's the correct link? It takes me to a page discussing large projects in a functional language

John Dudley said...

Acetech Information, a leading software development company of India offers Software Development,Custom Software Development ,Website Design, website development , search engine optimization, ecommerce and website maintenance services for its customers around the globe.