[jdom-interest] Performance regressions in JDOM
jdom at tuis.net
Wed Jun 10 03:56:56 PDT 2015
Your results are fascinating, and your feedback is very appreciated. It
is going to take me a little while to digest it all, but, at the same
time, yes, it is most likely that at a minimum I will duplicate your
setup and see how it can be integrated. I strive to produce
high-performing code and value any tools and feedback that can help with
I am delighted that you could use JDOM as a test subject for your tool,
and paper. Congratulations on it, by the way.
As for your results, I wish you would have contacted us before choosing
your individual test cases. I have some concerns about them. For
example, the one you have listed as being a regression (negative
Improvement) is this commit here:
That commit is a major update with a huge impact on many parts of the
code base... it is also from 2001 and is from when the API was changing
Most importantly, that commit did two things of note:
1. it added a Text class to the API which is what likeley does impact
and slow down the SAXBuilder.build performance
2. the performance changes it mentions are for "FilterList" which is not
related to the build process at all. The performance impact would not be
seen in build().
Your tool may have been useful to see the regression in the build, but
the developer(s) are "innocent" (or, at least there's no relevant
evidence) when it comes to a performance-reducing commit being called a
Also of significance with that commit, it's using Java 1.2 or
something.... it is not like we can actually go back and reproduce the
same performance environments that were used at the time. The commit
message may in fact be right....
The first Verifier example 4ad684a (
) you use for the verifier performance is great to see confirmed, but I
am also concerned with that because that commit is just the first of 5
that specifically target performance in the Verifier. As a developer
(actually, as one of the two developers in this case), I would have
preferred to see the results of the wider range of commits. Using the
tool I would have wanted to compare my "current" performance to the
"before" performance, and spanned the 5 commits, instead of just the first.
The remaining example also offers some insight. Unfortunately it is from
so long ago.... from 2000:
Regardless, I believe there is some significant value in here, and using
your tool as a day-to-day aid seems like a good approach.
On 10/06/2015 2:32 AM, Vojtech Horky wrote:
> Hello all.
> TL;DR version of this e-mail is: at our research group we run some
> tests scanning for performance regressions in JDOM. We mostly focused
> on finding out whether assumptions stated in commit messages (such as
> "it should be faster and consume fewer resources") were actually met
> and whether it is possible to capture them in automatically testable way.
> Now, we would like to know whether someone would be interested in the
> results; whether we should polish the tests to be able to move them to
> contrib/ or (better) whether someone would like to push this further.
> If you are interested in this, here is the full version.
> Our aim is to allow developers create performance unit tests, i.e.
> something to test performance of individual methods. For example, if
> you (think that you have) improved performance of a certain method, it
> would be nice if you could test that. Of course, you can run tests
> manually but that is time consuming and rather inconvenient in
> general. So our tool (approach) allows you to capture such assumption
> in a Java annotation and our program - still a prototype, but working
> pretty well - scans these annotations and runs the tests
> automatically, reporting violations. In this sense, it is like a
> performance equivalent of an assertion of a unit test.
> As a simple example, if we want to capture an assumption stating that
> SAXBuilder.build() function is faster in version (commit) 6a49ef6 than
> in 4e27535, we would put in the annotation the following string:
> SAXBuilder#build @ 6a49ef6 < SAXBuilder#build @ 4e27535
> and the tool would handle the rest. Well, more or less.
> Regarding JDOM, we went through its commits and identified almost 50
> of them that we found interesting. Interesting commits were those that
> mentioned that they improved performance or when the commit was a
> "refactoring" one. We measured mostly SAXBuilder.build() and several
> methods from the Verifier class.
> In the end, we found out that we were able to confirm lot of the
> assumptions about performance but there were also cases where the
> assumptions were not met, i.e. the developer thought that the commit
> improved performance while the opposite was true.
> We published our results in a paper  (PDF available as well );
> detailed results are available on-line .
> Right now, the tests themselves are in a separate repository  and
> the setup is rather complicated. However, if someone would find this
> interesting and potentially useful, we would gladly refactor the tests
> to fit the contrib/ structure and prepare a fork to be merged.
> - Vojtech Horky
>  http://dx.doi.org/10.1007/978-3-642-40725-3_12
>  http://d3s.mff.cuni.cz/~horky/papers/epew2013.pdf
>  http://d3s.mff.cuni.cz/software/spl/#jdom-case-study
>  https://sf.net/p/spl-tools/casestudy/ci/master/tree/
> To control your jdom-interest membership:
More information about the jdom-interest