Components & Versioning in .NET Core

This post covers the versions and components in .NET Core 1.x.

Components

.NET Core consists of multiple components that are each versioned
independently and can often be mixed and matched.

  • The Shared Framework contains the APIs and the Virtual Machine and other runtime services needed for running .NET Core applications.
    • The current .NET Core Virtual Machine is called CoreCLR. This executes the .NET bytecode by compiling it JIT and provides various runtime services including a garbage collector. The complete source code for CoreCLR is available at https://github.com/dotnet/coreclr.
    • The .NET Core standard APIs are implemented in CoreFX. This provides implementations of all your favourite APIs such as System.Runtime, System.Theading and so on. The source code for CoreFX is available at https://github.com/dotnet/corefx.
  • The Host is also sometimes called the muxer or driver. This components represents the dotnet command and is responsible for deciding what happens next. The source for this is available at https://github.com/dotnet/core-setup.
  • The SDK consists of the various tools (dotnet subcommands) and their implementations that deal with building code. This includes handling the restoring of dependencies, compiling code, building binaries, producing packages and publishing standalone or framework dependent packages. The SDK itself consists of the CLI, which handles command line operations (at https://github.com/dotnet/cli) and various subprojects that implement the various operations the CLI needs to do.

Versions

Each of the components listed above are versioned separately. You can find out the version of each of those components.

  • For the SDK, you can use the --version option to dotnet to see the version. For example: 
    $ ~/dotnet-1.1.1/dotnet --version
1.0.0-preview2-1-003176
  • For the Host you can run dotnet by itself without any arguments or options to see the version. 
    $ ~/dotnet-1.1.1/dotnet

Microsoft .NET Core Shared Framework Host

Version : 1.1.0
Build : 362e48a95c86b40cd1f2ef3d08741f7fed897956

Usage: dotnet [common-options] [[options] path-to-application]
...
  • For the Shared Framework there no command currently to display the version(s). I use ls /path/to/where/you/installed/dotnet/shared/Microsoft.NETCore.App which relies on internal implementation details. For example: 
    $ ls ~/dotnet-1.1.1/shared/Microsoft.NETCore.App/
1.1.1

Components in .NET Core Installations

Various official and unoffical packages, tarballs, zips and installers for .NET Core (including those available on https://dot.net/core) provide .NET Core in many variants. Two common ones are .NET Core SDK and .NET Core Runtimes. Each .NET Core SDK or .NET Core Runtime distribution contains a number (possibly 0) of hosts, sdk and shared framework components described above.

  • .NET Core Runtime contains
    • 1 version of the Host
    • 1 version of the Shared Framework
  • .NET Core SDK contains
    • 1 or more versions of the Shared Framework (varies depending on the version of the version of the .NET Core SDK)
    • 1 version of the Host
    • 1 version of the SDK

Selecting Versions

It’s possible to have multiple .NET Core SDKs and .NET Core Runtimes available on disk. You can select the versions easily.

To select the version of the SDK to use, use global.json.

To select the version of the shared framework to use, use the .csproj file (or project.json if you are still using that).

OpenJDK 9 for Fedora/EPEL

Looking to try out OpenJDK 9? If you didn’t know, OpenJDK 9 is going to be the reference implemtation for Java 9 (assuming the pattern for Java 7 and Java 8 continues). Pre-release packages for Fedora 23, 24, 25 and Rawhide as well as RHEL 7 are available on copr:

https://copr.fedoraproject.org/coprs/omajid/openjdk9/

Please note that these are not official Fedora (or RHEL) packages. Official Fedora packages will be proposed and added to Fedora closer to OpenJDK 9’s release.

To install on Fedora:

# dnf copr enable omajid/openjdk9
# dnf install java-9-openjdk-devel

To install on RHEL (or CentOS) 7:

$ wget https://copr.fedorainfracloud.org/coprs/omajid/openjdk9/repo/epel-7/omajid-openjdk9-epel-7.repo
$ su -
# cp omajid-openjdk9-epel-7.repo /etc/yum.repos.d/
# yum install java-9-openjdk-devel

Then use as follows:

$ /usr/lib/jvm/java-1.9.0-openjdk/bin/java -version

To make it the default Java version, use update-alternatives.

# update-alternatives --config java
# update-alternatives --config javac

If you are a user, making 9 the default may not be a great idea – OpenJDK 9 is in heavy development and lots of things are changing. There are also some known compatibility issues. On the other hand, if you are a Java developer, this will give you a good idea of how prepared your favourite libraries and tools are for Java 9.

-Wmisleading-indentation in OpenJDK

Inspired by this post by Mark Wielaard I started wondering how OpenJDK 9 would fare in a -Wmisleading-indentation test. So I built OpenJDK 9 using -Wall to find out.

First, though, a big heads up: the OpenJDK 9 build selectively enables and disables warnings and turns on -Werorr. There doesn’t seem to be a straight-forward way of enabling all compiler warnings everywhere. I ended up patching various makefiles to comment out various DISABLED_WARNINGS* flags. Hotspot needed a separate patch to initialize WARNING_FLAGS with -Wall in hotspot/make/linux/makefiles/gcc.make. Then I built OpenJDK 9 using:

mkdir build
cd build
bash ../configure --with-extra-cflags="-Wall -Wextra" --with-extra-cxxflags="-Wall -Wextra -fpermissive" --disable-warnings-as-errors

A grep misleading-indentation build.log was surprising:

jdk9-dev/jdk/src/java.base/share/native/libfdlibm/e_asin.c:103:17: warning: statement is indented as if it were guarded by... [-Wmisleading-indentation]
jdk9-dev/jdk/src/java.base/share/native/libfdlibm/k_rem_pio2.c:201:61: warning: statement is indented as if it were guarded by... [-Wmisleading-indentation]
jdk9-dev/jdk/src/java.desktop/share/native/liblcms/cmscgats.c:613:13: warning: statement is indented as if it were guarded by... [-Wmisleading-indentation]
jdk9-dev/jdk/src/java.desktop/share/native/liblcms/cmscgats.c:690:13: warning: statement is indented as if it were guarded by... [-Wmisleading-indentation]
jdk9-dev/jdk/src/java.desktop/share/native/liblcms/cmsintrp.c:959:29: warning: statement is indented as if it were guarded by... [-Wmisleading-indentation]
jdk9-dev/jdk/src/java.desktop/share/native/liblcms/cmsintrp.c:1023:29: warning: statement is indented as if it were guarded by... [-Wmisleading-indentation]
jdk9-dev/jdk/src/java.desktop/share/native/liblcms/cmsio1.c:717:9: warning: statement is indented as if it were guarded by... [-Wmisleading-indentation]
jdk9-dev/jdk/src/java.desktop/share/native/liblcms/cmsopt.c:557:13: warning: statement is indented as if it were guarded by... [-Wmisleading-indentation]
jdk9-dev/jdk/src/java.desktop/share/native/liblcms/cmsopt.c:1021:29: warning: statement is indented as if it were guarded by... [-Wmisleading-indentation]
jdk9-dev/jdk/src/java.desktop/share/native/liblcms/cmsps2.c:1015:9: warning: statement is indented as if it were guarded by... [-Wmisleading-indentation]
jdk9-dev/jdk/src/java.desktop/share/native/liblcms/cmstypes.c:2398:9: warning: statement is indented as if it were guarded by... [-Wmisleading-indentation]
jdk9-dev/jdk/src/java.desktop/share/native/liblcms/cmstypes.c:2679:9: warning: statement is indented as if it were guarded by... [-Wmisleading-indentation]
jdk9-dev/jdk/src/java.desktop/share/native/liblcms/cmswtpnt.c:109:9: warning: statement is indented as if it were guarded by... [-Wmisleading-indentation]
jdk9-dev/jdk/src/java.desktop/share/native/libfontmanager/layout/SunLayoutEngine.cpp:154:25: warning: statement is indented as if it were guarded by... [-Wmisleading-indentation]
jdk9-dev/jdk/src/java.desktop/share/native/libsplashscreen/libpng/pngread.c:3880:16: warning: statement is indented as if it were guarded by... [-Wmisleading-indentation]
jdk9-dev/jdk/src/java.desktop/share/native/libsplashscreen/libpng/pngread.c:3997:10: warning: statement is indented as if it were guarded by... [-Wmisleading-indentation]

libfdlibm, liblcms and libpng are all external libraries that are bundled with OpenJDK. In the case of Linux distributions, they are often replaced with dynamically linked system libraries. That only code that’s actually part of OpenJDK itself is libfontmanager.

Let’s take a look at that. The warning itself is:

/home/omajid/jdk9-dev/jdk/src/java.desktop/share/native/libfontmanager/layout/SunLayoutEngine.cpp:154:25: warning: statement is indented as if it were guarded by... [-Wmisleading-indentation]
   if (min < 0) min = 0; if (max < min) max = min; /* defensive coding */
                         ^~
/home/omajid/jdk9-dev/jdk/src/java.desktop/share/native/libfontmanager/layout/SunLayoutEngine.cpp:154:3: note: ...this 'if' clause, but it is not
   if (min < 0) min = 0; if (max < min) max = min; /* defensive coding */
   ^~

So gcc thinks the programmer meant to guard the second if statement with the first but failed to do so. But a quick look at the code suggests that these are indeed two independent checks – one checks and fixes min and the other fixes max. The developer just tried to save whitespace and write them in one line. Definitely a false postive.

Anyway, I am impressed with OpenJDK’s code quality: gcc doesn’t find any actual misleading indentation issues in OpenJDK.

IcedTea-Web and OpenJDK

I posted 3 patches today

The first patch is for OpenJDK 9. It creates a hole that allows alternative plugin implementations to plug into (pardon the pun)  OpenJDK and make use of existing plugin-specific classes in OpenJDK.

The second and third patches are for IcedTea-Web. They allow IcedTea-Web to (progressively) build against OpenJDK8 and OpenJDK9.

What does this mean? Soon we might actually be able to build and run IcedTea-Web against OpenJDK. Yes, an Open Source/Free Software plugin running on top of OpenJDK (without any custom patches). Of course, a lot of this is still up in the air. The OpenJDK folks might not accept the patch, though I am hopeful they will – it’s very small and conservative.

Fun times ahead!

OpenJDK6 B29 Released!

On December 5, 2013, the OpenJDK6 b29 source bundle was published at https://java.net/projects/openjdk6/downloads/.

This is a security bugfix release that includes all the security patches since the b28 release. A complete list of changes is available too.

The test results were not ideal.

The TCK failed, with all issues again basically boiling down to limited cryptography support. There are patches available to fix this. As far as I can tell this is not a regression.

The jtreg tests did not pass cleanly either.

A special thanks to Andrew Hughes for backporting all the security fixes to OpenJDK6.

Onwards to B30!

OpenJDK6 B28 Released

On October 4, 2013, the OpenJDK6 b28 source bundle was published at https://java.net/projects/openjdk6/downloads/.

This includes all the security patches since the b27 release as well as a major version bump in hotspot to hs23. A complete list of changes is available too.

The test results were not ideal.

The TCK failed, with all issues basically boiling down to limited cryptography support. There are patches available to fix this. As far as I can tell this is not a regression.

The jtreg tests did not pass cleanly either. Most of the failures were caused by tests trying to access “javaweb.sfbay.sun.com”, which is not accessible (from outside Oracle?). Some issues were caused by tests having a @build tag but no @run tag (perhaps older jtreg versions were more lenient). Two new tests, introduced by the 2013-04-16 security update, java/beans/Introspector/7064279/Test7064279.java and java/beans/Introspector/Test5102804.java both failed.

A special thanks to Andrew Hughes for all his contributions (especially with the hotspot 23 backport) and to Pavel Tisnovsky for testing.

Onwards to B29!

Contributing Patches to OpenJDK

Someone asked me earlier today about how to contribute patches to OpenJDK. OpenJDK is probably not the easiest upstream to contribute patches to – their patch approval system is a little different from most other projects. I have tried to summarize the basic steps here. Let me know in comments if anything is not clear.

The OpenJDK Developer’s Guide covers most of this, but some of it is in too much detail and some of it is outdated. The process I describe here is what I have seen others use and what I have been using myself.

The overall process for submitting patches to OpenJDK is:

  1. Identify what group or project the fix is for
  2. Generate a webrev
  3. Send it to the right mailing list for review
  4. Make changes as requested by upstream and update webrev
  5. A sponsor will push your fix for you

As you can see, it’s not a very complicated process. Most of the delays in patch acceptance have been when the patch is non-trivial or has unexpected issues (even if those issues are known only Oracle-internal and can only be guessed by those of us outside) or someone sufficiently qualified to review the patch is not available.

One thing to note: Oracle requires signing a Contributor License Agreement before they will accept non-trivial patches.

Identify the Group or Poject

The first thing to do if you want to send a fix upstream is to identify what Group or Project it is for. For this purpose, both groups and projects are more-or-less the same (groups are more general). They consist of a collection of people who care about something. For example, some folks care about the javac compiler, some others cares about the build system and others that care about ports to different architectures. There is a list of these groups and projects available on the OpenJDK web page, on the left side.

Simply look through them and try to pick what you think is the best project or group for your fix. Some groups and projects are obvious: if you are fixing something in swing, that’s the Swing group. If you want to fix something in the ARMv8 port, that’s the aarch64 project. If you can’t find the best group, you can fall back to Core Libraries if it’s a patch for the standard java library (that is, a fix for a java.* or javax.* class) or you can fall back to Hotspot if it is a fix for hotspot.

Make a note of the mailing list(s) and repositories used by the group.

Don’t worry if you can’t find the ideal group, project or repository; someone will point you in the right direction in later on, but it may mean (slightly) more work later.

Generate a webrev

Since you are trying to submit a patch upstream, I will assume you already have a fix for it. The first thing you need to do is to get the latest OpenJDK code using the most appropriate repository. And then make your fix. Once you have built and tested your fix, you can generate a patch for review. OpenJDK uses a custom tool called webrev to generate easier-to-reivew patches.

Assuming you are making a fix to hotspot, use it like so:

cd jdk8/hotspot
ksh ../make/scripts/webrev.ksh

This program will look for changes in the current repository and generate a bunch of files. You should see a new
webrev.zip file and a webrev directory. The webrev directory should contain an index.html file that should look roughly like this existing one

Upload the webrev directory somewhere publicly where anyone who wants to review the patch can see it. A public dropbox folder works just fine, as done any personal webspace that you may have. Regular contributors to OpenJDK generally use cr.openjdk.java.net.

Send mail with patch

Now you need to send an email describing the patch and what it’s fixing to the right OpenJDK mailing list. remember to subscribe to the mailing list first, though.

Use a descriptive subject in the email, prefixing it with “RFR” (or Request for Review). Link to the public webrev in the email.

Make changes and update webrev

Hopefully, someone will review the patch quickly (depending on the fix, you may get a response within the hour or it may take a few days).

The reviewer may suggest improvements or additional fixes. They may also request additional cleanup or testing as well.

A Sponsor will push the fix

Once upstream is happy with the patch, a sponsor will ask you for a changeset that they can push or perhaps they will generate one for you. The details of producing a changeset are available in the developer’s guide. Since you don’t have push access, a sponsor will push the fix on your behalf to the appropriate repository. It will probably take a few weeks until that change makes it from group repository to integration repository to the master repository (jdk8/jdk8).

And that’s it! So don’t be afraid to propose patches for OpenJDK 🙂

Complete details about all this process is available, of course, in the Developer’s Guide.

SharedSecrets in OpenJDK

I have always been impressed by the effort OpenJDK folks go to maintain API compatibility. I recently came across a nifty little trick that is used in the OpenJDK codebase to help achieve this goal. Behold the SharedSecrets class! The name of the class is a little misleading since it doesn’t have anything to do with the cryptographic concepts of shared secrets. The javadocs do a (slightly) better job describing its purpose:

/** A repository of "shared secrets", which are a mechanism for
    calling implementation-private methods in another package without
    using reflection. A package-private class implements a public
    interface and provides the ability to call package-private methods
    within that package; the object implementing that interface is
    provided through a third package to which access is restricted.
    This framework avoids the primary disadvantage of using reflection
    for this purpose, namely the loss of compile-time checking. */

In other words, it is used to make private methods public, without using reflection! AND it works under a security manager! At first, I thought this was too good to be true but it turns out that the concept behind it is pretty simple. Here’s how it works. Suppose we have a class with a fixed public api and we wish to add a new method to it that we would like to be able to use internally within the project but not provide as an api:

public class FixedPublicApi {
  private static doFoo() { /* do something */ }
}

We could choose to add the doFoo method to another class (in fact, it will primarily be accessed through another class) and make it public, but let’s assume that the only way to avoid duplicating lots of code is to keep doFoo in FixedPublicApi. First we create an interface that exposes the method we want to be invoked:

public interface FooHandler {
  public void doFoo();
}

Then we modify the SharedSecrets class (or it’s equivalent in our codebase), to set the implementation for FooHandler

public class SharedSecrets {
  private static FooHandler fooHandler;
  public static void setFoo(FooHandler handler) {
    fooHandler = handler;
  }
  public static void foo() {
    fooHandler.doFoo();
  }
}

Finally we modify the FixedPublicApi class to install a handler:

public class FixedPublicApi {
  static {
     SharedSecrets.setFoo(new FooHandler() {
       public void doFoo() {
         FixedPublicApi.doFoo();
       }
     }
  }

  private static doFoo() { /* do something */ }
}

And that’s it! Now we can use SharedSecrets to call the private method:

  SharedSecrets.doFoo();

Hats off to whoever came up with this great technique.

OpenJDK and Awesome

I have been playing around with awesome and ran into some issues with Java programs (apparently others users are running into those problems too). I hacked together a patch against OpenJDK7 that seems to solve all my issues:

diff --git a/src/solaris/classes/sun/awt/X11/XWM.java b/src/solaris/classes/sun/awt/X11/XWM.java
--- a/src/solaris/classes/sun/awt/X11/XWM.java
+++ b/src/solaris/classes/sun/awt/X11/XWM.java
@@ -102,7 +102,8 @@
         METACITY_WM = 11,
         COMPIZ_WM = 12,
         LG3D_WM = 13,
-        CWM_WM = 14;
+        CWM_WM = 14,
+        AWESOME_WM = 15;
     public String toString() {
         switch  (WMID) {
           case NO_WM:
@@ -131,6 +132,8 @@
               return "LookingGlass";
           case CWM_WM:
               return "CWM";
+          case AWESOME_WM:
+              return "awesomewm";
           case UNDETERMINED_WM:
           default:
               return "Undetermined WM";
@@ -573,8 +576,12 @@
 //                            getIntProperty(XToolkit.getDefaultRootWindow(), XAtom.XA_CARDINAL)) == 0);
     }
 
+    static boolean isAwesomeWM() {
+        return isNetWMName("awesome");
+    }
+
     static boolean isNonReparentingWM() {
-        return (XWM.getWMID() == XWM.COMPIZ_WM || XWM.getWMID() == XWM.LG3D_WM || XWM.getWMID() == XWM.CWM_WM);
+        return (XWM.getWMID() == XWM.COMPIZ_WM || XWM.getWMID() == XWM.LG3D_WM || XWM.getWMID() == XWM.CWM_WM || XWM.getWMID() == XWM.AWESOME_WM);
     }
 
     /*
@@ -754,6 +761,8 @@
                 awt_wmgr = CWM_WM;
             } else if (doIsIceWM && isIceWM()) {
                 awt_wmgr = XWM.ICE_WM;
+            } else if (isAwesomeWM()) {
+                awt_wmgr = XWM.AWESOME_WM;
             }
             /*
              * We don't check for legacy WM when we already know that WM


Given the effort it’s taking to push a fix for mutter (the default gnome3 window manager), I wouldnt hold my breath waiting for this code to make it upstream.