Marius Gundersen

Rotating earth-cube in CSS

2026-01-28T00:00:00Z

I just saw a small neat demo of a rotating earth cube using css and js, and thought to myself "I can do that in only css". So that is what I'll do here.

This article is the written log of how I have done this, so it might be a bit weird and rambly. Here goes.

The first step is to get the 6 images and create an earth div with the six images. Note: The blog I'm writing this wraps <img> in <picture>, so I had to make it work with that. I couldn't be bothered to figure out how to disable the wrapping for this code only.

<div class="earth">
  <picture><img src="./images/top.png"></picture>
  <picture><img src="./images/left.png"></picture>
  <picture><img src="./images/right.png"></picture>
  <picture><img src="./images/front.png"></picture>
  <picture><img src="./images/back.png"></picture>
  <picture><img src="./images/bottom.png"></picture>
</div>

The six images now need to be positioned into a cube. This can be done using the 3d css transform. For that to work we need to put the earth into a div that has some perspective applied, using the perspective: 1000px css rule.

This was done with roughly this css:


.space {
  perspective: 1000px;
  display: grid;
  place-content: center;
  height: 400px;
  background: rgb(1, 5, 25);

  .earth {
    display: grid;
    transform-style: preserve-3d;
    width: 200px;
    height: 200px;
    animation: rotate 20s linear forwards infinite;

    picture {
      grid-area: 1 /1;
      transform-style: preserve-3d;
      
      img {
        position: absolute;
        top: 0;
        left: 0;
        width: 200px;
      }
    }

    [alt="top"]{
      transform: rotateX(-90deg) translateZ(100px);
    }
    [alt="bottom"]{
      transform: rotateX(90deg) translateZ(100px);
    }
    [alt="left"]{
      transform: rotateY(-90deg) translateZ(100px);
    }
    [alt="right"]{
      transform: rotateY(90deg) translateZ(100px);
    }
    [alt="front"]{
      transform: rotateX(0deg) translateZ(100px);
    }
    [alt="back"]{
      transform: rotateY(180deg) translateZ(100px);
    }
  }
}

@keyframes rotate {
  from {
    transform: rotateY(0turn) rotateX(0turn);
  }
  to {
    transform: rotateY(4turn) rotateX(1turn);
  }
}

This css makes a cube and makes the cube rotate slowly around two axis. It's now a rotating cube, but it does not have any light effects added to it.

Lighting

My plan is to use some basic lighting effects for this demo, and therefore I looked into the Phong reflection model. This controls how much light a material reflects from the ambient surrounding and from the direct light. This can be simulated in css using the brightness() filter, which takes a value from 0 to 1, and beyond.

We could set this to a constant value for the six sides, and it would look like a cube with a directional light on it, but it would be pretty static. Look at this demo and see that each side has a different brightness, but the brightness doesn't change.

To solve this we need to imagine there is a strong light source somewhere in space, like a sun. We then need to see if each face points towards or away from the light source. The wikipedia page explains in more detail, but the simple solution is to take the dot product between the normal of the face and the direction to the light and to multiply this by some constants for the light source and the material. The brightness can be simplified to the sum of three components:

--brightness: calc(var(--ambient-light, 0) + var(--diffuse-light, 0) + var(--specular-light, 0));

In the code above I set it so that if they aren't defined they don't contribute. This way we can add them as we go. The first step is to add some ambient light. That is just a constant value

--ambient-light: 0.3;

In order to find the light we need to use vector calculations. That means making some changes to the code we have so far so that we can use vectors for both the rotations and for doing the calculations for the light. So, let's replace the rotation of the earths 6 faces with some other code:

.earth {
  img {
    transform: 
      rotateX(asin(var(--y))) 
      rotateY(calc(atan2(var(--x), var(--z)))) 
      translateZ(100px);
  }
  
  [alt="top"]{
    --x: 0;
    --y: -1;
    --z: 0;
  }
  [alt="bottom"]{
    --x: 0;
    --y: 1;
    --z: 0;
  }
  [alt="left"]{
    --x: -1;
    --y: 0;
    --z: 0;
  }
  [alt="right"]{
    --x: 1;
    --y: 0;
    --z: 0;
  }
  [alt="front"]{
    --x: 0;
    --y: 0;
    --z: 1;
  }
  [alt="back"]{
    --x: 0;
    --y: 0;
    --z: -1;
  }
}

This way we have a normal vector for each face that is also used to transform the face. This is a simplified calculation since we know that each face is orthogonal and points in one of the 6 directions. But we can use this to calculate the diffuse light of each face, for example by assuming it comes in from the top right corner:

.earth {
  --d: 0.5;

  img {
    --diffuse-light: max(0, var(--d) * calc(var(--x) + var(--y) + var(--z)));
  }
}

This is a hacky trick that we now need to improve. The next step is to find the dot product between the normal vector of the face and the vector indicating how the earth has rotated. We can start with a simplified version where the earth rotates around a single axis only, the y-axis. Then we can find the vector of the earth using the trigonometric functions sin() and cos(). The dot-product of the face vector (--x, --y, --z) and the earth vector (--earth-x, --earth-y, --earth-z) says how much the face points towards the initial earth direction. We can assume the sun is over there.

.earth.rotate-y {
  animation: rotate-y 5s linear forwards infinite;
  transform: rotateY(var(--rot-y));
  --earth-x: cos(var(--rot-y));
  --earth-y: 0;
  --earth-z: sin(var(--rot-y));
  
  img {
    --diffuse-light: max(
      0, 
      var(--d) * calc(
        calc(var(--earth-x) * var(--x)) + 
        calc(var(--earth-y) * var(--y)) +
        calc(var(--earth-z) * var(--z))
      )
    );
  }
}

@property --rot-y {
  syntax: "<angle>";
  inherits: true;
  initial-value: 0deg;
}

@keyframes rotate-y {
  from {
    --rot-y: 0turn;
  }
  to {
    --rot-y: 1turn;
  }
}

We need some more complex math to rotate arund two axis. For one, we need another rotation varible, and we need to set up the animation to rotate around both the x and y axis:

.space {
  .earth.rotate-xy {
    animation: rotate-xy 20s linear forwards infinite;
    transform: rotateY(var(--rot-y)) rotateX(var(--rot-x));
    --earth-x: cos(var(--rot-y));
    --earth-y: calc(0 - sin(var(--rot-x)))* sin(var(--rot-y));
    --earth-z: calc(cos(var(--rot-x)) * sin(var(--rot-y)));
    
    img {
      --diffuse-light: max(
        0, 
        var(--d) * calc(
          calc(var(--earth-x) * var(--x)) + 
          calc(var(--earth-y) * var(--y)) + 
          calc(var(--earth-z) * var(--z))
        )
      );
    }
  }
}

@keyframes rotate-xy {
  from {
    --rot-y: 0turn;
    --rot-x: 0turn;
  }
  to {
    --rot-y: 4turn;
    --rot-x: 1turn;
  }
}

This is cool, but the earth looks rather flat, it doesn't have any shinyness to it. For that we need specular light, which looks at how the light from the light source is reflected off the surface towards the viewer. For this demo I'm assuming the light source and the viewer is infinitly far away, so we don't need to worry about the position of the surface, only the rotation. This isn't entirely correct, but it doesn't matter much for this demo.

The code is getting rather complex, now that we need to do some more complex vector calculations to find how the light reflects off the surface and if this reflected light hits the camera or not.


.space.specular-light {
  --s: 1;
  --alpha: 20;
  .earth {
    --view-x: calc(0 - sin(var(--rot-y)));
    --view-y: calc(0 - sin(var(--rot-x)))* cos(var(--rot-y));
    --view-z: calc(0 + cos(var(--rot-x)) * cos(var(--rot-y)));

    img {
      --light-normal-dp: calc(
        calc(var(--earth-x) * var(--x)) + 
        calc(var(--earth-y) * var(--y)) + 
        calc(var(--earth-z) * var(--z))
      );
      --r-x: calc(2 * var(--light-normal-dp) * var(--x) - var(--earth-x));
      --r-y: calc(2 * var(--light-normal-dp) * var(--y) - var(--earth-y));
      --r-z: calc(2 * var(--light-normal-dp) * var(--z) - var(--earth-z));

      --specular-light: calc(var(--s) * pow(
        max(
          0,
          calc(var(--view-x) * var(--r-x)) + 
          calc(var(--view-y) * var(--r-y)) + 
          calc(var(--view-z) * var(--r-z))
        ),
        var(--alpha)
      ));
    }
  }
}

Control

The only thing missing now is a way to control the rotation. We can achieve that using scroll driven animation tied to the x and y scroll axis. Using a very large pseudo element we force the scroller to overflow. The space is positioned using sticky positioning so it doesn't move when we scroll. The animation is now tied to the scroller for both the x and y axis. Weirdly the y axis controls the x rotation and the x axis controls the y rotation.


.scroller {
  overflow: scroll;
  height: 400px;
  position: relative;
  scroll-timeline: --scroller-x y, --scroller-y x;

  &::after {
    content: '';
    display: block;
    width: 300%;
    height: 300%;
  }

  .space {
    position: sticky;
    top: 0;
    left: 0;

    .earth {
      animation: rotate-x 1ms linear, rotate-y 1ms linear;
      animation-timeline: --scroller-x, --scroller-y;
    }
  }
}

OK, that's enough for today. I'll admit that getting this to work wasn't easy, and it was really difficult to debug.

I had to cheat a bit and assume some simple positions and rotations. I put the light source to the right, and it's not moving. The earth only rotates around two angles. If it was allowed to rotate around three axis or if the light was allowed to move the calculation would be a lot more complex.

Another neat thing I wanted to try was to have only the water have specular highlights. This could probably be achieved with more filters stacked on top of each other and using the blue hue of the water to multiply with the specular highlight. Another cool thing would be to create a normal map of the surface and have some shadow effects.

I learned a few tricks though. For example, if you try to get the computed value of a complex css custom property calculation, it will just output the string of the definition. However, if you define it using the @property --variable declaration you can get the calculated value, as long as it's not using percentage. You can use window.getComputedStyle($0).getPropertyValue('--variable') to get the value of the variable.

Another thing I want to mention is that it's really nice to have typed variables, specifically the <angle> type. In other languages you need to make sure you use radians and have to convert to and from when using the sin() and other trigonometric functions. But in css I can declare --rot: 1turn, --rot: 360deg, --rot: 400grad or --rot: 6.2832rad and it would all mean the same thing. This way you can use the unit that best suits the situation, for example using 4turn to specify four complete rotations, but you could use 30deg for smaller angles. I have not used any other language that works like this.

I also wish there was a way to use the transformation methods for other things than just transforming an element. I had to do all the vector calculation myself, but it would be nice to say --vector: transform(rotateY(45deg), vector(1, 0, 0)), for example. Not sure there is any real use for it, except for this demo.

Parsing user-agent strings with CSS

2025-10-07T00:00:00Z

Which browser is this?

Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/142.0.0.0 Safari/537.36 Edg/142.0.0.0

Let's have a look at the clues:

It starts with Mozilla, which makes Firefox
It contains Windows NT 10.0, så probably it runs on Windows 10
It Then contains AppleWebKit, so is this an Apple device?
KHTML was the name of the KDE browser, but it has been discontinued since 2016
Gecko is the rendering engine in Firefox, the one by Mozilla
Chrome is created by Google
Safari is created by Apple
Edg is short for Edge, the browser that replaced Internet Explorer

So, given all these clues, which browser is it, and why? Luckily for us we have an ancient text that can help us determine it. If you haven't read the History of the browser user-agent string from 2008, go read it now.

Things have continued to evolve since 2008, so there is even more complexity in the user-agent string today. There are important parts and ignorable parts, and some parts are more important than others. So how do you take that long string and determine the browser? Using CSS of course.

Using CSS

This is from an actual webapp I developed at work. A list of requests contained information, like the path, the method, the remote IP and the user-agent string. I wanted a small icon for the browser with a hover tooltip telling which browser it was and containing the full user-agent string. So I rendered it to html like this:

<span data-user-agent="Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/142.0.0.0 Safari/537.36 Edg/142.0.0.0"></span>

Now that the user agent string is in the attribute we can create some clever css to target it.

span[data-user-agent]{
  position: relative;

  &::before {
    content: var(--user-agent-icon, '?');
    width: 1em;
    height: 1em;
    font: normal 400 1em/1 'Font Awesome 6 Brands';
  }
}

We select all <span> elements with data-user-agent attributes, and then create a pseudo element before it, where the content is whatever is in the --user-agent-icon variable, or ? if it hasn't been set. We now need to set --user-agent-icon for different browsers. Let's start with Safari, since almost all browsers are based on Safari today.

span[data-user-agent]{
  position: relative;

  &::before {
    content: var(--user-agent-icon, '?');
    width: 1em;
    height: 1em;
    font: normal 400 1em/1 'Font Awesome 6 Brands';
  }

  &[data-user-agent*='safari' i] {
    --user-agent-icon: '\f267';
    --user-agent-name: 'Safari';
  }
}

If the data-user-agent string contains safari, then this is Safari. But the example above also contained safari, but it's not safari.

span[data-user-agent]{
  position: relative;

  &::before {
    content: var(--user-agent-icon, '?');
    width: 1em;
    height: 1em;
    font: normal 400 1em/1 'Font Awesome 6 Brands';
  }

  &[data-user-agent*='safari' i] {
    --user-agent-icon: '\f267';
    --user-agent-name: 'Safari';

    &[data-user-agent*='chrome' i] {
      --user-agent-icon: '\f268';
      --user-agent-name: 'Chrome';
    }
  }
}

Ok, so if it contains safari and chrome, then it must be chrome!. Well, the example contains both safari and chrome, but it's not chrome...

span[data-user-agent]{
  position: relative;

  &::before {
    content: var(--user-agent-icon, '?');
    width: 1em;
    height: 1em;
    font: normal 400 1em/1 'Font Awesome 6 Brands';
  }

  &[data-user-agent*='safari' i] {
    --user-agent-icon: '\f267';
    --user-agent-name: 'Safari';

    &[data-user-agent*='chrome' i] {
      --user-agent-icon: '\f268';
      --user-agent-name: 'Chrome';

      &[data-user-agent*='opr/' i] {
        --user-agent-icon: '\f26a';
        --user-agent-name: 'Opera';
      }

      &[data-user-agent*='edg/' i] {
        --user-agent-icon: '\f282';
        --user-agent-name: 'Edge';
      }
    }
  }
}

I've added two more options now, so if it claims to be safari, chrome and edg, then it's Edge, and if it claims to be safari, chrome and opr, then it's Opera.

span[data-user-agent]{
  position: relative;

  &::before {
    content: var(--user-agent-icon, '?');
    width: 1em;
    height: 1em;
    font: normal 400 1em/1 'Font Awesome 6 Brands';
  }

  &[data-user-agent*='safari' i] {
    --user-agent-icon: '\f267';
    --user-agent-name: 'Safari';

    &[data-user-agent*='chrome' i] {
      --user-agent-icon: '\f268';
      --user-agent-name: 'Chrome';

      &[data-user-agent*='opr/' i] {
        --user-agent-icon: '\f26a';
        --user-agent-name: 'Opera';
      }

      &[data-user-agent*='edg/' i] {
        --user-agent-icon: '\f282';
        --user-agent-name: 'Edge';
      }
    }
  }

  &[data-user-agent*='firefox' i] {
    --user-agent-icon: '\f269';
    --user-agent-name: 'Firefox';
  }
}

One of the only browsers which isn't based on WebKit is Firefox. With this in place we can target the major browsers, and we can keep expanding this to target other browsers. The majority of them, like Samsung browser, is based on Chrome which is based on Safari. They can be added in the same way to the stylesheet.

In the end we can check for a few different browsers, for example:

Your browser is

Today there are even more interesting browsers and bots. Some, like Ladybird and Servo, are not based on any of the existing browser engines. Then there are bots, like Facebook, GoogleBot, ChatGPT etc. You can also detirmine which device you are on from the user-agent string, for example if you are on a Windows machine, an iPhone or maybe a Nintendo Switch.

Anchored definition popovers

2025-09-30T00:00:00Z

Note: As of January 2026 the code on this page no longer work correctly. A change was made in the css spec that reverted an earlier change that enabled pseudo elements to be positioned relative to the anchor. There is a discussion to fix this again.

For a project at work I've had to implement . These should pop up above the word and be easily dismissed, for example by clicking anywhere else on the page. For a while I've used JavaScript for this, but now it's (almost) possible to do it with only html and css. This page will explain how.

Definition popover

Underlined words in text that when clicked create a popover with the definition of that word, like this one

The first step is to make a popover that can be opened, and for that we need a button with popovertarget and a div with the popover attribute.

<p>
  Click <button popovertarget="id-0">here</button> to show a popover
</p>

<div popover id="id-0">
  <h2>Popover</h2>
  <p>This is the popover</p>
</div>

Click to show a popover

Popover

This is the popover

The popover is hidden by default, but is made visible when you click on the button. The button knows which popover to show because of the matching popovertarget and id attributes they both have.

Click on the button in the grey box above to see how it works. This is what the popover and button looks like without any special css; it produces a white box with a thick black border right in the middle of the screen. We need some css to convert this into the kind of popover we want.

Styling the button

Let's first style the button to not look like a button. It will be included in running text, so it needs to look like a link, not a button.

We style all buttons with a popovertarget that are direct children of paragraphs, and with the exact same html as before we get the following.

Click to show a popover

Popover

This is the popover

p > button[popovertarget] {
  all: unset;
  cursor: help;
  text-decoration: underline dashed;
}

Styling the popover

Let's customize the popover a bit by setting a max-width, adding some padding and a border-radius, replacing the thick black border with a box-shadow, and use a friendlier sans-serif font.

Click to show a popover

Popover

This is the popover

The popover looks nicer, but it is still opened at the center of the page. The browser sets a bunch of rules for the popover by default, like the black border that we removed in the previous step. The position of the popover is controlled by the browser using the rules margin: auto; and inset: 0px, which centers it in the screen.

p > button[popovertarget] {
  all: unset;
  cursor: help;
  text-decoration: underline dashed;
}

[popover] {
  max-width: 400px;
  padding: 0.5em;
  border: none;
  box-shadow: 0 0 1em 0 rgba(0, 0, 0, 0.2);
  border-radius: 0.5em;

  h2,
  p {
    margin: 0;
    font-family: sans-serif;
  }
}

By setting the margin to 0 vertically and 1em horizontally and unseting the inset (isn't that a fun combination of words) the popover is now placed in the top right corner.

Click to show a popover

Popover

This is the popover

Now it's time to position the popover relative to the button.

p > button[popovertarget] {
  all: unset;
  cursor: help;
  text-decoration: underline dashed;
}

[popover] {
  margin: 0 1em;
  inset: unset;
  max-width: 400px;
  padding: 0.5em;
  border: none;
  box-shadow: 0 0 1em 0 rgba(0, 0, 0, 0.2);
  border-radius: 0.5em;

  h2,
  p {
    margin: 0;
    font-family: sans-serif;
  }
}

This is the really interesting part, by using the new rules bottom: achor(top) and justify-self: anchor-center we place the popover centered above the button.

There is a subtle but very neat rule for popovertarget that there is an implicit anchor association between the popover and the button. This means we don't have to be explicit about which anchor we want the popover to use, since there is an implicit one already. All we have to do is tell it where to place itself relative to the anchor element. That is very useful for our css, as it can be reused for multiple popovers.

Click to show a popover

On the

Popover

This is the popover, it is very wide

CSS takes care of a bunch of edge cases for us, like when the popup is at the edge of the screen. Clicking on the word edge in the box above opens the popover, but it does not go outside the viewport, even though it is quite wide and at the edge of the screen.

This popover opens in the right place, but it can still look a bit nicer. I want it to look like a speech bubble, with a little downward notch pointing to the button that was clicked. That is also easy to add, using the psudo element ::before.

p > button[popovertarget] {
  all: unset;
  cursor: help;
  text-decoration: underline dashed;
}

[popover] {
  margin: 0 1em;
  inset: unset;
  bottom: anchor(top);
  justify-self: anchor-center;
  max-width: 400px;
  padding: 0.5em;
  border: none;
  box-shadow: 0 0 1em 0 rgba(0, 0, 0, 0.2);
  border-radius: 0.5em;

  h2,
  p {
    margin: 0;
    font-family: sans-serif;
  }
}

The first two lines inside the ::before pseudo-element control the position, just like for the popover. The rest of the rules are there just to style it to look like a downard notch. It needs position: fixed or else it cannot be positioned or have a size. It's made white and given a box-shadow and then rotated 45 degrees so that it looks like a downward arrow. The way it is placed and the shadow is offset makes it blend in with the rest of the popover. I also changed the popover to be 0.5em above the anchor element, to make room for the notch.

Click to show a popover

On the

Popover

This is the popover, it is very wide

One very neat effect is that this is also attached to the top center of the button, so while the popover itself will adjust its position to not go outside the screen, this one will stay stuck centered above the button. This is noticable when opening a popover close to the edge of the screen, where the popover is nudged sideways but the notch stays put, relative to the button. The result is an asymetric speech-bubble effect.

So far so good, but it's a bit jarring that it just appears and disappears, it would be nice with a subtle little animation.

p > button[popovertarget] {
  all: unset;
  cursor: help;
  text-decoration: underline dashed;
}

[popover] {
  margin: 0 1em;
  inset: unset;
  bottom: calc(anchor(top) + 0.5em);
  justify-self: anchor-center;
  max-width: 400px;
  padding: 0.5em;
  border: none;
  box-shadow: 0 0 1em 0 rgba(0, 0, 0, 0.2);
  border-radius: 0.5em;

  &::before {
    bottom: anchor(top);
    justify-self: anchor-center;

    transform: rotate(45deg);
    position: fixed;
    width: 1em;
    height: 1em;
    background-color: white;
    box-shadow: 0.5em 0.5em 1em 0 rgba(0, 0, 0, 0.1);
    content: '';
  }

  h2,
  p {
    margin: 0;
    font-family: sans-serif;
  }
}

Animating it

Let's first try to hide the popup slowly when you click away from it.

Ok, a lot going on here. The first thing I do is to change the bottom rules of both the [popover] and the ::before pseudo-element to add the value of a css custom preperty called --translate. This way we can move the two elements up and down by changing the value of this property. The second argument of var(), the 0em, is the default value if the property isn't set. The property is only set when :not(:popover-open), that is, when the popover is closed. Then it is also made transparent using the opacity: 0 rule.

Click to show a popover

Popover

Now click away, and see how this popover fades away upwards

Then it's time to animate these changes, by setting a bunch of transition rules. transition-property is set to bottom (so that it moves up), opacity (so it fades out) and then display and overlay. The last two are interesting, and are why there is also a transition-behavior: allow-discrete there. This rule makes it possible to transition properties that don't have intermediate values. opacity can easily transition between 0 and 1 but how can display: block transition to display: none? The answer is that it can't, but allow-discrete instead makes it wait for the transition-duration until it changes, therefore the popover is visible until it has completely faded out.

We also want the down arrow to fade out, so it inherits the entire transition property. Strangely it does not need to inherit the opacity, but it does neet to explicitly get the correct bottom property.

The last thing to do now is to make a neat animation when it appears.

p > button[popovertarget] {
  all: unset;
  cursor: help;
  text-decoration: underline dashed;
}

[popover] {
  margin: 0 1em;
  inset: unset;
  bottom: calc(anchor(top) + 0.5em + var(--translate, 0em));
  justify-self: anchor-center;
  max-width: 400px;
  padding: 0.5em;
  border: none;
  box-shadow: 0 0 1em 0 rgba(0, 0, 0, 0.2);
  border-radius: 0.5em;

  transition-property: bottom, opacity, display, overlay;
  transition-duration: .3s;
  transition-timing-function: ease-out;
  transition-behavior: allow-discrete;

  &:not(:popover-open) {
    --translate: 0 1em;
    opacity: 0;
  }

  &::before {
    transition: inherit;

    bottom: calc(anchor(top) + var(--translate, 0em));
    justify-self: anchor-center;

    transform: rotate(45deg);
    position: fixed;
    width: 1em;
    height: 1em;
    background-color: white;
    box-shadow: 0.5em 0.5em 1em 0 rgba(0, 0, 0, 0.1);
    content: '';
  }

  h2,
  p {
    margin: 0;
    font-family: sans-serif;
  }
}

The @starting-style rule makes it possible to animate how an element appears. It defines what the element looks like just before it exists or is visible. Since we have transition rules they make it possible to transition our element into existance.

Click to show a popover

Popover

This is the popover

p > button[popovertarget] {
  all: unset;
  cursor: help;
  text-decoration: underline dashed;
}

[popover] {
  margin: 0 1em;
  inset: unset;
  bottom: calc(anchor(top) + 0.5em + var(--translate, 0em));
  justify-self: anchor-center;
  max-width: 400px;
  padding: 0.5em;
  border: none;
  box-shadow: 0 0 1em 0 rgba(0, 0, 0, 0.2);
  border-radius: 0.5em;

  transition-property: bottom, opacity, display, overlay;
  transition-duration: .3s;
  transition-timing-function: ease-out;
  transition-behavior: allow-discrete;

  &:not(:popover-open) {
    --translate: 0 1em;
    opacity: 0;
  }

  &::before {
    transition: inherit;

    bottom: calc(anchor(top) + var(--translate, 0em));
    justify-self: anchor-center;

    transform: rotate(45deg);
    position: fixed;
    width: 1em;
    height: 1em;
    background-color: white;
    box-shadow: 0.5em 0.5em 1em 0 rgba(0, 0, 0, 0.1);
    content: '';
  }

  h2,
  p {
    margin: 0;
    font-family: sans-serif;
  }

  @starting-style {
    --translate: -1em;
    opacity: 0;

    &::before {
      --translate: -1em
    }
  }
}

Conclusion

This is a pure css popover that sticks to the word you clicked, and it is positioned correctly in the screen. Previously this would have required dozens of lines of js and very special care to make sure it didn't go outside the screen or did something else weird.

It should be noted where this works (or doesn't work). At time of writing the popover api has fairly good support and works in more than 85% of browsers. The anchor positioning is supported only in Chrome (and Chrome-derived) browsers, but it is part of the interop 2025 cross-browser effort to improve on web-standards, so is likely to be functional in all major browsers at the end of 2025. The animation is an enhancement, so even though not everyone supports allow-discrete or @starting-style that's not as important. The browsers that do support them get nice animations, others just get the basic popover experience.

Given that only about 70% of browsers today support these features you should be careful where you use this technique. If your users tend to use old browsers, then maybe wait a few more years. If you design for a backoffice system and know your users then it's easier to start uning new features. You can also use @supports to check if the current browser supports these features, and to use a fallback effect if they don't support it.

Webpack CSS LiveReload

2024-11-01T00:00:00Z

Webpack CSS LiveReload

In this article I will describe how to make a very simple but useful live-reload plugin for Webpack, something I recently needed for a project. There are more complex plugins available, but they rely on the dev server which I couldn't use. Therefore I hacked together a simple standalone live-reload plugin, and now I'm sharing my journey with you.

The goal is for the CSS in a website to automatically update whenever the source stylesheet is edited. To achive this we need to break the problem into smaller parts:

Detect that a CSS file has changed
Notify the website that depends on the CSS file
Replace the existing CSS file with the new version

This should all be implemented as a webpack plugin. The simplest way to create a plugin in webpack, perfect for hacking together a quick test, is as a function.

// webpack.config.js

module.exports = {
  entry: [ './src/index.js', './src/index.css'],
  output: {
    path: __dirname+'/output/assets',
    publicPath: '/assets/'
  },

  // Lots more configuration complicating things here...

  plugins: [
  ]
}

Given a simple webpack configuration file like this, where I have left out most of the interesting things, it's easy to add a new plugin

// webpack.config.js

module.exports = {
  entry: [ './src/index.js', './src/index.css'],
  output: {
    path: __dirname+'/output/assets',
    publicPath: '/assets/'
  },

  // Lots more configuration complicating things here...

  plugins: [
    (compiler) => {
      const plugin_name = 'live-reload';
      // This is now a simple plugin
      console.log("Look at me, I'm a plugin!");
    }
  ]
}

The plugin is a function that gets called with one parameter, the compiler. I've created a variable called plugin_name that we need several times later on.

// webpack.config.js

module.exports = {
  entry: [ './src/index.js', './src/index.css'],
  output: {
    path: __dirname+'/output/assets',
    publicPath: '/assets/'
  },

  // Lots more configuration complicating things here...

  plugins: [
    (compiler) => {
      const plugin_name = 'live-reload';

      compiler.hooks.assetEmitted.tap(plugin_name, (name, info) => {
        console.log('emitted', name);
      });
    }
  ]
}

Detecting changes to assets

Webpack plugins can be notified whenever an asset (any file generated by webpack) is emited (written to the filesystem) by subscribing to the assetEmitted hook, so let's set it up in the plugin.

// webpack.config.js

module.exports = {
  entry: [ './src/index.js', './src/index.css'],
  output: {
    path: __dirname+'/output/assets',
    publicPath: '/assets/'
  },

  // Lots more configuration complicating things here...

  plugins: [
    (compiler) => {
      const plugin_name = 'live-reload';

      const publicPath = compiler\.options\.output\.publicPath ?? '/';

      compiler.hooks.assetEmitted.tap(plugin_name, (name, info) => {
        console.log('emitted', publicPath + name);
      });
    }
  ]
}

The name is relative to the output.path property, so a file like /output/assets/main.css will only have the name main.css. To get the path that the client is interested in, the one that is in <link rel="stylesheet" href="/assets/main.css">, we need to concat the output.publicPath property with the name. If publicPath isn't set then we use / to get the root path.

Note that publicPath can be a function, but I'm not adding support for that here.

Notifying the client

Now that we know if a file has changed we need to notify the client. I'm going to use Server Sent Events to do this, because it's very simple to set up yet robust. This requires a small client side script that can connect to a small webserver and listen for events. The events are sent to the client as newline separated text, something like this:

event: message
data: Something here

event: message
data: Data is always a string

event: message
data: {"json": true, "text": "To send something more complex just serialize it as JSON"}

In the above example you can see that each event is separated by two newline characters, and each event has a name (message in the above example) and some data. There are some more options and technical details, but this is enough for us to implement a live-reload server.

// webpack.config.js

import http from 'http';

function createSseServer(port = 8789) {
  const server = http.createServer((req, res) => {
    res.writeHead(200, {
      'Content-Type': 'text/event-stream',
      'Cache-Control': 'no-cache',
      'Access-Control-Allow-Origin': '*',
      Connection: 'keep-alive',
    });

    req.on('close', () => {
      res.end();
    });

    res.write('event: open\n\n');

    res.write('event: message\n');
    res.write('data: Hello\n\n');
  });

  server.listen(port);
}

module.exports = {
  entry: [ './src/index.js', './src/index.css'],
  output: {
    path: __dirname+'/output/assets',
    publicPath: '/assets/'
  },

  // Lots more configuration complicating things here...

  plugins: [
    (compiler) => {
      const plugin_name = 'live-reload';

      const publicPath = compiler\.options\.output\.publicPath ?? '/';

      compiler.hooks.assetEmitted.tap(plugin_name, (name, info) => {
        console.log('emitted', publicPath + name);
      });
    }
  ]
}

Webpack runs in node.js so we can make a simple node http server that only sends messages. This little server will send two messages and then do nothing. Creating a client that listens to these events is even simpler.

// webpack.config.js

import http from 'http';

function createSseServer(port = 8789) {
  const server = http.createServer((req, res) => {
    res.writeHead(200, {
      'Content-Type': 'text/event-stream',
      'Cache-Control': 'no-cache',
      'Access-Control-Allow-Origin': '*',
      Connection: 'keep-alive',
    });

    req.on('close', () => {
      res.end();
    });

    res.write('event: open\n\n');

    res.write('event: message\n');
    res.write('data: Hello\n\n');
  });

  server.listen(port);
}

module.exports = {
  entry: [ './src/index.js', './src/index.css'],
  output: {
    path: __dirname+'/output/assets',
    publicPath: '/assets/'
  },

  // Lots more configuration complicating things here...

  plugins: [
    (compiler) => {
      const plugin_name = 'live-reload';

      const publicPath = compiler\.options\.output\.publicPath ?? '/';

      compiler.hooks.assetEmitted.tap(plugin_name, (name, info) => {
        console.log('emitted', publicPath + name);
      });
    }
  ]
}

// client.js

const es = new EventSource('http://localhost:8789', { withCredentials: false });

es.addEventListener('open', () => {
  console.info('Connected to live-reload server');
});

es.addEventListener('error', () => {
  console.warn('Disconnected from live-reload server');
});

es.addEventListener('message', ev => {
  console.log(ev.data); // this will log Hello
});

EventSource is part of the web platform, so this is a no dependency client. It connects to the server and then uses an event listener to get notified every time the server sends a message. This client will even reconnect to the server if the connection is lost, without me having to code anything.

// webpack.config.js

import EventEmitter from 'events';
import http from 'http';

function createSseServer(port = 8789) {
  const events = new EventEmitter();

  const server = http.createServer((req, res) => {
    const listener = (data) => res.write(data);

    res.writeHead(200, {
      'Content-Type': 'text/event-stream',
      'Cache-Control': 'no-cache',
      'Access-Control-Allow-Origin': '*',
      Connection: 'keep-alive',
    });

    req.on('close', () => {
      res.end();
      events.off('message', listener);
    });

    res.write('event: open\n\n');

    events.on('message', listener);
  });

  server.listen(port);

  return {
    sendMessage(data){
      events.emit('message', `event: message\ndata: ${JSON.stringify(data)}\n\n`);
    }
  }
}

module.exports = {
  entry: [ './src/index.js', './src/index.css'],
  output: {
    path: __dirname+'/output/assets',
    publicPath: '/assets/'
  },

  // Lots more configuration complicating things here...

  plugins: [
    (compiler) => {
      const plugin_name = 'live-reload';

      const publicPath = compiler\.options\.output\.publicPath ?? '/';

      compiler.hooks.assetEmitted.tap(plugin_name, (name, info) => {
        console.log('emitted', publicPath + name);
      });
    }
  ]
}

// client.js

const es = new EventSource('http://localhost:8789', { withCredentials: false });

es.addEventListener('open', () => {
  console.info('Connected to live-reload server');
});

es.addEventListener('error', () => {
  console.warn('Disconnected from live-reload server');
});

es.addEventListener('message', ev => {
  console.log(ev.data); // this will log Hello
});

I've expanded the server with an EventEmitter and made the function return an object with a sendMessage function. The data parameter is serialized as json and sent to the client. We can now use this in our plugin to send messages to the client.

// webpack.config.js

import EventEmitter from 'events';
import http from 'http';

function createSseServer(port = 8789) {
  const events = new EventEmitter();

  const server = http.createServer((req, res) => {
    const listener = (data) => res.write(data);

    res.writeHead(200, {
      'Content-Type': 'text/event-stream',
      'Cache-Control': 'no-cache',
      'Access-Control-Allow-Origin': '*',
      Connection: 'keep-alive',
    });

    req.on('close', () => {
      res.end();
      events.off('message', listener);
    });

    res.write('event: open\n\n');

    events.on('message', listener);
  });

  server.listen(port);

  return {
    sendMessage(data){
      events.emit('message', `event: message\ndata: ${JSON.stringify(data)}\n\n`);
    }
  }
}

module.exports = {
  entry: [ './src/index.js', './src/index.css'],
  output: {
    path: __dirname+'/output/assets',
    publicPath: '/assets/'
  },

  // Lots more configuration complicating things here...

  plugins: [
    (compiler) => {
      const plugin_name = 'live-reload';

      const sseServer = createSseServer();

      const publicPath = compiler\.options\.output\.publicPath ?? '/';

      compiler.hooks.assetEmitted.tap(plugin_name, (name, info) => {
        sseServer.sendMessage({ asset: publicPath + name });
      });
    }
  ]
}

// client.js

const es = new EventSource('http://localhost:8789', { withCredentials: false });

es.addEventListener('open', () => {
  console.info('Connected to live-reload server');
});

es.addEventListener('error', () => {
  console.warn('Disconnected from live-reload server');
});

es.addEventListener('message', ev => {
  console.log(ev.data); // this will log Hello
});

Instead of logging to the console the plugin now sends a message to all connceted clients.

// webpack.config.js

import EventEmitter from 'events';
import http from 'http';

function createSseServer(port = 8789) {
  const events = new EventEmitter();

  const server = http.createServer((req, res) => {
    const listener = (data) => res.write(data);

    res.writeHead(200, {
      'Content-Type': 'text/event-stream',
      'Cache-Control': 'no-cache',
      'Access-Control-Allow-Origin': '*',
      Connection: 'keep-alive',
    });

    req.on('close', () => {
      res.end();
      events.off('message', listener);
    });

    res.write('event: open\n\n');

    events.on('message', listener);
  });

  server.listen(port);

  return {
    sendMessage(data){
      events.emit('message', `event: message\ndata: ${JSON.stringify(data)}\n\n`);
    }
  }
}

module.exports = {
  entry: [ './src/index.js', './src/index.css'],
  output: {
    path: __dirname+'/output/assets',
    publicPath: '/assets/'
  },

  // Lots more configuration complicating things here...

  plugins: [
    (compiler) => {
      const plugin_name = 'live-reload';

      const sseServer = createSseServer();

      const publicPath = compiler\.options\.output\.publicPath ?? '/';

      compiler.hooks.assetEmitted.tap(plugin_name, (name, info) => {
        sseServer.sendMessage({ asset: publicPath + name });
      });
    }
  ]
}

// client.js

const es = new EventSource('http://localhost:8789', { withCredentials: false });

es.addEventListener('open', () => {
  console.info('Connected to live-reload server');
});

es.addEventListener('error', () => {
  console.warn('Disconnected from live-reload server');
});

es.addEventListener('message', ev => {
  const { asset } = JSON.parse(ev.data);
  // We only care about css files
  if (typeof asset !== 'string') return;
  if (!asset.endsWith('.css')) return;
  const stylesheet = Array.from(document.styleSheets)
    .filter(s => s.href)
    .find(s => new URL(s.href).pathname === asset);
  if (!stylesheet) return;
  // We have found the stylesheet that has changed and needs to be updated
});

Updating the CSS in the client

The client gets a message every time any asset changes, but so far it only console.logs it. We can filter it so it only reacts to css files (maybe this should have been done in the server?).

Next we want to update the stylesheet that has changed. We can look at all the stylesheets in the document using document.stylesheets and compare their href with the asset value. We can then use ownerNode to find the <link rel="stylesheet" href="..."> that imported it. This is slightly limiting in that it cannot replace @import url('./another.css') stylesheets.

// webpack.config.js

import EventEmitter from 'events';
import http from 'http';

function createSseServer(port = 8789) {
  const events = new EventEmitter();

  const server = http.createServer((req, res) => {
    const listener = (data) => res.write(data);

    res.writeHead(200, {
      'Content-Type': 'text/event-stream',
      'Cache-Control': 'no-cache',
      'Access-Control-Allow-Origin': '*',
      Connection: 'keep-alive',
    });

    req.on('close', () => {
      res.end();
      events.off('message', listener);
    });

    res.write('event: open\n\n');

    events.on('message', listener);
  });

  server.listen(port);

  return {
    sendMessage(data){
      events.emit('message', `event: message\ndata: ${JSON.stringify(data)}\n\n`);
    }
  }
}

module.exports = {
  entry: [ './src/index.js', './src/index.css'],
  output: {
    path: __dirname+'/output/assets',
    publicPath: '/assets/'
  },

  // Lots more configuration complicating things here...

  plugins: [
    (compiler) => {
      const plugin_name = 'live-reload';

      const sseServer = createSseServer();

      const publicPath = compiler\.options\.output\.publicPath ?? '/';

      compiler.hooks.assetEmitted.tap(plugin_name, (name, info) => {
        sseServer.sendMessage({ asset: publicPath + name });
      });
    }
  ]
}

// client.js

const es = new EventSource('http://localhost:8789', { withCredentials: false });

es.addEventListener('open', () => {
  console.info('Connected to live-reload server');
});

es.addEventListener('error', () => {
  console.warn('Disconnected from live-reload server');
});

es.addEventListener('message', ev => {
  const { asset } = JSON.parse(ev.data);
  // We only care about css files
  if (typeof asset !== 'string') return;
  if (!asset.endsWith('.css')) return;
  const stylesheet = Array.from(document.styleSheets)
    .filter(s => s.href)
    .find(s => new URL(s.href).pathname === asset);
  if (!stylesheet) return;
  const link = stylesheet.ownerNode;
  const clone = link.cloneNode();
  clone.setAttribute('href', asset + '?' + Date.now());
  link.insertAdjacentElement('afterend', clone);
  clone.addEventListener('load', () => link.remove());
});

We can find the html <link rel="stylesheet" href="..."> element for the given stylesheet by using the ownerNode property. We need to force this element to reload the stylesheet, so we append the current timestamp to the end of the url, which will bypass the cache. But instead of updating the existing link I create a clone and insert it, and I only remove the old element once the new one has loaded. This prevents flash of unstyled content (https://en.wikipedia.org/wiki/Flash_of_unstyled_content).

And that's about it, we have a simple but working live-reload implementation.

// webpack.config.js

import EventEmitter from 'events';
import http from 'http';

function createSseServer(port = 8789) {
  const events = new EventEmitter();

  const server = http.createServer((req, res) => {
    const listener = (data) => res.write(data);

    res.writeHead(200, {
      'Content-Type': 'text/event-stream',
      'Cache-Control': 'no-cache',
      'Access-Control-Allow-Origin': '*',
      Connection: 'keep-alive',
    });

    req.on('close', () => {
      res.end();
      events.off('message', listener);
    });

    res.write('event: open\n\n');

    events.on('message', listener);
  });

  server.listen(port);

  return {
    sendMessage(data){
      events.emit('message', `event: message\ndata: ${JSON.stringify(data)}\n\n`);
    }
  }
}

module.exports = {
  entry: [ './src/index.js', './src/index.css'],
  output: {
    path: __dirname+'/output/assets',
    publicPath: '/assets/'
  },

  // Lots more configuration complicating things here...

  plugins: [
    (compiler) => {
      const plugin_name = 'live-reload';

      if (!compiler.options.watch) return;

      const sseServer = createSseServer();

      const publicPath = compiler\.options\.output\.publicPath ?? '/';

      compiler.hooks.assetEmitted.tap(plugin_name, (name, info) => {
        sseServer.sendMessage({ asset: publicPath + name });
      });
    }
  ]
}

// client.js

const es = new EventSource('http://localhost:8789', { withCredentials: false });

es.addEventListener('open', () => {
  console.info('Connected to live-reload server');
});

es.addEventListener('error', () => {
  console.warn('Disconnected from live-reload server');
});

es.addEventListener('message', ev => {
  const { asset } = JSON.parse(ev.data);
  // We only care about css files
  if (typeof asset !== 'string') return;
  if (!asset.endsWith('.css')) return;
  const stylesheet = Array.from(document.styleSheets)
    .filter(s => s.href)
    .find(s => new URL(s.href).pathname === asset);
  if (!stylesheet) return;
  const link = stylesheet.ownerNode;
  const clone = link.cloneNode();
  clone.setAttribute('href', asset + '?' + Date.now());
  link.insertAdjacentElement('afterend', clone);
  clone.addEventListener('load', () => link.remove());
});

Getting the client code into the bundle

There are a few details I have skipped over so far to keep things simple. For example, we only want this plugin during watch. So let's check if the watch option is enabled, and if it isn't this plugin shouldn't do anything.

But we also want to add the client code to our entry bundle. When developing and testing this plugin I just added the code to my entry, but that means the code is included in the production build as well. A much better solution is for the plugin to add the client code when it is needed. This is quite easy to do, although the webpack documentation doesn't want to admit that.

// webpack.config.js

import EventEmitter from 'events';
import http from 'http';

function createSseServer(port = 8789) {
  const events = new EventEmitter();

  const server = http.createServer((req, res) => {
    const listener = (data) => res.write(data);

    res.writeHead(200, {
      'Content-Type': 'text/event-stream',
      'Cache-Control': 'no-cache',
      'Access-Control-Allow-Origin': '*',
      Connection: 'keep-alive',
    });

    req.on('close', () => {
      res.end();
      events.off('message', listener);
    });

    res.write('event: open\n\n');

    events.on('message', listener);
  });

  server.listen(port);

  return {
    sendMessage(data){
      events.emit('message', `event: message\ndata: ${JSON.stringify(data)}\n\n`);
    }
  }
}

module.exports = {
  entry: [ './src/index.js', './src/index.css'],
  output: {
    path: __dirname+'/output/assets',
    publicPath: '/assets/'
  },

  // Lots more configuration complicating things here...

  plugins: [
    (compiler) => {
      const plugin_name = 'live-reload';

      if (!compiler.options.watch) return;

      compiler.options.entry['main'].import?.push(require.resolve('./client.js'));

      const sseServer = createSseServer();

      const publicPath = compiler\.options\.output\.publicPath ?? '/';

      compiler.hooks.assetEmitted.tap(plugin_name, (name, info) => {
        sseServer.sendMessage({ asset: publicPath + name });
      });
    }
  ]
}

// client.js

const es = new EventSource('http://localhost:8789', { withCredentials: false });

es.addEventListener('open', () => {
  console.info('Connected to live-reload server');
});

es.addEventListener('error', () => {
  console.warn('Disconnected from live-reload server');
});

es.addEventListener('message', ev => {
  const { asset } = JSON.parse(ev.data);
  // We only care about css files
  if (typeof asset !== 'string') return;
  if (!asset.endsWith('.css')) return;
  const stylesheet = Array.from(document.styleSheets)
    .filter(s => s.href)
    .find(s => new URL(s.href).pathname === asset);
  if (!stylesheet) return;
  const link = stylesheet.ownerNode;
  const clone = link.cloneNode();
  clone.setAttribute('href', asset + '?' + Date.now());
  link.insertAdjacentElement('afterend', clone);
  clone.addEventListener('load', () => link.remove());
});

I'm adding the client.js file to the main entry, since in this example I only have one entry. But maybe it should be added to another entry? It would be nice if the user of the plugin could specify themselves which entry it should be added to.

// webpack.config.js

import EventEmitter from 'events';
import http from 'http';

function createSseServer(port = 8789) {
  const events = new EventEmitter();

  const server = http.createServer((req, res) => {
    const listener = (data) => res.write(data);

    res.writeHead(200, {
      'Content-Type': 'text/event-stream',
      'Cache-Control': 'no-cache',
      'Access-Control-Allow-Origin': '*',
      Connection: 'keep-alive',
    });

    req.on('close', () => {
      res.end();
      events.off('message', listener);
    });

    res.write('event: open\n\n');

    events.on('message', listener);
  });

  server.listen(port);

  return {
    sendMessage(data){
      events.emit('message', `event: message\ndata: ${JSON.stringify(data)}\n\n`);
    }
  }
}

class LiveReloadPlugin {
  constructor ({entry = 'main'} = {}){
    this.options = {entry};
  }

  apply(compiler) {
    const plugin_name = 'live-reload';

    if (!compiler.options.watch) return;

    compiler.options.entry[this.options.entry].import?.push(require.resolve('./client.js'));

    const sseServer = createSseServer();

    const publicPath = compiler\.options\.output\.publicPath ?? '/';

    compiler.hooks.assetEmitted.tap(plugin_name, (name, info) => {
      sseServer.sendMessage({ asset: publicPath + name });
    });
  }
}

module.exports = {
  entry: [ './src/index.js', './src/index.css'],
  output: {
    path: __dirname+'/output/assets',
    publicPath: '/assets/'
  },

  // Lots more configuration complicating things here...

  plugins: [
    new LiveReloadPlugin()
  ]
}

// client.js

const es = new EventSource('http://localhost:8789', { withCredentials: false });

es.addEventListener('open', () => {
  console.info('Connected to live-reload server');
});

es.addEventListener('error', () => {
  console.warn('Disconnected from live-reload server');
});

es.addEventListener('message', ev => {
  const { asset } = JSON.parse(ev.data);
  // We only care about css files
  if (typeof asset !== 'string') return;
  if (!asset.endsWith('.css')) return;
  const stylesheet = Array.from(document.styleSheets)
    .filter(s => s.href)
    .find(s => new URL(s.href).pathname === asset);
  if (!stylesheet) return;
  const link = stylesheet.ownerNode;
  const clone = link.cloneNode();
  clone.setAttribute('href', asset + '?' + Date.now());
  link.insertAdjacentElement('afterend', clone);
  clone.addEventListener('load', () => link.remove());
});

I've pulled out the plugin as a class so that an instance of it can be made in the plugin array with some option parameters. This way we can pass in different options depending on what you want to do.

// webpack.config.js

import EventEmitter from 'events';
import http from 'http';

function createSseServer(port = 8789) {
  const events = new EventEmitter();

  const server = http.createServer((req, res) => {
    const listener = (data) => res.write(data);

    res.writeHead(200, {
      'Content-Type': 'text/event-stream',
      'Cache-Control': 'no-cache',
      'Access-Control-Allow-Origin': '*',
      Connection: 'keep-alive',
    });

    req.on('close', () => {
      res.end();
      events.off('message', listener);
    });

    res.write('event: open\n\n');

    events.on('message', listener);
  });

  server.listen(port);

  return {
    sendMessage(data){
      events.emit('message', `event: message\ndata: ${JSON.stringify(data)}\n\n`);
    }
  }
}

class LiveReloadPlugin {
  constructor ({entry = 'main', port = 9876} = {}){
    this.options = {entry, port};
  }

  apply(compiler) {
    const plugin_name = 'live-reload';

    if (!compiler.options.watch) return;

    compiler.options.entry[this.options.entry].import?.push(require.resolve('./client.js'));

    const sseServer = createSseServer(this.options.port);

    const publicPath = compiler\.options\.output\.publicPath ?? '/';

    compiler.hooks.assetEmitted.tap(plugin_name, (name, info) => {
      sseServer.sendMessage({ asset: publicPath + name });
    });
  }
}

module.exports = {
  entry: [ './src/index.js', './src/index.css'],
  output: {
    path: __dirname+'/output/assets',
    publicPath: '/assets/'
  },

  // Lots more configuration complicating things here...

  plugins: [
    new LiveReloadPlugin({
      port: 1234
    })
  ]
}

// client.js

const es = new EventSource('http://localhost:8789', { withCredentials: false });

es.addEventListener('open', () => {
  console.info('Connected to live-reload server');
});

es.addEventListener('error', () => {
  console.warn('Disconnected from live-reload server');
});

es.addEventListener('message', ev => {
  const { asset } = JSON.parse(ev.data);
  // We only care about css files
  if (typeof asset !== 'string') return;
  if (!asset.endsWith('.css')) return;
  const stylesheet = Array.from(document.styleSheets)
    .filter(s => s.href)
    .find(s => new URL(s.href).pathname === asset);
  if (!stylesheet) return;
  const link = stylesheet.ownerNode;
  const clone = link.cloneNode();
  clone.setAttribute('href', asset + '?' + Date.now());
  link.insertAdjacentElement('afterend', clone);
  clone.addEventListener('load', () => link.remove());
});

This way we can supply other options, for example the port we want the SSE server to run on, which is passed into the createSseServer function.

// webpack.config.js

import EventEmitter from 'events';
import http from 'http';

function createSseServer(port = 8789) {
  const events = new EventEmitter();

  const server = http.createServer((req, res) => {
    const listener = (data) => res.write(data);

    res.writeHead(200, {
      'Content-Type': 'text/event-stream',
      'Cache-Control': 'no-cache',
      'Access-Control-Allow-Origin': '*',
      Connection: 'keep-alive',
    });

    req.on('close', () => {
      res.end();
      events.off('message', listener);
    });

    res.write('event: open\n\n');

    events.on('message', listener);
  });

  server.listen(port);

  return {
    sendMessage(data){
      events.emit('message', `event: message\ndata: ${JSON.stringify(data)}\n\n`);
    }
  }
}

class LiveReloadPlugin {
  constructor ({entry = 'main', port = 9876} = {}){
    this.options = {entry, port};
  }

  apply(compiler) {
    const plugin_name = 'live-reload';

    if (!compiler.options.watch) return;

    compiler.options.entry[this.options.entry].import?.push(require.resolve('./client.js'));

    const sseServer = createSseServer(this.options.port);

    new DefinePlugin({
        ___LIVE_RELOAD_PORT___: this.options.port,
    }).apply(compiler);

    const publicPath = compiler\.options\.output\.publicPath ?? '/';

    compiler.hooks.assetEmitted.tap(plugin_name, (name, info) => {
      sseServer.sendMessage({ asset: publicPath + name });
    });
  }
}

module.exports = {
  entry: [ './src/index.js', './src/index.css'],
  output: {
    path: __dirname+'/output/assets',
    publicPath: '/assets/'
  },

  // Lots more configuration complicating things here...

  plugins: [
    new LiveReloadPlugin({
      port: 1234
    })
  ]
}

// client.js

const url = new URL(document.location);
url.port = ___LIVE_RELOAD_PORT___;
const es = new EventSource(url, { withCredentials: false });

es.addEventListener('open', () => {
  console.info('Connected to live-reload server');
});

es.addEventListener('error', () => {
  console.warn('Disconnected from live-reload server');
});

es.addEventListener('message', ev => {
  const { asset } = JSON.parse(ev.data);
  // We only care about css files
  if (typeof asset !== 'string') return;
  if (!asset.endsWith('.css')) return;
  const stylesheet = Array.from(document.styleSheets)
    .filter(s => s.href)
    .find(s => new URL(s.href).pathname === asset);
  if (!stylesheet) return;
  const link = stylesheet.ownerNode;
  const clone = link.cloneNode();
  clone.setAttribute('href', asset + '?' + Date.now());
  link.insertAdjacentElement('afterend', clone);
  clone.addEventListener('load', () => link.remove());
});

We also need to tell the client js code what port to run on. A neat trick with webpack plugins is that one plugin can call another plugi, so we use the DefinePlugin to define a magic variable for the client code to use.

And that's about it! There are improvements that can be done, obviously, but this is the code I'm using in my project, and it is working perfectly well.

I have published this plugin to npm and on github.

Dynamic endpoint routing

2021-09-08T00:00:00Z

I want to share a bit about how the endpoint routing in asp.net can be made dynamic. I tried to look into this, but couldn't find any documentation and so decided that I had to write about it myself. I will use asp.net 6 in this article since I can then use the minimal apis and more easily fit everything into a single file, but the code shown here will work in asp.net 5 and will work with the classical Startup class too.

With a minimal api we have a program.cs file that looks like this:

var builder = WebApplication.CreateBuilder(args);

var app = builder.Build();

app.MapGet("/", () => "Hello World!");

app.Run();

This very simple example adds a single route that responds Hello World! to GET / requests, and we can add more such routes by copying and slightly changing the app.MapGet() line. But we are limited in that we can only statically add routes, we can't change routes at runtime. How would we add, remove or change a route while the application is running, without having to restart it?

The short answer is to use an EndpointDataSource. The slightly longer answer is that everything is already using EndpointDataSource, it's just hidden inside extension methods. The app variable is a class that implements IEndpointRouteBuilder interface which looks like this:

// Licensed to the .NET Foundation under one or more agreements.
// The .NET Foundation licenses this file to you under the MIT license.

using System;
using System.Collections.Generic;
using Microsoft.AspNetCore.Builder;

namespace Microsoft.AspNetCore.Routing
{
    /// <summary>
    /// Defines a contract for a route builder in an application. A route builder specifies the routes for
    /// an application.
    /// </summary>
    public interface IEndpointRouteBuilder
    {
        /// <summary>
        /// Creates a new <see cref="IApplicationBuilder"/>.
        /// </summary>
        /// <returns>The new <see cref="IApplicationBuilder"/>.</returns>
        IApplicationBuilder CreateApplicationBuilder();

        /// <summary>
        /// Gets the sets the <see cref="IServiceProvider"/> used to resolve services for routes.
        /// </summary>
        IServiceProvider ServiceProvider { get; }

        /// <summary>
        /// Gets the endpoint data sources configured in the builder.
        /// </summary>
        ICollection<EndpointDataSource> DataSources { get; }
    }
}

That's the whole interface, all the fun methods like MapGet() and MapControllers() are extension methods that can only work with these three exposed members. There's no list of endpoints here that the MapGet() extension method could add to, so how are new endpoints added? Instead of adding an endpoint directly the extension methods add data sources to the DataSources collection.

The DataSources is a collection of EndpointDataSource objects. This abstract class is defined below, and here you can see the Endpoints member that exposes a list of endpoints.

// Licensed to the .NET Foundation under one or more agreements.
// The .NET Foundation licenses this file to you under the MIT license.

using System.Collections.Generic;
using Microsoft.AspNetCore.Http;
using Microsoft.Extensions.Primitives;

namespace Microsoft.AspNetCore.Routing
{
    /// <summary>
    /// Provides a collection of <see cref="Endpoint"/> instances.
    /// </summary>
    public abstract class EndpointDataSource
    {
        /// <summary>
        /// Gets a <see cref="IChangeToken"/> used to signal invalidation of cached <see cref="Endpoint"/>
        /// instances.
        /// </summary>
        /// <returns>The <see cref="IChangeToken"/>.</returns>
        public abstract IChangeToken GetChangeToken();

        /// <summary>
        /// Returns a read-only collection of <see cref="Endpoint"/> instances.
        /// </summary>
        public abstract IReadOnlyList<Endpoint> Endpoints { get; }
    }
}

So instead of adding endpoints (aka routes) to app directly, we instead specify a list of endpoint sources, each of which specifies a list of endpoints. But as you can see from the abstract class above, there is also a GetChangeToken() method, which sounds interesting. Maybe we can change what endpoints are registered by playing around with this token?

var builder = WebApplication.CreateBuilder(args);

var app = builder.Build();

app.MapGet("/", () => "Hello World!");

app.Run();

Registering an EndpointDataSource

We'll start with the minimal api and then add stuff to it.

var builder = WebApplication.CreateBuilder(args);

var app = builder.Build();

app.MapGet("/", () => "Hello World!");

app.Run();

public class MyEndpointDataSource : EndpointDataSource
{
    // we have no changeToken
    public override IChangeToken GetChangeToken() => default;

    // we don't have any endpoints either
    public override IReadOnlyList<Endpoint> Endpoints { get; } = Array.Empty<Endpoint>();
}

Let's make our own EndpointDataSource. This on doesn't really do anything, we can make it more useful later on. How would we use this class?

var builder = WebApplication.CreateBuilder(args);

var app = builder.Build();

app.DataSources.Add(new MyEndpointDataSource());

app.MapGet("/", () => "Hello World!");

app.Run();

public class MyEndpointDataSource : EndpointDataSource
{
    // we have no changeToken
    public override IChangeToken GetChangeToken() => default;

    // we don't have any endpoints either
    public override IReadOnlyList<Endpoint> Endpoints { get; } = Array.Empty<Endpoint>();
}

This is the simplest way to do it, but that's not very pretty. We can hide this detail inside an extension method, just like Microsoft does.

var builder = WebApplication.CreateBuilder(args);

var app = builder.Build();

app.UseMyEndpoints();

app.MapGet("/", () => "Hello World!");

app.Run();

public static class MyExtensionMethods
{
    public static void UseMyEndpoints(this IEndpointRouteBuilder endpoints)
    {
        endpoints.DataSources.Add(new MyEndpointDataSource());
    }
}

public class MyEndpointDataSource : EndpointDataSource
{
    // we have no changeToken
    public override IChangeToken GetChangeToken() => default;

    // we don't have any endpoints either
    public override IReadOnlyList<Endpoint> Endpoints { get; } = Array.Empty<Endpoint>();
}

Very nice, this looks much more professional. We can now hide details of how MyEndpointDataSource is constructed inside this extension method, and all anyone else ever needs to know is that they should call UseMyEndpoints().

But now we have a new MyEndpointDataSource() in the extension method, can we improve this? What if we want to inject things into this constructor? To do that we need to get the MyEndpointDataSource from the services list:

var builder = WebApplication.CreateBuilder(args);

var app = builder.Build();

app.UseMyEndpoints();

app.MapGet("/", () => "Hello World!");

app.Run();

public static class MyExtensionMethods
{
    public static void UseMyEndpoints(this IEndpointRouteBuilder endpoints)
    {
        var dataSource = endpoints.ServiceProvider.GetService<MyEndpointDataSource>();

        if (dataSource is null)
        {
            throw new Exception("Did you forget to add MyEndpointDataSource to the services?");
        }

        endpoints.DataSources.Add(dataSource);
    }
}

public class MyEndpointDataSource : EndpointDataSource
{
    // we have no changeToken
    public override IChangeToken GetChangeToken() => default;

    // we don't have any endpoints either
    public override IReadOnlyList<Endpoint> Endpoints { get; } = Array.Empty<Endpoint>();
}

The extension method now relies on the service provider and dependency injection to create the MyEndpointDataSource. This means that we can inject whatever we want into its constructor, and we don't have to change anything here. We'll see later how that becomes useful. But, as the exception says, we also have to remember to add MyEndpointDataSource to the list of injectable services.

var builder = WebApplication.CreateBuilder(args);

builder.Services.AddSingleton<MyEndpointDataSource>();

var app = builder.Build();

app.UseMyEndpoints();

app.MapGet("/", () => "Hello World!");

app.Run();

public static class MyExtensionMethods
{
    public static void UseMyEndpoints(this IEndpointRouteBuilder endpoints)
    {
        var dataSource = endpoints.ServiceProvider.GetService<MyEndpointDataSource>();

        if (dataSource is null)
        {
            throw new Exception("Did you forget to add MyEndpointDataSource to the services?");
        }

        endpoints.DataSources.Add(dataSource);
    }
}

public class MyEndpointDataSource : EndpointDataSource
{
    // we have no changeToken
    public override IChangeToken GetChangeToken() => default;

    // we don't have any endpoints either
    public override IReadOnlyList<Endpoint> Endpoints { get; } = Array.Empty<Endpoint>();
}

By adding it as a singleton before calling builder.Build() we can get the service later by explicitly calling GetService or implicitly through constructor injection. But again, this is exposing some details that aren't so nice. We can make another extension method in order to clean up this a bit, making it more like the asp.net routing.

var builder = WebApplication.CreateBuilder(args);

builder.Services.AddMyEndpoints();

var app = builder.Build();

app.UseMyEndpoints();

app.MapGet("/", () => "Hello World!");

app.Run();

public static class MyExtensionMethods
{
    public static void AddMyEndpoints(this IServiceCollection services)
    {
        services.AddSingleton<MyEndpointDataSource>();
    }

    public static void UseMyEndpoints(this IEndpointRouteBuilder endpoints)
    {
        var dataSource = endpoints.ServiceProvider.GetService<MyEndpointDataSource>();

        if (dataSource is null)
        {
            throw new Exception("Did you forget to call Services.AddMyEndpoints()?");
        }

        endpoints.DataSources.Add(dataSource);
    }
}

public class MyEndpointDataSource : EndpointDataSource
{
    // we have no changeToken
    public override IChangeToken GetChangeToken() => default;

    // we don't have any endpoints either
    public override IReadOnlyList<Endpoint> Endpoints { get; } = Array.Empty<Endpoint>();
}

Moving the AddSingleton call into an extension method hides some details from the developer who is going to maintain our code.

And there we have it, we have set up our own endpoint data source, just like the routing that is part of asp.net does it. Now we just need to make it do something useful.

var builder = WebApplication.CreateBuilder(args);

builder.Services.AddMyEndpoints();

var app = builder.Build();

app.UseMyEndpoints();

app.MapGet("/", () => "Hello World!");

app.Run();

public static class MyExtensionMethods
{
    public static void AddMyEndpoints(this IServiceCollection services)
    {
        services.AddSingleton<MyEndpointDataSource>();
    }

    public static void UseMyEndpoints(this IEndpointRouteBuilder endpoints)
    {
        var dataSource = endpoints.ServiceProvider.GetService<MyEndpointDataSource>();

        if (dataSource is null)
        {
            throw new Exception("Did you forget to call Services.AddMyEndpoints()?");
        }

        endpoints.DataSources.Add(dataSource);
    }
}

public class MyEndpointDataSource : EndpointDataSource
{
    // we have no changeToken
    public override IChangeToken GetChangeToken() => default;

    // we don't have any endpoints either
    public override IReadOnlyList<Endpoint> Endpoints { get; } = Array.Empty<Endpoint>();
}

Providing endpoints

The endpoints data source we currently have is quite bare, it provides only an empty list of endpoints. Let's set it up so that it at least provides one endpoint, although it's hardcoded and not terribly interesting.

var builder = WebApplication.CreateBuilder(args);

builder.Services.AddMyEndpoints();

var app = builder.Build();

app.UseMyEndpoints();

app.MapGet("/", () => "Hello World!");

app.Run();

public static class MyExtensionMethods
{
    public static void AddMyEndpoints(this IServiceCollection services)
    {
        services.AddSingleton<MyEndpointDataSource>();
    }

    public static void UseMyEndpoints(this IEndpointRouteBuilder endpoints)
    {
        var dataSource = endpoints.ServiceProvider.GetService<MyEndpointDataSource>();

        if (dataSource is null)
        {
            throw new Exception("Did you forget to call Services.AddMyEndpoints()?");
        }

        endpoints.DataSources.Add(dataSource);
    }
}

public class MyEndpointDataSource : EndpointDataSource
{
    // we have no changeToken
    public override IChangeToken GetChangeToken() => default;

    public override IReadOnlyList<Endpoint> Endpoints { get; } = new []
    {
        CreateEndpoint("/myEndpoint", async context =>
        {
            await context.Response.WriteAsync("Hello World!");
        })
    };

    private static Endpoint CreateEndpoint(string pattern, RequestDelegate requestDelegate) =>
        new RouteEndpointBuilder(
            requestDelegate: requestDelegate,
            routePattern: RoutePatternFactory.Parse(pattern),
            order: 0)
        .Build();
}

The static CreateEndpoint method wraps the RouteEndpointBuilder and the RoutePatternFactory to make an endpoint from a string and a delegate. Making an endpoint is surprisingly difficult, so making your own simplified methods is recommended.

I use this method to create a single endpoint for the Endpoints list, so you can see how it works. The first parameter is the route pattern, which needs to follow the rules for route templates. The second parameter should be a delegate/lambda/method that handles the request.

You can change the code to add more endpoints if you want to. But all those endpoints will be static, that is, they will be defined before the application starts and cannot be changed. How can we add another endpoint while the application is running?

var builder = WebApplication.CreateBuilder(args);

builder.Services.AddMyEndpoints();

var app = builder.Build();

app.UseMyEndpoints();

app.MapGet("/", () => "Hello World!");

app.Run();

public static class MyExtensionMethods
{
    public static void AddMyEndpoints(this IServiceCollection services)
    {
        services.AddSingleton<MyEndpointDataSource>();
    }

    public static void UseMyEndpoints(this IEndpointRouteBuilder endpoints)
    {
        var dataSource = endpoints.ServiceProvider.GetService<MyEndpointDataSource>();

        if (dataSource is null)
        {
            throw new Exception("Did you forget to call Services.AddMyEndpoints()?");
        }

        endpoints.DataSources.Add(dataSource);
    }
}

public class MyEndpointDataSource : EndpointDataSource
{
    // we have no changeToken
    public override IChangeToken GetChangeToken() => default;

    public override IReadOnlyList<Endpoint> Endpoints { get; } = MakeEndpoints("myEndpoint");

    private IReadOnlyList<Endpoint> MakeEndpoints(string route)
        => new []
        {
            // routes have to start with /
            CreateEndpoint($"/{route}", async context =>
            {
                await context.Response.WriteAsync("Hello World!");
            }),
            CreateEndpoint("/setEndpoint/{**route}", async context => {
                Endpoints = MakeEndpoints(context.Request.RouteValues["route"]);
            })
        };

    private static Endpoint CreateEndpoint(string pattern, RequestDelegate requestDelegate) =>
        new RouteEndpointBuilder(
            requestDelegate: requestDelegate,
            routePattern: RoutePatternFactory.Parse(pattern),
            order: 0)
        .Build();
}

This is a bit of a silly example that lets you change the route using another route. Here you can go to /myEndpoint to get the "Hello World!" response, but if you go to /setEndpoint/yourEndpoint, then /myEndpoint will no longer work, and now you have to go to /yourEndpoint. Well, in theory at least. We need to implement the GetChangeToken for the changes to take place.

var builder = WebApplication.CreateBuilder(args);

builder.Services.AddMyEndpoints();

var app = builder.Build();

app.UseMyEndpoints();

app.MapGet("/", () => "Hello World!");

app.Run();

public static class MyExtensionMethods
{
    public static void AddMyEndpoints(this IServiceCollection services)
    {
        services.AddSingleton<MyEndpointDataSource>();
    }

    public static void UseMyEndpoints(this IEndpointRouteBuilder endpoints)
    {
        var dataSource = endpoints.ServiceProvider.GetService<MyEndpointDataSource>();

        if (dataSource is null)
        {
            throw new Exception("Did you forget to call Services.AddMyEndpoints()?");
        }

        endpoints.DataSources.Add(dataSource);
    }
}

public class MyEndpointDataSource : EndpointDataSource
{
    private CancellationTokenSource _cancellationTokenSource = new CancellationTokenSource();

    public override IChangeToken GetChangeToken() => new CancellationChangeToken(_cancellationTokenSource.Token);

    public override IReadOnlyList<Endpoint> Endpoints { get; } = MakeEndpoints("myEndpoint");

    private IReadOnlyList<Endpoint> MakeEndpoints(string route)
        => new []
        {
            // routes have to start with /
            CreateEndpoint($"/{route}", async context =>
            {
                await context.Response.WriteAsync("Hello World!");
            }),
            CreateEndpoint("/setEndpoint/{**route}", async context => {
                var oldCancellationTokenSource = _cancellationTokenSource;

                Endpoints = MakeEndpoints(context.Request.RouteValues["route"]);

                _cancellationTokenSource = new CancellationTokenSource();

                oldCancellationTokenSource?.Cancel();
            })
        };

    private static Endpoint CreateEndpoint(string pattern, RequestDelegate requestDelegate) =>
        new RouteEndpointBuilder(
            requestDelegate: requestDelegate,
            routePattern: RoutePatternFactory.Parse(pattern),
            order: 0)
        .Build();
}

Here we use a CancellationTokenSource to create a CancellationChangeToken (which implements IChangeToken). This is (kind of) the way it's done in the asp.net source code, and I haven't found a simpler or better way to do it, so I've copied it (with some slight changes) here. It's a bit weird to me that we use the concept of cancellation for notifying changes, but if it's good enough for Microsoft it's good enough for us.

The change token is used to notify asp.net that the Endpoints list has changed. If you trigger a change through the change token then asp.net will get the (read only) list of endpoints and update the route table. It seems like it also gets a new change token when it gets the list of endpoints, so the change token can only be triggered once. That is why a new one is made each time the Endpoints is written to.

One thing I haven't copied from them (yet) is the locks, which are needed to make it thread-safe. This class is getting a bit too big, having to deal with both the routes and the change notification code, and making it thread-safe will make it even bigger. What we can do is take all the change tracking code and put it in an abstract base class that MyEndpointDataSource can extend.

var builder = WebApplication.CreateBuilder(args);

builder.Services.AddMyEndpoints();

var app = builder.Build();

app.UseMyEndpoints();

app.MapGet("/", () => "Hello World!");

app.Run();

public static class MyExtensionMethods
{
    public static void AddMyEndpoints(this IServiceCollection services)
    {
        services.AddSingleton<MyEndpointDataSource>();
    }

    public static void UseMyEndpoints(this IEndpointRouteBuilder endpoints)
    {
        var dataSource = endpoints.ServiceProvider.GetService<MyEndpointDataSource>();

        if (dataSource is null)
        {
            throw new Exception("Did you forget to call Services.AddMyEndpoints()?");
        }

        endpoints.DataSources.Add(dataSource);
    }
}

public class MyEndpointDataSource : EndpointDataSource
{
    private CancellationTokenSource _cancellationTokenSource = new CancellationTokenSource();

    public override IChangeToken GetChangeToken() => new CancellationChangeToken(_cancellationTokenSource.Token);

    public override IReadOnlyList<Endpoint> Endpoints { get; } = MakeEndpoints("myEndpoint");

    private IReadOnlyList<Endpoint> MakeEndpoints(string route)
        => new []
        {
            // routes have to start with /
            CreateEndpoint($"/{route}", async context =>
            {
                await context.Response.WriteAsync("Hello World!");
            }),
            CreateEndpoint("/setEndpoint/{**route}", async context => {
                var oldCancellationTokenSource = _cancellationTokenSource;

                Endpoints = MakeEndpoints(context.Request.RouteValues["route"]);

                _cancellationTokenSource = new CancellationTokenSource();

                oldCancellationTokenSource?.Cancel();
            })
        };

    private static Endpoint CreateEndpoint(string pattern, RequestDelegate requestDelegate) =>
        new RouteEndpointBuilder(
            requestDelegate: requestDelegate,
            routePattern: RoutePatternFactory.Parse(pattern),
            order: 0)
        .Build();
}

public abstract class MutableEndpointDataSource : EndpointDataSource
{
    private readonly object _lock = new object();

    private IReadOnlyList<Endpoint> _endpoints;

    private CancellationTokenSource _cancellationTokenSource;

    private IChangeToken _changeToken;

    public MutableEndpointDataSource() : this(Array.Empty<Endpoint>()) { }

    public MutableEndpointDataSource(IReadOnlyList<Endpoint> endpoints)
    {
        SetEndpoints(endpoints);
    }

    public override IChangeToken GetChangeToken() => _changeToken;

    public override IReadOnlyList<Endpoint> Endpoints => _endpoints;

    public void SetEndpoints(IReadOnlyList<Endpoint> endpoints)
    {
        lock (_lock)
        {
            var oldCancellationTokenSource = _cancellationTokenSource;

            _endpoints = endpoints;

            _cancellationTokenSource = new CancellationTokenSource();
            _changeToken = new CancellationChangeToken(_cancellationTokenSource.Token);

            oldCancellationTokenSource?.Cancel();
        }
    }
}

Ok, that's a lot of code for the MutableEndpointDataSource, but it's going to make MyEndpointDataSource much simpler. This abstract class keeps track of endpoints and uses the CancellationTokenSource trick for notifying whenever the list of endpoints change. The only way to change that list is to call SetEndpoints with a new list of endpoints. That method is wrapped in a lock, so that only one thread can make a change to the list at a time. If multiple threads try to call SetEndpoints at the same time they will have to wait and run one at a time.

How would we use this abstract class?

var builder = WebApplication.CreateBuilder(args);

builder.Services.AddMyEndpoints();

var app = builder.Build();

app.UseMyEndpoints();

app.MapGet("/", () => "Hello World!");

app.Run();

public static class MyExtensionMethods
{
    public static void AddMyEndpoints(this IServiceCollection services)
    {
        services.AddSingleton<MyEndpointDataSource>();
    }

    public static void UseMyEndpoints(this IEndpointRouteBuilder endpoints)
    {
        var dataSource = endpoints.ServiceProvider.GetService<MyEndpointDataSource>();

        if (dataSource is null)
        {
            throw new Exception("Did you forget to call Services.AddMyEndpoints()?");
        }

        endpoints.DataSources.Add(dataSource);
    }
}

public class MyEndpointDataSource : MutableEndpointDataSource
{
    public MyEndpointDataSource()
    {
        SetEndpoints(MakeEndpoints("myEndpoint"));
    }

    private IReadOnlyList<Endpoint> MakeEndpoints(string route)
        => new[]
        {
            // routes have to start with /
            CreateEndpoint($"/{route}", async context =>
            {
                await context.Response.WriteAsync("Hello World!");
            }),
            CreateEndpoint("/setEndpoint/{**route}", async context => {
                SetEndpoints(MakeEndpoints(context.Request.RouteValues["route"].ToString()));
            })
        };

    private static Endpoint CreateEndpoint(string pattern, RequestDelegate requestDelegate) =>
        new RouteEndpointBuilder(
            requestDelegate: requestDelegate,
            routePattern: RoutePatternFactory.Parse(pattern),
            order: 0)
        .Build();
}

public abstract class MutableEndpointDataSource : EndpointDataSource
{
    private readonly object _lock = new object();

    private IReadOnlyList<Endpoint> _endpoints;

    private CancellationTokenSource _cancellationTokenSource;

    private IChangeToken _changeToken;

    public MutableEndpointDataSource() : this(Array.Empty<Endpoint>()) { }

    public MutableEndpointDataSource(IReadOnlyList<Endpoint> endpoints)
    {
        SetEndpoints(endpoints);
    }

    public override IChangeToken GetChangeToken() => _changeToken;

    public override IReadOnlyList<Endpoint> Endpoints => _endpoints;

    public void SetEndpoints(IReadOnlyList<Endpoint> endpoints)
    {
        lock (_lock)
        {
            var oldCancellationTokenSource = _cancellationTokenSource;

            _endpoints = endpoints;

            _cancellationTokenSource = new CancellationTokenSource();
            _changeToken = new CancellationChangeToken(_cancellationTokenSource.Token);

            oldCancellationTokenSource?.Cancel();
        }
    }
}

MyEndpointDataSource is now much simpler, it now focuses purely on how to create the endpoints we are interested in. It can call SetEndpoints with the new list of endpoints and let the abstract base class deal with the issue of notifications and stuff.

And there we have it, probably the simplest way to implement dynamic route updates in asp.net.

var builder = WebApplication.CreateBuilder(args);

builder.Services.AddMyEndpoints();

var app = builder.Build();

app.UseMyEndpoints();

app.MapGet("/", () => "Hello World!");

app.Run();

public static class MyExtensionMethods
{
    public static void AddMyEndpoints(this IServiceCollection services)
    {
        services.AddSingleton<MyEndpointDataSource>();
    }

    public static void UseMyEndpoints(this IEndpointRouteBuilder endpoints)
    {
        var dataSource = endpoints.ServiceProvider.GetService<MyEndpointDataSource>();

        if (dataSource is null)
        {
            throw new Exception("Did you forget to call Services.AddMyEndpoints()?");
        }

        endpoints.DataSources.Add(dataSource);
    }
}

public class MyEndpointDataSource : MutableEndpointDataSource
{
    public MyEndpointDataSource()
    {
        SetEndpoints(MakeEndpoints("myEndpoint"));
    }

    private IReadOnlyList<Endpoint> MakeEndpoints(string route)
        => new[]
        {
            // routes have to start with /
            CreateEndpoint($"/{route}", async context =>
            {
                await context.Response.WriteAsync("Hello World!");
            }),
            CreateEndpoint("/setEndpoint/{**route}", async context => {
                SetEndpoints(MakeEndpoints(context.Request.RouteValues["route"].ToString()));
            })
        };

    private static Endpoint CreateEndpoint(string pattern, RequestDelegate requestDelegate) =>
        new RouteEndpointBuilder(
            requestDelegate: requestDelegate,
            routePattern: RoutePatternFactory.Parse(pattern),
            order: 0)
        .Build();
}

public abstract class MutableEndpointDataSource : EndpointDataSource
{
    private readonly object _lock = new object();

    private IReadOnlyList<Endpoint> _endpoints;

    private CancellationTokenSource _cancellationTokenSource;

    private IChangeToken _changeToken;

    public MutableEndpointDataSource() : this(Array.Empty<Endpoint>()) { }

    public MutableEndpointDataSource(IReadOnlyList<Endpoint> endpoints)
    {
        SetEndpoints(endpoints);
    }

    public override IChangeToken GetChangeToken() => _changeToken;

    public override IReadOnlyList<Endpoint> Endpoints => _endpoints;

    public void SetEndpoints(IReadOnlyList<Endpoint> endpoints)
    {
        lock (_lock)
        {
            var oldCancellationTokenSource = _cancellationTokenSource;

            _endpoints = endpoints;

            _cancellationTokenSource = new CancellationTokenSource();
            _changeToken = new CancellationChangeToken(_cancellationTokenSource.Token);

            oldCancellationTokenSource?.Cancel();
        }
    }
}

A more realistic usecase

When I needed to implement dynamic endpoint routing and went hunting for some documentation, without finding any, it wasn't for the use case in the code above. What I needed to support was reading a routing table from the appsettings.json file and reacting to that file changing while the program was running. I'll quickly walk you through how I did that, based on the code we have already written.

Asp.net has support for listening to changes to the settings files and updating the settings while the program is running, so that part was straight forward to do.

var builder = WebApplication.CreateBuilder(args);

builder.Services.AddMyEndpoints(builder.Configuration);

var app = builder.Build();

app.UseMyEndpoints();

app.MapGet("/", () => "Hello World!");

app.Run();

public record MyConfig
{
    public IReadOnlyDictionary<string, string> Routes { get; init; }
}

public static class MyExtensionMethods
{
    public static void AddMyEndpoints(this IServiceCollection services, IConfiguration configuration)
    {
        services.Configure<MyConfig>(Configuration);
        services.AddSingleton<MyEndpointDataSource>();
    }

    public static void UseMyEndpoints(this IEndpointRouteBuilder endpoints)
    {
        var dataSource = endpoints.ServiceProvider.GetService<MyEndpointDataSource>();

        if (dataSource is null)
        {
            throw new Exception("Did you forget to call Services.AddMyEndpoints()?");
        }

        endpoints.DataSources.Add(dataSource);
    }
}

public class MyEndpointDataSource : MutableEndpointDataSource
{
    public MyEndpointDataSource()
    {
        SetEndpoints(MakeEndpoints("myEndpoint"));
    }

    private IReadOnlyList<Endpoint> MakeEndpoints(string route)
        => new[]
        {
            // routes have to start with /
            CreateEndpoint($"/{route}", async context =>
            {
                await context.Response.WriteAsync("Hello World!");
            }),
            CreateEndpoint("/setEndpoint/{**route}", async context => {
                SetEndpoints(MakeEndpoints(context.Request.RouteValues["route"].ToString()));
            })
        };

    private static Endpoint CreateEndpoint(string pattern, RequestDelegate requestDelegate) =>
        new RouteEndpointBuilder(
            requestDelegate: requestDelegate,
            routePattern: RoutePatternFactory.Parse(pattern),
            order: 0)
        .Build();
}

public abstract class MutableEndpointDataSource : EndpointDataSource
{
    private readonly object _lock = new object();

    private IReadOnlyList<Endpoint> _endpoints;

    private CancellationTokenSource _cancellationTokenSource;

    private IChangeToken _changeToken;

    public MutableEndpointDataSource() : this(Array.Empty<Endpoint>()) { }

    public MutableEndpointDataSource(IReadOnlyList<Endpoint> endpoints)
    {
        SetEndpoints(endpoints);
    }

    public override IChangeToken GetChangeToken() => _changeToken;

    public override IReadOnlyList<Endpoint> Endpoints => _endpoints;

    public void SetEndpoints(IReadOnlyList<Endpoint> endpoints)
    {
        lock (_lock)
        {
            var oldCancellationTokenSource = _cancellationTokenSource;

            _endpoints = endpoints;

            _cancellationTokenSource = new CancellationTokenSource();
            _changeToken = new CancellationChangeToken(_cancellationTokenSource.Token);

            oldCancellationTokenSource?.Cancel();
        }
    }
}

This code assumes our appsettings.json looks something like this:

{
  "Logging": {
      // the usual stuff here
  },
  "Routes": {
      "/home": "Hello from home",
      "/about": "This is the about page"
      "/some/other/page": "This page has many segments"
  }
}

I've created a record called MyConfig that contains a dictionary of routes. I would like to bind this to the appsettings.json file, and that's what the Configure<MyConfig>(configuration) call does. I need to pass the configuration into the extension method here, so a small change is needed right at the top of the code too.

The Configure<MyConfig>(confiuration) line also registers IOptionsMonitor<MyConfig> in the services, so we can inject it into MyEndpointDataSource.

var builder = WebApplication.CreateBuilder(args);

builder.Services.AddMyEndpoints(builder.Configuration);

var app = builder.Build();

app.UseMyEndpoints();

app.MapGet("/", () => "Hello World!");

app.Run();

public record MyConfig
{
    public IReadOnlyDictionary<string, string> Routes { get; init; }
}

public static class MyExtensionMethods
{
    public static void AddMyEndpoints(this IServiceCollection services, IConfiguration configuration)
    {
        services.Configure<MyConfig>(Configuration);
        services.AddSingleton<MyEndpointDataSource>();
    }

    public static void UseMyEndpoints(this IEndpointRouteBuilder endpoints)
    {
        var dataSource = endpoints.ServiceProvider.GetService<MyEndpointDataSource>();

        if (dataSource is null)
        {
            throw new Exception("Did you forget to call Services.AddMyEndpoints()?");
        }

        endpoints.DataSources.Add(dataSource);
    }
}

public class MyEndpointDataSource : MutableEndpointDataSource
{
    public MyEndpointDataSource(IOptionsMonitor<MyConfig> options)
    {
        SetEndpoints(MakeEndpoints(options.CurrentValue));
    }

    private IReadOnlyList<Endpoint> MakeEndpoints(MyConfig config)
        => config.Routes
            .Select(route => CreateEndpoint(
                route.Key,
                context => context.Response.WriteAsync(route.Value)))
            .ToList();

    private static Endpoint CreateEndpoint(string pattern, RequestDelegate requestDelegate) =>
        new RouteEndpointBuilder(
            requestDelegate: requestDelegate,
            routePattern: RoutePatternFactory.Parse(pattern),
            order: 0)
        .Build();
}

public abstract class MutableEndpointDataSource : EndpointDataSource
{
    private readonly object _lock = new object();

    private IReadOnlyList<Endpoint> _endpoints;

    private CancellationTokenSource _cancellationTokenSource;

    private IChangeToken _changeToken;

    public MutableEndpointDataSource() : this(Array.Empty<Endpoint>()) { }

    public MutableEndpointDataSource(IReadOnlyList<Endpoint> endpoints)
    {
        SetEndpoints(endpoints);
    }

    public override IChangeToken GetChangeToken() => _changeToken;

    public override IReadOnlyList<Endpoint> Endpoints => _endpoints;

    public void SetEndpoints(IReadOnlyList<Endpoint> endpoints)
    {
        lock (_lock)
        {
            var oldCancellationTokenSource = _cancellationTokenSource;

            _endpoints = endpoints;

            _cancellationTokenSource = new CancellationTokenSource();
            _changeToken = new CancellationChangeToken(_cancellationTokenSource.Token);

            oldCancellationTokenSource?.Cancel();
        }
    }
}

This simplified example takes the dictionary and maps it so that the key is a route and when you go to it you get the value as a response. For example, you can now request /home and the application will respond with Hello from home.

There is just one small line of code needed to make it react to appsettings.json changes.

var builder = WebApplication.CreateBuilder(args);

builder.Services.AddMyEndpoints(builder.Configuration);

var app = builder.Build();

app.UseMyEndpoints();

app.MapGet("/", () => "Hello World!");

app.Run();

public record MyConfig
{
    public IReadOnlyDictionary<string, string> Routes { get; init; }
}

public static class MyExtensionMethods
{
    public static void AddMyEndpoints(this IServiceCollection services, IConfiguration configuration)
    {
        services.Configure<MyConfig>(Configuration);
        services.AddSingleton<MyEndpointDataSource>();
    }

    public static void UseMyEndpoints(this IEndpointRouteBuilder endpoints)
    {
        var dataSource = endpoints.ServiceProvider.GetService<MyEndpointDataSource>();

        if (dataSource is null)
        {
            throw new Exception("Did you forget to call Services.AddMyEndpoints()?");
        }

        endpoints.DataSources.Add(dataSource);
    }
}

public class MyEndpointDataSource : MutableEndpointDataSource
{
    public MyEndpointDataSource(IOptionsMonitor<MyConfig> options)
    {
        SetEndpoints(MakeEndpoints(options.CurrentValue));
        options.OnChange(config => SetEndpoints(MakeEndpoints(config)));
    }

    private IReadOnlyList<Endpoint> MakeEndpoints(MyConfig config)
        => config.Routes
            .Select(route => CreateEndpoint(
                route.Key,
                context => context.Response.WriteAsync(route.Value)))
            .ToList();

    private static Endpoint CreateEndpoint(string pattern, RequestDelegate requestDelegate) =>
        new RouteEndpointBuilder(
            requestDelegate: requestDelegate,
            routePattern: RoutePatternFactory.Parse(pattern),
            order: 0)
        .Build();
}

public abstract class MutableEndpointDataSource : EndpointDataSource
{
    private readonly object _lock = new object();

    private IReadOnlyList<Endpoint> _endpoints;

    private CancellationTokenSource _cancellationTokenSource;

    private IChangeToken _changeToken;

    public MutableEndpointDataSource() : this(Array.Empty<Endpoint>()) { }

    public MutableEndpointDataSource(IReadOnlyList<Endpoint> endpoints)
    {
        SetEndpoints(endpoints);
    }

    public override IChangeToken GetChangeToken() => _changeToken;

    public override IReadOnlyList<Endpoint> Endpoints => _endpoints;

    public void SetEndpoints(IReadOnlyList<Endpoint> endpoints)
    {
        lock (_lock)
        {
            var oldCancellationTokenSource = _cancellationTokenSource;

            _endpoints = endpoints;

            _cancellationTokenSource = new CancellationTokenSource();
            _changeToken = new CancellationChangeToken(_cancellationTokenSource.Token);

            oldCancellationTokenSource?.Cancel();
        }
    }
}

Just one little line of code and now the application reacts to changes to the appsettings.json file and updates the routing without having to restart.

The full code can be found below. It should be possible to paste this into a brand new net6.0 project and run it.

using Microsoft.AspNetCore.Routing.Patterns;
using Microsoft.Extensions.Options;
using Microsoft.Extensions.Primitives;

var builder = WebApplication.CreateBuilder(args);

builder.Services.AddMyEndpoints(builder.Configuration);

var app = builder.Build();

app.UseMyEndpoints();

app.MapGet("/", () => "Hello World!");

app.Run();

public record MyConfig
{
    public IReadOnlyDictionary<string, string> Routes { get; init; }
}

public static class MyExtensionMethods
{
    public static void AddMyEndpoints(this IServiceCollection services, IConfiguration configuration)
    {
        services.Configure<MyConfig>(Configuration);
        services.AddSingleton<MyEndpointDataSource>();
    }

    public static void UseMyEndpoints(this IEndpointRouteBuilder endpoints)
    {
        var dataSource = endpoints.ServiceProvider.GetService<MyEndpointDataSource>();

        if (dataSource is null)
        {
            throw new Exception("Did you forget to call Services.AddMyEndpoints()?");
        }

        endpoints.DataSources.Add(dataSource);
    }
}

public class MyEndpointDataSource : MutableEndpointDataSource
{
    public MyEndpointDataSource(IOptionsMonitor<MyConfig> options)
    {
        SetEndpoints(MakeEndpoints(options.CurrentValue));
        options.OnChange(config => SetEndpoints(MakeEndpoints(config)));
    }

    private IReadOnlyList<Endpoint> MakeEndpoints(MyConfig config)
        => config.Routes
            .Select(route => CreateEndpoint(
                route.Key,
                context => context.Response.WriteAsync(route.Value)))
            .ToList();

    private static Endpoint CreateEndpoint(string pattern, RequestDelegate requestDelegate) =>
        new RouteEndpointBuilder(
            requestDelegate: requestDelegate,
            routePattern: RoutePatternFactory.Parse(pattern),
            order: 0)
        .Build();
}

public abstract class MutableEndpointDataSource : EndpointDataSource
{
    private readonly object _lock = new object();

    private IReadOnlyList<Endpoint> _endpoints;

    private CancellationTokenSource _cancellationTokenSource;

    private IChangeToken _changeToken;

    public MutableEndpointDataSource() : this(Array.Empty<Endpoint>()) { }

    public MutableEndpointDataSource(IReadOnlyList<Endpoint> endpoints)
    {
        SetEndpoints(endpoints);
    }

    public override IChangeToken GetChangeToken() => _changeToken;

    public override IReadOnlyList<Endpoint> Endpoints => _endpoints;

    public void SetEndpoints(IReadOnlyList<Endpoint> endpoints)
    {
        lock (_lock)
        {
            var oldCancellationTokenSource = _cancellationTokenSource;

            _endpoints = endpoints;

            _cancellationTokenSource = new CancellationTokenSource();
            _changeToken = new CancellationChangeToken(_cancellationTokenSource.Token);

            oldCancellationTokenSource?.Cancel();
        }
    }
}

I want to share one extra little thing that I had trouble finding. The code below shows how to make an endpoint that reacts to only POST, unlike the endpoints we have made so far that react to any HTTP method.


    private static Endpoint CreatePostEndpoint(string pattern, RequestDelegate requestDelegate) =>
        new RouteEndpointBuilder(
            requestDelegate: requestDelegate,
            routePattern: RoutePatternFactory.Parse(pattern),
            order: 0)
            {
                Metadata =
                {
                    new HttpMethodMetadata(new []{ HttpMethods.Post })
                }
            }
            .Build();

Template Literal Types

2021-01-06T00:00:00Z

With the release of TypeScript 4.1 we get a really powerful feature called template literal types. This is based on the template literals that are in JavaScript, but now for the type system in TypeScript. Together with conditional types it makes it possible to parse strings and represent them as types at compile time. An early example that I saw was of ts-sql, which is an sql database implemented purely in TypeScript types. This might seem like a silly example, and while clever not really useful. But I found a place where I needed this kind of functionality, and set about trying to ~~understand~~ implement template literal types.

WebGL and shaders

The scenario I wanted to explore is that of WebGL shaders, specifically the uniforms in the shaders. If you don't have much experience with WebGL (or GL shaders in general), then what you need to know is that ~~working with WebGL is a horrible experience~~ there are many small libraries that make it easier to use WebGL. A shader is a small program written in a language very similar to C which is compiled by the browser and run on the graphics card. This small C program takes input, and the output of the C program is what you see on the screen. Here is a very simple example of what it could look like:

const vertexShader = `
  precision highp float;

  // here is an attribute
  attribute vec3 position;

  varying vec2 uv;

  void main() {
    gl_Position = vec4(position, 1.0);
    uv = position.xy;
  }
`;

const fragmentShader = `
  precision highp float;

  // and here is a uniform
  uniform float time;

  varying vec2 uv;

  void main() {
    gl_FragColor = vec4(0.5*(uv+1.0), 0.5*(cos(time)+1.0), 1.0);
  }
`;

const canvas = document.querySelector("canvas");
const gl = canvas.getContext("webgl");

const programInfo = twgl.createProgramInfo(gl, [vertexShader, fragmentShader]);

const bufferInfo = twgl.createBufferInfoFromArrays(gl, {
  // this is the attribute again
  position: [-1, 0, 0, 0, -1, 0, 1, 1, 0],
});

requestAnimationFrame(function render(time) {
  twgl.resizeCanvasToDisplaySize(canvas);
  gl.viewport(0, 0, canvas.width, canvas.height);

  gl.useProgram(programInfo.program);
  twgl.setBuffersAndAttributes(gl, programInfo, bufferInfo);
  twgl.setUniforms(programInfo, {
    // and this is the uniform again
    time: time / 1000,
  });
  twgl.drawBufferInfo(gl, bufferInfo);

  requestAnimationFrame(render);
});

This code results in this fantastic looking WebGL demo:

This is about as simple of a WebGL program you can have, but don't worry if it looks intimidating, we don't really need to understand how it works. The important bits are the ones I have commented. There is one attribute (position) and one uniform (time), and they show up twice, once in the shaders and once in the js code. For the WebGL code to work the attributes and uniforms used in the shaders have to be given a value in the js code, and they have to be given the correct type of value. If the value isn't set, the name is spelled wrong or the wrong js type is given, then the code will either not behave correctly or not work at all. So, it would be useful to have TypeScript check for us if we have set the right attributes and uniforms for us, by comparing the TypeScript code with the shader program. But that means we have to teach TypeScript to understand a C program string. So let's do that!

type Uniform = `uniform ${string} ${string}`;

// this is ok
const success: Uniform = "uniform vec2 something";

// this is not ok
const failure: Uniform = "not a uniform";

In this code we have a type called Uniform and it is defined using the template literal syntax. We can use this type on a variable, and that makes it only possible to assign values to that variable that matches the string pattern. That means variables of type Uniform have to start with the string uniform and then contain some more stuff (with at least one space between any string values). In the code example at the beginning of the article you can see that in the fragmentShader there is one line with uniform float time which will match this pattern.

You can find more information about template literal types in the TypeScript documentation.

type UniformType =
  | "float"
  | "vec2"
  | "vec3"
  | "vec4"
  | "mat2"
  | "mat3"
  | "mat4";

type Uniform = `uniform ${UniformType} ${string}`;

// this is ok
const success: Uniform = "uniform vec2 something";

// this is not ok
const failure: Uniform = "not a uniform";

// this is also not ok
const failure: Uniform = "uniform unknown something";

Now we have limited the types our uniform can have. The last line now has the correct format, but it uses an unknown uniform type, one that isn't in the list of UniformType, so it's not allowed, and the TypeScript compiler will complain.

type UniformType =
  | "float"
  | "vec2"
  | "vec3"
  | "vec4"
  | "mat2"
  | "mat3"
  | "mat4";

type Uniform<
  Type extends UniformType,
  Name extends string
> = `uniform ${Type} ${Name}`;

// this is ok
const success: Uniform<"vec2", "something"> = "uniform vec2 something";

// this is not OK
const failure: Uniform<"float", "something"> = "uniform vec2 something";

Here the Uniform is made generic, so we now have to specify the type and the name that it can have. This might not seem very useful, the example gets quite verbose, but it will be useful in the next step.

type UniformType =
  | "float"
  | "vec2"
  | "vec3"
  | "vec4"
  | "mat2"
  | "mat3"
  | "mat4";

type Uniform<
  Type extends UniformType,
  Name extends string
> = `uniform ${Type} ${Name}`;

type GetUniform<T extends string> = T extends Uniform<infer Type, infer Name>
  ? [Type, Name]
  : unknown;

declare function test<T extends string>(text: T): GetUniform<T>;

const result: ["vec2", "something"] = test("uniform vec2 something");

Ok, now we have introduced conditional types as well. Things are starting to get weird. The GetUniform type is a conditional type that checks if its generic parameter T matches the pattern of a Uniform. If it does, it is able to infer what the two missing generic parameters Type and Name are. The really weird thing here is that we say that if it matches the pattern, then it will be a tuple that contain the type and name, and if it doesn't match, then it will be unknown. You can read more about conditional types and inference in the TypeScript documentation.

I have left out the implementation of the function test here. It will use some deep webgl functionality to be able to return what we declare it will return. Sometimes when working with TypeScript you have to ignore what happens inside a function and just say what it will return, and this is one of those cases. This is when the conditional types are useful, as we can say what it will return depending on the parameters without having to say how it will work.

type UniformType =
  | "float"
  | "vec2"
  | "vec3"
  | "vec4"
  | "mat2"
  | "mat3"
  | "mat4";

type Uniform<
  Type extends UniformType,
  Name extends string
> = `uniform ${Type} ${Name}`;

type GetUniform<T extends string> = T extends Uniform<infer Type, infer Name>
  ? Record<Name, Type>
  : unknown;

declare function test<T extends string>(text: T): GetUniform<T>;

const result: { something: "vec2" } = test("uniform vec2 something");

// now result is an object, and we can assign to something:
result.something = "vec2";

Getting a tuple isn't that useful, it's much more useful to get an object that we can use. The type Record is a built-in type in TypeScript, and it is the type for an object with keys and values. The way it is used here produces an object with a single key, which is the Name, and a value that is the string Type. Therefore result is now an object, which we can assign a value to! The value we can assign to it is currently limited to the string value vec2 though, since that is what we got from the line of C code. What we really want to assign here is either a number or an array of numbers, since that is what the shader program expects to get. So we need to somehow map from the types in UniformType to the correct TypeScript type.

type UniformType =
  | "float"
  | "vec2"
  | "vec3"
  | "vec4"
  | "mat2"
  | "mat3"
  | "mat4";

type GetRealType<Type extends UniformType> = Type extends "float"
  ? number
  : ArrayLike<number>;

type Uniform<
  Type extends UniformType,
  Name extends string
> = `uniform ${Type} ${Name}`;

type GetUniform<T extends string> = T extends Uniform<infer Type, infer Name>
  ? Record<Name, GetRealType<Type>>
  : unknown;

declare function test<T extends string>(text: T): GetUniform<T>;

const result = test("uniform vec2 something");

// now result is an object, and we can assign to something:
result.something = [12, 43];

// this will fail, since foo isn't the name of the uniform object
result.foo = [12, 43];

// this will also fail, since something is vec2 and not float:
result.something = 12;

One more conditional type. Here we check if the the Type is 'float', and if it is, the real type should be number, but if it is one of the other options (vec2, vec3, ..., mat4) then should be an array of numbers. The last line shows that we can now set an array of numbers to our uniform something, because it was of type vec2. We might have done something a bit more clever to limit the lengh of the arrays assignable, so that a vec2 becomes a [number, number], while a mat4 becomes an array with 16 numbers, but let's leave it like it is for now. The type of result is inferred from the return value of test(), so we don't need to specify it. TypeScript will know what the type is based purely on the string we passed to test() and it will warn us if we do something wrong, like in the last line, where we assign a number to something that should be an array of numbers.

type UniformType =
  | "float"
  | "vec2"
  | "vec3"
  | "vec4"
  | "mat2"
  | "mat3"
  | "mat4";

type GetRealType<Type extends UniformType> = Type extends "float"
  ? number
  : ArrayLike<number>;

type Uniform<
  Type extends UniformType,
  Name extends string
> = `uniform ${Type} ${Name}`;

type GetUniform<T extends string> = T extends Uniform<infer Type, infer Name>
  ? Record<Name, GetRealType<Type>>
  : unknown;

type GetAllUniforms<
  T extends string
> = T extends `${infer Statement};${infer Rest}`
  ? GetUniform<Statement> & GetAllUniforms<Rest>
  : GetUniform<T>;

declare function test<T extends string>(text: T): GetAllUniforms<T>;

const result = test("uniform vec2 something;uniform float color;");

// now result is an object, and we can assign to something:
result.something = [12, 43];

// this will also work:
result.color = 12;

The limitation we have had until now is that we only support a single line of code. That's not very useful, so let's have a look at working with a multi-statement codesnippet. Here there is a new conditional type called GetAllUniforms. It looks for semicolons in strings and infers what comes before and after the semicolon. It then takes the Statement and passes it to GetUniform, and then it passes the rest of the string to itself recursively. This way it is able to handle multiple statements and merge it all together as as single object using the & operator. There is a limitation here that our code has to be on one line, which we want to change.

type Trim<T extends string> = T extends `\n${infer Rest}`
  ? Trim<Rest>
  : T extends ` ${infer Rest}`
  ? Trim<Rest>
  : T;

type UniformType =
  | "float"
  | "vec2"
  | "vec3"
  | "vec4"
  | "mat2"
  | "mat3"
  | "mat4";

type GetRealType<Type extends UniformType> = Type extends "float"
  ? number
  : ArrayLike<number>;

type Uniform<
  Type extends UniformType,
  Name extends string
> = `uniform ${Type} ${Name}`;

type GetUniform<T extends string> = T extends Uniform<infer Type, infer Name>
  ? Record<Name, GetRealType<Type>>
  : unknown;

type GetAllUniforms<
  T extends string
> = T extends `${infer Statement};${infer Rest}`
  ? GetUniform<Trim<Statement>> & GetAllUniforms<Rest>
  : GetUniform<Trim<T>>;

declare function test<T extends string>(text: T): GetAllUniforms<T>;

const result = test(`
  uniform vec2 something;
  uniform float color;

  void main() {
    gl_FragColor = color;
  }
`);

// now result is an object, and we can assign to something:
result.something = [12, 43];

// this will also work:
result.color = 12;

I've introduced here the Trim type, which uses a combination of conditional types and template literal types to trim away newlines and spaces. This way we can format our two uniform statements on separate lines. Note also that we can have more than just those two statements, since the GetUniform type will turn strings that don't match what it expects into unknown, and this is ignored by the & operator. That is, {a: number} & {b: number} & unknown is the same as {a: number, b: number}.

type Trim<T extends string> = T extends `\n${infer Rest}`
  ? Trim<Rest>
  : T extends ` ${infer Rest}`
  ? Trim<Rest>
  : T;

type UniformType =
  | "float"
  | "vec2"
  | "vec3"
  | "vec4"
  | "mat2"
  | "mat3"
  | "mat4";

type GetRealType<Type extends UniformType> = Type extends "float"
  ? number
  : ArrayLike<number>;

type Uniform<
  Type extends UniformType,
  Name extends string
> = `uniform ${Type} ${Name}`;

type GetUniform<T extends string> = T extends Uniform<infer Type, infer Name>
  ? Record<Name, GetRealType<Type>>
  : unknown;

type GetAllUniforms<
  T extends string
> = T extends `${infer Statement};${infer Rest}`
  ? GetUniform<Trim<Statement>> & GetAllUniforms<Rest>
  : GetUniform<Trim<T>>;

interface ProgramInfo {}

interface Twgl {
  createProgramInfo(
    gl: WebGL2RenderingContext,
    shaders: [string, string]
  ): ProgramInfo;
  setUniforms(
    programInfo: ProgramInfo,
    uniforms: Record<string, number | ArrayLike<number>>
  ): void;
  // ... more stuff here ...
}

This is the interface of Twgl that we have to implement. The first method, createProgramInfo, takes a tuple of shaders and returns something, we don't really know what it is, but it is ProgramInfo. The second method takes the programInfo together with a uniforms object. I've indicated here that it is an Record where the name is a string and the type is either a number or an ArrayLike<number>. Currently this isn't very useful, there is no way to ensure that the uniforms passed to setUniforms contains all the right keys with the correct types. So let's replace some of the types with the clever stuff we have already written.

type Trim<T extends string> = T extends `\n${infer Rest}`
  ? Trim<Rest>
  : T extends ` ${infer Rest}`
  ? Trim<Rest>
  : T;

type UniformType =
  | "float"
  | "vec2"
  | "vec3"
  | "vec4"
  | "mat2"
  | "mat3"
  | "mat4";

type GetRealType<Type extends UniformType> = Type extends "float"
  ? number
  : ArrayLike<number>;

type Uniform<
  Type extends UniformType,
  Name extends string
> = `uniform ${Type} ${Name}`;

type GetUniform<T extends string> = T extends Uniform<infer Type, infer Name>
  ? Record<Name, GetRealType<Type>>
  : unknown;

type GetAllUniforms<
  T extends string
> = T extends `${infer Statement};${infer Rest}`
  ? GetUniform<Trim<Statement>> & GetAllUniforms<Rest>
  : GetUniform<Trim<T>>;

interface ProgramInfo<VS extends string, FS extends string> {}

interface Twgl {
  createProgramInfo<VS extends string, FS extends string>(
    gl: WebGL2RenderingContext,
    shaders: [VS, FS]
  ): ProgramInfo<VS, FS>;
  setUniforms<VS extends string, FS extends string>(
    programInfo: ProgramInfo<VS, FS>,
    uniforms: GetAllUniforms<VS> & GetAllUniforms<FS>
  ): void;
  // ... more stuff here ...
}

The big change here is to make everything generic on the two shaders, the vertex shader VS and the fragment shader FS. This way the setUniforms method can declare that the second parameter, the uniforms, depends on those two shaders. Note that it uses GetAllUniforms<VS> & GetAllUniforms<FS> to extract all the uniforms from both the vertex shader and the fragment shader and combine them into a single object. TypeScript is able to infer all of these generic parameters, so we don't have to specify them in our code.

type Trim<T extends string> = T extends `\n${infer Rest}`
  ? Trim<Rest>
  : T extends ` ${infer Rest}`
  ? Trim<Rest>
  : T;

type UniformType =
  | "float"
  | "vec2"
  | "vec3"
  | "vec4"
  | "mat2"
  | "mat3"
  | "mat4";

type GetRealType<Type extends UniformType> = Type extends "float"
  ? number
  : ArrayLike<number>;

type Uniform<
  Type extends UniformType,
  Name extends string
> = `uniform ${Type} ${Name}`;

type GetUniform<T extends string> = T extends Uniform<infer Type, infer Name>
  ? Record<Name, GetRealType<Type>>
  : unknown;

type GetAllUniforms<
  T extends string
> = T extends `${infer Statement};${infer Rest}`
  ? GetUniform<Trim<Statement>> & GetAllUniforms<Rest>
  : GetUniform<Trim<T>>;

interface ProgramInfo<VS extends string, FS extends string> {}

interface Twgl {
  createProgramInfo<VS extends string, FS extends string>(
    gl: WebGL2RenderingContext,
    shaders: [VS, FS]
  ): ProgramInfo<VS, FS>;
  setUniforms<VS extends string, FS extends string>(
    programInfo: ProgramInfo<VS, FS>,
    uniforms: GetAllUniforms<VS> & GetAllUniforms<FS>
  ): void;
  // ... more stuff here ...
}

declare const twgl: Twgl;

// This is how the two methods will be used

const vertexShader = `
  precision highp float;
  attribute vec3 position;

  varying vec2 uv;

  void main() {
    gl_Position = vec4(position, 1.0);
    uv = position.xy;
  }
`;

const fragmentShader = `
  precision highp float;
  uniform float time;

  varying vec2 uv;

  void main() {
    gl_FragColor = vec4(0.5*(uv+1.0), 0.5*(cos(time)+1.0), 1.0);
  }
`;

const programInfo = twgl.createProgramInfo(gl, [vertexShader, fragmentShader]);

twgl.setUniforms(programInfo, {
  time: 10,
});

So now, given the vertexShader and the fragmentShader the twgl.setUniforms() method will complain if the time: 10 line is missing.

This is of course just a simple demonstration, and it does not handle all scenarios and it can be improved quite a bit. For example, since it splits on semicolon only, it's not able to handle comments. Comments start with // and end at a newline, but we don't split our code on newlines, only on semicolon, so the statement on the line after a comment is included with the comment. Note also that since we split on semicolon we don't actually parse the code very well, but that's not necessary since we are only interested in getting the uniforms, and they are all declared at the top level. Therefore we don't care that the parsing doesn't understand { and }.

The code currently only extracts the uniforms, it would be useful to extract the attributes as well. It can be done in almost exactly the same way, just producing a different shape of the object. And I have skipped the uniform types sampler2D and samplerCube, but it shouldn't be too difficult to implement them too.

A lot of the code above is based on work done by others, for example the ts-sql I mentioned earlier. There is a curated list of awesome template literal types and examples by very clever people. TypeScript has become very powerful as of version 4.1.

I think this could be a very useful way for TypeScript code to suddenly understand other languages, like CSS or SQL. I've created a feature suggestion for the TypeScript language to support importing other files as strings, and then using template literal types and conditional types be able to parse the string and produce useful types that can be used in projects.

Type-dictionary trick in C#

2019-06-30T00:00:00Z

This article explores a trick in C# for looking up values based on types, much like a Dictionary<Type, T> only it's almost 10x faster! You probably don't need this trick and even if you need it, it will only work in a few very specific scenarios. But it's a neat trick that might be fun to read about. I don't know if this pattern has a name, and I'm not very good at naming things, so maybe I'll just refer to it as the private static generic inner class trick. If you know of a better name, please let me know.

It's a bit tricky to explain exactly how and when to use this trick, so I'll look at a specific use-case and work our way towards a specific solution. From there we can generalize it until we hit on the limitations of this pattern and find other specific use-cases for it within those limitations.

An imaginary inefficient ORM

Let's imagine we are working on an ORM library, one that will be really good. It will work like many other ORMs, where a C# class maps to a table in our database. Let's ignore if this is a good or bad idea and focus on just a tiny part of the implementation, the way we map the class to the table. We can use the [Table] attribute like so:

[Table("BlogPosts")]
public class BlogPost
{
  public int Id { get; set; }

  public string Title { get; set; }

  public string Content { get; set; }

  public DateTime Published { get; set; }
}

The class BlogPost maps to the table BlogPosts in the database through the attribute [Table]. How can our code find out this? We need to use reflection.

public static class Reflections
{
  public static string GetTableName<T>()
    => typeof(T)
      .CustomAttributes
      .OfType<TableAttribute>()
      .First()
      .Name;
}

This looks up all the attributes applied to a type, finds the Table attributes and takes the first one and gets the Name property from it. If you have used reflection in C# then this will look familiar and straight forward. But you will then also know that it's not very efficient.

Reflection is slow. If we run this method every time we query the database then we are not being as efficient as we could be. The Name property of the Table attribute applied to a specific type will not change while the code is running, so we have a method that will slowly find the same answer every time we call it. We can speed it up by saving the result in a dictionary, so that we don't need to use reflection every time:

public static class Reflections
{
  private static ConcurrentDictionary<Type, string> TableNames = new ConcurrentDictionary<Type, string>();

  public static string GetTableName<T>()
    => TableNames.GetOrAdd(typeof(T), type => type
      .GetCustomAttributes(typeof(TableAttribute), true)
      .Cast<TableAttribute>()
      .FirstOrDefault()
      ?.Name);
}

(I use ConcurrentDictionary here only to get the very useful GetOrAdd method, which doesn't exist on IDictionary. In a real implementation of this I would probably use just a normal dictionary and a lot more error checking and fallback.)

This way we only use reflection for a type once, after that the result is stored in the dictionary TableNames. Benchmarking, which is often wrong, will show that this is 100s or 1000s of times faster for repeated lookups! We have reduced the CPU time a lot by slightly increasing memory usage, a good tradeoff. But we have only reduced the CPU usage, we haven't removed it. That is, there are still further improvements, and this is where the finely named private static generic inner class trick can be used.

So let's do what the name implies and add a private static generic inner class:

public static class Reflections
{
  public static string GetTableName<T>()
    => TableName<T>.Name;

  private static class TableName<T>
  {
    public static readonly string Name = typeof(T)
      .GetCustomAttributes(typeof(TableAttribute), true)
      .Cast<TableAttribute>()
      .FirstOrDefault()
      ?.Name;
  }
}

Here finally we see the trick in action. The inner class TableName<T> is generic, so the static field Name will be created once per type T. The way generics work in C# is that at runtime a class is created for each specific type of the generic class. The initializer of this field will only be called the first time, just like the way we set up our dictionary in the previous example.

Is this faster than the dictionary? Yes, from simple benchmarks it seems to be almost 10x faster! This is because we have replaced hashcode calculation and dictionary lookup with a single static field read from a class. It's unlikely that we can make this any faster.

Benchmark you say?

Here is the full code sample for the benchmark, written in BenchmarkDotNet. This is very simple, probably not the best way to do it, but I think it gives a good indication of how this trick works:

[CoreJob]
[RankColumn]
public class Program
{
  [Benchmark]
  public string Reflection() => GetFromReflection<BlogPost>();

  [Benchmark]
  public string Dictionary() => GetFromDictionary<BlogPost>();

  [Benchmark]
  public string Inner() => GetFromInner<BlogPost>();

  public static void Main()
    => BenchmarkRunner.Run<Program>();

  private static string GetFromReflection<T>()
    => typeof(T)
      .GetCustomAttributes(typeof(TableAttribute), true)
      .Cast<TableAttribute>()
      .FirstOrDefault()
      ?.Name;

  private static Dictionary<Type, string> _dictionary = new Dictionary<Type, string>();

  private static string GetFromDictionary<T>()
    => _dictionary.TryGetValue(typeof(T), out var result) ? result : _dictionary[typeof(T)] = GetFromReflection<T>();

  private static string GetFromInner<T>()
    => InnerClass<T>.Name;

  private static class InnerClass<T>
  {
    public static readonly string Name = GetFromReflection<T>();
  }
}

This is the result I get on my machine:

BenchmarkDotNet=v0.11.5, OS=Windows 10.0.18362
Intel Core i7-3517U CPU 1.90GHz (Ivy Bridge), 1 CPU, 4 logical and 2 physical cores
.NET Core SDK=2.2.300
  [Host] : .NET Core 2.2.5 (CoreCLR 4.6.27617.05, CoreFX 4.6.27618.01), 64bit RyuJIT
  Core   : .NET Core 2.2.5 (CoreCLR 4.6.27617.05, CoreFX 4.6.27618.01), 64bit RyuJIT

Job=Core  Runtime=Core

|     Method |         Mean |       Error |      StdDev | Rank |
|----------- |-------------:|------------:|------------:|-----:|
| Reflection | 5,426.419 ns | 120.0703 ns | 328.6905 ns |    3 |
| Dictionary |    32.038 ns |   0.1739 ns |   0.1627 ns |    2 |
|      Inner |     3.844 ns |   0.0532 ns |   0.0471 ns |    1 |

While the Dictionary code takes about 32ns to complete the Inner class version runs in 3.8ns, or more than 8x as fast! It varies a bit, and this is a very simple scenario where the dictionary contains only a single item, so I suspect the Inner method will be even more performant, compared to the Dictionary method, in the real world.

Where does this trick come from?

I found this trick first used in the Array.Empty method, which relies on the internal class EmptyArray. In this scenario it is used to create a single instance of an empty array of a specific type for the duration of the program. This prevents the wasted CPU and memory of creating a new instance of an empty array every time that is needed. All empty arrays are exactly the same, so we don't need a new one for every scenario, we can just reuse the same empty array (of a specific type) everywhere it is needed. I stole this trick for the Empty class in our extension methods nuget package at work.

I haven't seen it used any other place, probably because of its many limitations.

Limitations

I've called this article the type-dictionary trick and compared it against the performance of a dictionary, but the code above doesn't much behave like a dictionary. In particular:

Unlike a dictionary it can only work with Type as its key
Not just that, the type has to be passed in as a generic type, it can't be a normal parameter to the function
It is not possible to remove entries from the dictionary, they will be there forever
- Well, you could set the value to null, or create a wrapper type with a special "no-value" value...
- ... but even then the memory for the entry will not be released, so the dictionary will only ever grow in size
You can only have a single instance of this dictionary; it has to be static
- ... or do we? We'll look at a workaround below
This limitation means that probably you want it to be read-only, since mutating static variables leads to pain and suffering

The result is that you probably only want to use this trick to store some values that don't change and are fixed at compile time, like the value of the attribute in the example above. Reflection caching seems to be a good scenario for this trick, since it works on types and we often use reflection on types, and the result won't change at runtime.

Another (crazy) example

A place where reflection is often used is for dependency injection, like the IServiceProvider in .NET core. We can use the trick to make a super fast service lookup, like this:

public class FastServiceProvider
{
  public void AddService<TService>(Func<TService> factory)
    => Inner<TService>.ServiceFactory = factory;

  public TService GetService<TService>()
    => Inner<TService>.ServiceFactory();

  private static class Inner<TService>
  {
      public static Func<TService> ServiceFactory;
  }
}

This is a very limited example that only lets you register a factory function and look up a service using the generic GetService<TService>() function. But it has a bigger problem, namely that you can't have multiple instances of FastServiceProvider with different services, since it all relies on the static inner class. How can we work around this problem? First, we need some way to differentiate different instances of FastServiceProvider, and we can do that by making it too generic. The inner class is now unique per two generic type parameters, so as long as we create a new FastServiceProvider<TypeA> and a FastServiceProvider<TypeB> then they will have different inner classes and different "dictionaries" of services.

public class FastServiceProvider<T>
{
  public void AddService<TService>(Func<TService> factory)
    => Inner<TService>.ServiceFactory = factory;

  public TService GetService<TService>()
    => Inner<TService>.ServiceFactory();

  private static class Inner<TService>
  {
      public static Func<TService> ServiceFactory;
  }
}

But, we have to make sure to always use a new type every time we create a new instance of FastServiceProvider<T>, or else two instances will share services! We need some way to create the generic FastServiceProvider<T> with a different type T each time. We can do that by dynamically creating types, at runtime, when we need one. The code for doing just that look something like this:

public static class MyTypeBuilder
{
  private static int Counter { get; set; } = 0;

  private static AssemblyName MyAssemblyName { get; } = new AssemblyName("MyAssembly");

  private static AssemblyBuilder MyAssembly { get; } = AssemblyBuilder.DefineDynamicAssembly(MyAssemblyName, AssemblyBuilderAccess.Run);

  private static ModuleBuilder MainModule { get; } = MyAssembly.DefineDynamicModule("MainModule");

  public static Type CreateType()
    => GetTypeBuilder($"MyType{Counter++}")
      .CreateType();

  private static TypeBuilder GetTypeBuilder(string name)
    => MainModule.DefineType(name,
        TypeAttributes.Public |
        TypeAttributes.Class |
        TypeAttributes.AutoClass |
        TypeAttributes.AnsiClass |
        TypeAttributes.BeforeFieldInit |
        TypeAttributes.AutoLayout,
        null);
}

Don't worry too much about this code, it is mostly copied from a Stack Overflow answer. It creates a new Type in an assembly created at runtime. The type we create here is empty, since we don't need it for anything apart from being unique. The Counter that increments ensures that the name is unique.

Now that we have a way to create new unique types whenever we need them we just have to combine it with the generic FastServiceProvider<T> somehow. We've used inner classes to solve all kinds of problems so far, so why stop now? We can take the generic class we have and put it inside a new class, one that is static and only has a single public method, Create(). This creates a new instance using a new type. The way it does it is through reflection and to make it all easy to use I've created an interface that we can rely on.

public interface IFastServiceProvider
{
  void AddService<TService>(Func<TService> factory);

  TService GetService<TService>();
}

public static class FastServiceProvider
{
  public static IFastServiceProvider Create()
    => (IFastServiceProvider) typeof(FastServiceProvider)
      .GetMethod("CreateInner", BindingFlags.NonPublic | BindingFlags.Static)
      .MakeGenericMethod(MyTypeBuilder.CreateType())
      .Invoke(null, null);

  private static InnerFast<T> CreateInner<T>()
    => new InnerFast<T>();

  private class GenericServiceProvider<T>
  {
    public void AddService<TService>(Func<TService> factory)
      => Inner<TService>.ServiceFactory = factory;

    public TService GetService<TService>()
      => Inner<TService>.ServiceFactory();

    private static class Inner<TService>
    {
        public static Func<TService> ServiceFactory;
    }
  }
}

Ok, that's quite a bit of convoluted code. I don't know if this is at all useful or just too complicated for any practical use. Since this uses reflection in the Create method it is not very fast to create new instances, but once the instances have been made they are lightning fast!

We can work around some of the other limitations in similar convoluted ways, but this is already getting a bit out of hand, so I'll leave that as an excercise for the reader.

Conclusion

I've shown here how to use a generic inner class to store data per type in a program. I think this is a neat way to get high performance code, and I've seen it used in the wild, but only a few places. I don't know if this trick has a name, or if it is used other places, but if you do, please let me know. Oh, and if you find mistakes or spelling errors, let me know as well by creating an issue or pull-request here.

Duck-typed Extension methods in C#

2018-10-21T00:00:00Z

C# might have started as a static object oriented language but over the years it has ~~stolen~~ implemented a lot of features from other very different languages. It has dynamic types, SQL-like queries, lambdas, async-await and so much more. In this article we'll see what kind of magic we can create if we take extension methods and mix them up with generics and duck-typing. Extension methods are ways to add methods to a class without modifying the class and duck-typing is (in C# at least) when the compiler doesn't require an implementetion of an interface, it just looks for a method with the right name, return type and parameters. Extension methods are quite common, but duck-typing is still pretty rare in C#. There are a few places where it's avaliable, and that's what we will look at in this article.

GetAwaiter

Let's start off with some async stuff. Since version 5.0 C# has had support for async code where we can await a Task. But why only Tasks, why can't we await other things? It turns out that we can, we aren't limited to awaiting only Tasks, we can await anything that has a GetAwaiter method that returns a TaskAwaiter (well, not really, it should return an awaiter. What is an awaiter? It's complicated, so let's not worry about that just yet, we can just use the TaskAwaiter class for most (most) scenarios. If you are curious you can read more about it in this excellent article). The cool thing is that there is no restriction that the GetAwaiter method be declared on the class, it can be an extension method! C# even tells us this if we try to await something that isn't awaitable, for example a string:

await "Hello world";

// error CS1061: 'string' does not contain a definition for 'GetAwaiter' and no
// accessible extension method 'GetAwaiter' accepting a first argument of type
// 'string' could be found (are you missing a using directive or an assembly reference?)

So let's do what C# says, and make an 'accessible extension method GetAwaiter accepting a first argument of type string', and then see what happens.

// Example 1: await strings

public static TaskAwaiter<string> GetAwaiter(this string text)
=> Task.FromResult(text).GetAwaiter();

// Now it doesn't complain when we await the string
await "Hello world";

(Try it online)

Cool, but let's take it one step further and make this method generic, so we can await anything.

// Example 2: await anything

public static TaskAwaiter<T> GetAwaiter<T>(this T nonAwaitable)
=> Task.FromResult(nonAwaitable).GetAwaiter();

// Now we can write code like this
await "Hello world";
await 100;
await true;

(Try it online)

Ok, this was a bit useless maybe, can we find a type to extend that can be useful to await? How about Lazy<Task<T>>? Instead of making an AsyncLazy<T> class, as described in this article (from all the way back in 2011), we can just make an extension method for Lazy<Task<T>>

// Example 3: await Lazy<Task<T>>

// We need a lazy variable to work with
var lazySomething = new Lazy<Task<string>>(() => GetSomethingAsync(10));

// Before
Console.WriteLine(await lazySomething.Value);

// Then we introduce this extension method
public static TaskAwaiter<T> GetAwaiter<T>(this Lazy<Task<T>> lazyTask)
=> lazyTask.Value.GetAwaiter();

// After
Console.WriteLine(await lazySomething);

(Try it online)

Success! We got rid of the .Value! Ok, still not impressed? How about getting rid of Task.WhenAll? We can do that by extending IEnumerable:

// Example 4: await IEnumerable<Task<T>>

// Before (notice the Task.WhenAll)
var manyThings = await Task.WhenAll(Enumerable.Range(0, 10).Select(GetSomethingAsync));

// Then we introduce this extension method
public static TaskAwaiter<T[]> GetAwaiter<T>(this IEnumerable<Task<T>> manyTasks)
=> Task.WhenAll(manyTasks).GetAwaiter();

// After (no more Task.WhenAll)
var manyThings = await Enumerable.Range(0, 10).Select(GetSomethingAsync);

(Try it online)

This magic extension method lets us await an IEnumerable of Tasks directly, without having to wrap it in Task.WhenAll. It ends up converting an enumerable of tasks (IEnumerable<Task<T>>) into a task of an enumerable (Task<IEnumerable<T>>). This is great when we have many Task<T> for the same type T, but what if we have a few async functions that return different results, and we want to run them in parallel?

// Example 5: await a tuple of Task<T>

// Before
var task1 = GetSomethingAsync(10);
var task2 = GetAnotherThingAsync("some parameter");
await Task.WhenAll(task1, task2);
var something = task1.Result;
var anotherThing = task2.Result;

// Then we introduce this extension method
public static TaskAwaiter<(T1, T2)> GetAwaiter<T1, T2>(this (Task<T1>, Task<T2>) tasks)
=> Task.WhenAll(tasks.Item1, tasks.Item2)
       .ContinueWith(_ => (tasks.Item1.Result, tasks.Item2.Result))
       .GetAwaiter();

// After
var (something, anotherThing) = await (GetSomethingAsync(10), GetAnotherThingAsync("some parameter"));

(Try it online)

This example uses ValueTuples, introduced in C# 7.0. ValueTuples are just structs, so we can write extension methods for them too. Here we convert a tuple of tasks into a task of a tuple. The problem with tuples is that we have to write one extension method for each tuple length. This example only works for tuples with 2 values, we have to write a similar method for tuples with three values and so on. It's not very difficult, but it's not so fun to copy paste the same code into each project you work on. Luckly there is a nuget package called TaskTupleAwaiter which contains extension methods for tuples with up to 10 values. It works slightly differently from the solution here in that it doesn't return a TaskAwaiter. If you are curious how it works you can have a look at this open GitHub issue which resulted in the nuget.

Add

C# 3.0 added the syntax for object and collection initializers. We'll skip object initializers beacuse it's boring, and focus on collection initializers, which use duck-typed magic. How does collection initializers work? Let's say we want to make a list with three items, then we can either call Add three times, or we can use the curly brace and list the items we want to add:

// Without collection initializer
var list = new List<int>();
list.Add(1);
list.Add(2);
list.Add(3);

// With collection initalizer
var list = new List<int>
{
  1,
  2,
  3
};

While these may look very different these two scenarios are exacly the same; when we use the collection initializer it is just syntax sugar for calling Add multiple times. That's great, as it means we can define an Add extension method and then we can use it with the collection initializer.

For example, JavaScript, as of ES2015, got the ability to spread an array into another array, and this is something I've wanted to do in C#. But it has not been possible, until now!

// Example 6: Add a list

// With this extension method...
public static void Add<T>(this List<T> list, IEnumerable<T> items)
=> list.AddRange(items);

// ...this is possible
var result = new List<int>
{
  1,
  2,
  GetAnotherListOfInts(),
  3
};

(Try it online)

The result will be a list of ints, starting with 1 and 2, followed by all the ints returned by GetAnotherListOfInts and ending with 3. This is quite useful when we are working with IEnumerable<T>, IReadOnlyCollection<T> where we can't modify a list directly.

Now you may have seen the following systax for initializing a dictionary, and wondered what the special sytax with the curly braces is

var populations = new Dictionary<string, int>{
  { "China", 1_409_517_397 },
  { "India", 1_339_180_127 },
  { "USA", 324_459_463 },
  { "Indonesia", 263_991_379 },
  { "Brazil", 209_288_278 }
};

Is this a syntax made specifically for dictionary initialization? No, this is the syntax for passing multiple arguments to the Add method! IDictionary has an Add(key, value) method which takes two arguments, the key and the value. So we aren't limited to creating an Add extension method with only one (additional) parameter, we can make one with as many as we want! But before we look at the opportunities this enable, let's implement the Add method that spreads a dictionary into a dictionary, just like we did for a list:

// Example 7: add a dictionary

// This is the dictionary spread method
public static void Add<TKey, TValue>(this IDictionary<TKey, TValue> dictionary, IDictionary<TKey, TValue> items)
{
  foreach (var item in items)
  {
    dictionary[item.Key] = item.Value;
  }
}

// It makes this possible
var populations = new Dictionary<string, int>
{
  { "China", 1_409_517_397 },
  { "India", 1_339_180_127 },
  { "USA", 324_459_463 },
  { "Indonesia", 263_991_379 },
  { "Brazil", 209_288_278 },
  GetAfricanCountries(),
  GetEuropeanCountries()
};

(Try it online)

Neato, this is sure to be useful! One fancy feature of this extension method is that it will not fail on duplicate keys, it will overwrite previous keys. This means that the order of the items matter, and you can pass in two dictionaries with the same keys and have the latter one override the previous one.

Ok, enough about that, let's get back to the curly braces, can we make an extension method that takes multiple arguments? Sure, no problem:

// Example 8: add new instances

// Before
var cities = new List<City>
{
  new City("Chongqing", 30_751_600, "China"),
  new City("Shanghai", 24_256_800, "China"),
  new City("Delhi", 11_034_555, "India"),
  new City("Beijing", 21_516_000, "China"),
  new City("Dhaka", 14_399_000, "Bangladesh")
};

// Then with this very specific extension method
public static void Add(this List<City> list, string name, int population, string country)
=> list.Add(new City(name, population, country));

// After
var cities = new List<City>
{
  { "Chongqing", 30_751_600, "China" },
  { "Shanghai", 24_256_800, "China" },
  { "Delhi", 11_034_555, "India" },
  { "Beijing", 21_516_000, "China" },
  { "Dhaka", 14_399_000, "Bangladesh" }
};

(Try it online)

Hey, would you look at that, we don't have to write new City for each item to add! Instead we pass in three arguments to the Add method and let it create a new city for us.

That was a very specific extension method, since it works only for City and it takes exactly three arguments. Can we make an Add extension method that takes arbitrary many arguments? Yes, using params!

// Example 9: multi or single dimensional array?

// We define this extension method
public static void Add<T>(this List<T> list, params T[] items)
=> list.AddRange(items);

// And then we can make a flattened matrix
var matrix = new List<double>
{
  { 1, 0, 0 },
  { 0, 1, 0 },
  { 0, 0, 1 }
};

(Try it online)

This might be useful when we are working with multi dimensional arrays, for example matrices and images. Maybe.

Now we have curly braces that don't do anything, they are just there to look pretty. Well, curly braces aren't really that pretty, not as pretty as parentheses anyways. And if we go back to the example with the cities above, isn't a bit strange that we use curly braces to hold a list of arguments? It looks like object initialization, but it's not. Can we use parentheses instead? Will it make us more like lisp programmers?

// Example 10: add a tuple

// We can gather up all the parameters in a tuple
public static void Add(this List<City> list, (string name, int population, string country) city)
=> list.Add(city.name, city.population, city.country);

// Now we have parenthesis, not curly braces
var cities = new List<City>
{
  ("Chongqing", population: 30_751_600, "China"),
  ("Shanghai", population: 24_256_800, "China"),
  ("Delhi", population: 11_034_555, "India"),
  ("Beijing", population: 21_516_000, "China"),
  ("Dhaka", population: 14_399_000, "Bangladesh")
};

(Try it online)

Ah, much better. Here we just wrap the Add(name, population, country) method we created in the original example, so we can pick and choose which one we want to use. We could even do every other line with parentheses and curly braces, if we are in a bad mood. Notice that we can name the arguments, like I've done with the population, an option we don't have with curly braces.

Oh, BTW, did you notice the fancy digit separator in the numbers above? The syntax for using _ in numbers like 7_550_262_101 was added in C# 7.0 as is pretty neat, but a bit weird when we are used to seeing numbers represented like 7,550,262,101. Can we fix this using extension methods? Yes, we can hack it!

// Example 11: add with params

// Use params to take several ints, each representing 3 digits
public static void Add<TKey>(this IDictionary<TKey, int> dictionary, TKey key, params int[] values)
=> dictionary.Add(key, values.Aggregate(0, (s, v) => s*1000 + v));

// Now we can write it like this
var populations = new Dictionary<string, int>
{
  { "China", 1,409,517,397 },
  { "India", 1,339,180,127 },
  { "USA", 324,459,463 },
  { "Indonesia", 263,991,379 },
  { "Brazil", 209,288,278 }
};

(Try it online)

OK, even I realize the examples are getting pretty silly now, so let's stop.

The future

In the examples with dictionaries you might have noticed that I didn't use the index initializer, which looks like this:

var populations = new Dictionary<string, int>{
  ["China"] = 1_409_517_397,
  ["India"] = 1_339_180_127,
  ["USA"] = 324_459_463,
  ["Indonesia"] = 263_991_379,
  ["Brazil"] = 209_288_278
};

This might look very similar to the collection initialization we have used in many of the examples, but it's not, it's the closely related object initializer. This code ends up calling the index operator which is defined as an operator on the class. As of C# 7.2 it's only possible to extend instance methods, not overloaded operators, so we can't use this syntax for magic.

But there is a proposal called Extension Everything that will make it possible to extend not just instance methods, but also properties and operators, including the index operator! It is marked for C# 8.0, but that seems unlikely as it hasn't gotten any attention lately, at least not compared to some of the other features that are planned, like nullable reference types and patterns and ranges. We'll just have to wait and see when it will be possible to extend other fun stuff and what kind of cool and silly things might be possible in the future.

Ekkiog followup

2018-04-07T00:00:00Z

Almost two years ago I wrote about Ekkiog, and since then I haven't really written anything here, about it or anything else. In coding I follow the philosophy of agile/scrum/xp/mvp/ci where a program should always be usable and useful, but also under continuous development and improvement. And so I have worked on Ekkiog for almost two years, adding new features, improving existing parts and refactoring code, but always trying to have a working product. But with writing, I never feel ready to post about it, as the article is never good enough, never really done. And so I've started writing about what I've done multiple times, but never completed it. It's weird how I'm comfortable publishing code that is usable but not done, but not comfortable publishing an article as it never feels finished.

What is Ekkiog?

Ekkiog, the name, is a combination of two Icelandic (or old norse) words; ekki (not) and og (and), which describes the NAND gate that is at the core of the mobile webapp (Ekkiog is made for mobile devices, and therefore only works on touch screens). A NAND gate is a simple logic gate, with two inputs and a single output, and the output is logic high (aka true, 1) unless both inputs are high, in which case the output is low (aka false, 0). This is one of the simplest logic gates, but it has a special property in that it can be used to make all other logic gates. A single NAND gate can be used to make a NOT gate, two NAND gates can be used to make an AND gate, three NAND gates can be used to make an OR gate and four NAND gates can be used to make an XOR gate. This is called NAND logic, and using enough NAND gates you can make a computer.

For example, combine an AND gate and an XOR gate, and you have a half-adder. Combine two half-adders and an OR gate, and you have a full-adder. Four full-adders combined will give you a circuit that can add two four bit numbers together.

The components I have linked to above are interactive; try tapping a button (labeled "PUSH ME!") and you'll see the wires and (ugly) bulbs light up. You can double tap the components to open them up and see how they work, recursively going deeper, from the large and complex components into increasingly simple components, and then go back out of them using the back button that shows up in the upper left corner. These examples are in read-only mode, in that you can play with them and see how they work, but you can't edit them. Ekkiog is primarily about making, so of course you can make your own components. For example, click here to make a NOR gate.

Sharing components

About a year ago, after I got the editing and the components to work well enough, I started working on a way to share components that I had made. But I didn't want the components to be files that had to be sent around, I instead wanted links that were short and friendly that could open the components. This meant storing the actual data somewhere, and retrieving it when linked to. Instead of making a backend with a database, I decided to use git and GitHub/BitBucket/GitLab/etc as the storage mechanism. This way editing while offline would not be a problem, as the changes could be synced later, and I had a way to store the entire edit history, with undo/redo, of a component.

So to share your component with others you need to synchronize (git push) it to an online git repo. I've only implemented GitHub for now, so if you have a GitHub account then you can log in by clicking the person icon in the upper left corner. After logging in you can synchronize with the GitHub repo and then you will have a branch for each component you have made. Click on the sliders icon in the top right corner to reveal (among some other buttons) the paper airplane icon, which will give you a shareable link. When others follow this link they will do a shallow clone of the git repo to download the component (branch) that you have made.

What's next?

Back in 2016 I wrote that I had three ideas, or goals, for Ekkiog: to make some kind of programming game to use while away from a PC, to make something big from something small, and to make a logic circuit sandbox game. At the time I only had a very simple simulator and editor, but I had a rough idea of what I wanted, and I've worked on it in my free time since then. I've used Ekkiog as a way to not only implement something I want, but also to learn and teach myself new things. I have learned how to use React and Redux (and when not to use them), RxJs and WebGL, and I've learned how to use tools like Webpack and TypeScript. I've also taken some detours to implement other things I've needed (or wanted), like ReaxJS, w3c-manifest-loader and for the past year or so, ES-git.

Ekkiog is fairly usable, but far from done. It can really need some polish to remove some small annoying bugs, to make it more robust and user friendly. I'm working on some small features, like labeling buttons, lights and component inputs and outputs, and I want to redraw the tiles and icons. Next I want to add a 7-segment display, to make the output of components easier to read, and I want to make some small improvements to the editor. But the big milestone that is left to do is to make it into a game. That is, have a challenge or puzzle to solve and then a way for the game to verify that the component you have made completes the challenge and solves the puzzle.

In addition I want to share what I have learned while making Ekkiog, and so I hope to write a few articles about how it works. For example, I want to write about how I'm using WebGL not for 3D graphics, but for simulating logic gates and as a tile engine and text renderer. I'd like to write about the persistent data structures and how using immutable data and diffing gives huge advantages and works well with redux, WebGL textures and git. I want to write about how git works and how it can be implemented in JavaScript and made to work in the browser. And I have an idea for how to implement the puzzle solution verification code, and I'll try to write about it while I'm implementing it.

But most of all I have to get comfortable with publishing what I've written, even if it doesn't feel perfect. So I apologize for clumsy writing and rough drafts; I hope to improve my writing by sharing what I'm coding.

Ekkiog: Turing Complete NAND-Simulator

2016-08-03T00:00:00Z

Today is the end of my summer holiday, and so I've decided to publish the status of my 2016 summer hobby project. This year I've been working on a webapp called Ekkiog, which is a NAND gate simulator for smartphones, using WebGL to concurrently simulate 65536 gates at 60 fps. If you are on a smartphone, click the link below to test it out. You can find the source code at GitHub.

Click here for demo

Ekkiog is based on three ideas I've had for some time, and at this state it contains two of them. I'll get back to the third one at the end of the article.

The first idea is to produce for entertainment while on the go. For a long time I've wanted a way to program while on the move, away from a propper PC, just for fun. Most games are destructive, but I more enjoy constructing things, so for entertainment on a long flight, train journey or on the bus I wanted a game where I could produce something. Ekkiog is Turing complete, so it can be used to make anything, just like a programming language. But it couldn't just be a text based programming language, since they are not very well suited for touch screens. Ekkiog is a visual language, where you build the program from lines and boxes connected together in a 2D space. The continuing goal is for it to be intuitive and user friendly to use on a touch screen device.

The second idea is to make something big from something small, by creating your own abstraction layers. I heard about an MIT course a few years ago where students would build their way from a transistor to a GameBoy. The idea was that you would start with something really primitive (a transistor) and then build something a bit more complex from that (a binary gate), use those to build something even more complex (an integrated circuit) and keep going (micro controller, operating system, video game) building upon what you had already made. I wanted to do the same thing, letting the player go from something very basic and learning by combining what they had made themselves.

The third idea I got from the JS1k competition, where Alex Patton made the NOT Logic demo, inspired by Minecraft's Redstone. This was the perfect fit for the two other ideas, since it was visual and used very primitive components to build something more complex. I initially copied the logic, making it a bit more user friendly and visually nicer, but quickly ran into performance issues. Simulating several NOT gates in software isn't fast!

Ekkiog

So what I've made over the past few weeks is a game/simulator where you can place NAND gates and wire them together. It will only work on touch devices, and since it uses NAND logic, you can build any other binary gate, and from there build more complex gates, like Flip-flops or Adders. It's currently not terribly user friendly or intuitive, but it has reached a point where it is at least usable. Pick one of the three primitives (wire, gate or underpass) from the radial menu in the bottom right corner. Tap the screen to place them, and long press a gate to remove it. Pan and zoom like you would any other touch based app. The game should work on modern smartphones, but I've only really tested it on Firefox and Chrome for Android, so no guarantees.

WebGL

Modern browsers have direct access to the GPU, and the GPU on modern devices is really incredible. WebGL lets you control the GPU using some very low level APIs, which means you can build some really performant code. The part of the GPU that I primarily used for Ekkiog is the fragment shader. This, very simplified, is a small C-like program that is run once for every pixel on your screen, 60 times a second. This should give you an idea of how powerful the GPU is, when it can run this program millions of times per second. The GPU is really good at running the same program with different inputs both in parallel and repeatedly, which is what makes it so fast.

I got the idea of using WebGL from Brandon Jones and specifically his Tilemap Demo. Ekkiog is also a tile engine, so it made sense to use the principles from his demo. But the less obvious use of the GPU is to run the simulation of the binary logic. Each gate in the simulation takes two inputs, which can be either true or false, and produces either a true or false output. I wanted to make a fragment shader that could take two inputs and produce an output, just like a NAND gate. To understand the fragment (and vertex) shader you can't think of it in terms of imperative programming. Think of it instead as FRP, or like a spreadsheet. The code in the cell decides on the result in that cell, and it can only control that cell. It's the same for the fragment shader; it controls the color of its pixel, and cannot control anything else.

So my simulation fragment shader would then decide on the color (true is white and false is black) for one pixel based on the color of two other pixels. Since each pixel was one NAND gate, I could scale the texture I wanted the fragment shader to run on to decide how many NAND gates I wanted to simulate. I'm now running it on a 256x256 pixel texture, which ends up being 65,536 pixels (or NAND gates). I think it will still run quite well on a 1024x1024 texture, which would be a million NAND gates!

The future

Over the past three weeks I've gotten the simulation, the rendering and the editing to work fairly well. The one thing that isn't possible (yet) that I originally wanted was to build abstractions. This is up next in my mental backlog of things to implement. The idea is to be able to select a few gates and the wires that connect them and group them into a component. Then there would be a way to add components (just like you can add NAND gates today) to the board, and group those components together. There is quite a bit of UX that is needed for this, but it should be fairly easy to implement from a technical point.

Before this though I want to add some buttons/switches (for input) and LEDs (for output). I want to make the simulation speed adjustable and I have some ideas for how to make editing easier and more powerful. The graphics are no longer horrible, but I've reached the limit of how well I can make pixel art, so unless I find someone to help me, I don't think the graphics will be improved. The code is on GitHub, if you want to help or just have a look.

I've learned quite a bit over the summer from this project, not just about WebGL but also about WebPack, ECMAScript 2015/Babel and matrix math.