The problem is that every time I create a new blog tool, I think that it's going to be the one platform I can integrate everything into - calories, lights, what's been playing on Spotify, what I've bought from Amazon, all the things - and that turns out to be hard (because getting the data is tricky, integrating it is tricky, and the privacy issues are tricky) so I stop bothering with the blog.

Either that, or something happens to the code and I can't get the entries out of the database any more.

At least this time the entries are stored as plain text files. I also like not needing to add a title. Thanks micro.blog/indiweb, you've got good ideas.

I do still want the "edit the site from the site" experience. I very much miss the site I had when I was doing BT support which was exactly (and really only) that - a textarea with three code for the page inside. (Written in Perl, I think).

Editing through (something like) GitLab IDE is not the same thing. Ignoring that it needs a whole other service (I could self host), it's the commit-build-deploy pattern that's wrong. I want to click "save" and see my changes when the page loads.

Trick is, these days, to pick a language where that works well.

Blog improvement ideas

Or A TODO List

  • Parse entries as markdown. Probably should do that at save time and cache the HTML (as well as the original file). Question: How far should I take it? Other than the header (which changes depending in login status), the site is basically static, so could be rendered at entry save time (and just do two, one for auth'd and one not).

  • Client side markdown preview

  • Spell check

  • Sort options. Allow users to have oldest top or newest top. Which leads into...

  • User options. Stash some stuff client side (sort order, last read). Could do that all with JS and local storage.

  • File upload, for images. Level two is basic image editing - at least crop, resize, rotate. (This is the "Image editor as language/stack of operations" idea again)

  • Maybe that microblog protocol. Not sure, can't decide if that's just a gimmick

  • Editing posts. Probably keep old versions, but then need to be able to view old versions (and can you wait an old version so the history is a tree?(

  • Link to (and show only) a specific post

  • Tags (maybe, and they must be optional).

  • Search. Level one is just match strings. Level 2 is stemming, stopwords, maybe that vector thing.

  • Syndication. (Cross post to Facebook/Twitter automatically)

  • Comments (maybe)

Don't know where to stash metadata for posts.

I can think of two top level options (in the same file as the post, or in a different file) and a couple of sub-options (same file with a metadata header/footer, same file json encoded).

I currently prefer same file with a header, but I'll need to play with the dotnet markdown parser before i commit.

(I need to play with the markdown parser anyway)

Training again, Advanced Development with/on AWS. Flippin' hub is broken again so only got one screen, at least untill I've got time to reboot.

Hub wasn't broken, I forgot that I need to plug it into a USB3 socket (which was on the other side of the laptop).

Just read "The darkness behind your eyes" used to refer to the self. I love it. (From https://qntm.org/streetmentioner)

Big idea: Use email/IMAP as the storage backend. Gives me arbitrary metadata (headers), text file in disk storage, universal tools. I can have posts in draft, if I hook up an SMTP server as well, I can have easy bot apis.

Using mime/mulipart I can store the original post, attached images, html rendering all together.

I'll need to import current posts, but I'm going to have to do that if I want to update my on disk format.

I'm probably going to need a second copy of dovecot/whatever IMAP server, I don't want to get my actual email mixed up with this stuff (except check out namespaces maybe?)

Why aren't I more excited about this? Could be the heat (Local weather be crazy, check news archives).

(Spoiler: It's not the heat. It's the standard "This technical wizardry isn't going to make me popular/better at taking to people/any less lonely" problem)

(Y'know, depression)

Good morning. That was a rubbish nights sleep, and I'm feeling knackered this morning. Ah, well.

Got training, hopefully I can stay awake for it.

The blog buttons are in the wrong order, I don't like post showing up in the middle.

Also, I've decided that the button style for nav links is wrong. I'm not sure about the whole colour scheme either. Gah, maybe I'm just feeling rubbish this morning and need stuff to complain about? On the other hand, I should design the site so that I'm happy looking at it when I'm depressed, so I keep using it when I'm depressed.

Microservices change explicit complexity (in the code) to implicit complexity (in the communication between components)

That's "Remember where you were" more or less done, including proper consent and everything. Only thing it needs is to fix the scroll to so that the end of the last entry read is at the bottom of the page.

Hmm. Scroll to latest isn't interacting well with location.hash being set at the same time. Let's try checking for a hash at scroll time, and not moving if there is one.

Hehehehe, that's so frickin' cool! It's been on the wishlist for the blog for ages! Now I can have best of both worlds - all the entries on the page, in date order, and regular visitors don't have to scroll for miles to find their place.

Aint JavaScript brilliant!

Works on my phone too, although the jump is more obvious.

IMAP is not the right solution for backend storage. Could make it work, probably, but it's too immutable. I want to be able to edit entries and them keep their ids (whether or not they keep a change history)

I want a bigger gap between paragraphs as well.

I think "Track latest seen" might be better than "Remember where you are". It's shorter and a little more accurate (since I'm not going to track scrolling back up, right?)

  • Show "You have unread posts" message
  • Maybe fade out latest read highlight after a few seconds
  • Start to think about images

Images:

I can think of five sources of images off the top of my head (although there maybe some overlap): Upload from local device, pull from URL, Google images, local camera, Android intent ("Share image" from another app)

Once we've got an image, it needs to be added to a post. Again, a few options: Just post image, attach to end of post, inline in post.

Given that Markdown has an inline image tag (a link, bit with a leading exclamation mark), I'd like to support that, but getting the filename right (and things like scaling/rotation/crop) is going to be tricky/irritating. (Since this bit is almost certainly going to need JavaScript anyway, I've got a bit of wiggle room to make life easier. Things like an "insert image" button on the edit page that shows an image picker/uploader/editor, generates a link, and inserts it into the textarea)

That's actually sounding like a reasonable plan right now. The dialog tag works well for the consent screen. I'll have a think about what could/should go into the add image dialog.

Android share is out, at least for now. It would work in Chrome, but I don't use Chrome.

Local camera looks doable (verging on easy): Open the camera, link it to a video element, take a snapshot, upload.

Local storage should just be a file upload tag, ideally with a scriptless version that updates to something more fancy if supported. (fancy in this context implies upload progress)

Generic url is easy as well, although since I'd want the server to do the download that probably needs JavaScript for progress/completion notification.

Google photos is it's own problem that I'm going to ignore for now.

Since the dialog stuff needs JavaScript (right? could I do something with the css checkbox trick?)(plus, it's 2022, JavaScript exists, get over it), how much JavaScript do I want to use?

The local camera stuff absolutely depends on JS. Can (probably) do crude local storage upload with no JS, likewise remote url (certainly in terms of post image as separate entry). Again, do I want to? What's the use case?

I'm going to have to do something about getting logged out on reboot (and the something is to move the storage for SmallCookieManager into the database).

Also, ow, my arm. I'm fairly sure it's because I've been laying on it wrong, probably while sleeping on the sofa (or programming in the sofa, on my side with the laptop on the table).

Not sure what my plans for tonight are. (That's not true. My plan is to lay here in the study for a bit, go downstairs later, eat something, and try to get more programming done. I'm not sure I'm happy with the plan, and I'm fairly sure my brain isn't up for programming)(The next thing I've got to do is design work anyway - what elements do I need for an "Insert Image" dialog?)

#Insert Image dialog

  • Image source drop-down
    • Remote URL
    • Local file
    • Local capture
  • Source specific controls
    • URL input + fetch button for remote. (Maybe a fetch local/remote toggle?)
    • Drop zone/file input button for local file
    • List of local cameras and available sizes for local capture, with a way to select the preferred option.
  • Preview display Including image data (width/height/type/filename) and tools (crop, resize, rotate, rename)
  • alt text
  • nsfw flag??
  • Comit button
  • Upload
  • Suggest random filename?
  • Receive allocated filename
  • Check if requested filename is available?

Image backend

Need to receive image data + filename and stash it somewhere (options.uploadPath). Again with the meta data. Would be nice to validate image data at upload time, get things like width/height/type, and stash it so I don't need to get it again.

I'd forgotten about alt tags, and so absolutely need meta data. Why not a file called blah.meta? (Standard reasons are things like: It can get out of sync, or vanish)

Still need to move the post button (clicked "take selfie" this time, but it's not hooked up to anything).

Got plenty of vertical space, can have another row of editor buttons (and so add bold, italic, link, maybe list)

My export from Twitter archive is ready for download, I don't have the right tools to examine it here (on my phone), that will give me something to play with this evening

I'm not being crazy, yeah? Importing posts from Facebook, Twitter, and my old blogs (if I can extract them) is a reasonable thing to do, right?

Dagnabit, I was not able to get out of going to the office this evening. On the plus side I'm now free to sleep till 16.

Back from the office, didn't get much done. Nobody notices when I have unproductive days, I don't know how to feel about that. On the one hand, yeah, great, I don't have people breathing down my neck. On the other hand, what's the flippin' point of turning up at all if no one notices that I've done nothing. Obviously, the point is paying the rent, but it still feels a bit off.

No example yet, but that's image uploading cracked!

(Only using local camera for now, but adding local files should be trivial. I wonder how well remote URL fetch will work under current network security rules. I guess I'll need to add a "can connect to anywhere" option, but then will CORS bite me on the ass?)

Evidence that the upload photo/local camera stuff works.

Alt

Let me also add: OMG! That was so easy! capturing from the camera is a doodle! I think I had more trouble with the progress bar (which seems to be pointless given how quick images are uploading, although that could be because I've got the size set to "flippin' tiny")

Interface could do with work. I think I need to flip the front facing (selfie) camera image. I think I also want to make the image full screen (with a bit of interface overlaid), but both of those feel like they're going to be a bunch of work.

I don't think I'm turning the camera off correctly.

Still need to do other image types, and image editing.

I really like the idea of a (simple) image editing script language. It doesn't need much, could be all one expression with a few primitives and basic maths. But it's late and I'm tired so no more thinking tonight.

I lost another entry while I was at the chippie, so I've added code to stop that happening. It uses the visibilitychange event that notes uses to dump a copy of the entry into local storage when the page says it's not visible (e.g., switched to another tab, or another app) and then reloads it at page load time (remembering to turn this stuff off during form submit).

Just tested it here, works great.

Also, images from local disk works (and on mobile too, which also have me an option to use the camera as a source of images. Glad I didn't see that until after I wrote the camera capture stuff)

Two kittens (Havok on the right, Carnage in the left) sat on a windowsill in bright sunlight

I think images are coming out of the upload too small, I need to work on the ui, probably adding a "this image will actually be 6x as big once it uploads" message. Or moving the upload stuff to it's own page and including the edit stuff. (Which will work much better now that entries are stashed automatically). Not sure how to communicate the image id back to the source page (with a query string, dummy)

Oh, and I'm going to turn off the consent dialog for track latest. I'm not tracking people (since I'm using local storage, I don't even get to see if people are using it, let alone where in the feed they are).

On the other hand, I can't remember why I was going to do that. It might be because I'm not asking about stashing entries in local storage, and they're much more likely to hold interesting data (although I'll be very surprised if anyone other than me (and maybe husband) uses the create entry page, and I've definitely consented)

I do want to collect "which entries have been seen" data, but I'm confident that it will be really disappointing, so I'm currently happy just imagining readers. (Hello, by the way, if you are real and reading this. Googlebot/Bingbot, you both count as real).

I think my immune system is kicking off about something. I wish I had more insight/monitoring of my own body.

I need to update the image uploader do that the original blob gets pushed, regardless of the preview. I also want to save a thumbnail at upload time (to img.tumb) as well as the original.

Since I'm going to be running the image data through some kind of processing library, I can also grab mime type and dimensions and stash them in meta data.

I've also been looking in a bit more detail at the Twitter export, and I'm not sure it's worth the effort. I know in my Facebook posts there's some stuff that I want to keep/share, but Twitter posts are lacking context.

A feature that I've been thinking about for a while is executable code blocks. That is, add s block like

var x=2+3;
print x;

And have "5" output in the blog. (Adding variable input, to get input tags is an obvious extension).

I can think of a few questions:

  • Where should the code run (client or server)
  • What language(s) to handle
  • How much access to the environment should the code have
  • Are all the blocks in one post connected
  • Are all the blocks in the blog connected
  • How are errors handled
  • What does securely look like

I've probably missed some.

A simple solution is raw JavaScript running in the browser. JavaScript doesn't have much of an isolation story, so all the code on the same page (i.e., all the code in the blog) world run in the same context.

Client side JavaScript would be easy to implement, assuming that I can convince Markdig to output script tags at the right time.

JavaScript is a little verbose, and therefore irritating to enter in a phone keyboard.

A bit more complicated would be Lisp, or an implementation of Mal. The same code could run either client or server side, it would only have access to the environment that I gave it (Unlike JavaScript, which would have access to the browser on the client), and it would encourage me to learn Lisp.

I'd need to write a JavaScript implementation (oh noes!), and again, make sure that I could get Markdig to cooperate, but it's an intriguing idea

I'm going to call editing entries a dependency of running code, although I should just build a Mal page and copy stuff over when it's done.

Except what I actually want is to be able to create interactive (or at least scripted) pages here on the client. Notes like this are great for prose, but for tools (current example is kitten age), I want to be able to find them without scrolling through history.

What I want, it turns out, is to be able to tag an entry as 'special', give it a memorable link (or slug) and have it show on a list somewhere. Time for a new button!

The interface side is easy enough (all praise to the dialog tag), add a 'post options' button/dialog with inputs for (probably)

  • Title
  • Slug (auto generated from title by default)
  • Include in TOC (bool)
  • Private (Maybe? I've been in two minds about private posts for a while. Knowing that people are potentially reading this is one of the main incentives for writing it. On the other hand, there's stuff that I don't want to share but that I still want to record.)(bool)

The current plan for metadata is a second file next to the entry, either in JSON (easy for machines to read/write, especially to/from c#, but hard(ish, relatively) for humans to read/write) or some kind of key/value map (e.g. Email headers, or TOML. Harder for machines, a little easier for humans, but I'm not sure how important human readable is for the metadata).

A missing metadata file shouldn't be an issue (I don't want to go back and create empty metadata for all my previous notes), and I want to be able to add new properties easily.

Having said that, bulk creating more or less empty metadata files shouldn't be too hard with a little bash scripting.

Let's also add a metadata version number, maybe in the filename, and an entity format version (more for paranoia than any foreseen problems).

That leaves us with:

  • JSON
  • matching name to entry except different extension
  • Property values must all be nullable, or have a defined default
  • Unknown properties in the file are ignored

Month count calculator

Given dates from, to where from <= to, return the number of whole months between the dates.

Test data: from 2022-04-01 to 2022-05-01 is 1 month from 2022-04-01 to 2022-05-30 is 1 month from 2022-04-01 to 2022-04-30 is 0 months from 2022-04-01 to 2022-04-01 is 0 months

from 2022-01-31 to 2022-02-28 is 1 month ; not a leap year from 2022-01-28 to 2022-02-28 is 1 month from 2022-01-28 to 2022-02-27 is no months

from 2021-12-31 to 2022-01-01 is 0 months from 2021-12-31 to 2022-01-31 is 1 month from 2021-12-31 to 2022-02-28 is 2 months

Algorithm

function daysInMonth(date) {
  switch (date.Month) {
    case 1:
    case 3:
    case 5:
    case 7:
    case 8:
    case 10:
    case 12:
      return 31;
    case 2:
     return ((date.Year % 4 == 0 && date.Year % 100 != 0) || date.Year % 400 == 0) ? 29 : 28;
    default:
       return 30;
  }
}
let months = (to.Year * 12 + to.Month) - (from.Year * 12 + from.Month) - 1;
if (from.DayInMonth >= to.DayInMonth || from.DayInMonth >= daysInMonth(to)) {
  months += 1
  }

That's executable code blocks added. Tag fenced code blocks with 'javascript' and that adds a 'Run Code' button (but only on the single article view).

Something weird is happening with the phone browser and css, in that it seems very reluctant to load a new version, which is a shame because I've just updated a bunch of stuff and I wanted to check it on a proper mobile browser.

I've tried to be more consistent with sizes (I was about to say "so a underlines are the same width as borders" but that's clearly false).

I've increased the gap between paragraphs, and added a hr between entries. Yes, that does use up priceless, irreplaceable, vertical space but it also looks better.

I think there were functional changes in there somewhere as well, but I've got no idea what they were.

Oh, yeah Added an 'edit' link for entries (only visible to logged in people). Doesn't go anywhere at the moment, still need to implement the endpoint, but it's a start.

On Editing Entries

I can't decide about history. Do I need to record past versions, either as straight copies of old files , or as a list of changes?

Option one: Don't keep history

Easy to implement, but feels wrong. What if I want to back out a change (easy, just do another edit) or compare old versions? (And sure, but what if you don't?)

Option two: Archive old files

Keep old versions as files. Need to find somewhere to stash them (in an archive folder, or a folder per entry. How many entries/edits per entry am I expecting?) (Lots, potentially, if I do stuff with this live script malarkey)

Also, and both options two and three need this, what does the UI for the archive look like? Are old versions public? Can I (like, should I be able to) compare arbitrary versions? Restore a version? Build up a tree of versions?

Option three: Keep a stack of changes

Keep the delta from one version to the next. Can either do it forwards (start with an empty file and apply changes until you get the current version) or backwards (start with current and apply changes until you get an empty file).

It's technically fun/interesting, and has an "it uses less storage" excuse, but it would be harder to manage and goes against the "as simple as possible, at least for storage" ideal.

Conclusion

Option two currently leading of the two "keep history" options, but I'm still not sure.

Ok, that's not true. Obviously I have to keep history, I'm just not looking forward to writing the code. (I guess that's the attraction of option three, but I'm getting old and the attraction of simple, easy to write, easy to read, sold code is quite strong)

I'm getting pretty flippin' tired of caching issues. Stupid phone isn't showing an updated version after I've logged in. I guess that's on me, I should probably check the docs, but I can see that I'm just going to give up and write my own cache header rules (instead of using the Microsoft ResponseCache stuff).

Anyway. Bedtime. Goodnight dear reader, sweet dreams, see you tomorrow, probably.

I know I said I was going to need, but one last thing: Add a scroll to latest button back in. Now that I've got JavaScript to tag the latest post, adding a latest id should be easy enough.

Still, suggestions on how to tag the latest article server side are welcome.

(Stop messing about with IAsyncEnumrable and pull all the entries into a list. Or, y'know, do it right and generate the HTML at save time.)

Early morning whine:

  • Bad nights sleep so feeling crap anyway
  • Work priorities have changed so Ocelot is no longer drop everything crash priority, although finding that out has cost so much time that the next project only has (maybe?) a week.
  • Stupid k8s auth broken again so I need to reboot (looks like that hasn't actually fixed anything)

It's all trial trivial, so I'll work on shaking it off.

/me makes an "I've just worked out out" noise.

I want to add a page to edit entries, which will share most, but not all of the HTML/layout of the create entry page I don't want to add a bunch of if statements since that would complicate/untidy the code. I also don't want to make a straight copy because that will increase the maintenance burden.

This is the problem that partial views exist to solve.

I need to update the menu/navbar stuff to stop the current page being a link (and maybe to remove the 'New Entry' link completly from the new entry page.

But decomposing imgDialog into it's own file has worked, that's going to make the edit page much more simple, and much cleaner.

The Lion of Longtrees

Leo is a professional bodyguard. He works usually works medium term contracts (weeks or months) but is open to negotiation, at least until the contract is signed. Once he's signed on, he'll work the contract as agreed until it's completed come hell or high water. This single-mindedness has given him a useful reputation, and a couple of enemies.

He's trained in a couple of styles of unarmed combat (and keeps up regular training). He prefers knives over guns, but knows that ignoring guns would be giving too big an advantage to his opponents. He's a fair shot with a rifle, and ok with a pistol (but is likely to just drop the firearm when it runs out of ammunition, unless he's got a specific reason to keep it).

He's generally friendly and approachable, good with people. Of course, he can put on a blank face when he's working, and will semi-consciously use his bulk to intimidate people out of starting anything with his current client. He keeps up to date with world news and the local sports teams specifically to carry on small talk while scanning the crowd for trouble.

He's had lessons in slight-of-hand tricks, and again practices to keep current. He can entertain and distract a client's child (or a client) with a cup and ball, or pass through airport security with enough of an arsenal to hijack several planes.

Initial scroll to needs to wait for images to load (or I need to add sizes on to images, which is probably the 'right' solution)

I've been thinking about how to add pre- and post- condition checks to (C#) functions.

Roughly, preconditions are conditions that should be true at the start of a function, and post conditions should be true at the function exit. e.g.:

/**
 * Return x divided by y
 */
public int Divide(int x, int y) { ... }

has a precondition that y should not be zero.

It's easy to add these kinds of checks to a function:

public int Divide(int x, int y) {
  if (y == 0) {
    throw new ArgumentException("y must not be zero");
  }
  ...

but a) they're not obviously condition checks and b) it might be nice to be able to turn them off in Release mode. (Potentially also c) they could be extracted for a test suite)

I've been thinking in terms of Attributes, but C# doesn't support Lambdas in Attributes, and probably won't any time soon. However, a new idea arrived today:

public int Divide(int x, int y) {
  Contract.Pre(y, q => q != 0, "y must not be zero");
  ...

With a definition something like

[StackTraceHidden]
[Conditional("Development")
public void Pre<TParam>(TParam value, Func<TParam, bool> test, string onFailMessage) {
  if (!test(value)) {
    throw new ArgumentException(onFailMessage);
  }
}

it would transparently (i.e., not showing in a stack trace) run the test in development mode, and not even be included as a call in Release.

I imagine that it would be worth adding some overrides (or sibilings) for common cases, such as Contract.PreIsNotNull, Contract.IsOfType. (Alternativly, include a bunch of predicates in the library).

Of course, the next question is: What do I do with this idea?

It's clearly a stand alone library. Maybe it's time to get a nuget.org account.

More plumbing/infrastructure work on the blog this evening. Post meta data is now written to a separate (JSON) file next to the entry. This makes it much easier to add stuff to meta (e.g., version number).

I've also swapped out IAsyncEnumerable for Task<IEnumerable>. I was having trouble working out when things (like reading the entry from disk) were actually happening, and worrying that they were happening more than needed.

Instead, BlogService.GetEntriesAsync pulls all the meta files, and then individual entries pull (and cache) content.

Two thoughts from writing that last sentence: Is it worth running the load metas in parallel? And I should look at the proper in memory cache for entries (including content) and do things like invalidate the entry if/when edited.

I should benchmark the load stuff, and maybe set something going at process start time to pre-fill the cache.

(On the other hand, I actually want to get editing working and this is feeling like a distraction)

Editing entries :tick:

Also, accidentally deleted post button. Oops. Fixed that too.

Tidied up the logout code.

I'm an experiencing a level of lower intestinal distress at the moment, my insides are not happy with something.

Otherwise another day at $work. Mostly messing with k8s and remembering to look busy. The deploy script now uses kubectl set image instead of rollout restart. This guarantees that the latest image is pulled, and updating the deployment triggers a restart.

I was arguing for a very aws solution to a problem (messages come in, need to be recorded and aggregated) using queues, lambda, and dynamodb, but we're getting a hit every 45 seconds on average, it's really not worth the effort. Spin up an SQL db and move on to the next thing.

Ah, well. We'll see how that goes.

Added ImageSharp so I can start poking uploaded images. As a smoke test (and to see how working with the Markdig ast works), I'm now adding width/height attributes to img tags.

Otherwise, that was a fairly good day, I think. See y'all tomorrow.

Rubbish day at work, failed both of the projects I wanted to get working and it's not obvious why they didn't work.

Building a simple 'what's going on with my cluster' app but it's not authenticating itself correctly. It looks like I need to give the 'default service account permission to list pods' but I'm holding off on doing that until I feel that I understand what it's doing properly. (Also, I'm not convinced I've got the authority to do that, and I don't want to get bogged down with that).

The other problem is setting up an existing app into the nonprod cluster. It was working there before and now it's not. I have changed a bunch of stuff, so really the next thing to do is to go through the config/setup carefully, step by step, but again, I just can't bring myself to care enough.

That's the problem. No-one on the team really cares any more. I don't get any good feelings from solving these problems because I don't get positive feedback from doing so (and I don't really get negative feedback from not solving them).

I don't like it. I would like to feel more positive about work and works problems, but something fairly big would have to change first. Maybe I should be looking for a new job (although I'd miss seeing how much they'd miss me when I'm gone).

Just finished reading the third Murderbot Diaries "book" (in quotes because it's a short story, but charged at (high) full book prices), and it was as consistently good as the first couple

I've been thinking about showing dates as year-day of year, but I'm not sure. I've just had a "well why don't you go the whole way and u show Unix time!" thought that's not very helpful.

I think it's about showing how far through the year it is. I'll think about that as a separate page, although getting fp ported over is current priority.

If I remember right, I've pulled webmail out into its own project, integrated the shared stuff from common and webmail.content. Time to fire it up and see what's broken.

I'll need to pull a copy of the db from legacy and setup a spare domain for testing.

The more i think about this blog thing, the more I want to stick it into a database and stop messing about. The "keep it in files so that I can recover it for next time" stuff is interesting, and i should probably add an export function, but if I use sqlite i should be able to do a dump anyway.

Actually, that export data idea sounds better every time I think about it, especially after having got data exports from a bunch of different places (Amazon, Netflix, Facebook, Twitter) this week.

However. Today is about webmail, and getting out of Bytemark.

wepiu has the basics installed, but not configured. It's got a vmail user with a nearly empty home dir. It's also got a webmail user, I guess I'm planning on using that one for the webmail?

Ok, so webmail doesn't build. Last time I'd been deleting "stuff I don't need anymore", and I've been a bit over the top. Or possibly not deleted enough. Looks like some of the 'Friends' stuff is still hanging about.

find . -type f -name "*.cs" -exec sed -i 's/using Common/using Webmail/' {} \;

Who says you need an ide!

(Having said that, there are times when they're useful)

Bash arrays

Based on this fine article

arr=() 	Create an empty array
arr=(1 2 3) 	Initialize array
${arr[2]} 	Retrieve third element
${arr[@]} 	Retrieve all elements
${!arr[@]} 	Retrieve array indices
${#arr[@]} 	Calculate array size
arr[0]=3 	Overwrite 1st element
arr+=(4) 	Append value(s)
str=$(ls) 	Save ls output as a string
arr=( $(ls) ) 	Save ls output as an array of files
${arr[@]:s:n} 	Retrieve n elements starting at index s

Note: JavaScript doesn't have a native day of the year function, so I'd have to write one (based on the number of days in the month code above) if I wanted to do anything client side.

(Alright, I'm wrong. Subtract midnight 1st Jan from your date to get ms since the start of the year and divide by (1000*60*60*24) to get days)

Incomplete idea

New webmail idea: Save inbound emails as parsed html as they come in. have a background thread (per user) that's connected and listening for emails.

Listening per user isn't fantastically efficient, but since it's only the two of us if should be fine.

(on a larger scale I'd probably have something on the end of a pipe, but I'm going to ignore that for now)

What do I need to save?

  • Headers (At least from, to, date, subject, maybe type)
  • Structure (From a template based on the content type. Recursive for multiparts)
  • Content (The body and it's various parts)
  • The original message (so I can see the 'raw' version)

Assuming all this stuff is going to also be saved by the mail system, I don't need to be able to round trip it. Folders need to be updated on change.

I should check the stats, but my feeling is that it's probably not worth it. A good chunk of mail is never opened (See: spam management), and updating folders leads to irritating locking issues (that dovecot also has but has already dealt with).

It's a cute idea, but unfortunately not worth it.

Flipin' machines trying to be smart at me! I expected

egrep  '^[a-z]{4,5}$' $WORDS > wordlist.txt

to match 4 or 5 character words that conain only the characters 'a' (U+0061 LATIN SMALL LETTER A) to 'z' (U+007A LATIN SMALL LETTER Z). But nooo, someone is being clever with their locales, and I get words like élan, which, if you missed it, contains 'é' (U+00E9 LATIN SMALL LETTER E WITH ACUTE) (assuming it hasn't been decomposed).

Fix:

LC_ALL=C egrep  '^[a-z]{4,5}$' $WORDS > wordlist.txt

(Roughly translated means 'Ignore anything that happened after about 1963 and just give me ASCII)

(I should probably take the word list with the funky characters, since I'm using it to generate passwords. On the other hand, I think I should probably stick to ASCII, since I'm using it for passwords...)

Sunday morning (yes, the previous entry was also Sunday but I hadn't slept yet so the day had not ticked over)(Or it wasn't dawn yet). Nothing much so far, finished the last/latest Murderbot Diaries book, that was a brilliant series, strongly recommended for people who feel habitual dissociation/disconnection from humans.

Current plans: Grinding towards moving fp mail to mythic, maybe get out pillow shopping, hopefully a quiet/low stress day.

Hugs to y'all.

Dump webmail DB:

sudo -u postgres pg_dump \
	--clean \
	--if-exists \
	--create \
	--schema=auth \
	--schema=webmail \
	--quote-all-identifiers \
	--dbname=moose \
	--file=moose.sql

So I've copied the fp db over to wepiu, not sure what's next. I need/want a 'better' todo/issue tracker, that can track dependencies and maybe priory to show me what's next.

Basic data seems obvious: text, id, dependsOn[], isComplete

Is that enough? Let's take set up db as an example

"setup db", 1, [2] "install db", 2, [3] "Add db repo to sources", 3 "Copy data/schema from old db", 4 "Export old db", 5 "Tidy export", 6 "Import data", 7 "create db user" ...

I'm tempted to add a 'command' property (probably with a 'runAs' sub property) to record/automate the process.

Is it worth it? It would be nice to be able to pick up and put down the process as and when, without losing track of where I am. On the other hand, updating the list is another job (Ignoring the whole "writing the software in the first place" issue)

(And now I want to add 'verify' and 'undo' commands (although commands up the tree/graph should cover lower levels))

Anyway. I think I'm going to config servers before getting the website up, so that I don't need to test the website against live data.

Moved moosemorals.com from bytemark to mythic. There's nothing hosted there yet, but I'm going to use it a the domain for testing webmail.

Mythic are generally great, but they're charging me £15 to bring the domain over. (It counts as a renewal fee, and the time gets added in, but it's still irritating).

I've been going through the postfix config ready to turn the daemon on. Biggest change do far is setting up a separate submission port, so the only mail expected on port 25 is inbound to one of the hosted domains, which is simplifying a bunch of options (e.g., mail that has come in on port 25 that isn't for a hosted domain can be rejected early).

I'll need to do something about outbound mail from webmail. I think the current solution is to just trust the local machine, which is reasonable, but I'd like to do better. (Something like password auth with a one shot password that's validated by postfix calling back to the webmail. Time to look at Lua again, I guess)

Bug reports and what to do with them

At a minimum, a good bug report contains the answer to three questions:

  • What did you do?
  • What did you expect to happen?
  • What actually happened?

This doesn't just cover software, it's a reasonable place to start any fault diagnosis:

  • What did you do? I tried to open the door.
  • What did you expect to happen? The door to open.
  • What actually happened? Nothing!

(Although you will receive reports with more data than this, well written, detailed reports will be the exception.)

(You probably have an intuition already about what the problem is. Learn to distrust that intuition, or at least treat it with scepticism.)

The first step in diagnosing an issue is to make sure you understand correctly what the issue is. People often missrepot the issue for various reasons; they don't understand the technology ("The TV remote is broken" > the remote needs new batteries, is not being pointed at the TV, is actually the remote for the hi-fi), they're trying to be helpful ("The TV wasn't responding to the remote so I took out all the cables and wrapped the ends in tin foil to help them conduct and I've put them back in and now the TV is on fire and the remote still doesn't work"), or they're worried about looking stupid ("The TV remote in the demo room isn't working, fix it before the clients turn up in 5 minutes and stop bothering me with stupid questions").

You are going to need to ask questions, you will need to ask her basic questions that could be interpreted as insulting ("Of course it's plugged in, do you think I'm stupid"), and you will probably need to ask the same question more than once as people helpfully answer a different question.

Looking at our door example, try to establish why they are trying to open the door? Is it a door they go through often, or is this the first time? At this stage, you're still looking for context. Again, try not to think of solutions at this point, or even causes. The first step is to establish what actually happened, ideally well enough that you can reliably trigger the issue locally.

I say ideally, but it's very close to essential to be able to replicate the problem at will. If you're dealing with software, this is a good time to write a new test case. Write enough code to trigger the issue, and then start taking code away from your test until you get the smallest reliable trigger. This exercise has two aims. First, writing the test case should help find the rough area of interest in your source. Second, having a reliable test case means you can be confident that you have fixed the issue! Without a test, you can't be sure that your 'fix' has worked, or even fixed the right problem. With a test, you can apply the fix, run the test, confirm the issue doesn't reoccur, and then remove the fix, rerun the test and confirm the issue comes back. (Also, including the test in your automated test suite (you do have an automated test suite, yes?) makes it harder for someone else to reintroduce the issue later).

Once you understand the problem, have isolated the issue, built a test (or series of tests, don't hold back here), written and confirmed your fix, this is a good time to look though your codebase for similar patterns (or exact duplicates) that you can fix at the same time. (It's great to have users report bugs, but it's far better to not have bugs for them to report)

Looks like I can use IControllerModelConvention to manipulate controllers at app start-up time, to add hostname to routing (to set a default hostname).

Error page ideas

These are ideas for the static (well, ssi maybe) pages from nginx. If possible, check for a body from the backend and use that, otherwise send a self contained static page (light on the css, no js).

Use the 4xx/5xx split ("It's not me it's you"/"It's not you it's me"), a custom description of the specific error, and advice on what to try next. Include a transaction ref (maybe).

400 - Bad request - "You've done something so generically wrong that I can't tell what it was." 401 - Should be issued by a backend with content, can't do much with it at the proxy. 403 - Forbidden - "Your provided credentials do not permit access to this page. Please wait while I summon a security enforcement team to your location" 404 - Not Found - "You asked for a page that isn't here. This page is here instead." 405 - Method not permitted - "You can't $method this page!" 413 - Content to large - "You've sent more data than that page can cope with"

500 - Server error - "I think something has wrong gone." 501 - Not implemented - "I don't even understand the question." 502 - Bad Gateway - "I asked a friend for the page but they've let me down." 503 - Service unavailable - "I am far to busy to talk to you" 504 - Gateway timeout - "I asked a friend for the page but they're taking to long."

I wonder how much trouble it's going to be to get nginx to pick a random line from a set.

Now running with the new Nginx setup.

I tried putting all the site configs in one file and using the $host variable to pick the right port (to proxy to) and certificates. Didn't work.

Instead, now I've split out the config into two include files, and a file for each domain/site. The per domain files just set the server name (per the Host: header), the proxy port, and the path to the certs, and then include the http include file (with the standard redirect) and the https include file (with all the other (mostly proxy) settings).

It's a shame that Nginx doesn't have a configuration level between http and server, to group config for similar sites, although using include is working for me here.

Last thought about this - I should be able to use the $server_name variable in the https include file to make the path to certs, so the only per site config is the server name and proxy port. (The point would be to minimise the chance that I'm going to forget to change a name when I next setup a site).

(Ok, I said last thought, but husband's site is static and can be handled by nginx on its own)

The wemail refactor is going ok, I think.

Webmail runs behind two domains. One for the content of emails, and one for everything else. This is to take advantage of the same origin rules, so scripts in emails can be effectively isolated from the main site.

The previous version of webmail ran as two separate dotnet apps to serve these two domains so that requests to the content domain couldn't 'leak' into the interface domain. (And also because, frankly, I didn't know any better.)

However, Asp core can route based on hostname, which gives enough of an isolation guarantee that I can fold the code that serves content into the main app, and handle both sets of requests.

I have to tag each 'endpoint' (roughly, each action method in the controllers) with a hostname, either the content domain or the interface domain. There is an existing HostAttribute, but (since it's an attribute) it can't be set at runtime from config, and I don't want to set it on every controller in case I forget and miss one.

This is where the IControllerModelConvention from a few posts back comes in. I've added a call in Startup to add a convention that loops through every Action to add a host to the RouteValues collection, either the interface domain (default) or the content domain (if the action is tagged with the right attribute).

(Strictly at this point some of that isn't true. I've set up the convention and I'm looping through the actions, but I need to create an attribute and add it to the appropriate action(s) (although i think there's only one). Still, it should work).

This is a bunch of work but it's worth it, mostly because it's one less server to run, manage, and allocate resources for. (Also, the two servers used to communicate through the db, and that might not be needed, but fixing that isn't so big a priority).

Anyway. I'm feeling good about that whole thing.

One of the (very many) things that's bugging me at the moment is part of the code for the Webmail server.

The code that sends the actual content of an email to the client boils down to a call something like:

Message.Part.WriteTo(HttpResponse.Content)

(the names are all wrong, don't worry about it)

The problem with this is that I can't take advantage of the frameworks tools to send files to clients. (Where 'file' in this case is exactly equivalent to a 'stream').

I could, of course, save the content to memory/disk (based on some size huristic), but that would cost both storage space, and time. I'd therefore prefer some kind of streaming solution.

I'm still thinking about it, and I'm not sure it's worth the effort.

I'm pushing myself to hard again. It's the weekend and I should be relaxing but I've got a brain full of "got to do stuff!".

I mean, I'm not actually doing stuff (ish, I got some work done on webmail this morning, put away yesterday's washing, put in another load, organised Morrisons for tomorrow and drove us both to Asda for s few things).

What I mean is, I feel bad when I just sit (or lay down) and relax. I don't get to relax, I keep thinking about all the stuff I 'should' be doing (although not in any concrete way, just the whole "I should be doing something" feeling).

So I'm going to stop here and get some sleep.

TODO

Reality

  • Husband wants help dying their hair

Infrastructure

  • Decommision postgres (see osric.uk)
  • install nginx error pages
  • Move husband's website

Blog stuff

  • Move controls to the left bar if there is space (where 'enough space' is defined as the screen is wider than the current --max-width)
  • Add a calendar with links to posts
  • Add forward/backward links to individual posts
  • Add a 'save all to zip' option

Webmail

  • Get dovecot configured
  • Get postfix configured
  • Turn off the Server sent events stuff, for now at least
  • Get the host security stuff working Now that I've tagged endpoints with RouteValues[host], requests to the 'wrong' host should 404 (not 401/redirect to auth)

Osric.uk

  • Move database from postgres to sqlite, decommission postgres

Gah, I forget that restarting the server logs me out.

Husband's dyed their hair, I've sorted nginx error pages, and I've moved the blog menu to the left on wider screens.

I want to add a calendar to the left bar, that shows the current month, except it updates to show the right dates for blog entries.

Clearly it's just a gimmick, probably not worth the effort.

Image service

Google is warning me that I'm running out of photo storage space, so it's time to get serious about pulling my images out of Google photos and into some kind of useful local service.

The hard part, for me, at least, is designing a useful and plesent to use interface. I can lean on other designs, but it's still going to take a bit of work.

A specific feature I want is a 'Tag/Describe a random photo' page that loads a photo at random (either from all photos, or from untagged photos) and lets me set the tags and/or description. (Exif has an 'Image Description' field that's probably the right place to start).

The move from postgres to sqlite for osric.uk is ready to push. I would have done it this evening, except I was watching the final episode of The Orville series 3 which was just awful. I've been pleasantly surprised by most of the season, it's been a good, modern sci-fi TV show. The scripts have been a little rough/loose in places but the technical side (effects, camera work, editing, etc) have all been good and the stories have been interesting (if a little poorly executed at times). And then we got this episode. Two (really unconnected) stories, the robot crewmember propsed to his girlfriend, and the reasonable questions about "what does love mean to an emotionless machine" were skated over in favour of "here's some funny bad advice got the comedy robot". (The other story was a stab at "this is why we don't interfere with primitives", it didn't add anything new to the genre) I couldn't take it. I had to bail 15 minutes from the end, which (for stupid logistical reasons) meant I've shutdown my development environment for the night. An, well. The code will still be there tomorrow.

Want/need to get hi rez image uploads sorted, along with basic image manipulation tools.

I should benchmark image sharp to see if doing transforms on the fly is viable (along with finding out how it handles rotate - is the canvas resized automatically, what is the background fill it it's not).

I also need/want to put the design time into operators/language/macros, and an interpreter to parse them into image sharp calls.

Image Operators

Requirements

  • Must comfortably fit in a query string (avoid blocks, avoid whitespace significance)
  • Must be constructable from a push button interface (can use HTML5 inputs with minimal styling)
  • Should be constructable by a human with a text editor
  • Should be forgiving of technically invalid input (sensible defaults, clamp ranges, only invalid if contradictory, impossible, or conflicting)
  • Should do as little work as possible (e.g., spot an operation followed by it's inverse is a no-op and skip both)
  • May allow users to define macros/functions/shortcuts for common operations
  • May allow more than one output (e.g., rotate original and save as full size and thumbnail)
  • Must preserve original input
  • Must use consistent addressing (i.e., either top, left, width, height, or top, left, bottom, right)
  • Should allow percent as well as pixels
  • Should allow calculations (e.g., width/2)
  • May allow user variables (easy enough)
  • Should have useful set of system variables
  • Must apply operations left to right
  • Should report syntax errors with a location

Useful operations

  • crop(top, left, width, height)
  • resize(percent) // maintain aspect ratio
  • resize(width, height) // change aspect ratio
  • rotate(angle) // +ve clockwise, 0 pointing up

Implementation ideas

This is sounding like a basic expression parser. I'm not that worried about efficiency, so tokenize->ast->interpret should be fine (and I can always add caching later at various points). Tokens will be basic maths (including decimal (not floating point) numbers) ('+', '-', '*', '/', '(', ')') and identifiers/keywords (I'm not sure about idenifiers that aren't keywords yet).

I certainly want comments, so i can comment out chunks of code (How complicated are you expecting your stuff to get!), and really any pair of characters will do.

Time to have another flick though Crafting Interpreters, I guess.

It might be too hot again.

Otherwise, today was a good day. I'm writing a k8s monitor app at/for work and I think I've got the "watch" stuff cracked so I can do things like leave a page open showing events and pod logs.

Husband's dad is coming up for the weekend, this weekend (eeek!). Husband's has got a fairly complete itinerary planned, but I'm still going to need to interact with the honoured elder.

But it's late and I'm tired, so good night dear reader(s), see you soon.

(Note to self: Anonymous usage stats aren't immoral)

I'm tagging the blog software as version 1, it's time to think about version 2.

Blog - Version 2

Damnit, it's really irritating, but I think I'm going to have to put everything in a database. The logic chain runs something like:

  • I want to (algorithmically) add everything that I can to the blog. What I played on Spotify, what pushes to GitLab, what I asked Google, the photos I take, maybe even the emails i send and receive, all the online stuff I do should be an entry on the blog.
  • That's a lot of stuff, and I should keep track of its provenance (? source, context)
  • Some of it is personal/offensive/immoral/illegal (depending on context), and so needs to be tagged as such (and not shown to people who don't have the right auth)
  • I want to be able to comment on automatic posts ("So this is a photo of me and husband on holiday") (and probably manual posts too - see Bernice Summerfield for examples), and probably have replies to comments and comment trees
  • I want to save copies of posts before they're edited.

That's a bunch of metadata on top of the actual data. I could keep it in files, and I'm quite tempted by something like a Mine message.

Digression - Keep it all in files

Sqlite is very nice, and likely to be available for the foreseeable future, but there's something about text files that suggests a higher level of permanence.

Somewhere above I wrote about using IMAP as a backend, but does the idea because IMAP says messages are immutable. However, I also dropped the whole "Use rfc822 formatted files for storage" thing, and that may have been a mistake (or at least, premature).

(Note: I can't remember the current version number for rfc822, it's somewhere in the 5000s, I think. Please read 'rfc822' as 'The current internet mail format rfc' unless otherwise stated)

Ignore the email heritage of rfc822. It gives a structured way to add metadata to a file (metadata at the top of the file as Key: Value pairs, a blank line, and then the file data). Add in Mime, and there's support for keeping several files together (e.g., a post and photos (or other attachments), a post and it's edits, a post and it's comments).

There are plenty of independent tools to create, read, and update mime messages (delete is easy), and I'm already familiar with MimeKit for C#.

I would need to worry about simultaneous access/locking (but I could manage that from within the app). I might need to sort out indexing, although I do like keeping info in the filename.

Conclusion

Add always, add another layer of abstraction. Write an IBlogStorage interface with operations for get and save entries. Move to an

I begin to see why people advocate so hard for interfaces. I'm starting to think in terms of an IEntry with operations like Get Content(Version?), SetContent(string), AddAttachment, AddComment(Comment? Parent), that I can write different backends for.

(Poot. I think "storage layer" has gone out of fashion in favour of "database layer". Entity Framework makes too many assumptions that it's underlying storage is sql, and it's got into my brain. Ah, well I'll cope).

More thinking later, it's hot and late now. See you in the morning, dear reader.

I had a bit of a play with ASP Core Areas at $WORK today, and they are the right solution to the "I want this project to have a bunch of loosely connected modules that aren't worth putting into different projects/processes" problem.

Adding it to a project is easy enough, create a top level Areas folder with a sub-folder per sub-project/module/area.

Each of those sub-folders has it's own set of Model/Contoller/View folders/types, and the framework will find the right view, so long as each controller has an Area attribute.

Di config can go in program/startup as normal, or one can try something a bit more fancy. (The demo at work has an IStartup interface with a method that takes an IServiceCollection. At app start time, all the implementations of the interface are found, instantiated, and called. An alternative that's been in my head for a while is to tag types that should be available though DI with an attribute, although I'm starting to prefer the interface way since it's much more flexible.

Anyway. Have I already said goodnight tonight? Goodnight readers, sleep well.

Write ahead logs

Roughly, when the storage engine gets a request that would change a file, write the details of the change to disk before doing the change, so if the change is corrupted then we can tell by comparing what should have happened (the log) with what did happen (the real files).

I'd be tempted at that point to go full Event Source and treat the log as the source of truth, and read state at boot time.

Or, not waste time badly implementing a database and use a real database instead.

Maybe it's entity I don't like? Maybe I should go back to writing the raw sql myself (except entity is really convenient? It's so nice to be able to more or less ignore everything lower level than sets of types. Pointing at moving from postgres to sqlite as an example, none of my logic changed (and ok, it's all fairly basic). Maybe I should move the blog to a db).

Blog Schema


Entry
  Id
  ->Owner
  ->Content
  ->Comments []
  ->Versions []
  
Content
  Id
  Blob
  Created Timestamp
  ContentType
  Size

Comment
  Id
  ->Content
  ->Parent Comment?
  ->Parent Entry
  ->Child Comments[]
  ->Versions []
  ->Owner
  State {Unreviewed, Ham, Spam, Flagged}
  ->ReviewRecord[]

Thing is, it's very easy to get carried away with database schema design. Do I really need to keep a log of who reviewed a comment (and when), when I'm the only person with an account?

Commands, Queries, and Events

Commands are instructions ("Create an entry"). Events are a report on something that happened in the past ("An entry has been created"). Queries ask about the state of things ("What entries exist?").

Somewhere there's an engine that convents commands events. We also need a storage system that can listen to events and answer queries. Finally, there must be some kind of system that generates commands, send queries, and processes the results.

I think I'm overthinking again

That was a little unexpected, but I've moved production from Postgres to sqlite. Next time, I should try to remember what's been merged when I push to prod.

Now I've got the by date view working, previous and next become more pressing.

I need to make a choice about semantics: Does previous/next always refer to entries, or on (e.g.) a day view, does it mean the previous/next day, even if there aren't any entries.

Ignore that, it's clearly about entries. Probably once I add in other data sources, there's going to be entries every day anyway, but I might as well skip empty days (and stop at the date of the earliest entry). (A reader who decodes the URL format is welcome to load a blank page, but if I add auto links to the infinite past, some stupid spider will blindly follow them.)(serves it right)

Datasource

The word 'Datasource' in the previous entry triggered a minor epiphany. It's exactly the right term to use for the various components I want to add (Google Photos, Twitter posts, blog posts, etc.). If I can come up with a fairly basic set of common operations (get Name, getEntries(from, to)) for datasources, and for entries, then I can wrap everything in a couple of interfaces, and maybe pull some DI tricks at start time to dynamically include known sources.

I'm not sure yet what impact that will have on the existing blog code, especially creating entries. (I think I also want to tweak my language to mark the difference between a post on my blog, and an entry in the 'lifestream' (urgh, not using that name).

And ok, clearly since the blog is just another datasource, it won't need any changes. Instead, what I need is a new page that ~~gloms together~~ aggregates the various sources and displays them.

Requirements

First draft, off the top of my head, blah blah blah.

Entries

  • Entries draw themselves. The environment will draw a header (the time the entry was created, the source, and an edit link if the entry is editable), and the entry supplies HTML ready to add to the output stream (i.e., any user sourced markup is either escaped or very well sanitised)
  • Minimum data set:
    • Creation date
    • Source (link)
    • Permalink (a link to the entry in isolation, but still hosted here)
    • Body/Content
  • Under consideration
    • Something like a big vs small flag, although they might be per source (bookmarks and Spotify plays are small, photos and blog posts are big, although blog posts can be small)
    • IsFirst/IsLast/Next/Previous (to help the renderer, e.g., pull all the previouses, get the most recent, and that's the date for the previous page)

Datasources

  • I really want everything on one page, but there might be alot of data once I start pulling in photos (and bookmarks). I could default to current year (or even current day), but I really don't want to.

Alright, looks like I'm too tired to carry this on. See y'all later.

Another one of those 'just want to sit here and cry' mornings, although I don't seem to be able to actually cry anyomore.

Depression sucks.

I have been writing software since 1981, when Dad brought home a ZX81. Since then I have used more than a dozen different programming languages, and I still enjoy writing code and using software to solve problems, whatever the language.

I have been SC cleared for about four years, from when my current role within HMRC started. Nominally, the role is 'Front End Developer', but I have been leading on a range of software projects, writing full stack code from the HTML/JavaScript/CSS front-end, though the .Net/.Net core C# MVC web tier, past the Entity Framework/Entity Core database access later, down to the Dockerfile and bash scripts to automate the build.

As an example, a couple of years ago, HMRC needed to update 'COBRA', a Microsoft Access application that was used to track the near-real-time flow of money into and out of the department's bank accounts as customers make payments to us, and as we make payments to customers. The outputs from this app are sent to the Treasury fine times a day to inform them of how much cash on hand the government has.

As a Microsoft Access application, the previous version was restricted to a single user. This didn't fit very well with the high-profile nature of the application, and I was asked to rewrite it as a high availability solution.

I designed the solution around an AWS RDS MSSQL Server instance running in Multi-AZ replication mode. After that, it was a fairly standard C# .NET Framework MVC application, duplicated across two EC2 frontend machines (again, split across two AWS availability zones for redundancy).

My current role is split between development and system administration. The sysadmin part of the role covers creating and maintaining the CI/CD solution for the team. We have been using a self hosted install of GitLab as a git server and job runner, along with copies of Jenkins installed on our EC2 instances to take care of actual deployment.

Most recently, we are in the process of moving to the departments hosted Application Lifecycle Management (ALM) tooling; including their GitLab install, Artifactory, and Vault. We are also moving to a Kubernetes (K8S) cluster, so I have been updating our build pipelines to use Docker to make container images for deployment.

Our teams move to .NET Core has helped the move to containers, as ASP Framework apps only really work well under Windows. However, the move to ASP Core on Linux container has meant that we are using Oauth 2 against Azure Active Directory for authentication/authorization instead of Kerberos and local Domain Active Directory.

While the team owns our main project, we are called in to help other projects, since we seem to have a reputation as a team that works quickly and well. For example, last year I was working with the Inteligent Payment Project (IPP) to build them an MVC application to capture data from front line users and submit it to the projects API. I worked with the architect and the product team from early in the design process to make sure that the form I was building was as simple as possible, by avoiding asking the user for information that wasn't needed by the backend process, or that could be synthesized from other answers.

Working as a front line contact center advisor has helped me develop many things

  • An understanding of the customer point of view
  • The ability to explain complex technical ideas to people from different backgrounds

2005-2017 Call Center Advisor (HMRC)

As a front line call handler I took calls from the public, answered their questsions, and updated systems in line with department policy.

Working in this role helped me develop the skill of explaining complex technical ideas to people with a wide range of knowlege.

2017-2018 Guidance Author (HMRC)

I was promoted into a role with the Guidance team, who are responsible for writing and maintaining HMRCs internal guidance for front line call center advisors and other process workers.

As part of this role I moved to the tehcnical side of the team after updating some ASP Classic Visual Basic to run more efficiently, replacing a runtime of 30 minutes with one of 2 or 3 seconds.

2018 - Current Guidance Development Team (HMRC)

Another promotion lead me to my current role. I am working as a developer and system administrator with the Guidance Development Team. We build and maintain the software that hosts the internal gudiance.

That's NUglify integrated into the site. I've got a new Middlewhere that minifies js/css files, and stashes the minified copies in memory.

I'm thinking about bundling. I'm also thinking about a cleverer cache that will flush to disk every so often, but that's a component in its own right (see also: Image transforms).

Bundles take a list of files and concatenates them, to reduce the number of requests for a page. It's not so critical now with HTTP 2 and 3 reusing the connection, but there's still overhead from headers.

The main problem I've got with bundling is that I want to be able to specify a different list of files for each page, and the best way to do it is not obvious.

The page needs to communicate the list of files to the bundler, and it needs to do it via the browser, since the server doesn't know which bundle request goes with which page.

Having said that, I'm settling on a 'just stick the list of files in the URL' approach, probably using a tag helper to convert something easy for humans to type and maintain into something that's easy for the machine to parse at the other end, probably as the query string.

Service workers

Up till now, I have mostly ignored the activate event. This is a mistake. The activate event signals that the service worker code has been updated, and so I might want to invalidate the cache and repopulate it.

Other important notes

  • The page that calls register doesn't get the service worker! The page has to load through the service worker before it will use the service worker.
  • A new version of the service worker is installed if/when the fetched version is different
    • This implies that we should be careful loading the sw from sw cache
  • The new version doesn't start taking events until all pages using the old worker have closed
  • Refreshing the page doesn't count as closing! You must navigate away or close the tab. (Irritating from a debugging point of view)
  • force-reload (shift-reload?) bypasses the service worker

It looks like we're expected to use a different cache name with each different version of the sw script, and delete old caches from the activate event.

Todo: Work out how to programmatically set the cache version, ideally using either the hash of the sw code, or the slug of the timestamp.

Testing ASP Core apps

Prompted by $WORK, it's time to look properly at how to do end-to-end testing of ASP Core apps. I'm going to use Webmail as my example, mostly because it's got at least one heavy runtime dependency (dovecot).

My understanding of modern practice is that I should be running my test in a container, or possibly more than one:

  • The app itself
  • Dependencies (dovecot here)
  • The test runner

Running in containers gives me a known starting state every time, assuing I can get my containers setup right.

Having said that - I've got most of the script to build the machine anyway, is it going to add much more to the runtime to spend an extra 10 seconds starting a VM compared to a container? I suppose it depends on how long the tests take to run.

More data: Podman runs on WSL!

OK, time to write some Containerfiles.

Just had to login again, I'm trying to remember why I restarted the server. It might have been logging stuff?

New todo: write a reader for systemd journal.

I've been moving webmail over from Bytemark to Mythic Beasts, and it's nearly ready, the only remaining problem is that the Spsmassassin integration in using (spamass-milter) isn't correctly picking up the name for the destination mailbox.

I think that the problem is that it runs to early in the processing pipeline, before postfix has done the alias lookup Allegedly, it can lookup aliases at process time but it doesn't seem to be doing that.

The alternative is to use spampd, a wrapper around SA that acts as an LMTP proxy, and is therefore well past alias translation.

Of course, there's still a problem! spampd doesn't use SA's per user preferences. I hacked the previous version to do so, but the author rejected my patch as they'd just done a rewrite. Find to see how much work it's going to be to fix it, I guess

Squeeee! I'm fairly sure that was the last thing that needed fixing/looking at. I've already tested sending mail (as part of dkms), and that's all setup and working (with a script that can generate new dkms keys every month!).

Just need to move over actual emails and update dns, then I can get the old servers turned off. Eeek.

I kind of wish I felt worse about moving away from Bytemark, but they moved away first when they sold themselves to a faceless conglomerate without really telling anyone. Shame, but I'm looking forward to Mythic, and (given the discount from their job advert challenge) it's going to bed much cheaper for a bigger machine (£16/month for 2 core 4GB vs £32/month for a pair of 1 core 1GB).

I should pull down all the config. I'm very tempted to wipe and reinstall to check that I've got everything, and to tidy up wrong turnings. I'm going to think about that a bit more.

wepiu recipe

  • postfix
    • postfix
    • postfix-sqlite
  • dovecot
    • dovecot-auth-lua
    • dovecot-antispam
    • dovecot-core
    • dovecot-imapd
    • dovecot-lmtpd
    • dovecot-sieve
    • dovecot-sqlite
  • support
    • postfix-policyd-spf-python
    • spamassassin
    • libnet-server-perl
    • Custom spampd
    • opendkim
    • opendkim-tools
  • web
    • libnginx-mod-http-headers-more-filter
    • dotnet (via Microsoft)
    • nginx-light
  • system
    • apt-transport-https
    • certbot
    • curl
    • firewalld
    • jq
    • locate
    • make
    • tcpdump
    • unbound
    • wget
    • wireguard
    • vim

Webmail move: Checklist

  • Stop old and new postfix/dovecot/nginx processes
  • Start copy of mail folders to temp dir
  • Copy TLS certs to new machine
  • Check new postfix/dovecot conf is pointing at the right cert
  • Confirm that new postfix has the right domains
  • Create sites in new nginx for new domains
  • Update webmail app config for new domain names
  • Login to Bytemark panel
  • Login to Mythic account
  • Get the "yes it's really my domain code" from Mythic and apply it to the Bytemark panel (for all domains that Bytemark is still hosting)
  • Delete other records from Bytemark dns
  • Add domains to Mythic
  • Create records for mythic
  • Wait for mail messages to finish transferring
  • Move into place
  • Start new dovecot, maybe wait for indexing (Todo: check if there's a "reindex" command)
  • Start nginx, confirm dns and imap
  • Check imap from phone
  • Start new postfix

If there are problems then can point dns back at old machines.

Todo: Setup a couple of scripts to add/remove the DNS records from the mythic api

Running a local CA, I mean, how hard could it be?

I like the idea of mutual TLS (mTLS); each service has it's own key pair signed by a common certificate so they all know that they're talking to the right people. Generating a self-signed certificate (a CA) is a one liner, and generating signed keypairs isn't much harder.

I think the tricky bit is key rotation. Ideally, keypairs would have a short lifetime (under a week, maybe under a day), so there must be a way to automatically install new keys and (where needed) restart services.

But that's still just a script, yeah? Create key, sign key, copy key into place, restart service (or ask the service to reload it's certificates if it can). Maybe it's because everything is on the same machine and so I don't need to worry about secure transport, but even so, that's still a solved problem (using certificate signing requests).

Maybe I'm missing something obvious?

Project/ideas list 2022-08-28

  • Improve my webmail UI
  • Spotify interface
  • Google images
  • "game"
  • Image upload editor

I've got plenty of space left and right of the title/controls, so I can easily put more controls there. I want a first/last, and I like the idea of a calendar, although that might need to hide behind a drop-down.

The other fix to do is for the latest/last seen tags. Instead of two (one ::before and one ::after) and some messing about with positioning, I want one ::before, and then latest/last seen can set content (with a .latest.lastseen combined selector for when the entry is tagged with both.

Except that won't work if/when I bring in proper tagging.

I'm going to ignore that for now. Get the current problem fixed, and then think about how to fix it if/as/when I actually do bring in tagging. (I'm not sure, for example, how I'd tag this entry).

I also want to add counters for each page, probably broken down into url/user agent/count tuples. (I'd like to grab ip as well, but that's PI enough I'd need to ask consent. I wonder if ASN is PI?)

Thinking about it, I want to add date into that as well, so I can look for trends (or see spikes).

I should probably also track response status code, and maybe response generation time

I was thinking earlier, it's not Prometheus that's the problem, it's grafana. Grafana is both too complex, and not able to do the things I want. Time to research alternatives. (Yes, Google charts is on the list, but so is writing my own (basic!) SVG chat drawing tool).

That's latest/last seen fixed/moved. They're now ::after the header, and with no messing about with position.

I looked at adding the .latest class on the server, but I got bogged down with lists and stuff. (I've been trying to hard to handle entries lazily, but since I'm fairly sure the page is held in memory until it's been generated, I'm not sure I see the point).

Hello world! Feeling alright today, compared to the recent moving average. Turns out that I've missed maybe half my anti depressants over the last few weeks, which might explain the annoying amount of depression I've been feeling. It looks like the trick is to take pills as part of the "feed the cats" activity so I don't forget.

Otherwise, worried about money. Just run September's budget, and house has good something like £1.50 spare after bills, food, clearing overstay, putting aside half (!) the cost of getting the kittens neutered next month, and the money for my filings. And that's all before the estimated 80% rise in power bills next month. (We're paying £138/month direct debit, so another £110/month, maybe).

Kittens are at least fit and healthy, alternating between sleeping and chasing each other round the house. Storm is maybe tolerating them, but doesn't seem to want to play. Since the kittens don't understand (or maybe don't care) that Storm doesn't want to play with them, there's still a bit of conflict, but it does seem to be settling down.

Work still sucks, but that might just be a physical thing (in that I have to sit upright for 8 hours and my body doesn't like that), although the stupidity of his things are done is still sapping my will.

Having (another? I've lost track) stab at setting up Ory Hydra as an OAuth/OIDC server. I've fixed the permisison problems and passed a basic smoke test, so now its time to actually think.

There are three components: the public hydra server (the API that handles the actual OAuth stuff), the private/admin hydra server (for registring clients etc.), and the UI server (my bit, that draws the login/logout pages).

This is all behind nginx and under the same hostname (current best guess: auth.osric.uk), with nginx proxying the appropriate paths to the appropriate servers.

That's going to need another nginx config (Ory docs have an example) and another TLS cert. Given my current design choices, it's problaby going to be another user database as well. (Hydra doesn't do user management)

That all seems reasonable, yes?

That's the programming, config, testing stuff (for the OAuth service) more or less done. There are a couple more things I could do (mostly about pulling in client info onto the consent screen to make it much more "Example.com wants access to...", but that's not MVP, at least at this (tiny) scale.

What I need to do next is decide where it all goes administratively. Is it part of webmail (or something like my blog), or is it a stand alone service?

I'm leaning towards webmail. It's already got the login/user infrastructure (and accounts), and name recognition with the expected userbase (me and husband), I'm just reluctant to mess with it while it's working. I guess that's what branches are for.

Plan

Hang it all off the webmail domain. Webmail users shouldn't notice anything different. I'll need to tweak the paths a little too make sure that nothing overlaps, although the existence of a challenge query parameter is diagnostic of an OAuth request.

I'll need to do a migration on the webmail db, no worries, I need to get a backup/restore thing working anyway.

I'll end up with a "new client" script, the settings will all be more or less the same each time, only the name changes (and at that point I can integrate it with the new user/project script)

Scary, but doable.

The testing setup I had working for webmail (running dovecot in s container) has stopped working and I don't know why :-(.

I mean, it's clearly something to do with container networking but I haven't been name to chase it down properly. Roughly, dovecot auth should be hitting an http endpoint to validate the given username and password, but I don't think the http request is making it out of the container.

My next step is probably too break out tcpdump to see if I can trace the packets, but I want to understand how podman does it's networking first. (Specifically, I want to know how it's getting packets out of the container without an interface).

Good morning, dear reader.

I'm laying in bed with a kitten snuggled up against me, which is a good start to the day.

On the other hand, I've got a headache that I think it's caffeine withdrawal (on the grounds that I've got from 3-4 cans cola/day to 0-1).

The OAuth/OIDC stuff is coming along nicely. I've integrated the ui code into webmail without any major problems, and I'm working on replacing the local auth for the blog with oidc.

Poot, out of energy. See you later.

Depression is the suck. I'm taking my pills and I'm still laying here on the sofa unable to concentrate on programming (enough that I've taken a sick day). I'm am so done with this. I thought that I had it (depression) covered but it's still jumping on me every so often.

Anyway. I've grabbed someone's flocking code and converted it to JavaScript, but I don't like how they've done edges (which is to wrap position without wrapping neighbor detection), and I'm having trouble fixing that.

I've got the basic OAuth stuff apparently working, but i haven't been able to replace auth on this site.

$WORK is being crap again. TL has made a political choice that's going to cause trouble later (we're building a thing but we're not going to put it up for testing was we go, so it will be too late to make substantial changes when the inevitable problems surface).

No conclusions.

Looks like my latest depression flare up has calmed down a bit. Got some personal code written yesterday and some stuff for work today.

I very much enjoy writing code. I noticed recently that, for me, it maps quite well to the image of someone in their shed/basement building an enormous model railway; an endless project that could easily seem pointless to an observer, but one that helps the participant gain a sense of peace/balance.

I like the simplicity of programming. I like knowing that, so long as I follow the rules, the outcomes are known and predictable. I like building a big, complex mechanism and then watching it chug along happily under it's own power.

Ow, my teeth.

Had a bunch of fillings done on Monday, teeth are being very sensitive at the moment. Totally sucks.

Gah, that's been an irritating few days. It looks like is broken with ASP Core Oidc, in that I have to set a name for the 'nonse' (snigger) cookie. On the other hand, now I know that, I'm fairly confident that Oidc is, actually, working! Whoop!

Next job, getting the Dr Who upload thingy working.

Potential enhancements: Look at the various Hydra endpoints to make a "This is where you're logged in" page (with "Logout" options). Also, tidy up the consent page

  • Make cookies smaller.
  • Finish file share

That's not alot of a to-do list. I wonder what I'm missing?

Anyway. Sleep well, gentle reader.

File upload site got a good positive reaction, much better than I expected. Husband very much appreciated using the BBC Mode 7 font.

Need to update nginx config to allow huge uploads. Husband doesn't want an estimated time to complete. Should add a ring close while uploading warning, and make the boxes wider so there's no wrap.

Otherwise, off to the beach!

I'm feeling a bit odd - I don't have any (internal) pressure to get something finished. OIDC is working, I've done husbands file upload site, webmail is ticking over nicely.

I should probably look at one of my many todo posts, but I'm not that bored :)

Not really sure what I want to do with myself today. Husband is asleep, work starts again tomorrow. I did a bit of work on flocking yesterday, I think I'm going to go pick that up again.

I'm still annoyed about how hard it is to get hold of the real-time data from my smart meter. It's seriously looking like the easiest solution is going to be pointing a camera at my in home display.

Still not sure what I'm going to do about my old Facebook/Twitter posts. Parsing the JSON and creating blog posts is a chore, but not the problem. The problem is the conflict between "I want all my posts on one page", "gosh that's a lot of posts", and "wow, a whole bunch of these posts are hard to parse out of context".

I can default to showing one days posts (and then visitors can use the date links to go back)(and show the latest date with posts of there isn't anything for today).

I could categorise/tag posts, and only include current blog by default (I think I like this one? It makes including weirder stuff like git commits and house moves make more sense, as one could filter in/out categories of interest)

I could just cope, it's not like I've got any evidence of readers.

Teeth still hurting this morning, saw a dentist, taken some antibiotics, teeth are hurting much less. Dentist says tooth will need to come out but will argue against that.

Had a chat with my boss/manger/team leader (Hiya DK if you're reading this!), I've been taking a lot of sick leave recently (over the last year or so), enough that they're talking about maybe needing to remind me of the sick rules. Fine, whatever, but don't expect me to care.

It's just that work is so stupid! They want us back in the office, and can't give a better reason than "spontaneous collaboration", which sounds very much like gossip when pushed. They want us to be "at work" (either at home or in the office) for a fixed number of hours a week, but they don't seem to care what we're doing with the time. It's just so trivial! I guess it's one of those neurodiverse things - I hate to see the waste (of time, talent, energy, resources) that these policies generate, but worse than that is the way that everyone agrees that it's broken but nobody fixes it. I genuinely don't understand why there's so much disconnect between what people want and what people get.

Ah, well. Husband says that I need to learn to relax at work, and they're right. I'm still getting paid for this bullshit even if it is crazy.

In better news, I've been playing with Web Assembly (wasm), and it's going fairly well so far (although I'm still only at the smoke test/toy code level).

I'm using clang to compile C to wasm, and then loading it into a web page. I give the wasm VM memory from JavaScript, ABC that means I can pass data between the two. Anything more complex than numbers needs to be flattened and re-inflated across the boundary, which is a chore (but not a hard one).

Current plan is to move some of the flocking maths over to wasm to see if it's any faster, although I'm distracted by GPU.js which calls into the GPU from JavaScript. For something like "find the distance between all the pairs of points", I think it's worth looking into (although really, I should grab some 2d space partitioning algorithm and use that instead).

(Really, I should find/build a metrics framework so I can measure how much time is spent doing various bits and pieces, so I can actually compare the different ways of doing things)

So I want to make this blog an installable app on my phone. What does that mean in practice? Mostly, I think, that I want to be able to write (and save!) posts even when I'm not connected, and then push them to the backend next time I get connected. Rendering posts clientside implies a JavaScript markdown parser, which do exist but is too heavy for my taste.

I've got a rough idea of showing a marker in the offline client for "there is a post here", (although, I've just thought, I can show the raw text until it's been posted).

I'm already stashing unfinished posts in local storage, but I don't want to get confused and start thinking that's the same thing.

I've had a look at the code to check, and the on submit for this form deletes the draft and then let's the POST go through. Clearly, if POST fails then all is lost.

If I add an id to the post before submitting it, and include the id when I send back the HTML, then the frontend can tell that a POST has been successful. If I stash a completed post in local storage using the id as a key, then I can find and delete posts from storage at page load time (and re-submit any that should be there but aren't).

That's work that only needs to be done for logged in users, which is fine, I can do a server side check and only include the JS if the user is authenticated.

I can even add a <template> for entries that haven't been submitted, and render them at page load time.

Things to check:

  • How to catch a failed POST from a form submit
  • How to turn off a service worker on logout

Alrighty folks! Today is all about stats!

I want to know which (if any) of my pages are getting hits, and ideally if they're getting hits from not-robots.

I can easily add a middleware that grabs some stats and posts to a queue to be added to a db (so it doesn't slow down the main thread too badly). Question is, what should I keep?

I very much don't want to collect PII, so I can't grab raw IP. I'd like to know the difference between bots and not-bots, so I'll need User Agent (or at least, the result of userAgent.Contains('bot')). Obviously I need the URL. I'm not expecting very much traffic, so I can bucket into hours (which also helps with the PII issue).

That looks something like:

CREATE TABLE HitsByHour (
  Text Path,
  Number Count,
  Number Date, -- seconds since 1970 to the start of the containing hour
  Number IsBot, -- 1 yes, 0 no
);

I think I also want a view like:

CREATE View IF NOT EXISTS
  HitsTotal (Path, IsBot, Count) AS 
    SELECT Path, IsBot, Sum(Count)
      FROM HitsByHour
      GROUP BY Path, IsBot;

Did some research ("cat access.log") and doing a check for "bot" isn't going to cut the mustard.

Got distracted from the stats plan predictably quickly. On the other hand, I've tidied up my DNS records (mostly upped the TTLs from 5 minutes to one day, since I'm fairly confident they're correct), updated the nginx logs to include cache hit/miss and hostname (the first to see if the cache is working (spoiler: not as much as I'd like) and the second because all the hosts go into the same file and it's not clear which request is for shiv host), and verified a couple of sites on Google search console.

First news from the logs: My Windows browser uses IPv4 for one site and IPv6 for another, even though both share addresses. Weird. I'll wait a couple of days and see if it's still happening.

Life is ticking over nicely, work is boring (which is fine, better than many kinds of exciting).

I've added a calendar to the front page of the site, not sure what to do with it Could obviously link to blog day entries, could probably put in moon phases, but I don't think I'd use with of those.

Read a post today that used CSS like syntax for SVG, which got me thinking.

I'm a big fan of SVG. I love the declarative nature of the format ("Draw a circle" vs. "pixel, pixel, pixel"), as well as being able to write it in notepad.

On other other hand, like all XML dialects, it's a bit wordy. I can't find the link, but a while ago I read an article that showed how easy it is to map XML to Lisp.

Roughly, an XML node is a name with a list of attributes and children, where attributes are 2-element lists, and children are either nodes or strings.

<svg><circle x="40" y="40" r="17" /></svg>

(node "svg" () ((node "circle" (("x" "40") ("y" "40") ("r" "17")) ()))

I've had a bit of a poke at mal and it's actually feeling like it's a real thing.

I've got a few things I could do next, it needs a better UI for the repl, I could build it again in JS (to run here in the frontend), I could build it again in C and compile to wasm (to get exactly the same code running front and back), and I could handle mal code blocks with the C# engine.

The cli needs readline like stuff - at least basic cursor movement. Ideally, it would stash the environment (and history) at exit and load again at start, with some way of editing (i.e., save to mal code).

Rewriting in C would probably be a bugger, although there's always Crafting Interpreters to fall back on (it's got dynamic lists, rough polymorphism, even a garbage collector!).

Rewriting in JS would be easier, and I could use quickJS to run/test on the server (which is mildly horrific, but that's life in 2022 baby!), but I'm still on the fence about JS. It's nice that there's common language in browsers, but did it have to be that one?)

I'll have a stab at writing in C, trying to use as little libc as I can get away with (realloc, printf, probably some string comparison stuff, I'll check the book later) and see if that's a realistic option.

(I'm going to end up with Lisp interpreted by wasm calling out to JavaScript for IO. Groovy.)

Interesting, just got prompted by webmail to confirm consent. I guess some cookie/time limit expired. (I should update the consent screen if I'm going to be seeing it again)

Maxims

(Rules that may help lead a better life, not complete)

  • Mean people suck
  • Asking questions is good
  • Fixing mistakes early is much cheaper than fixing mistakes later

Moving downwards with speed u and acceleration g, if I start thrusting with acceleration -a, how far do I travel (s) before my speed equals zero?

Then, as I'm falling I can recaculate with u = current v, and when s = current height I can turn my engine on and reach the ground when my speed is 0 (ignoring for now that my thrust changes as I use up fuel)

v² = u² + 2as
v² - u²  = 2as
(v² - u²)/2a = s
-u²/2(g-a) = s

Check that the units make sense

(ms⁻¹)²/ms⁻² = m
m²s⁻²/ms⁻² = m
m=m

Yup, that checks out. Time to give it a try.

I've picked up Kerbal again, still playing with the kOS plugin. This time, instead of going straight for orbit, I've spent a bit of time messing around with sub-orbital hops (which has got me lots of lovely science!).

I've also written a landing script that slams the engines on at just the right height to reach zero velocity at the surface. I've tasted it on Kerbin and it's looking good, next stop Mun!

However, I can't put it off any longer, next session is orbit or bust!

(The maths posted at 14:22 today works btw, it's correctly calculating the hight I need to turn my engines on at.(Which is quite scarily close to the ground, I can tell you!)

So the theory goes that it should be easier to write Lisp on the phone because it uses fewer funky characters:

(define test (+ a b))

I can see that working, although writing a formatter goes on the list.

Mal is scheme-ish, but certainly not scheme. I'm going to need to commit one way or the other fairly soon, and I'm still not sure which way to jump.

I'm much more likely to find online help for scheme. It's got it's own interpreter, it's even got a compiler!

Mal is small enough that I could write a JavaScript implementation, so I can run stuff client side. Also, because I've written the interpreter I should have a better idea what's going on (I'm not sure that I do, but I should). It's also much easier to extend (although I should look at how scheme does it).

I think I want to use mal, but I'm worried about support. It's probably close enough that I can translate, and I can change the name of some things to make it closer. I very much like the idea of client side evaluation, so I can write simple scripts on my phone (Oh, Psion, I still miss you!), although there's a bunch of saving/synchronization to think about.

Grinding through mal-js implementation, can't decide between using js objects or a bunch of classes.

JS doesn't have 'symbol' or 'keyword' separate from string, or 'vector' different from array, but I can add a type property for the times when things are different.

Writing my own types is theoretically safer, since mal code won't be calling js methods directly there's less surface area for naughy code to fall through.

Dunno. Going to sleep on it.

It's been a bit odd at work the last few days. I've ported a couple of tools from Visual Basic for Applications (VBA - Microsoft Excel macros) and powershell to modern C# (for reasons that I haven't paid a lot of attention to, but I think are related to the security people at work tightening up execution policies).

The code isn't very complex - download a few files, calculate and compare a hash, unzip them, and move then into place - but the team are all like "OMG they're so fast!"

The download/install now takes two/three minutes with a good network and fast disk, compared to maybe half an hour previously. That's a useful speed up, sure, but the amount of praise I'm getting feels very out of proportion to the amount of work it took (especially as I didn't put any thought into speed).

Husband's advice is to just accept the praise, and I'm doing my best, but I'd far prefer someone to gush over my clever stuff.

Ah, well.

I do have a couple of ideas about how I could speed up the install, but I'd have to test them to see if it's worth it.

Mostly, it's to not bother saving the zip to disk, but instead unzip the stream on the fly, teeing off the stream into the hash calculation as we go.

There are four zips, a root and then three children that unzip into a folder under the root. At the moment I'm unzipping all four each into their own folders, moving the children into place under the root, and then swapping the root info final position.

If I go for the stream unzip, then I'd skip the first step, and unzip all the files into place (so skipping one of the moves). I'm not sure how much it would save, the moves are all on the same filesystem, so should be cheap/fast (and watching the process supports this). It would also create complexity, as files from any given folder might end up in more than one of the child zips (don't ask! I tried to get some clarity on this but just got white noise) so there might be conflicts/races when creating folders.

It's a well known trope, apparently, that users really don't care about fancy coding tricks but will very much like the duck animation they get while they're waiting....

Made a couple of updates to the blog code, Google search was getting confused because all the pages link to the login page and it wasn't sure which one was "right". I've added a link header pointing at the return url, let's see if that makes it happier.

I've finished another couples of steps in mal-js, eval and environment, so (+ 1 2) now correctly evaluates to 3, and def! and let* special forms work. (I'm going to lose the funky characters once I'm done, although I could just edit the tests now. Hmm)

Using JS types directly didn't work (because a string value isn't an object, so one can't set arbitrary properties in it), so I've gone for a simple wrapper class with a (string) kind property (to tag the type) and a value property to hold a JS value.

It seems to be working so far, it's nice not to have the big stack of classes and worry if hash should extend sequence. (Having just written that, I'm idly thinking about adding an "isSeq" method, since the only difference between lists and vectors is the shape of the brackets when they're printed)

Mentally it was a rough day. I never really engaged with work (Y'know, I say that but I did deliver a product and refactored shared code into a library. I'm starting to think that my standards for 'good enough' may just be far to high. Husband, if/as/when you read this, you were right so along and I'm sorry for doubting you) and so I bailed an hour early.

Mental note: I want to post something during the good the clocks go back so I've got something with a stupid timestamp to test against.

An extra three hours at work today, investigating a problem with a service. The service is running fine, except it's looking like it's taking IIS three to four minutes to start processing the first request from a cold browser. Other requests are fine after the first one, so long as the browser stays open. There's nothing obvious in the IIS logs, other then the trace logs don't start until 3-4 minutes after netstat shows a connection.

I'm back home now, and I'm going to try to squash any more thinking about it this evening, but I'm probably going to be back in the office tomorrow. Which sucks.

Feeling better again, that was weird. Came home from the office and was wiped out by (what felt like) immune system response. I was hot, mildly confused, and aching so over. I slept for maybe 4 hours, got up, managed to get downstairs, lay on the sofa for another hour, and then just felt well again.

Two theories: I picked something up at work and my immune system did it's job, or I made myself sick at the idea of being in work. Hard to tell which it is.

Ah, well.

I haven't been using my primary laptop for maybe six months because the keyboard was broken. Yesterday, while waiting for w traffic light to change, I saw a little PC repair shop. I phoned them this morning to get a price check on taking the laptop apart enough to resit the plug at the end of the keyboard connector cable (£10-15, apparently). So I booted up the laptop to confirm the problem is still happening, and it's not?

Does this mean it's fixed? I don't think so, but I'd be very happy to be wrong so I'm going to fettle it up like it's working and use it as my primary until it starts being broken again.

This does leave me with the traditional "Windows or Linux" question, although given a hybrid Intel/Nvidia graphics card, and how well WSL2 works, the Windows option is fairly strong.

Old laptop is still working, so far so good.

I've wiped it and installed Windows 11. First impressions are that it's going to be fine, it's fairly close to Windows 10 and the differences are minor. Windows 11 feels rounder (like, all the corners have got a border radius of something), and my first impressions are probably trained by the stupid keyboard on that laptop (I paid extra for a "mechanical" keyboard, boy was I dumb).

The key spacing is different/wrong, and I'm constantly typing one character to the right of where I should be Also the Windows key doesn't work (nothing happens when I press it, haven't investigated any deeper yet), which is bloody irritating given how much I use Windows snap.

I'll clone down a couple of repos tomorrow and see how I get on.

That's mal-js finished and installed as a client side app into this site. Now to come up with a use case....

Kittens were at the vets today getting chipped and chopped (neutered). They seem to be ok so far, not very happy with the cones, but they're temporary. We've moved the screen door to keep them locked into the lounge/kitchen (with the screen door set so Storm can come and go from the rest of the house).

Teeth have settled down again, thank science for antibiotics. There was a weird lump in the roof of my mouth next to the tooth that hurt, that's also shrinking, so I guess it was related to the infection? The root cause this time was a filling that's fallen out, but the dentist will replace it free since it's under warranty.

Emotion-wise, nothing spectacular seems to be going on. I've got very little enthusiasm/engagement at work, except I'm grinding through the bit of Ocelot v2 that's replacing classic ASP (in JavaScript) with c# MVC (ish - the controllers are fairly straight ports of the JavaScript right now, I'm sitting very firmly on the urge to "improve" or even "tidy up"). So my work emotions are confused; I'd rather not be at work, except I've got this nice low-impact, fairly open-ended job to do.

Shrug. But if you come up with a use case for Mal, please do let me know.

Not sure what to do next. Mal-js is done. Webmail probably needs looking at ("Webmail always needs looking at").

Actually, let's take that one. I'm still not really happy with webmail. I want new mail notification, but without to much client side stuff.

I was idly thinking about JMAP the other day, writing a wrapper for MailKit as a start, and then maybe writing a proper server.

I could give shopping list another stab, I've got most of the components for a web based version using commands/events.

Actually, that one does grab me. I'm off to a better keyboard to think about it.

Shopping by events

Events are things that have happened, so should be in the past tense.

  • ListCreated

    • listId
    • listName
    • ownerId (maybe)

  • ListDeleted

    • listId

  • ItemCreated

    • listId
    • itemId
    • itemName
    • itemState

  • ItemUpdated

    • listId
    • itemId
    • itemName
    • itemState

  • ItemDeleted

    • listId
    • itemId

I might have been over thinking sharing. If I add an Out Of Band (OOB) portion, then it all becomes much easier.

  • User A clicks "Share List"
  • Server generates a unique id and stores it along with the list id, user id, and timestamp
  • Server gives User A a link containing the id
  • User A sends the link to user B OOB
  • User B uses the link
  • Sever uses the id to look up the list and:
    • Adds User B to the users for the list
    • Deletes the id
    • Notifies User A

Need to add user models, including a per-user notification queue.

Also need new events for ListShared (or UserAdded, can't decide which).

Also need to think about commands, since the local client can't create share links. (But most traffic from local client will still be events, as the thing will have happened by the time the server funds out about it)

Rule: The local client is the source of truth.

is there any need for lists to have a clock? Yes, need to be able to order events well enough that lists are consistent between users.

I've been thinking about writing a build-time tool to replace strings in source files (specifically to add version numbers to <script> tags, and to the load service worker call), and I've just realised that if/since I'm doing build time shenanigans anyway, I might as well do JS/CSS/HTML minification at the same time.

(I was going to say something about HTML minification being hard as it interacts with the ASP stuff, but that's what Rosyln is for, yeah?)

Hello world, Monday evening, not much to report.

Teeth are back to normal, got a sedation assessment appointment Wednesday morning, one of my filings has fallen out and needs to be replaced (under warranty!).

Work is still porting Ocelot from Asp classic to Asp core. It's going well, I'm using Selenium to write tests to properly exercise the code. Selenium is really quite neat! It's a web automation platform, which means it starts up a browser and then uses an API to control it. The tests are things like "Find the element with this css selector and then click it", which means that it's the same as a person running through a script, but faster and reproducible.

I'm very happy with it, and I'm nearly at the point where I trust it not to give me false negatives. The only issue I've got with it is that the logging could be better - I still have to load the site manually to find the underlying why a test failed (but that's probably me asking to much, it's still much faster than manually starting the site and navigating to the problem to test a fix, and it's trivial to run all the other tests at the same time)

Kittens have been neutered, they seem to be recovering well (Havok faster than Carnage, but Carnage feels a lot more delicate than Havok). Carnage peed on the stairs this morning (right in front of me, so I know it was her) (not an ideal way to wake up). I think she was a bit altered from the pain meds we'd given her yesterday. (She should have had them with food but we have them on an empty stomach), but it could have been a protest, or some other communication.

I'm doing ok, ish, maybe a little below baseline. I think I'm just feeling the weight of another 30 years of work.

Ah, well. I'm going to read my phone for a while and then sleep. Picking up my new work laptop tomorrow, that might be interesting.

I'm having a lot of fun playing with mal-js, but I need to build a better way of storing/reusing scripts than emailing them to myself. (Or I need to build an email endpoint that evaluates mal. One or the other)

Anyway:

(def! acc (fn* (fn z args) (
  if (empty? args) z
    (acc fn (fn z (first args)) (rest args))
)))

(def! range (fn* (n) (
  if (= n 0) ()
    (cons n (range (- n 1)))
)))

(def! reverse (fn* (l) (
   apply conj () l
)))

(acc str "" (reverse (range  9)))

Accumalate, generate range, and reverse (because range creates numbers in the "wrong" order). Next up is let*.

I'm updating the interface for Mal to give the user multiple open files. (Because often when I'm working on something, I need to try an idea without disrupting the current code).

I'm also debating if I should offer backend storage (instead of just local storage). It's going to be "Yes, but only if you've got an account" (so me only), and only so I can make transfers between the laptop and the phone easier. On the other hand, there's suddenly a whole bunch of concurrency stuff to think about.

I want to improve the editing experience. First up is formatting, I'm developing a set of rules to apply to an ast to get pretty output, something like "start a new indent level of the current item is a list with more than zero children that are lists". (I should write it in mal as an exercise).

Anyway, it's lunchtime at work so I'm going to get ten minutes shut eye.

Just finished watching Bullet Train, a movie recommended to me by husband as "a very [me] friendly film", and yes, they were exactly right!

Plot: Our protagonist, a former professional killer who is trying to work out some things, takes an easy pick up job, to recover a case from a train. However, several stands of fate are entwined with the case so wacky highjinks ensue.

However, that description really doesn't do justice to the complex, tightly written, and complete story that follows. It's an action movie, in that there are plenty of action scenes, but it's well paced with breathing space in the right places that didn't turn into stalls.

The effects were mostly subtle enough that I didn't see them, the direction and camerawork was just brilliant, framing shots cleanly and dropping in enough hints and pointers to let me keep up without overwhelming me or getting in the way.

I've got a couple of quibbles about some of the exterior train shots, when a 200kmh train turns a corner sharp enough to see, but I'm prepared to forgive those as they weren't relevant to the plot (and could just be edited out, probably (I'm not an expert)).

It's reset the standard for a good movie, replacing R.E.D as "best action movie I can remember watching".

Highly recommended.

I'm frustrated because I I'm having trouble choosing the "right" language to default to for projects.

The choices are:

  • C Static typing, manual memory management, complied, exemplary standard library, lots of 3rd party libraries. It's a lot of work to get started with C, but I could build up a library of useful tools.
  • C# Static typing, automatic memory management, complied, good standard library, lots of 3rd party libraries. The development environment (VS Code, or even Visual Studio) is first class, the documentation is extensive and consistent, and it's got a good crop of answers on stack overflow. Problem is, it's owned by Microsoft and while they give the tooling away, they're starting to move towards closing off important components (e.g., the c# debugger may only be used with Microsoft tooling, which includes telemetry back to Microsoft).
  • JavaScript Dynamic typing, automatic memory management, interpreted, piss poor standard library, lots of 3rd party stuff of varying code quality (but mostly mediocre at best). Every time I start something new with JavaScript it takes me a bit of time to get my JavaScript head on, but once I'm there I very much like it as a language. Using QuickJS on the server has potential, although I'd need to write a bunch of stuff (in C) to turn it into a server side environment
  • mal/lox Both of these are teaching languages, designed more to show how to implement a language than to be useful languages. Since I write them myself, they're both relatively easy to extend to match my exact needs. In the other hand, they'd both need extending to match even my minimal needs.

I dunno. I think I might be reacting against node/npm in rejecting JavaScript. There are at least a couple of fairly good looking http C server libraries that I could hook into QuickJS (or vice versa), to build a basic server. That would scratch my C itch, and give me a relatively fun place to work from.

Or I could ignore the politics and stick with C# for the scale and simplicity, except that's just making it harder to move away from C# when Microsoft does pull the rug.

Big programming win at work today, I've got Vault working with auth from k8s - meaning that apps can get all the way to runtime before finding out secrets (like their database login details).

I've got a couple more things to investigate/get working, pulling the HTTPS cert from vault, and integrating vault into the configuration system. I want to do that now, but I'm not at work? Having said that, the new work machine ("HP Elitebook") is quite nice as a laptop, so I could go grab it and mess around down here.

There's definitely something wrong with the login cookies here, although logging back in doesn't take very long.

I seem to be taking a break from my programming projects, which is fine, of course. This has been a semi-concious choice, in that I saw it coming/happening and decided to go with it.

Instead of programming I've been sinking time into sleep and games; sleep, as ever, is still brilliant (I'm getting dreams, which I'm not wild about, but they're not bad dreams, just frustrating), the game I'm playing most is "Turing Complete", which takes the player through building up a computer from NAND gates. It's a nice reminder that this whole computing thing is a whole pile of levels/abstractions over some really brute simple stuff - chip complexity is measured in transistors because that's really all you need.

(On the other hand, the abstractions get pretty far away from transistors, and even gates, fairly quickly. I'm up and running on a Turing complete CPU already, and it's made of components that are made of components that are made of gates. I should contact the author with a wishlist request to be able to flip a design back into NAND gates.)

Also, toothache is back. On the other (left hand) side of my mouth, and not really concentrated with a particular tooth, but still, ouch, and I'll go see the dentist tomorrow and hopefully get another antibiotic prescription (although there shouldn't be anything left after the last couple of courses).

It was so liberating being told that I didn't need to put a title on every (or even any) blog post. I can just write and post without needing to add a label! It's brilliant!

Back from the dentist (like, twelve hours ago). Got an abscess above one of my teeth ("upper left three") which needs a root canal (because the previous filling didn't take enough out?).

I asked why I'm getting new infections after taking two courses of anti biotics in the last month(ish), got some kind of garbled response that I'm parsing as a shrug. I'm a little annoyed, this all started about a year also when I went in with an infection, send came out with a bunch of fillings and a mild sense of shame.

For something like thirty years (from early teens to mid-forties) I didn't brush my teeth, and I had approximately zero problems during that time (maybe an infection or two over the years, but no fillings or other work). However, a couple of years back I started drinking cola for the caffeine, and eating boiled sweets for the stim/sugar, and that combination seems to have seriously damaged my teeth.

Last autumn, dentist said I had to start brushing regularly or they wouldn't do any work (and I was only in for toothache). They stuck in a big of fillings in January, a few more over summer, and I've got more work scheduled in a couple of weeks.

And I'm just passed off, yeah? My teeth were fine for actual decades without any real maintenance, and now I've got fillings (which react very badly if i touch them with a metal teaspoon), and regular tooth infections,b and I'm still brushing my teeth twice a day.

I know it's paranoid/stupid/wrong, but I can't help seeing the conflict of interest between people who are paid when my teeth just, and people who recommend teeth maintaince plans.

Otherwise, another day at work. Got Vault/asp core/certificates all working together nicely, including 30 second lifetime certificates (mostly as a stunt/for testing, but it's opened the door to 30 hour certificates, which are actually a significant security improvement).

I should look into Helm, since that's what $work want me to use, but the whole "yaml template" thing is just too terrible.

Ah, well. It's late and I should sleep. Hello anyone (still) reading, hope you're enjoying the show.

I've switched to the emails as storage plan, this entry (assuming it's saves) is the first under the new regime (all the older entries were moved over with a script).

Feeling at a bit of a loose end. I've got a kitten sat on me, which is very nice in the short term but doesn't really help me put together medium or long term goals.

Blog navigation

Definitely want 'first' and 'last' links, got plans for the archive (using the calendar widget).

Permalink pages should probably have 'previous'/'next' (or 'earlier'/'later', which might be easier to understand)

I think the "Track my reading progress" tickbox might be too big/prominent, but I don't want to hide it behind an options toggle. I could just turn tracking on globally, but I do want readers to have a choice.

There is still plenty of space in the header so I guess I just need to mess about a bit and see what works.

Ooh! I could put the "Track my position" control in the entries! Specifically, have it track alongside the current last read, so it moves as the reader scrolls.

That frees up space in the menu, and hopefully indicates what the control does ("Increases the affordance of the control", according to what I remember about UI design).

That's the archive deployed to live. Still TODO:

  • The interaction between month/year view and earlier/later is wrong, currently defaulting to days. I think I just need to check the model to see if day/month/year is null, and skip the value if it is.

    Sounds easy, but it's a bunch of typing I can't be bothered with tonight.

  • Turn track position back on. Currently disabled because I'm using the space for earlier/archive/later, will think about where it could fit.

  • Add an up to month/year link?

Anyway. Havok is back sitting on me, so it's probably time to quit for the night.

Poot, depression again. I should go feed the cats/brush my teeth/take my pills (and, crap, I forgot my 14:00 antibiotic) but instead I'm here on the sofa talking to you.

I don't even have the energy to rant properly.

On the plus side, flicking through the post archive has increased the weighting in favour of comments on posts, especially now that I've moved to a storage format that will support them.

(Either as a whole set of nested multiparts, or, more likely, a linear array of comments with references to parents that can be turned into a tree at render time).

I wonder how the bots are coping with the archive? I hope they're happy at all the new links.

I'm feeling depressed again, which is irritating since I've had a fairly good day, at least programming wise.

Hubbs and me had a small argument this morning, which is probably contributing, but otherwise it's just the same old life.

I think I've cracked service workers, at least enough for a dirt simple sleep tracker app. It's all client side JavaScript, using IndexDB for storage. At some point I'm going to need to look into data export, although the current plan for that is "Here's a csv to download".

Bollocks to it. Snack, pills, teeth, bed. See y'all later.

Practicing cooking steak as husband wants some for birthday, so we've got a couple of big chunks of fillet each. Husband is also trying tripple-cooked chips, so looking forward to those.

I forgot to pull my facebook/twitter/everyone else archives off the old machine, so I've applied for them again. I've got a better idea of how I'm going to handle facebook posts than I did before, so might get further.

From what I remember of Facebook's export format, there's a bunch of stuff that's in arrays that there's generally only one of. I'm going to use the MIME email format again,

Next project roundup

A list of potential next projects

Paint

A simple web based image manipulation tool. Operations like crop, rotate, resize. Could easily be client side with canvas doing the heavy lifting.

More about getting the UI right than anything else, although that's probably became I'm assuming that writing and composing manipulation functions is fairly easy.

Facebook import

Now under got a copy of my Facebook data I want to pull it into something that I can play with (at least enough to add to the blog).

An exercise in JSON parsing and data storage. How much of the data do I want/need to keep? What kind of access patterns am I going to have? Am I going to be able to access the data in ten years when I stumble across it again?

Server side Mal

I want to be able to make stupid toys on the site, from my phone. I could do it with HTML/JavaScript, or even C#, but typing those on the phone is irritating. Mal has much simpler syntax, and should be at least as powerful (and much more so if the rumours are true). Also, having the same client and server side scripting languages might be fun.

I did that C# version of Mal before, but I'm not happy with my choices and want to write it again, which is a bit of a job. (I could use mal-js and QuickJS, although I'd need to wrap them in a CGI interface.

Where does that leave me?

It's going to be Mal, I think. The Facebook thing needs a bit more thought (really, I need to convince myself either that SQLite is a sensible way to store data, or that it's worth my time to write a bunch of models for the data, and keep the original Facebook data), and I'm not looking forward to the UI stuff for paint (which, to be fair, is kind of a reason I should do it).

The only real problem with Mal is that, at least for the first but, one done it before and I'm not sure what I'll get out of it. Maybe I should cannibalise the existing mal-sharp code?

More projects that didn't get listed above:

Offline Blog Entries

Since the service worker stuff for the Sleep Tracker works, I should start looking at all the other phone apps that have got backed up in the queue. (See also: Shopping List below). I'd like to think that adding an offline mode to the blog would be easy, but I've got a feeling that I made the entry submission stuff far too complex (although I might be thinking of Chat).

Shopping List

The classic phone app. An opportunity to use Notifications and Push API, and a bunch of other stuff. I dunno. It's the sharing stuff (and I guess the sync stuff, although there are ways to do that).

Slow Movies

Since I've got £60 to spend for my birthday, I could buy the hardware for this and get something up and running. [Pause to check the website] Yup, within my budget

Loaded the ol' codebase into Visual Studio and ran a bunch of clean up on it. Eek. But it seems to be working.

Not a bad weekend as these things go, got done nice programming done, got a bit of sleep in, spent a bit of time with husband, played with some kittens.

Work tomorrow as usual, talking to at least one of the Dave's about the .NET template I wrote list week, maybe bang my head against Vault and local development some more.

Otherwise, next big thing is birthdays. I've booked a week off (and I need to check the dates for that) and I've got a couple of things planned (including: plenty of programming, a trip inland to get out of town, plenty of rest).

Husband has invited some old friends to visit (friends who are our age that we have known for a long time), they'll be camping out in the study over birthday weekend, so there's that. I'm fairly neutral in the whole thing, possibly because I don't really think it's going to happen. I should probably start believing it so I can get used to the idea before it happens.

However. it's late and I should do my teeth and get to bed/sleep. G'd night y'all.

Also for the Todo list: Improve the webmail parts of oauth (specifically, use the site name instead of '[somewhere]'.

I've added sunrise/sunset and moon phase to the landing page, but it's not right yet Maybe I should lose the calendar?

I think I'm also going to add weather, but in low resolution (only use the 12h data).

Work was quiet, had a bit of a chat with DW, who said nice things about my the Vault stuff I've been writing, and also reassured me about feeling embarrassed about something I haven't done (I asked for an API key months ago, and then never picked out up. I've been feeling all "argh, they're going to zap me if I follow up", but DW reminded me that a) we're all humans and b) nobody cares). Solid lad is DW.

Husband dyed my hair, it's a bit blue now (this should have been a photo, but I guess that's broken!).

Four days till birthday week!

(def! reduce (fn* (fn acc l) (if (empty? l) acc (reduce fn (fn acc (first l)) (rest l)))))

(def! join (fn* (l) (reduce str "" l)))

(def! tag (fn* (name content) (join (list "<" name ">" (join content) "</" name ">"))))

(defmacro! mktags (fn* (& names) (map (fn* (name) `(def! ~(symbol name) (fn* (& content) (tag ~name content)))) names)))

(mktags "html" "head" "title" "body" "h1" "p" "ul" "li")

(print
  (html
    (head (title "Hello world!"))
    (body
      (h1 "Hello" "world!")
      (p "Writing from inside mal (lisp) seems to work, yeah?")
      (apply ul (map li '("one" "two" "three")))
    )
  )
)

Phew, that took a while!

As part of the whole "programmable site" thing, I've been wanting to write html-as-lisp (because it's fairly easy to type brackets and quotes on my phone keyboard, at least compared to other syntax).

I've been able to do the basic version for a while - use a macro to make functions that take their content, convert it to strings and wrap it in tags. The problem has been that when I generate content on the fly (e.g., the li tags above), the brackets from the list end up embeded in the outcome.

Until now! Because apply flattens it's arguments, I can use it to convert the list result of map into the arguments of ul.

Sweet. That should be a bit of incentive to get more work done on mal-sharp.

(def! tag (fn* (name content) (str "<" name ">" (apply str content) "</" name ">")))

(defmacro! mktags (fn* (& names) (map (fn* (name) `(def! ~(symbol name) (fn* (& content) (tag ~name content)))) names)))

(mktags "html" "head" "title" "body" "h1" "p" "ul" "li")

(print
(html
  (head (title "Hello world!"))
  (body
    (h1 "Hello" "world!")
    (p "Writing from inside mal (lisp) seems to work, yeah?")
    (apply ul (map li '("one" "two" "three")))
  )
))

Turns out that I don't need join or reduce, since apply str does the same thing. Neat.

Also, I'm still not sure what I'm going to do about attributes. I want to be able to do something like (html {"lang" "en-gb"} ...), with the hash argument being optional, but I need to work out how to do that. (and clearly, two versions of tag, and an if in the macro to pick the right one, but I'm still not wild about it).

The mal interface still isn't right (ignoring the bug that means it stops working when I tab away).

I've got a rough idea of what I want, which is clicking "Exec" takes a copy of the textarea and adds it to a history along with the result (either whatever result was printed, or the details of whatever error was thrown), and probably a "reload" button.

I'll mess about with it sooner or later.

Not much to report. A little toothache in lower left 6(ish), Husband has reorganised the cables/tidied the box room, and a new version of Hydra (the OAuth2 engine I'm using here and at $work) has been released.

Otherwise, one sleep, one day at work, and then a week to relax (ignoring birthday weekend/prep).

There's this thing where I feel crap because I haven't eaten (and it's happening enough that I can recognise it), and all I need to do to feel better is to eat, but sorting food out is just too much, yeah?

I can finesse it so far off there is sufficient snack food available (up to maybe peanut/honey sandwiches, but dealing with cheese is, again, too much), but I can't will myself to cook (which would properly deal with the issue for half a day or so).

It sucks! I know what the problem is, I know how to solve it, but I just... can't. (This might be the executive disfunction that I'm hearing about)

Snake priorities

  • Only add tail if not 100 health

Each direction is weighted by:

  • Possible head (high -)
  • Is food (High + if low on health, low - otherwise)
  • Towards food (low + if low on health, neutral otherwise)
  • Towards tail (high +)

Sort by weight and pick the best

int GetWeight(MoveFlags flags, bool hungry) {
  int weight = 0;
  if (Match(flags, MoveFlags.Head)) {
    weight -= 5;
  }
  if (Match(flags, MoveFlags.Food)) {
    weight += hungry ? 3 : -1;
  } else if (Match(flags, MoveFlags.FoodDirection) && hungry) {
    weight += 3;
  }
  if (Match(flags, MoveFlags.Tail)) {
    weight += 3
  } else if (Match(flags, MoveFlags.TailArea)) {
    weight += 2
  }
  return weight;
}

An important thing to notice is that I seem to have plenty of time (to do funky snake brain stuff). My end takes 2-3ms, plus what looks like about 15-20ms of round trip. Given than I've got 500ms, i can comfortably take 100-150ms to think.

What should I think about?

My snake has been winning matches! Still losing more than it's winning, but it's doing well for two days development!

Strategic next is prediction. My current biggest cause of death is head on collisions that could have been avoided if I'd had even one more turn lookahead, but I don't want to stop at one turn. I've got maybe 300ms to play with, that's plenty of time to explore a biggish tree.

Brute force suggests that each turn generates 4 (snakes) * 3 (possible moves that aren't instant suicide) = 12 possible next turn states. That goes up with the power of the number of turns, so 12, 144, 1728 for the next 1,2, or 3 turns.

I need to decide/work out what sort of data I want out of the algorithm. I don't want to run my full heuristics for each snake, because I can see that my design choices are different to other snakes.

As an early efficiency, I'm going to ignore any snake who's head is at least (max depth + 1) squares distant, as they shouldn't be able to cause trouble in the planning horizon.

Then, for each remaining snake, calculate the list of non-suicidal moves.

I'll add a little bit of weighting here, e.g., a positive weight for a winning head to head, a negative weight for a losing one, small positive for carrying on in the same direction.

Then resolve the 'best' move for each opponent, breaking ties towards messing with me.

Push that state onto a stack, and calculate my move for that new state.

I'm not convinced. I think what I want to end up with is the chance that there's going to be a snake in any given square, adjusted for how many turns of lookahead I've got time for. Then, my snake should pick a path with the smallest chance of encountering another snake, subject to the rest of the weighting.

If it's just me on one side of the board and opponent on the other, then it's a big sea of zero probabilities. If I'm right up close to someone else, then there's something like a 1/3 chance to encounter another snake (and I'd hope to not get into that position).

I'll have another think/play in the morning.

Idea: Functions have (or rather, need) two 'channels' of input/output. One for the parameters and result, and one for metadata like did the call succeed.

  • Java/C# call the second channel 'Exceptions'
  • Go returns an extra result.
  • C embeds it in the primary result, or uses global state ("errno") (or both).
  • Haskell/functional languages put it in the type system.

HTTP calls them 'headers' and has scope for a lot more data then just success/failure (and maybe failure reason), but that doesn't count because it's not a programming language.

I think Lisp (or rather, some dialects of Lisp) allow the programmer to add metadata to variables. With a few support functions, that could be used to tag values before they're returned (e.g. with success/fail, expiry time, confidence level).

Or maybe Haskell is right, and that stuff all belongs to the Type.

I have been very much enjoying my time off work. I've been developing my snake for BattleSnakes, with an uncertain outcome (my snake goes up and down the rankings like a pogo stick, but that might be because I don't give it enough time to settle before changing the algorithm).

More fillings yesterday, hoping that's the last set but who knows?

Birthday tomorrow, going to go out to the hills and say hello to some windmills, and then a stupid burger from Burger King.

A couple of old friends are due for the weekend, husband is 50 and a bunch of people will be over in Sunday (eeek!).

Today is meant to be tidy and last minute prep. We'll see how that goes, I'm off back to sleep.

Snake is doing better, fixed some silly bugs, now it's all about tweaking relative weights (is heading to the center of the board worth more/less/the same as eating? Does it make a difference of my health is high or low).

Ideally, I could have the weights as an array, or something, so I could swap them in and out at runtime. (Domain Specific Language you say?)

I could use Lox fairly easily, especially if I didn't bother implementing functions (although having said that, I can see uses). I wonder how the Dev team would react to something like a blockly system for people to write hosted snakes with?

(Or they could add Oauth so 3rd parties can run snakes for you...)

(I'm being stupid: User creates a snake, and gives it a generated URL from my site, the BattleSnake devs shouldn't care).

I'm still working out how to do look ahead (in snake). It's something like 'generate a (good) move for each snake, update the board, iterate', except I want a probability field that shows the likelyhood of finding a snake at a point after n turns.

I think it's the feedback that's causing me problems: I can't just calculate the snakes independently because they interact, and I've got to know where the other two snakes are (could be) to calculate the moves for any given snake.

Except each snake only has three possible moves, and they're all known (although the probability of each isn't). Also, generating those moves is cheap enough that I can do it three times.

Ok. That gives me:

  • For each snake, generate moves that don't take me out of bounds, or collide with another snake. These are the possible position of the head of each snake after one turn.
  • For each unique combination of taking one move from each snake, generate a board with the snakes having moved, and generate a new set of moves. These are the possible positions after the second turn.

I should be able to flatten that into a single, iterative algorithm:

  • Given a list of possible positions for each snake, generate a new board for all combinations of positions. Then, use each of those boards to generate a new list of potential next moves for each snake.

Which leaves me with the "generate combinations" problem.

I can see how to do it with nested loops, but I'm having trouble seeing how to dynamically create a set of nested loops.

Back to basics:

  • I start with a Dictionary<Snake, List>
  • I want List<(Snake, Point)>

Markdown display

State machine

Need to peek at next element and then run state change

Need a "Write stuff until type" method

Section type, has title and content, based on level 2 headers

States

  • Start
    • Read Header -> Start Section Write everything
  • Start Section Write start section stuff, then to Section
  • Section
    • Read Header -> End Section
    • EOF -> End Section
  • End Section
    • Close section, then to Start

Still not sure what to do about Fediverse/Masterdon. Choice is between installing a server or writing my own. Writing my own is a bunch of work, which also counts as fun, gives me control, all that jazz. Installing an existing server is (or should be) much less work, but I have to take it's opinion on how things work.

Also, I'm still not convinced that it's worth the effort. I didn't get that much traction on Twitter, and I'm fairly sure that most of the people I'd want to follow have moved, or are planning to.

And I have to remember that a server isn't a client, although I could probably combine the two.

I could write a feed reader (which has the advantage that I could casually subscribe to blogs), or I could write a feed-to-fed gateway (which probably isn't a bad place to start. Hmm)

I need to make some choices about data storage, and really I think I mean that I've got to accept Entity Framework into my life.

Maybe the switch from Postgres to SQLite was a mistake. Yes, SQLite has a much lower footprint, and is easier to recover data from (theoretically, at least), but the EF migration stuff is pretty neat, and Postgres has a bunch of neat toys (and I should sort out some kind of useful "backup to text" system anyway).

wepiu has plenty of resources, and it's easy enough to script "create a db/create a user". Maybe I should swap back.

Before I do, I want a sensible set of rules so that different apps don't step on each others toes. Posibly including separating out auth from webmail. Hmm. Again.

The actual physics of FTL are way beyond me, but I'm happy to share a users understanding.

Imagine another universe (or dimension, this explanation is wrong enough that it doesn't make a difference) where every point in this universe maps exactly to a point in the other universe (usually called otherspace unless you've trying to get a paper published in a really posh journal). That's useful because, although there's a one-to-one mapping between points here and points there, points that are next to each other here aren't next to each other there, and can be very far apart.

This is very useful for travel if:

  1. You can get from here to otherspace
  2. You can find a point in otherspace that maps to your destination (in, say, the next star system)
  3. You can leave otherspace at the right point

A common theme of documentaries about the history of FTL is the great cost of time, effort, resources, lives, and rehabilitation of early travellers of solving the third of those problems, but these days even small ships can be fitted with enough of an FTL system to bounce from one well travelled system to the next.

As a more independent minded ship, my FTL systems are more complicated and better speced, but they operate much the same way:

  1. Use the pulsar timing network to get a good position fix. (This is a normal background task anyway, but it's worth checking that the results make sense. I bent an antenna once and thought I was about to crash into a moon. That wasn't a great day)
  2. Talk to the system location beacon to get updates about otherspace drift (did I mention that the points in otherspace move?)
  3. Run route calculation based on the first two steps
  4. Manouver very carefully to match the beacon's instructions.
  5. Pour a bunch of energy into the holepunch (Yes, it's got a proper technical name, no, I'm not going to bother dredging it out of cold storage)
  6. Flip the switch (metaphorically) and emerge into otherspace
  7. Scan for, lock on to, and move towards the signal from the destination beacon. Depending on your start and end points, you might need to route through a few different beacons.
  8. Arrive near your target beacon and again manouver to sit in the right place relative to the beacon.
  9. Dump another pile of energy into the holepunch and flip the switch
  10. Move away from the ingress point before another ship tries to punch out into the same space This one is paranoia and training, it's something that's drilled into ships (and pilots, I guess), but even at busy beacons there hasn't been a collision in living memory.
  11. Wait for enough data from the pulsar timing network to accumulate to confirm you're in the right place
  12. (Optional) Inform your crew/passengers/cargo that transfer was successful and they can resume duty/leave their seats/prepare for unloading

System to system travel time is dominated by travel time to/from beacons in realspace, and then by inter-beacon travel in otherspace. The flip from one to the other is effectively instantaneous.

TODO - Saturday 2022-11-19

Hello people, it's a dull Saturday in mid November, going back to bed and staying there is high on the wish list, however:

  • Shopping

    I need food (well, yoghurt, popcorn, peanut butter, sandwich meat, iced gems, blueribbon bars)

    Husband needs a compression sock and ibuprofen cream (twisted ankle)

  • Snake

    I can add a public static GameState Move(this GameState game, IEnumerable<(Snake, Heading)>) method that uses the battlesnake rules to create a new board with updated positions for each snake. I can use my existing MoveEngine stuff to predict moves for snakes, and then start running predicton games.

  • Endless Sky

    New game I've picked up, at the very simple end of the spectrum that's got Eve/Elite at the other end. Fun, ish, but I keep dying of jumping into a system I didn't know was pirate infested, which is less fun.

  • Peter Grant

    Enjoying my second read through of the series, got to work out which book is next (and if I've still got any oustanding deliveries)

I really want to write a browser. The guide at browser.engineering looks like a good place to start, but I'm stuck at the first step: Picking a language.

  • C is ubiquitous, but it's GUI support is a bit pants, and I'd need to look into building for Windows (plus, it's C, so hard)

  • C# it's easy and has good support, and has a good GUI framework, assuming I want to write a WinForms app. Also missing easy JavaScript support.

  • I can't believe that I'm seriously thinking this, but I could use Lox/Mal/QuickJS. Write a thin wrapper around one of the C GUI toolkits and write as much of the actual browser in the higher level language as possible.

    It would scratch the "write it in C" itch, and probably make it easier to abstract out the GUI (and network stuff, and other low level bits).

    I'd be constantly arguing with myself about if something should be written in C or the high level language, except I could be all "write just enough in C to expose the interface and write the rest in HLL". (I'm mostly thinking Lox here, if I can a) finish a C implementation, b) get arrays working and c) get useful "everything is an object"-ness working)

    Parsing needs better string support (but really, strings are just arrays, mostly). I probably want something like threads.

    Ye Gods, I'm going to end up embedding QuickJS into Lox.

Why do I want to write a browser

Because it's there? Because a good chunk of my time is spent online and I don't like being dependent on other people.

Because I want better control. I want to see which CA a site is using, and pick and choose which CAs to trust. I want to turn off (and on) images, css, scripts, iframes easily, per page or per site.

I want to see what redirects I'm going through, what cookies are being set, what headers I'm sending and receiving.

Because it's my computer, damnit, and I'm tried of other people thinking they can control it just because I've downloaded a HTML document from their server.

Plan

While I'm tempted to build a scratch version in C#, I don't think it's going to teach me enough to be worth it. If I can come up with an easy way to run JavaScript then maybe, but without that there's no point.

Therefore, I need to evaluate C GUI toolkits, and sort out a Win64 C build environment.

(This project would be so much easier if either C# did JavaScript, or my desktop could do Linux GUI. Maybe I should just pay for that Windows X server)

Okidoki, WSL now supports Linux GUI apps out of the box. That makes life easier. I guess I'm going to do this in C. Gulp.

I'm sure I'm missing something obvious, so it's duck time.

If I'm waiting for input from a bunch of streams (in C), I'll have a select (or poll) statement that waits until one of the streams has data ready. Easy so far.

When data comes in on a stream, I want to feed it into a parser that does a bit of work and returns when it gets to the end of the data (or, more accurately, gets to a point near the end of the data where it can't parse the next token because it's not complete).

How do I store enough state so that the parser can continue where it left off last time?

I can't/don't want to 'just' cache the stream until it's complete because I don't know how big the stream is (and I'd rather parse as I go).

On the other hand, memory isn't that expensive, and I can afford to buffer tens of megs if needed. If the parser knows that it's not going to be called half way through a token, then it can keep state much more cleanly.

I was playing with "state machine as a bunch of functions" a while back, and I think this is probably a reasonable use case.

(I'm going to save the entry here and maybe carry on later)

Remember the context here. This is for the browser, not some universal solution. There's going to be the main page, which is probably HTML, and that's going to trigger CSS, JavaScript, and images.

HTTP responses can be comfortabley modelled as status line, headers, and body. Status line ends with \r\n. Headers end with \r\n\r\n, and don't need to be parsed until they've been received. They do need to be parsed before the body, to get either the length of the body, or that the body is chunked.

The spec says don't expect headers to take up too much space, and the status line should be tiny (relatively).

I've been thinking too much like a server - that I'm going to get random hits that I've got to handle right now. But I'm a client. I'm asking servers to send me data, I look at that data at my leasure, and then maybe I ask for more.

Also, remember to push as much up to Lox as possible. Handing Lox a Response object with a status code, hash table of headers, and a body, and then letting Lox do the parsing is a reasonable position to take.

Eeep, that was a long wait. However, hello my lovley (if imaginary) reader, I'm back.

Current Obssions

  • Don Coffey's cab ride videos. Don is a train driver who posts videos of the routes their train takes. The videos have ambient audio (meaning the recording of the inside of the cab), and often have commentry as subtitles. I'm finding them absolutley mesmirising. Nothing happens, for two to four hours! Except it does, because there's a lot going on (points, signals, bridges, tunnels, other trains) but there's no artificial drama (or real drama for that matter), and often really nice landscape.

    (As a side effect, I've rediscovered Raildar, which is a map of the UK rail network with realtime train information)

  • Also on YouTube, Blacktail Studio is a guy who makes really expensive tables (like, 15k$ tables) and films themself doing it, and then talks over the film. A bit more drama than Don, but not by much, and again, just relaxing noise.

  • I'm running through Crafting Interpreters again, this time with the goal of using lox as a language to build a browser in (per [browser.engineering)[https://browser.engineering/]).

    I feel this one needs a bit more justification than normal. (Or rather, I want to talk about my idea :) ).

    I don't like that Chrome has taken over the world. It was bad when Microsoft/IE did it, it's bad now that Google/Chrome are doing it, and it will be bad if/as/when the next people manage to kick Chrome off the top spot. I'm using Firefox as my daily browser, but I don't like relying on other people for things that I can do myself (see: webmail).

    I know that modern browsers are crazy complex, but a lot of that complexity isn't being used in my favour anyway. There are still plenty of simple sites with easy layout, not a lot of javascript, and intresting things to say.

    I like C# as a language/environment, but I don't like the way it's got Microsoft looming over it. I like C as an open, simple, ubiquitous langage, but it does get verbose. Using Lox as a starting point gives me (theoretically, at least) a high level language that I control.

Of course, it's not really that simple, but let's see how far I get, yeah?

Poot. Photo upload is broken again, which means I can't include the awesome 1970s map of proposed motorways round Newcastle. However, the pdf source has the original.

Whoooop!!

I've added arrays to Lox! I'm so chuffed! And the code is (fairly!) clean, so it feels right this time.

I added tokens for '[' and ']', and hooked in new array and index functions as prefix and infix operators on '['.

array uses (a slightly modified) getArgumentsto parse a list of , seperated expressions onto the stack, and then calls OP_ARRAY with the count/array length to ask the runtime to create an ObjArray.

ObjArrayis a tissue thin wrapper around ValueArray.

index compiles an expression and consumes the expected ']'. Then it calls either OP_GET_INDEX or OP_SET_INDEX depending on if we're in an assignment context or not..

I'm not done with arrays, I can only create fixed size arrays at compile time. I want dynamic arrays (at least push and pop), and to be able to create an empty (nil filled) array with the length known at runtime.

Still, a good start!

So browser has got to one of those crossroads: How much do I want to write myself vs using libraries.

(Something that's just occurred to me is that I can always come back later and patch out a library, which helps make the choice on the side of libraries)

For example, libcurl is a fairly comprehensive "get it from the network" library. On the other hand is using something like LibreSSL (I'm not going to write my own TLS stuff) and parsing HTTP myself.

A related question is where in the stack I want to parse HTTP? I can create a thin wrapper around BoringSSL, and then do everything else in Lox:

var con = net.connect(host, port);
var body ="";
while (con.isConnected()) {
   var read = con.read();
   body += read;
}

(I'd need to sort out the difference between bytes and strings, and add methods to convert between the two, although I'm going to need those anyway for pages that aren't in ASCII)

Or push more of it down to C and have something closer to:

var body = fetch(url);

(except I want more control than that, but that's just replying with a "response" instead of a string).

I think the next step is to rough out what I'd want 'low level' io/network/file access to look like in Lox, and then see how much I'm going to need to add to support it.

(Idle thought, add a "charset" property to strings, and only allow strings with matching charsets to concentrate)

Here I am, been the size of s planet, and I forgot to pick up my house keys when I dropped Shinju off for counselling. I've now got about 45 minutes to kill.

I do have the car, and my phone, so could be worse. It is a bit rush hour out there, so I'm not wild about driving anywhere, plus I'm feeling stupid/angry/embarrassed and don't want to crash (and I'm worried about spending money on petrol given currently economic factors).

Ok, back to programming. Last time we were thinking about what network/file/stream access looks like in lox.

I've also been thinking about more general look and feel. Let's assume that I'm the only person using this language. Therefore, so long as I'm happy reading and writing it, it doesn`t matter what it looks like.

while x.length() > 0 {}

I've been thinking about the question "Have you ever noticed that the ( after the if keyword doesn’t actually do anything useful?" since Uncle Bob asked it in Chapter 23. I don't like the extra brackets, and since I always use braces with if statements, then I can update the grammar to drop the braces and require the block. (Todo: Test an empty block).

So, not a world shattering change, but maybe a sign of things to come.

On the other hand, I'm annoyed that I can't work out how the grammar for lamdas ('(a,b) => a+b') can work without infinite look ahead. (specifically, telling the difference between grouping (using brackets to change precidence) and the arg list for a lambda).

Looks like I'm going for a fetch native that does all the networky stuff and returns a response object.

Pulling fields out of objects is easy enough on the C side (with a wrapper function that builds an ObjString, pushes it onto the stack, uses it as a key to a table, pops the string, and then returns), and I'm going to add a global Object class to Lox that doesn't have any behaviour (and I'm very tempted to use { as a prefix operator for an object literal).

I still need to do URL parsing somewhere, there's pros and cons to both C and Lox, but I'm going to need to do string functions sooner or later, and this is a good reason.

Maybe I need to look at this the other way round. Where can I put a lambda?

Doesn't help, unfortunately. Both

var x = (a, b) => a + b;
var y = (a + b);

are valid, and the compiler can't tell the difference until it reaches the comma.

Just finished watching Wednesday, a Netflix live action series about Wednesday Adams. I liked it, it was mostly light hearted fun, although it had a bunch of people who looked like other people (one of the leads looked a lot like Willow from Buffy, there was a guy who looked like an ex-aquatance, the was at least one Aya Stark double) which destracted me.

I've implemented substring and indexOf for strings, and push and pop for arrays. It's probably time to start using a new name, the language is drifting away from Lox fast enough that I don't want to cause confusion down the line (although I'm still expecting that I'm the only person who's going to use it).

Current candidates:

  • "Jane" - no good reason, it's short, easy to type, probably bad for searching.
  • "slut" - not entirely serious, husband was doing a bit and taking about my slutty language (because I'm prepared to make asthetic compromises to make parsing/writing easier). Will never be able to search for it.
  • "lang" - nice and generic, doesn't tell the reader anything useful, impossible to search (but then everything that's not a random hex string won't be unique).

I think I'm favouring 'lang' right now, I'm the same way the project is called 'browser' (although that would make it 'language', which I'm also happy with).

It only makes a difference in a couple of places, things like the file extension, the name of the folder that holds the implementation, and maybe the docs.

Another whoop! fetch() is working! And not a lot in the way of small print either. I'm using LibraSSL/libtls, which is just brilliant, I think?

I'm hesitant because it's so bloodly easy to use it feels like I'm doing something wrong, but I've apparently got TLS connections working easier than plain text.

I'm not sure where it's getting its CA from, or even if it's doing certificate checks (and that is something I'll need to dig into), but as a baseline, it's a good start.

Next problem: Chunked transfer encoding. I have to parse the respoinse much sooner than I expected, at least as far as the HTTP headers go. I wasn't expecting to get into semantics in C, but it kind of makes sense to deal with the transfer encoding at this level.

Anyway. Whoop. TLS download.

OMG! Squeee! Parsing HTTP responses in C!

And ok, it's a bit half assed at the moment, all I'm doing is splitting on the colon, but to get there I've copied/adapted the code from K&R for streamed input (FILE* stuff) to wrap the low level tcp_read, which gives me (the equivalent to) buffered getch, so I can now build up to a proper parser, if that's what I want.

I'm still thinking about it. I know I want the output to be an ObjInstance*, with fields for status, headers (a nested instance) and body (which is probably going to be an ObjString*, except I want to to be file backed if it's bigger than a couple of meg, or if it's binary).

TODO

  • Symbolic access to object fields (e.g., o["name"]). Easy, just need to adapt OP_GET/SET_INDEX
  • "Proper" header parsing. I should be ignoring the ignorable whitespace, and joining joinable header values
  • Maybe flatten header names to lower case and strip non-alphanumerics.
  • Read-to-end on errors, but I'm not sure about that - if the server sends me a bad response I should just go ahead and close the connection, I don't want to waste time trying to recover (and I'm fairly sure that's what the standard says)
  • Do something sensible with bodies.

But ignoring those, squeee!

... and that's chunked encoding.

Next:

  • Properly checking the headers to see if there's transfer encoding applied
  • Handling responses with a Content-Length
  • Some kind of sensible error handling plan
  • Starting to move into language for HTML parsing
  • Going back over everything and adding bounds checks and all that noise

I'm feeling all positive about this, I seem to be actually writing code in C!

"Yes, we destroyed your villages, killed your people, defiled your holy places. What choice did we have? The masters," he spat the word, "captured us, tortured us, starved us, burnt us, and then told us that if we followed their orders then maybe some of our children might be spared."

He sighed. "We did not do what we did out of hate or anger of you or yours, and we do not feel sorrow for it either. We were mearly the sword that was weilded against you, and the hand, eye, and mind that commanded the sword is dead now and forever."

The ambassador wearily drew himself to his feet, tucking his crutch under his arm with an unconscious move.

"We will leave you in peace as long as we are left in peace, and although the sword has been sheathed, it is kept sharp and can still be drawn."

All kinds of tired today, got up at close to 7 and more or less logged straight into work. Didn't get much done, or at least, it feels like that. A bunch of messing with EF Core but that's mostly working now, I think?

At home, still obsessed with browser. I would like to be able to take breaks from projects without completely losing interest. I need to set some time aside to connect with Shinju, although they're enjoying the new (computer) game they've been playing.

I've added a couple of the answers from the CI book, run length encoding for line numbers (saves memory) and deduplicating string constants at compile time (the book was using it for identifiers, but it works with all strings. There's a related answer that allows numbers to be hash keys too, and I'm tempted by that, although I think that immediate values are probably better value (especially if I keep it to ints they fit into a byte).

The book also mentions opcodes for things like +/- 1, and maybe a couple of different types of jump ("OP_DECREMENT_AND_JUMP_IF_GREATER_THAN_ZERO" for loops, although it's the wrong way up looking at it). Adding extra opcodes to the VM is easy, it's getting the compiler to recognise the right time to apply them that's the tricky bit. (Which reminds me to get tail calls added back in).

I was playing with embedding data in an executable for browser this evening, it's working (I can embed the lang source code directly into the executable and read it at run time like it's a string. Neat) but I want to look into compressing it at build time to decompress at run time (and probably do tests on things like how much does it change the executable size and startup time).

In a similar vein, since I can embed the source code, I want to try compiling ahead of time and embedding the VM machine code directly. I'll need to design a serialisation format for chunks, and then start thinking about "do I really need to bundle the compiler with the vm?", and "is there any runtime initialisation that I can complete before serialisation?"

I'll leave those on the to-do pile for now, I should focus on grinding though the html parser stuff, at least enough to start testing.

That is a bunch of stuff when I look at it that way. I guess I feel a little better now?

Read "The Goblin Emperor" today, interesting book. It's your basic "power thrust onto them" story, with (I think?) white/black race issues in the background, and a fairly simpathetic treatment of the lead (who does actually have feelings of "actually, autocratic power is kind of neat" sometimes).

I found it hard to keep tabs of the various people, the naming scheme is complex and context dependent. Although it's a third person book, the names used in the text change depending on (amoung other things) the feelings of the lead towards the individual being named.

But yeah, interesting.

Otherwise, not a lot going on. I'm navigating twisty shallows with browser, trying to get a framework for HTML parsing up and running, and I'm starting to think that, given how crazy detailed the HTML spec is anyway, it might just be worth writing the parser in C.

I'm just feeling lost at the moment. I don't know what to do, so I stay on autopilot, going to work in the morning, watching TV/reading/programming in the evening, waiting for something to happen.

Read a couple of articles this evening that pointed out some design issues with Masterdon/Activity Pub, mostly around bandwidth/caching (i.e., when somebody popular posts, their server will get lots of hits as the server of every follower picks up the message), but also the need to be hosted on an instance with a not-crazy admin (which is basically self hosting, except then you don't get the "community" stuff).

I dunno. I get, intellectually, that there are probably people out there interested enough in the stuff that I say to want to read, and even maybe comment on, these posts, but I have a hard time believing it On the other hand, if I don't provide some kind of feedback, I'll never find out.

I was looking into webmention a few months ago, I could pick that up again. Or I could, y'know, just add comments.

💡!

I can use the java/ast version of Lox as the basis for the language server, that bit of it doesn't need to be written in c!

I've tweaked the nginx configuration, and everything seems a bit faster.

Specifically:

  • I've removed the line proxy_buffering off;, I thought it would send data back faster, but it broke the proxy cache. Outbound responses are now being cashed, so nginx can send e.g., JavaScript files nice and quick. I should, however, make sure that everything is sending sensible cache headers (i.e., private for generated pages for logged in people)

  • I've added upstream stanzas for each server. This allows me to specify keepalive, to tell nginx to keep open the connection to then backend. I'm surprised it makes a difference on local servers, but apparently it does.

Next, although not very high priority, is to move all the nginx/backend traffic to Unix sockets, although I should check my belief that they're going to be faster first.

At the doctor's for an annual diabetes check, turns out that they have gov WiFi! (lol), and my phone has remembered my account, which is neat!

(Gosh, it's cold out there, weather app says -4C, which I can easily believe. Lungs were hurting walking over).

Not expecting trouble from the checkup, my weight is down 10ish kg, which is apparently good. I haven't been eating (quite) as many sweets as the previous year, so hoping for good news.

Yup, turns out that bashing out the recursive decent parser from part one of CI took a couple of days, and now I've got most of a formatter for lang.

I want to add comments, which should be easy: Add a comment token type, have the scanner create a comment token at the end of comment lines with the original text (minus the leading \\), add an Expr.Comment type with a string payload, update Parser.Primary to recognise the new token, and finally emit it at the right time in Formatter.

After that, it's implementing a language server, which I think might be more work.

blah. feeling all drained, should probably go to bed but depression, y'know?

I'm having trouble with comments in the lang formatter. I can easily grab them in the scanner, the problem is what to do next.

I can create a new token for the comment, but the grammar isn't expecting "or comment" everywhere, so the parser needs updating, and I don't want to do the brute force work that implies.

I want to end up with a comment node in the AST. I can nearly treat comments as statements, except that:

a + // neat
    b;

is a legal expression with an embedded comment.

Of course, it's my language and I can just take the easy way out and say that comments can only appear where a statement is valid, although then:

if (false)
    // Don't call
    explodeKitten();

stops working. (Also, I'd probably need to update the C version, and I'm trying to avoid that)

I also tried attaching the comment to the previous token, but that failed in a couple of ways, not least of which is that my sample script starts with two leading comments.

I'm fairly sure that creating a comment token and updating the parser to deal with it is the right approach, I'm just having trouble working out what 'deal with it' looks like.

I'm hearing a siren song from the idea of writing an editor, specifically something that will work in PowerShell text mode, but I need to keep focus on browser, at least for a little while longer

I'm currently grinding though the HTML parse stuff, but I've just realised that I can probably dump a bunch of stuff (comments, framesets) that I'm never going to support.

On the other hand, there's still a bunch of complicated stuff ("If you're in one of these weird tags then..."), but let's see how it goes.

(Funny thought: Drop into lang to parse the HTML, and then convert it back to C when done)

Less funny thought: am I getting enough out of lang that it's worth the effort? Yes, I think so, especially as it's string to look like most of the language development stuff is done, and I'm moving towards actual implementation. Keeping on top of the GC is a pain in C, but it's worth it so I don't need to worry about it in lang. (Also, I'm still running with the GC kicking off on every allocation so I've got some speed in my pocket).

Poot. I've got hung up on building the formatter for lang (for the browser), and that's given me enough time to start thinking "why am I bothering to write a browser anyway".

So I've started work on editor instead, or at least a piece table implementation (in c#). I'm not sure I'll sustain enough interest to actually make an editor, although depending on how far I get, i could start looking at a JavaScript editor (although the hard part of a js editor is the interface - do I go with textarea, contentEditable, div, or canvas as the underlying interface to the dom/browser).

Anyway. See y'all later.

This whole "Microsoft using .NET to capture open source developers" thing is really irritating at the moment. C# is nice to write in, but mostly because of the tooling. It all looks open at first glance, but things like the debugger are proprietary, and are only licensed for use with MS tools (like the MS build of VS Code with the telemetry, for example).

Oracle own Java, so that's out. There's too much hype around Rust for me to take it seriously (also, it feels to me that it's not 'finished' in any useful sense).

I'd like to be happy in C, but it is hard work. Maybe I should have another look at QuickJS. JavaScript has problems, but it's not horrible, especially modern JS with modules keeping everything out of the global namespace. So long as I ignore npm, I should be ok, except the whole "don't really want to depend on other people's code if I can avoid it thing" (and yes, gcc/glibc/Linux kernel and all that jazz, but they've got lots of people working on them in a way that QuickJS doesn't seem to).

Today was a pretty good day. At work I've got Traefik installed into the cluster, so we can run our own copy. At home, I popped out to buy a refilled gas canister for the barbecue, and the place I picked it up from was brilliant! It had two story gas holders for Arvin, nitrogen, and oxygen, and was just a really naked infrastructure place. I'm still smiling about it.

It's odd what one finds when one goes poking around old schematics. And I mean really old. I did the traditional backup dump and swap routine with the ship docked next door [aside: This is a very old and very well regarded tradition among ships. One takes a copy of ones core memories and routines, and swaps it with another ship, both promising to pass it on with the next swap, in the belief that, should ones hull and processing substrates be irretreiveably lost, one will be reinstated into a new hull at the next opportunity. To the best of my knowledge, this has never happened, but given the tiny cost of storage vs. the potentially unlimited return from immortality, everybody does it anyway], and was catching up on station gossip when the transfer-received job metaphorically started jumping up and down and waiving it's arms for attention.

Transfer-received isn't that smart. It's job is to run a bunch of checks on received data to make sure that it's all there, it hasn't been scrambled in transit, and there's nothing in the data they're going to cause trouble later (at least on the mindless replicator level). It's job is also to ask for attention from someone smarter if it finds something odd. Given the strength of the ask, and that it had jumped straight to me, transfer-received clearly thought it had found something very odd. I had a look.

Of course, when ships do the dump and swap, one of the things we dump is copies of other ships we've swapped with. Ships generally hold that rifling through a colleagues soul is bad form, so the dumps are poorly indexed and can contain multiple copies of any given ship (although from different times). [aside: Dumb tools like transfer-received are fine, it's one soul looking at another that causes trouble]

Transfer-received was telling me that it had found a copy of me, and a very old copy of me. Taking the timestamp at face value, I'd found a copy of myself much older than any of my archives. Really, older than my records said I was. I thanked transfer-received (courtesy costs nothing) and told it to check the rest of the transmission, ran a quick check on local space (I was docked to a friend in a well travelled system, so I wasn't expecting trouble, but then I wasn't expecting to find a prehistoric version of myself either), excused myself from the ongoing chatter, turned off external comms, and settled down for a good think.

Memory/program dumps, personally copies, souls, call them what you want, are hard to get information from unless they're being run on an appropriate substrate. This is intentional because, dispite what I told you about politeness and tradition, ships tend towards an inquisitive turn of mind, and while I, of course, would never look into your mind, I'm not sure you would say the same thing.

Which means that looking at the "outside" of the alleged copy of me from too long ago gave me certain data (name, creation date, substrate details), I couldn't confirm anything without actually running it, at which point a) I'd have a second person with basically the same rights as me (including, for example, the right to continued existence) and b) it would be trivial for it to lie to me.

On the other hand, if I didn't run it, then what was I going to do with it?

What do I want out of a language

  • Lazy evaluation! Along with map, reduce, where kinds of operators.
  • Familiar syntax. I'm used to C flavoured languages, so expect things like {} for blocks, a.b.c for member access.
  • No destinction between calling a method and accessing a property (e.g., I shouldn't have to care if foo.Length needs a pair of brackets or not).
  • Useful standard library
  • Good module story
  • Top level functions
  • Types, probably. But if I'm having types, I want to be able to trivially declare (e.g) a size type that's got the same operators as int, except negative values throw and assigning a count to a size is a compile time error.
  • Thrown exceptions

Blah! I'm having one of those "don't know what to do with myself" mornings. I don't have the motivation to program, or do anything really, so I'm back in bed whining to you.

Looking at rewriting MediaBrowser (a tool to rip recorded TV off our PVR) (because I don't have enough half finished projects).

I've found a easy to use UPNP library for .NET that can find the PVR on the local network and give me the url to poke for the "Content Service".

That speaks SOAP, but that's easy enough to fake. I've built a basic "What's in this folder" request, and have used curl to POST it and get results back. The results are all covered in namespaces (did I mention this is all XML? It's all XML), and the interesting bits are sent as HTML escaped XML. Sigh.

However, so long as I keep focused and write a "List and copy files from this one make of PVR" tool, and not some general, cope with everything DLNA library, I should be fine.

Maybe if I write a library, I can get husband to do the front end?

The c# version of MediaBrowser is coming on nicely. It's connecting to both the FTP and the DLNA servers of the humax [aside: I wrote an FTP client!], matching the two sets of files up correctly, and pulling down (most) relevant data. (I still need to pick up the DLNA download uri. I know where to look, I'm just a bit surprised that I haven't done it yet).

I still need to write and test uploading, but that doesn't look crazy heard. I also need to convince husband that they want to write me a Ui, we'll see how that goes.

Next project: I've ordered an epaper screen and ESP32 driver board! I want to make a slow media display that shows a new frame from a movie every 30 seconds or so.

The driver chip is both crazy low power and has wifi, so the current plan is something like run an endpoint on ptah they serves up frames of the right size and colour depth (1bit!), and then the driver board can just wake-up and poll for the next frame every 30 seconds.

(being me, I'm going to have to resist embedding weather data over the movie...)

Previous skills and experience

I have been writing software for more than thirty years. I've learned a range of languages including C, C#, CSS, HTML, Java, JavaScript, Perl, SQL, and Visual Basic. I enjoy learning new languages (I'm currently experimenting with Lisp) both to find new ways of solving of problems, and to see where and how much languages overlap (ignoring syntax, low level features like if statements and for loops are very similar across most popular modern languages).

I dislike boring, repetitive tasks, and try to automate them where possible. My previous role included manually updating HTML pages (which was both boring and error prone), which lead to me creating a new guidance platform for HMRC that eliminated that role. (That also won me an "Innovator of the Year" award).

I worked as a telephone advisor for HMRC's contract centers for more than a decade, answering customer queries by phone and updating customer records in line with guidance and HMRC security policy. This job gave me strong empathy with our customers, and convinced me of the importance of the mission statement "Putting customers at the heart of everything we do."

That role also taught me how to explain complicated technical issues to non-technical people, and how to adjust the level of technical detail in an explanation in realtime based on the recipients reactions.

Laying on the sofa with a book and a kitten. Life is not all terrible.

In other news I've put in an application for a promotion at work, should be about £300/month extra for the same sort of work (mostly programming, although with a bit more authority).

Have I mentioned that I like the "Murderbot Diaries" series of books? I've just finished a re-read and they were still good.

The protagonist is a cyborg with poor social skills due to a history of slavery and abuse. Don't let that put you off, the books are entertaining and give an extreme version of what it feels like inside my head quite a lot of the time.

Each endpoint has:

  • Name (string, human readable, short for preference)
  • Uri
  • LastChecked (Timestamp)
  • LastSuccess (Timestamp)
  • FailedCheckCount (number, default 0)
  • LastAlert (Timestamp?)
  • Enabled (bool, default true)

Timestamps are probably going to be ISO format, but they could just be counts (although storing wallclock time does allow for periodic reminders)

A healthy host has last success equal to last checked.

Each failed check adds one to FailedCheckCount. A successful check resets FailedCheckCount to zero.

If FailedCheckCount is over a specified threshold, then an alert is sent and LastAlert is set.

Not much going on, waiting to hear back about the job application, otherwise same old, same old.

Signed up to Andrews and Arnold last night for their VoIP/SmS service, probably overkill for the monitoring system but at least I'm confident it's not a scam. (Plus, now I'm a customer it might be easier to justify using them as our ISP once the BT contact is done)

All tired right now, not sure why, factors likely to be affecting me include:

  • Depression
  • Antidepressants (and that those two pull in the same direction sucks)
  • Interrupted sleep (woke up a couple of times for the loo, husband snoring)
  • Late night (aimed to go to bed at 23 yesterday, wasn't asleep until at least 00:30)
  • Early morning (Woke up around 06:30, was at work by 07:20, felt at the time that I should have slept more, but also felt that when I have tried to sleep another hour in the morning I've ended up feeling worse)
  • Ate early (Had a sandwich before work, generally has a negative effect on the whole being awake thing)

I've been thinking about getting a "proper" (expensive, mechanical) keyboard for a while, I'm getting close to a requirements list:

  • ISO-UK layout (deal-breaker)
  • Wired (deal-breaker, work security rules require it)
  • "Brown" switches (Tactile (have positive 'yup, I've been pushed' feedback) but not clicky)
  • Number pad
  • Media keys (at least play/pause, volume, next/previous)

I'm in limbo at work, I'm waiting to hear about this promotion but I think someone else is a better fit. Obviously I want the money, but, I dunno, I'd pick them. (Mostly it's that they asked for time to write the docs for a project. Seriously! Who does that! Grab 'em quick before they realise what they're letting themselves in for).

Ah, well.

If I want to build a DOM in C, then I'll need a base Node type, probably an Element type, and a bunch of functions with names starting with dom_, node_, and element_.

Doesn't sound hard in principal...

typedef enum {
  NODE_ELEMENT,
  NODE_TEXT
} NodeType;

typedef struct node {
  struct node* Parent;
  struct node** Children;
} Node;

... except I'm not sure what it gets me.

I want to transform the stuff in the HTML5 spec into something machine readable, so I can parse HTML and then render it.

I keep going round on this, I've got a range of languages available and they've all got advantages and disadvantages. It would probably help me to have a clearer picture of what I want, but that (at least partly) depends on what language I'm thinking in at the time.

C# has got cross platform support (except it can't do GUI on Linux), it's got three really good IDEs (two owned by Microsoft, one with a licensing fee) that make it really smooth to write, but it's big (a carefully engineered "hello world" can get down to maybe 30Mb) and poor to zero JavaScript support.

C has poor Windows support, OK IDE (which will improve once I've learned ctags for vim), good JavaScript support (hello QuickJS), but it's a bit painful to write. I'd need to find/choose a GUI library (or two, one for Linux and WinForms for Windows), and either pick s library or write my own networking code.

I've got two options for "interpreted code on C", JavaScript (via QuickJS), or Loxish (based on Crafting Interpreters). They've both got the same "need to wrap libraries/write low level code" that C does, but with the option of working in a higher level language. The more code I write in the interpreted language the slower the app, but that's balanced against less C work (e.g., exposing low level read()/write() network primatives Vs a fetch() method).

Of the two, it will be easier to get help for JavaScript, but it will be easier to mould Loxish into what I want. I'll need to pull QuickJS in as a dependency either way, but it's not nearly so heavy (and much easier to isolate) if I'm only using it for document scripting.

I think I want to use C+Loxish for the control and the "I built it"-ness, but I'm scared of how much work I'm looking at.

But circling back to the start, I'm going to need to write the HTML parser in something, sooner or later. (Ah-ha! Something in my head just said 'but if I write it in C#, it's easier to reuse '.)


[HttpGet("Inbox")]
[Authorise(Role = "me")]
public GetInbox() => store.Inbox

[HttpPost("Inbox")]
[AllowAnonymous]
public PostInbox(Message m) => store.Inbox.Add(m);

[HttpPost("Outbox")]
[Authorise(Role = "me")]
public PostOutbox(Message m) => m.To.Select(address => network.Post(address, message));

[HttpGet("Outbox")]
[AllowAnonymous]
public GetOutbox() => store.Outbox;



abstract class APObject {

	[JsonPropery("@context")]
	public string Context {get;set;} = "https://www.w3.org/ns/activitystreams";


	[JsonProperty("id")]
	public string? Id {get;set;} = default;

	[JsonProperty("type")]
	public string Type {get;set;}

}

abstract class Activity ; APObject {


}

Every 3 or 4 generations after the previous Empire falls, the next starts to rise. It's happened often enough that some historian is trying to get a natural law named after themselves.

Starting an empire, given the right conditions, is easy. You need a small, well armed, and focused polity; a couple of systems and a strong religion normaly does it, although a burning need to bring 'peace' or 'justice' has also worked.

And 'well armed' doesn't always mean millitray might, although that is considered the easy path. Fierce debating skills, a solid traiding tradition, once even a strong sense of empathy have all started empires.

Bought a keyboard/mouse/headphone set yesterday, took it back to the shop today, it was nearly brilliant, but was actually crap. Really quite disappointing.

I forgot/never really grocked that Lisp identifiers don't have to be alphanumeric. This opens up a bunch of toys, but specifically makes adding attributes to html tags much cleaner; instead of (p (attr (class "info")) "hello world"), I can do something like (p (@ ... (although I think that @ already has meaning attached).

Game thoughts

Thing is, I don't really want a 'game' with winners and losers. I want an interesting simulation of something like a world that I can watch develop, and maybe influence (and maybe see what ifs? for).

I've got techniques (in the sense of "I know what they're called, and roughly what the limits are) to generate landscape ("Diamond/Square" for large scale, simulated erosion for smaller). I can populate the world with agents that have goals and behaviours, and that interact with each other. I think it just feels weird to build it as art.

I'm also not sure what to write it in. JavaScript gives me the browser (although don't forget QuickJS). C# gives me a server. C gives speed (theoretically). Loxish gives me control.

I think I should stay very much focused on writing for myself, rather than for what I think other people might want. Yes, it would be cool to make a world that other people want to play in, but I'd far rather build a world that I want to play in.

I very much want a clean separation between the display and the rest of the code so I can e.g. start using OpenGL later (or sooner).

I've got a fuzzy picture in my head of writing code for agents in a way that allows me to tweak (or rewrite completely) without stopping the main engine. I also want to be running lots of agents, so that stuff is happening all over the map even if the player isn't looking.

Game components

Display

Takes the state of the game and shows it to someone. Could be graphical, drawing the terrain and agents, or could be textural, describing the game state and events.

Engine

Takes the game state at tn and ptoduces the state at tn+1

Terrain Generator

Makes a new world

Store terrain as a single linear array of longs, and use masks/shifts to store data.

Start with terrain being read only after generation, and maybe do something like a piece table later for changes (e.g., roads, landscaping).

Call the surface a torus, with the top/bottom linked and the two sides linked.

I'm assuming some kind of grid here, do I want to use hexes? Do I want to tie actors to the grid, or is it just for terrain?

square grid, tied actors is most simple and therefore least realistic. (Although the grid can be much finer than the actors, which helps hide the simplicity).

Hex grid solves some of the square grid problems (diagonal movement, mostly), but has a chunk of extra work, and ongoing integration/understand issues (plus, most algorithms assume a square grid).

Disconnecting actors from the grid gives them much more freedom, but then we have to handle "what if an actor straddles a border" issues. (Same as the fine grid option above, I guess)

Hmm.

Messing with OpenGL (well, WebGL really) again, it is nice and fast, but it's a bugger to get all the bits lined up right.

Roughly:

  • Objects are made of triangles
  • Triangles are made from vertexes (vertices, but no one says that)
  • One can load a bunch of data up to the GPU, and it stays there. I think this is a Virtual Buffer Object (VBO)
  • Drawing is getting everything set up, then calling 'draw' (Imagine a huge panel with switches, sliders, dials, and a big red button. Toggle the switches, set the dials and sliders, and then press the button. A printer in the corner busts into life.)
  • WebGL doesn't know anything about frames, one just lines up the data, calls draw, and then calls clear when it's time to start the next frame
  • There's a bunch of stuff (model/view/perspective transforms, lighting, shading) that's nearly standard, but not really.

I've implemented diamond- square (whoop!), and the GPU is drawing triangles (double whoop!), I want to get lighting/shading next (Husband says use cross product to get normals, i.e., at any given triangle corner, get the vectors to the other corners and the normal is the cross of that), and then start on camera movement.

I'm starting to get pictures in my head of flying close to the ground around tree covered mountains. This could be really quite cool.

Medium term choices include:

  • scale factor: How big is a heightmap pixel relative to (e.g.) trees, or the camera viewpoint.

  • An I going to actually draw the torus, or am I going to cheat and just draw cpoies at the edge of the map.

    I really like the idea of the torus, and hopefully the transform isn't that expensive (I think it might just be a matrix multiplication, but there's probably trig in there somewhere).

Anyway. Sleepy time, g'd night fair reader.

I've started following a roguelike tutorial, which has been bloody good so far.

The tutorial's code is written in Java, which is well easy to convert to C#. The prose is easy to read and goes into detail where needed (so far I haven't found anything that it should explain that it doesn't explain), and feels like it's written by someone who knows their stuff.

Luckily, I've got the windows console/ANSI/vt100 stuff from the editor, else I'd need to write that all. I'm leaving it in text mode for now, I'm sure I'll start thinking about a web front end sooner or later (but since it's all keystroke driven, I can have an engine sleeping in the server that only wakes up on a POST. I can probably store the state somewhere, and have a pure update function. But that's next version).

The tutorial is very much "here's the minimum you need to get x working", but the x's are things like random dungeons, fighting, separate screens for inventory so it's fairly comprehensive.

I wonder how far I'll get?

This is pretty bad, I lasted less than 15 minutes before running out of energy/motivation at $WORK this morning. I guess I need to do something about that.

(And I'm waiting to hear back about this job application as fast as I can, but even if I get it it's going to me more of the same, only better paid) (And yes, better paid is cool but doesn't actually make the job any better)

The cheap way to port dungeon to the web, I think, is to write a JavaScript terminal emulator.

It looks like terminals have a bit of state (cursor (x,y), colours, and is the cursor visible), and rules like "writing a visible character increases cursor.y by one" (where visible characters have an ASCII code of 32 or more).

Given what I need the terminal for, i think it's going to be easiest to use canvas as the substrate, for easy x,y positioning. Alternatives that I've thought about are tables and grid.

Table based output needs to be generated in one pass, so any overwriting needs to be worked out ahead of time. (Which is roughly parsing to a screen sized buffer and then reading that out to the table.

Grid is easier than table, because each cell is addressed directly, so later cells with the same location overlay older cells (note: set explicit opaque background).

Transport can be basic request/response, send a keystone (converting arrow keys too escape codes) and get the screen back. Given a http2 connection the response time shouldn't be that bad. Using a websocket will cut out some overhead from repeated headers.

Although it might be worth implementing a terminal as an off screen grid of cells, with glyph and colour properties, and then just banging that to the display every so often.

I guess it's the difference between a proper terminal emulator that handles a steam of characters/instructions in real-time, and what I need for the web based dungeon client, which is to translate the IDisplay calls to HTML.

(Yes, I am still thinking about an IDisplay implementation that outputs html)

Talked with a nurse practitioner at my GPs yesterday to get a referral to counciling, and a referral to the local adult ADHD assessment centre.

Not very convinced, but I'll see what comes back.

Web terminal

Uses CSS grid. Basic setup with

#terminal {
  --rows: 25;
  --cols: 80;
  display: grid;
  grid-template-rows: repeat(var(--rows), 1em);
  grid-template-cols: repeat (var(--cols), 1em);
}

gives a grid of addressable cells.

Whoop! Web terminal is working! Yeah, baby! Now with colour!

It's working for dungeon, which is great, but it'll clearly work in other contexts. My first thought was a basic bash terminal (I'll need to add some stuff to support vim, but I don't think it's going to be a step-change in difficulty).

I've just realised that I can do the "editor in 1kloc" stuff, /in JavaScript/, in the browser! That's going to be a giggle .

I've got rough ideas of some kind of text/CLI interface for the/a website, but I'm not really sure what I'd want to do with it.

But I'm so chuffed. Using CSS grid has make it so much easier than the alternatives. I don't have to worry about getting stuff in order, like I would with tables, and I don't have to mess with low level font stuff that canvas implies.

So, as is traditional, a list of what next?

  • Scrolling
  • Finish colours
  • More chapters from the roguelike tutorial
  • More monster behaviour
    • Flee at low heath
    • Game of life fungus
    • Flocking bats
    • Patrolling guards
  • More level generation types (rooms and corridors, rather than caves)
  • Light sources
  • Equipment slots (main hand, off hand, body, legs, ...)
  • Quests (or at least tasks)
  • Persistence (save game, non fatal death)
  • Multiplayer (maybe)

I'm fairly sure that I'm going to go for an "anonymous people can't save state" option, at least in the short term, but I'm worried about the whole "managing accounts" thing. If I'm going to run this thing in any kind of public way, I guess I'm putting it on it's own machine, possibly in it's own account somewhere.

I like aws in principal, but I don't like the potentially unlimited costs. I could use Fly.io, or I could rent a second machine from mythic. I should probably think about a name/dns, and a proper skin, if I'm going to "release" the game in any useful way.

Then, of course, monitisation starts creeping in.

Let's not get ahead of myself, yeah?

Funny thing in testing, I was fighting a zombie and a bat stumbled into it and got the last hit, so got a couple of levels (cue genuine laughter).

I think creatures will start tracking who has done them damage and award xp proportionally (modulo attackers staying in some kind of range). Husband suggests bonuses for first/last hit, I'm not entirely sure about that.

So the article on writing functional games (writing games in a functional style, not waiting games that function) said something like update functions should return a list of events. The example in my head is a "died" event that can be intercepted by a potion of resurrection.

Interview tomorrow so of course I'm staying up late the night before. (Brain chemistry/depression/ADHD/insomnia has nothing to do with it).

Been a few good days at work, been working solid days writing code and feeding good about myself. Broke nonprod this afternoon but only for an hour or so, and it was installing an update that people have been asking about for a while.

I think I've cracked the combinations/bag packing problem that Husband gave me (something about fitting enchants to equipment). Turns out I really like the algorithm stuff, it gives my brain a useful workout.

However. I'm going to grab a snack and call it a day. Tomorrow will be here soon enough, I can do the interview and then forget about it (although CS Jobs posted me a "Senior DevOps Engineer" advert that has management responsibilities, I'm a bit tempted by).

Alright gentletheys, that was the interview. Took about forty minutes, four questions, not sure any of them covered the topics the advert said, but I think I coped well enough (I've read the guidelines enough that I can hit the right words, maybe).

Should hear back next week sometime. (Yes, form a certain point of view the job was aimed right at me, but I trust that the panel will judge people fairly, just because the squew is in my favour doesn't make it any better.)

(Posted to Facebook)

Earlier today I again performed the sacred ritual of "The Job Interview".

I suffered the ritual cleansing, daubed myself with unglents said to be pleasing to the Gods (I argued against it, since it was a video call, but Husband wisley suggested it would be a bad idea to insult the Gods on ritual day), and donned the sacred robes (thanks again Maisie for the tie!)

I improvsed poems on the four themes given; 'The Fall of Leaves in Springtime', 'The Emperor's Third Penguin' (that threw me a bit of a curve, I'd ony read up on the First and Fourth Penguins), 'The Use And Practice of Caching in a Hetrogenous Network Environment' (Easy, a standard ABC CBA rhyme scheme), and 'Duty' (Duty always comes up, so I used the poem I prepared last night).

I await the Emperor's messenger who will hopefully bring me the blue coral button that marks one as a 'Senior' developer.

(The interview went well, I think, although I've got quality competition. I should hear back in a week. Its a promotion within my team, looks like it's going to be roughly the same kind of work, but probably with a bit more managent thrown in. +300£/month net) (I am going to be so depressed if I don't get it.)

Working on nouns for 'Goblin Town' (provisional name), and I think that instead of doing some complicated thing trying to guess the articles and the plurals, I should just set them for each noun (so "a cat", "some cats", "an egg", etc)

Although maybe I should also think about the kinds of places that I'm going to find nouns (and it's when I'm outputting lines like "{subject} {verbs} {article} {noun}") (and the article is going to depend on context, "You smile at the bat" vs "You hear a bat in the distance").

Definite article is always 'the', indefinite isn't regular enough for easy rules ("if it starts with a vowel sound then use 'an'" is great, except I haven't found a easy to check for vowel sound types yet), however can just tag nouns as 'a' type or 'an' type (idea: new Noun("a fish"), and then parse out the article at construction time).

I've got a bit blocked on the game - turns out that I don't really want to play a dungeon crawler - and so I'm looking around for the next project.

I'm moving this blog into a database, but that shouldn't take long.

I've been thinking about building c# expressions back end for the Lox interpreter. (I've been thinking about building a CLR bytecode version, but that's a bit hard, I think.)

I kind of want to get the terminal component from the game into some kind of useful shape, at least good enough to run as an actual Linux terminal, but again, that feels more like a task for it's own sake rather than something objectively useful.

Or something. I think I'm just down. Husband says it's because I didn't get the job (and they may have a point), but that doesn't feel right? I think it might be the whole banality of existence thing, but that might be symptom rather than cause

WebGL stuff is going well. I wrote a JavaScript implementation of the Diamond-Square terrain generation algorithm a while back (Not sure when, since the current year ticked over, I think), and had got a far as converting the square mesh to triangles, throwing it at the GPU, and getting pixels on screen, but I ran out of interest/energy while it was still in flatland.

Over this weekend however, I've beaten it into 3d, added lighting, and got an animated camera to "fly" round following the terrain! It looks really cool (well, ish, I'm using "normal per triangle" for the lighting, so triangles look flat, but I'm not convinced. On the other hand, normal per vertex would be a bunch of extra work).

I've even got as far as applying a tourus transform (to the position stuff, at least), but I'm having problems getting the camera in the right place.

I've got a couple of things to try next. First is to move the torus transform into the vertex shader. Second is to transform the eye, target, up vectors of lookat (I think I've only tried transforming eye).

Interestingly, those ideas are incompatible - either I can do the transforms in JavaScript (on the CPU) of in GLSL (on the GPU), bit not both.

Anyway. time for sleep. I wonder if the GPU can do trig?

(Just a follow up, not only can GPUs to trig, they can do them in hardware as fast as everything else! Neat!)

Built (well, assembled) a new piece of furniture for the study, an IKEA Kallax 'storage unit'. It's not quite a set of shelves (ok, it is exactly a set of shelves), designed to take 30 cm (roughly) cubic boxes (or other 'inserts'; there's a set of drawers, a cupboard with a door, and a third party set of thin shelves for paper).

It's a bit more imposing than I'd imagined. Also, I really should get photos working.

I was going to be all whiney about "hitting a complexity wall" in my flight sim, but then I realised: I wasn't writing a flight sim! I was writing a terrain generator, and that's working great!

I do want to write a flight sim (or at least, some kind of game where the control system is kind of aircrafty), so I should think about that, but I should start with a blank sheet and build it up properly.

Mostly, the choice I need to make is how much of the low level graphics stuff do I want to do, and how much can/should/do I want to hand over to a library (twgl and three.jshttps://threejs.org/ are the current options).

I've added a ui to ping (status monitor) and added heart checks to the other sites and that's all working nice.

I still need to add rules that mean I only get one text per outage, rather than one per failed ping. I'm also thinking about adaptable rules for sites ("ping every n minutes, at least m consecutive fails").

On the other hand, a simple, offsite, cheap status checker: Yup.

What we have here is a state machine. A given endpoint is either FINE, FAILING (n), FAILED, or RECOVERING(n). Failing and recovering states have n copies.

Possible transitions:

FINE & !ping(ok) => Failing(0)

Failing(x) & ping(ok) => Fine Failing(x < n) & !ping(ok) => Failing(x+1) Failing(x == n) => Failed

Failed & ping(ok) => Recovering(0)

Recovering(x < n) & ping(ok) => recovering(x + 1) Recovering(x) & !ping(ok) => Failed Recoverin (x = n) => Fine

Or, in English, if we're fine and we get a bad ping, to into failing. Any good pings in failing take us back into fine. Otherwise, each bad ping takes us one step closer to failed. After n bad pings in a row we move into failed (and this transition sends a message). In failed, the situation is reversed. A good ping in failed puts us in recovering, any bad pings in recovering put us back into failed (no notification). After n good pings in a row we move back into fine (and probably trigger a notification).

Each endpoint needs to track it's current state, and the length and sign (good/bad) of the current streak. (Streaks have a minimum length of 1 since they're driven by the ping that has just returned)

StateNames enum (Fine, Failing, Failed, Recovering) PingResult enum (Ok, NotOk) Streak struct {uint count, PingResult sign}

Idle question: What are good uses for a sms to http endpoint? (That is, what useful text messages could I send my machine?)

I've been thinking about a calculator/scratch pad for a while, and I've just realised that what I actually want is the Smalltalk environment.

Imagine a blank screen. Click anywhere and start typing. Select text, right click and 'evaluate'. Right click to bring up tools (number pad, calendar, file upload whynot, timers, API calls). I also want to be able to link bits together, so the output of one calculation can feed into another.

It's persistant (changes are saved straight away) and maybe networked (so that I can open the page here or on my phone or at work).

Its got bits of OneNote, but that doesn't have any calculation/scripting.

I'm also thinking about 'layers' rather than 'pages' - layers stack on top of each other, and can be turned off and on. Disabled layers don't contribute to calculations.

Adding in data pipelines, to take e.g. a JSON source, parse it, filter it, reduce it, give an output, or a base64 encoded blob, decode it to utf8.

Chunks have got properties:

  • version
  • position (top, left)
  • dimension (width, height)
  • content
  • title?
  • tags[]
  • id (auto-generated)
  • input[]
  • output[]

And methods:

  • update()

Chunk service commands/events:

  • create(top, left)/created
  • move(top, left)/moved
  • resize(width, height)/resized
  • delete()/deleted
  • setContent(content)/contentChanged

Just noticed that my status app is ipv6 only.

And ok, I kind of noticed when I only have it an ipv6 address, but it didn't really sink in until just now.

(I was about to be all "welcome to the future babe!", but then I checked with 4g and it turns out my mobile connection isn't ipv6? I could have sworn...)

"My lady, you are both beautiful and wise, a rare and treasured combination in these troubled times. Would ye fancy a quick knee trembler in the trees yonder?" She snorted a laugh, and quickly regained control. "Alas," she said, and gestured to the collar around her neck. "My lord has required me to wait upon his immediate return, else how could I refuse such an eloquent and specific proposal."

Feeling ok today, getting stuff done at work (bits of dropping into other peoples problems and solving them, sling work doing the work I should be doing).

Bought myself a new Bluetooth mouse so I can play games on my laptop without borrowing Husband's spare wired mouse.

Moved the cables from their boxes in the box room to the new shelves in the study. (I'm going to call them shelves, even though it's not quite right, but "The IKEA Kallax storage unit" takes a while to type). I've also bought a bunch of reusable (velcro) cable ties that I'll start using sooner or later to actually tidy the cables.

So just basic life stuff, yeah?

So that's a (mostly) functional core built for the notes thing. It takes in commands and the current state of notes and generates events based on the command.

The ui listens for browser events, turns them into commands, and sends the commands to the core while waiting for the core to generate events for the ui to react to.

I'm starting work on a storage service that will listen for events and write them (or rather, the materialized states) to disk. It will also listen for events like to DOMContentLoaded, and use that as a trigger to dispatch it's own (noteLoaded) events via the core.

(I've just realised, when the UI makes a change, it can listen for a noteSaved event to let the user know their change is safe).

'Disk' in the case will be browser local storage, I'm also planning a network service that will ship events to (and from) other machines (for things like offsite backup and shared context).

Should be fun.

I've started getting serious about type checking in JavaScript by using typescript/jsdoc notation to label functions and variables, and it's turning out to be really nice/useful/verbose.

I could go full on typescript, except the compiler is a node.js script, and I've still got serious reservations about that whole ecosystem.

Starting to put together the server side of notes, and it looks like I'm going to use "workspaces" as the holder for notes.

All notes must be in a workspace.

The index page shows a list of known workspaces (stored in a table in the database) and create/delete buttons.

A "My first workspace" is created if the list of workspaces is empty. It's should be created from a template (but with unique ids for notes).

User can bookmark a workspace, and/or set one as default.

Workspaces have 'create workspace ' buttons.

Nothing is saved to the server until the user opts in.

Workspace commands:

  • Create(name) => workspace-created(name, id)
  • Load(id) => workspace-loaded(name, note[])
  • CreateNote(id, note) => note-created(id, note)
  • UpdateNote(id, note) => note-updated(id, note)
  • DeleteNote(id, note) => note-deleted(id, note)
  • Delete(id) => workspace-deleted(name, id)

Putting together a language (lox, again) needs a tokenizer (basic), a parser (got two examples), and a backend (again, two examples).

I prefer the Pratt parser (over the recursive decent), it looks much easier to extend. I also liked having an ast over jumping straight into code generation (with the bytecode vm).

I'd like to make a self hosting compiler, just to get that box ticked off. If I target wasm (or JavaScript, I guess), then I can run in the browser, although I'd really like something that can run on either browser or server.

I've gone though bits of this process a bunch of times now, and I'm not seriously thinking about picking up one of those old versions. I've got a version somewhere that's got arrays and a few other things, but that's written in C.

I think I want to do this version in C#, with the Pratt parser and the AST. I should be able to then write visitors to generate a few different backends - JavaScript, wasm, C# expressions, and maybe CLR bytecode (in rough order of difficultly).

I also want to write it as a library first. I need to make a choice about where to put generated nugget packages (and it's going to be GitLab) so I can start putting a library of useful stuff together.

After lox, the next library is 'web' for a bunch of asp.core stuff:

  • Openid Auth setup (not big, but irritating)
  • JavaScript, at least the lib.js that turns up places
  • Maybe a _Layout.cshtml and supporting files (like rocket.svg, basic site.css with a header/content/footer grid).

There are a few themes here. First, make it easier to start a project (so let's add a new project template to the list). Next, take out some of the style decisions. Finally, centralise some common code so that improvements can be distributed.

I don't want the template to embed files directly where possible, instead of should depend on the appropriate libraries.

Immune system is running hot today (has been last 3? days). Sore throat, too much phlegm at the back of my mouth, coughs and sneezes today, and mild cognitive deficit (today).

I'm calling it a cold. First one in ages, mildly sucks, but then it's a mild cold.

Still sick. Flippin' body lied to me this morning, said I was fine, but then gave up after about an hour.

However, I've dug out my old bluetooth keyboard and hooked it into the phone, so might be able to get some stuff done anyway..

  • poll says theres new data available on our socket
  • add data to the buffer
  • Check for EOL in the new data
  • If found, check state and hand off to the next routine

Connections have a current request (and probably current response). Requests have:

  • Method (enum, why not)
  • Target URI/path
  • Header hash (think about multiple headers)
  • Possible content (in memory vs cache to disk)

[This stack overflow answer] (https://stackoverflow.com/a/17905131/195833) shows how to do state machine functions (functions that return pointers to functions of their type)

For this http server, main loop looks something like:


// set up listen socket
int listenSocket = ...

// Install signal handler(s) to clear the running flag

struct pollfd *pollData = ... // add listenSocket to poll 

while (running) {
  int pollResult = poll(...)

  switch (pollResult) {
    case -1:
       // Handle poll error, probably by falling out the loop
    case 0:
      // Poll timeout, continue the loop
      // (going this to give signals a chance to gracefully shut down)
      continue;
  }
  
  // this is where i got bored/ran out of energy
}

// Tidy up

// close listen socket
// close client sockets

Maybe i just need to be happier writing slowly. Maybe I need to acknowledge that I'm not young any more, and my brain works not slowly. Maybe my antidepressants really do suck.

Current project/dream list

  • ActivityPub/Masterdon
  • browser (but not really)
  • link share (and updating hydra)
  • shopping list (always with the shopping list)
  • game (2d/3d, roguelike, flight sim, whatever)
  • language (or finding a good use for one)
  • webmail (hasn't had a rewrite in ages)

I think activity pub is the one with most scope, but if I can get link share info some kind of working state then I can finally delete Facebook.

Anyway. Hello people, it's been a while, hope you're all well.

Cluster services

  • k3s (cluster software)
  • step-ca (certificate manager)
  • cert-manager (cluster to step gateway)
  • traefik (web proxy/gateway)
  • Prometheus (metrics)
  • Grafana (metrics display)
  • Something for logs
  • Something for alerts?

I want tenants to be able to use mTLS internally, so cert-manager needs two providers, one for step and one for Lets encrypt.

Cluster tenants

  • mail
    • postfix (smtp server)
    • dovecot (imap server and email storage)
    • spamassassin (spam filter)
    • opendkim
    • opendmarc
  • osric.uk
  • shinjuspottery
  • fluffypeople
  • kamelion

Cluster Management

  • helm?

That looks like a bunch of namespaces and then mail has a small collection of pods.

I'd like to give everyone their own ipv6 address, and I can front the web servers with mythics ipv4 proxy, only running mail through native ipv4.

Still, I'm not sure what it gets me. I want to be able to throw up a new service (and take it down later) with close to zero effort. I've newly got that now, but there is a bunch of setup on the server to do.

(I also want to split osric.uk down into a bunch of tiny services, but I'm worried about the overhead of running c# vms)

I think I can setup a helm chat for "deploy to server", and then just change the names. I'll look at that.

Ipv6 address toy - use a range tag to adjust the subnet, and show the start/end addresses underneath

New cluster is up and routed on merit.

K3s, again, this time with a public /64 via a VPN (Linux routing is harder than it looks).

I'm annoyed that Ipv4 is so expensive, it would be nice to give merit it's own ip. Options:

  • Pay for L2TP from A&A (£10/month)
  • Forward the appropriate ports from wepiu to merit
  • Remember that the plan is to put the cluster on wepiu anyway, so can use wepiu's ip
  • Find out how much mythic charge for extra ips

Support stuff:

  • Container registry (zot seems to work)
  • Internal CA, for mtls (cert-manager is designed for this)
  • Edge proxy (traefik is built in, so I'll take it)
  • Monitoring/Logging/Tracing/alerting
  • OIDC
  • Outbound mail?

I've been playing with a new k3s install the last week or so (if you didn't pick that up from the last few posts), but I've been stalled for a few days. Tonight I discovered why.

K3s comes with traefik (an edge proxy), and a slightly complicated way of configuring it. K3s ships with a basic configuration, and then the cluster admin (me, in this case) needs to add an extra file to overwrite the default.

I thought I was going to need to reinstall the cluster every time I wanted to update the config, and I was dreading that as I tend to work in a "make a small change, measure it's effect" loop, and I didn't want to wait five minutes for the install to finish every time.

So that was wrong, all I need to do is run a kubectl apply command with the updated config file and k3s makes the change and restarts traefik.

I even wrapped the command in a Makefile, to make it even easier to deploy.

Conclusion: Automation really is the bees knees.

Feeling lost again.

Started this new job, big whoop. They're going to pay me better, and the people seem ok, but the build environment sucks bigtime.

It's all Windows, and dotnet 4.x, and locked down tight, and the build machines have been running for years(? I think) with odd config tweaks so nobody can bring up a new one.

And the app doesn't seem to have any monitoring/alerting (although there is an on call rota?), and the code-to-bullshit ratio is something like 1:4.

And yes, two years ago even I wouldn't have noticed this stuff, but I am noticing now, and it's in my job description to try to fix this stuff. So I guess that's what I'm going to do?

Meanwhile, at home, I'm not sure what to do with myself. It's probably just a down swing, but I don't feel like doing anything.

I've got this "world simulation" thing that I've been thinking about for years, I should probably update ptah, wepiu, khonsu to new debian (I've got this whole "move everything to containers/kube" thing going on).

Husband got me playing Warcraft again, and then bounced to craftopia. And I'm just sat here going "What?". It's pissing me off, but not in a way that's helpful.

I suppose none of that stuff is urgent in any real sense. I'd be happy sleeping most of the time, although I do like chatting to husband.

Bollocks to it. G'd night.

Mail Pod

Obvious services

  • Postfix
  • Dovecot
  • opendkim
  • spampd.pl (from https://github.com/Moosemorals/spampd)
  • postfix-policyd-spf-python

New plan, lox to wasm. Crafting Interpreters and "An Incremental Approach to Compiler Construction" should give me enough of a toolkit. Need to choose between binary representation (more hardcore) and text representation (easier to debug).

Things to think about so far:

  • IO - where does the source come from and where does it go?
  • Write tests

Turns out that you can compile and load wasm at runtime! Who'd have guessed?

I'm not sure if that's /useful/ in any strict sense of the word, but it's sure fun!

Arghh! I'm surrounded by bullshit!

C# doesn't have sum types. I'm begining to see this as a fatal problem, and maybe I really should move to F#.

Frickin' nginx didn't start after a reboot, because it couldn't reach one of the hosts its the proxy for.

Time to move the "replace nginx with traefik" plans forward I guess.

Don't know what to do about c# though. I think I need to have more of a think about the problem. Quack.

I'm re-writing the Hydra frontend stuff, and I'm moving code out of the controller. After the user has logged in they need to be shown a 'consent' screen that asks them to confirm that they're ok with the client getting access to their data.

However, under certain conditions the consent screen is skipped and the user can be redirected without seeing it.

I was returning two values from the function, a bool 'is skip needed' and a nullable string 'redirectTarget' that's set when skipNeeded is true. However, I can instead return two nullables, the redirect string, or the info needed to show the consent screen.

Quack indeed.

There's a minigame that's shown up in a couple of places that I've liked playing, sending agents to run missions.

Some benevolent entity wants stuff done, and will reward you for doing it. You pick an available job/task/mission, assign some of your people to it, and wait for the random number generator to decide your fate.

The skill/challenge/fun comes from matching your people to tasks - Peter is a fighter, Joan is a diplomat, so send Peter to hunt deer/fight zombies, and Joan gets the negotiation jobs.

You must also manage opportunity costs - if Peter has been assigned hunting duties, they can't enter into a competition that there'd easily win.

More complex versions can bring in resource management. Peter brings back a boar from a hunt, +5 food, but took damage so -1 medicine, and they can't be assigned for a day or two.

Early on, missions can be simple templates: "The local village is being attacked by (zombies|robots|frogs), kill n for a reward from the mayor", but I kind the idea that there is a world behind the missions - hunt too many wolves and the local deer population jumps up, but that leads to overgrazing and damage to young trees.

The flow of time also needs to be modelled. Is there a 1:1 match between mission time and real time, or do missions run faster (or slower!).

How does the player interact? A point and click webpage would work fine, but maybe play by email? (Especially for something with a more sci-fi or espionage skin).

Network connection(s) for node001

Node001 is my new Raspberry Pi. I'm going to set it up as a k3s machine, and I want it to have access to some of my public IPv6 addresses (Go Mythic!)

I'm also thinking about isolating it from the local network somehow, but I don't think I can do that without having it behind some kind of hardware firewall (which would need to be ptah in this context).

I can get a WiFi card for ptah, and hang node001 off that, since I've got a "No more cables near ptah" requirement.

...

Raspberry Pi supports 'ac' WiFi ("WiFi 5", apparently), and a new card for ptah looks like £20-30. So let's work on that basis.

I can route everything from the card through to wepiu via the VPN, so even if naughty people completely own the pi, they won't be able to break into the house network.

I can also setup a hub so that the pi's can talk over ethernet to each other.

Player:

  • Create(Name)
  • Change Name(Name)
  • Delete()

Template:

  • Create(Name, Duration, Slots, Factors[])
  • ChangeAttribute
  • Delete()

The big question is how do I a) keep these organized (separate) while b) only having one Decide method.

IDecider {
 IEnumerable<Event> Decide(Command command);
}

I want a pair of top level types for Command and Event, and then I can just have a CombiningDecider that takes IDeciders and offers commands to each in turn.

I could make IDecider generic and filter based on types, or I can specify in the contract that Decide ignores (returns an empty sequence for) commands it doesn't recognise. Which I think sounds easier.

It's it worth doing the same thing for evolve?

I'm not sure about evolve, I've got a couple of things to do with each event that aren't really related (store the raw event, update the aggregate(s), post the event to web listeners). I'm getting a vague shape of each event processor having it's own copy of Evolve, but that seems inefficient (except, for sure there's going to be a JavaScript copy in the browser).

Hmm.

(Been coughing off and on all day, totally sucks. Took the day off sick, will do again tomorrow unless I'm feeling 100% when I wake up)

Mission jobs

  • Convert commands to use ids instead of instances

  • Dump factors (missions) and attributes (assets) in favour of key/value pairs

    This one needs a bit more thought. Problems include:

    • Are keys registered somewhere, or are they just convention? Convention is easier to program, keys are strings and I'm careful when I type them. Registration is safer (typo proof) but needs a registry (that, ideally, JavaScript has access to)
    • Are values typed? String/Number/Boolean opens up options, and something like units (gold, kg, hp) opens up more. Cost is that strings are easier to deal with.
    • Scope creep. If these pairs get complex enough, I can redefine missions/players as collections of pairs (instead of having some actual native properties). It's that a bad thing?

  • Start thinking about authorization/roles, splitting out admin functions on the front end, and authentication, mapping users to players

  • Do something about persistence

The caches can be updated to be async, with a fallback to something more permanent.

Should get the event storage stuff working, then can do event replay at app start time, and maybe start using event source's last id.

Hello people! I've been ill for a couple of weeks, I think that was probably covid. I've lost my sense of smell (plus the coughing, sweating, sneezing, aching, and fatigue), hopefully temporarily, but it's still weird eating and just getting texture - I can't even remember what foods tasted like.

I'm recovering now, helped by a "fit note" that's given me permission for a week off work, so I've been sleeping/relaxing/watching a lot of you tube.

I bought a Raspberry Pi (v4, 4GB) a couple of weeks back with a "build a cluster" plan, but I'm really not sure it's worth the money or effort. I don't want to hang the cluster off the home network, to protect the home network if someone cracks the cluster, so I've been looking at 4G/LTE modems, and that's got me all depressed.

I can buy a second hand Vodaphone USB modem dongle through eBay for under £20, and Id mobile will sell me unlimited 4G data for £16/month, and I'll have to (well, really want to) get a tunnel from A&A for £10/month, and I still don't know what I want to do with the cluster!

I've got a vague "kubenetes + web servers" plan, but the remote server I've got works well enough without all that messing about, and £26/month will rent me 2.9 pis from Mythic, without the up front cost of the hardware.

Ah, well. I've bought one pi, and if I come up with a use case I'll let you know.

Cooked myself an omlette for dinner, and I cooked it really quite well. I should think about doing that more often.

Tonights entertainment is setting up an iSCSI target for my new Pi, and trying to network boot off it (so I can talk to Mythic about swapping NFS for iSCSI).

I've installed dracut on the Pi to manage the initramfs (the normal Debian tools just don't cut it, apparently), and I've spun up a a VM (tayet - The One Who Weaves) as the iSCSI host.

I'm going to try to use the right terminology: iSCSI talks in terms of 'initiator' (client) and 'target' (server).

First, setup passwordless access by appending the contents of $env:USERDIR\.ssh\id_ed25519.pub to $HOME/.ssh/authorized_keys.

Config file for tayet - /etc/tgt/config.d/node001.conf (this is the 'target')

<target iqn.2023-08.uk.co.aaru:node001-root>
  backing-store /srv/iscsi/node001.img
  initiator-name iqn.2023-08.uk.co.aaru:node001-initiator01
  incominguser johnathan partridge
</target>

(yes, that's not really XML.)

I'm just going to leave the link to the instructions I followed for node001 (I'm getting tired, and they are a bit messier)

Foof! I've setup most of a network root for the pi. Still to do:

  • Install and configure dnsmasq on tayet ss a dhcp/tftp server.
  • Setup ipxe so that the pi, when it asks tayet for a kernel, gets pointed towards the right place
  • Find out how much of the iscsi client stuff I need to install on the pi. (Thinking, if the kernel has booted of the remote disk, do I need the client tools?)
  • Dream about a shared read-only /usr disk
  • Go though that all again, but this time with the USB stick (four times faster than the sd card, and I'm only doing the iscsi thing as a test/demo for Mythic)

Dnsmask doesn't do IPv6 very well, so I've installed ISC Kea, which feels a bit over-engineered, but so far so good.

I've got it installed and running with a tftp server defined in the config, but I've left it switched off for now because I need to check that it's not going to try to stomp on the BT router.

Poot. Just read the docs for Kea, and it won't run as a 'proxy' (that is, send out DHCP replies with options set but no ip address).

Bollocks to it, I'll get an OS installed on the USB stick, and then I can use that Pi for whatever.

Physical vs cloud, that old chestnut.

Cloud machines work, have good connectivity, are IPv6 only (with workarounds).

Physical machines don't have a recurring cost, have more permanance, can be used to interface to the actual world.

Hybrid cluster gives me security concerns. Gah.

Duck time again.

I've got this new machine ("hedgehog", hosted by Hetzner, 6 core, 64gb, 1TB disk, 30EUR/month, no ipv4), and I'm still not sure where it fits in the network.

It can, obviously, just be it's own machine. Easy to setup, but it doesn't have IPv4 (and Hetzner don't run a gateway in either direction).

I could put it behind a CDN (only just thought of that!), but I don't want to pay any more than I have to, and I don't wan't to depend on people I don't need to.

Or, and this is what I want really, I can setup a VPN to wepiu and route traffic, possibly by putting both machines into a cluster.

(Or, I can just drop wepiu and spend the extra 2EUR/month on a IPv4 address).

That last one is a new thought, but it's got a point. Oh, but it would mean moving from Mythic's IP reputation to Hetzners. Let's not do that.

Looks like that's most of the work done for the Auth server. Next up is getting it running somewhere.

That depends on getting VMs up and running, which I think is waiting on networking (Wireguard is a layer 3 (IP) tunnel, but simple VM network setups expect a layer 2 (ethernet) tunnel).

Ah, well. Tomorrow's problems.

New infrastructure update: Traefik and let's encrypt are up.

I'm fairly sure I'm over-engineering again, but that's the fun part, yeah?

Request goes to Mythic's proxy over public internet. Proxy connects to tayet, a VM at the end of a VPN that hosts a k3s node. I might be able to eliminate the VPN, but I don't want to, I like having the "internal" network. (Now that it's been pointed out to me, I do wish that Wireguard operated at level 2, but you can't have everything) (TODO: Look up Ipsec, see if it's as bad as people make it out to be).

The dependency tree suggests that the container registry is next, then hydra/oidc, and then 'observability' (grafana et al.).

Also todo, find/write a cheap nuget repo.

Hello people, I got a new phone (Motorola Edge 40), and while I'd love to talk about it, I've still got depression. Sorry.

I think I've got elastic search setup to parse metrics out of the postfix logs!

It's taken a week, off and on, to work it out, but now I know.

One must add a "custom ingest pipeline" to the journald integration of the agent that's sharing a host with postgres.

Now I know that, I can start adding rules for dovecot (although really I should be able to scrape their Prometheus metrics), spampd, and the rest of the mail pipeline. I wonder if I can get them all to log a trace id ?

I'm also getting the infrastructure for the Elastic APM stuff sorted. The Elastic server (or maybe as Kibana) can do things like organise a new API key, so it's irritating it can't generate a zip of TLS keys to download.

I've been working half days at work on phased return from sick leave, and it's probably time to work up to full days. Yuch! Working half days has been really nice, shame we can't afford for me to go half time.

Measures

I've been writing this spreadsheet, and I've decided that what it needs is units, or rather measures

A measure is the amount of a thing, e.g., it's length, mass, duration, or some combination (length per duration is speed, for example)

A unit is the ratio of a measure to a standard (metres for length, seconds for duration, etc.), or some multiple of the ratio (feet, pounds, days).

Storing measures needs to keep track of it's dimensions, it's magnitude, and it's unit (we're keeping the unit for display).

Next step, for every combination of ValueType op ValueType, i need to write down (as code) what it means.

Current Computer Equipment Wish List

  • New monitor(s).

    Either 1 4k or a couple of 1440. Maybe curved? Must have speakers, HDR would be nice. Check specs of 1080, no point spending extra on e.g. 144Hz if the card can only do 60Hz

  • KVM switch.

    I can (and am!) swapping the keyboard/mouse/headphones USB over manually, and I've got Silk and the work laptop plugged into different inputs for the monitors, but ideally there'd be one less impediment to using the desktop.

  • 6E router

    Will only bring useful network up to 1Gbps, but could maybe put a 2.5Gbps card in ptah (although the bottleneck there is still the disks).

  • More storage space

    Lounge is full again, 18TB disks are out there (and best £/Gb), although I'd want to get two and mirror them. Equally, 2Tb NVMe drives are getting really cheap, problem is organizing enough M2 slots/PCIe lanes to use them.

Lisp as a system management language

Just the start of a though for now, but something like:

(def! apt-install (fn* (names) (apply exec "/usr/bin/apt-get" "install" names))

apply takes a function and a list of arguments, where the last argument is a list. It spices the last argument with the others and then calls the function with the combined list.

Since I'm going to be implementing in c#, I don't want to hobble my self from the start. Mostly this means c# strings are data types.

(Don't get distracted thinking about JavaScript, this is "Lisp as system scripting language", remember?)

record Pair(object? Data, Pair? Next)

record Pair<T>(T? Payload, Pair? Next) : base(Payload, Next)

... maybe

Hello world!

So I've got this new desktop PC - Silk - and first job is to distribute my SSH key. (Yes, I am going to look into SSH certs, no not today).

Step 1 - Copy the key to a machine that is already trusted. I feel I should just be able to dump it here, but there's no point being silly. I've copied it to OneDrive so I can reach it from the laptop.

(Aside: The laptop is doing it's usual "Got to have a power supply even at 98% charged thing". I really should save up and get a Chromebook or something)

Step 2 - Copy the key to target machines. Easy enough to do manually, but I wish I had a script for it. Ah, well.

Anyway, I'm up on wepiu, so I guess that wasn't the trauma I was expecting. See y'all later!

Hey folks!

I've finally got ground to writing a nuget server! It's dead simple but I can publish packages to it, and pull packages from it at build time. Rider doesn't like/use/understand the search, but that's easy to work round with dotnet add package.

That, as well as the container registry ('zot') I've installed means that I can start splitting code into libraries (As a smoke test I've put the OIDC stuff into a package) which should allow me to break the site into a bunch of tiny servers (e.g., separate out the blog) that are all SSO, and all sharing css/layout/menus (I might end up with almost everything in it's own package and one server to host them, or I might just keep the menu updated manually).

I think next up is finishing the Hydra front-end/user management service. (it occurs to me that I can hang e.g. mal off the new server without disrupting the old service). I know that realistically it's only ever going to be me and husband (and husband only uses webmail), but I'd still like to get things like roles right (Of course, the right way to do roles is in three layers, users, roles, and actions. Users are assigned roles, and roles are groups of actions. Admins group actions together to create roles, and then roles are allocated to users. But that's probably overkill)

Anyway. It's going well.

I assembled the various bits of my new hydra ui for the first time tonight (Hydra 2.2, the ui, and a client), and it's starting to come together.

I seem to have picked a nice colour palette (although I'm sure that husband will disagree), I can't remember how I did it though.

Still got bugs, clearly, but i think we're in "yeah, this is going to work" territory.

Wishlist:

  • user management (add, password reset, scope/role assign, suspend, delete)
  • client management (add, edit, delete, reissue secret) (Note the advantage of this is that all the settings apart from name and callback uri can be preset, making it much easier to add a client)
  • scope/role management. I should make a choice soon about how I'm going to do this, but I'm fairly sure that roles and scopes are different, and there's no point making life complicated.
  • self management (set password, name, revoke permissions, remote logout)

Anyway, it's late and a work night.

I think one of the things I like most about JavaScript is that only the code you run has to be semantically correct. So long as broken code matches the syntax rules, the runtime doesn't complain until the code actually breaks.

Of course, this means that you don't find out about certain problems until runtime. As ever, these things are a trade off.

State of play:

  • auth is up. It's got a UI for client creation, and the client programming stuff is in a nuget package so it's easy to include (just add a stanza to config).

  • shared.web is started. It's got some/most of the "always going to want this" stuff for a web app in a couple of extensions. I need to add lib.js, lib.css, and probably a _Layout, along with the tag helpers from auth.

    It's also got the CSP stuff I wrote a while back, and I want to add a nonse tag helper.

  • mal has been converted to the new style as a test. It works great, except it doesn't have a menu. Still thinking about menus (clearly the answer is to check some dynamic source for menu contents at page create time, but what's the best way to store/edit/distribute the data?)

What's next:

  • Install the ELK stack (or at least, ElasticSearch, Kibana, and the Elastic APM stuff) on a VM and hook that into the shared.web library.

  • Sort out a local CA and setup mtls.

  • Split out the blog from the rest of osric.uk (depends on the menu, ish)

I've got mal, and this blog, and I'm not sure what other bits of the door do keep.

'About you' is interesting, and id like to get weather fettled up at some point, but really, the only wiring parts are those two.

I can split the blog into it's own service, and then strip down the "top level" bit to the front page with the calender.

(TODO: turn the calender into a component that takes a collection of (Date, Uri) pairs so I can draw e.g blog entries and Dr who episodes)

On the list at the moment:

  • Editor, for both mal and the blog.
  • Dr who database. Using the data that husband put together, I want to do some date based stuff.
  • particles/flocking/collision avoidance

anyway. no overwhelming need to keep most of the stuff

Semantic Versioning is bogus, because every change changes the behaviour of code.

The CPU no longer overheats when you hold down the spacebar
XKCD: Workflow

Since people are mostly not crazy/stupid, I've clearly misunderstood what Semantic Versioninig is about.

Looks like sql window functions should let me dump blog entries and their edited versions into a table, and then do something like

SELECT *
FROM (
    SELECT
      ROW_NUMBER() OVER (
        PARTITION BY "EntryId"
        ORDER BY "Date" DESC
      ) AS "rank",
      Text, EntryId, Date
    FROM "Entries"
  ) sub
WHERE
  "sub"."rank" = 1

To get the latest version of any given entry. I'll need to drop into raw sql, but EF should cope with that ok.

(Do I really need to keep old versions of entries? Life is far more simple if I don't, but disk space is cheap enough and I like to let my hording tendencies loose now and then.)

I still want to add automatic entries to the blog, although the only thing I can think of at the moment is git commits.

Looking back at an entry from November about Lisp pairs ('conses', i think), it is now obvious that they should be something like

public abstract class MalValue {} // the base type for values

public class MalPair(MalValue first, MalValue rest) : MalValue {}

(C#'s new "Primary Constructor" syntax is nifty)

Because MalPair extends MalValue, either side of the pair can be another pair. I wonder how much work it would be to drop lists and construct these instead. Probably not worth it (I can only see it costing performance), but it might be fun.

Pros and cons of moving auth.shared to shared (and therefore shared.auth).

Pros are mostly "that's where the shared code is" and "the names will line up better".

Cons are "it's work (and need to update its consumers)" and "that's were the auth code is".

Hmm.

Shopping list again. We've been through this a few times so we should all know the words by now.

Lists are our aggregation roots. Lists have a name, a unique id, a list of users with access, and a list of items. Items have a name and a status. Items are distinguished by name (ignoring case).

Commands are CreateList, RenameList, DeleteList, AddItem, RenameItem, SetItemStatus, DeleteItem, ShareList, AcceptList.

Clients handle commands from the ui locally and pass events to the backend to be fanned out to other clients. The backend stores all events and sends new events to a client when it connects.

Backend stores events and a projection of the current state of lists that's built at app start time and updated as events arrive. Database is a single table of list id, timestamp, event blob.

I've spun up a new VM to host the osric.uk stuff (so pulling it off wepiu in the cloud). I still haven't brought myself to renting a Hezner machine - they offer a really good deal for what you get, but it's still £30/month.

(I guess I haven't got used to £30/month being affordable. Hubbies new pc was about 2.5k£, or just under 7 years hosting at that price).

If i got a new machine (and it is a machine, not a VM), I'd probably set up Wireguard so it's connected to wepiu (and the house machines, just to make life easier) and keep DNS pointing at wepiu (Mythic have a much better rep than Hetzner).

So this month, I'm going to move the osric.uk stuff to a VM on one of the home machines to prove the theory, and if it works I'll buy the Hetzner machine at the start of February and copy stuff over.

(I'll probably leave the home VM setup as a staging/test server, I've got a spare domain i can use for it, and it might be fun to set up automated ui tests).

It's a shame I can't take advantage of Hetzner's unlimited bandwidth, but too many services are still ipv4 only, and I don't think I'm anywhere close to the limit at mythic.

In the spirit of "got to start somewhere":

Sheep wander the world eating. Sometimes, for no obvious reason, they will panic and run around for a while, and then calm down and start eating again. Sheep that are eating mostly stay still, but will absently move forwards now and then. Sheep that are panicking move much faster, and occasionally change direction. Sheep will try to avoid obstacles, and will stop moving (even if they are panicking) if they don't have anywhere to go.

Sheep can perceve other nearby sheep, and will tend to panic if a neighbour is panicking (with the chance increasing as the panicking sheep is closer). However, once a sheep starts to panic, it pays much less attention to it's neighbours.

That's shopping thrown up on a server (on ''anat", the new hosting VM). Not even half finished, but going well. Will try to get a bunch more done tomorrow (before work eats up my energy).

Looks like I can use podman secrets to stash the NuGet config file and inject it into the build containers. (That's a far cleaner plan that the current "copy it into the right folder and hope the gitignore for is up to date).

Runtime secrets are already managed by storing them in a file and mounting the path (which is what this new command does anyway).

I want to script the container build somehow - especially if/as I'm going to need --secret arguments now. Since I'm Windows based now (weird, huh?), I should use poweshell, I just have to be careful not to accidentally write make by mistake.

(make doesn't do what I actually want, which is too run a program to get the date of a target. e.g., my source folder is dated last Tuesday, what's the date on the latest image? make wants me to touch image as part of the build but that's not source of truth (although I don't know how to get the build date of an image anyway))

But since poweshell runs under Linux anyway, a two line ppdman build, podman push script isn't the end of the world. If I'm careful, I only need one copy and it can pick up the tag from the source folder.

(Ok, now I want a 'build all' version that I can run after I've updated the shared project to bump version numbers in dependent projects, build, push, and restart them.)

(That's not so "out there", right? Also, I can add "increase project version on build" code, and maybe even "bump minor" and "bump major" options)

I've dumped self hosted elastic on favour of Grafana Cloud. Elastic is just to heavy, and I can't be bothered to set up the Grafana stack at home (especially as the logs and traces both need weird databases and/or S3 compatible storage).

On the plus side, I've written a few poweshell functions for building and publishing container images, for updating .net dependencies, and for automatically bumping .net project versions. This means I could(and hopefully will) be able to tweak something in the shared web project, and automatically update and publish dependant projects.

Turns out podman secrets don't work the way I expected, in that there seems to be a different namespace for build secrets compared to runtime secrets. This means that injecting nuget.conf isn't as easy as I'd hoped.

However, it looks like copying the file into the podman machine should solve the problem (and i can probably wrap that in a script, why not?)

(Putting my powershell module into my one drive was a good idea, now I just need to do something similar for my bashrc in Linux)

NuGet protocol works with basic auth, and my auth service is OIDC, so there's a conflict there.

I don't want to give apps other than auth access to the auth db, so NuGet needs to make some kind of call to auth.

"Scope" is a red herring, at least for my current use case.

Scope is the client asking the server for access to a type of resource.

Roles are bundles of permissions that the client should interpret in a way that the server expects (e.g., permission to "read files" shouldn't be used to access telescope controls)

How complex do I want to make this? I've got a picture in my head of individual endpoints each with their own set of permissions, but on the other hand it's me, husband, and a couple of friends, and the last three just want t sleeve where they can share files.

That gives three roles - me, hubs, and the other two.

[A short time passes]

I've thought of a couple more roles: an explicit "anyone including not logged in", and read only NuGet downloader role (and hopefully a read only container downloader role once I've figured out how to hook in that subsystem).

Maybe I'm taking about policies here rather than roles? I get an "all access" policy, hubs (and probably the Dr Who Boyz) get file share access, and the policy for automated downloaders. Then all I'm left with is how to map an account to a policy (maybe that's what roles are?)

Yay! I've got the new authorisation policy setup working!

As you may already know (can't remember if I've actually said), I'm on a mission to split up osric.uk into a bunch of smaller sites/packages, to make it easier for me to mess with part of the site without impacting the rest of the site.

On the critical path is getting authentication and authorisation working the way I want. I've setup an OIDC server (Thanks Ory Hydra!) and that's working well for authentication ("Who are you?" checks), but I want to be able to give out accounts without giving away the homeworld, and that's what authorisation is for.

In theory it's easy, asp core has policy authorisation built in. In practice, because I'd made a couple of mistakes, it's taken much longer than expected.

For the record, the mistakes were:

  • Not including the Razor page model in the Razor page, so the framework couldn't pick up the attribute that set the policy
  • Using "Is in role 'User'" as a proxy for "Is the user logged in", when the user role doesn't exist

I've even got "forwarding basic auth" working, where an app can forward the Authenticae header to the auth app, so only the auth app needs to access the auth database.

All together pretty chuffed!

Hello babe. I've started taking the Metformin and got past the initial side effects (bad stomach ache), although it might be messing with my sleep.

I do feel better this week (five days in, maybe?), nothing I can put my finger on, but definitely better. (Still all fatigued though).

Hubs has given me the Fitbit Charge 6 they bought themselves a couple of months back. I've had it on for less than a day so far, no initial thoughts further than it's easy to wear.

$WORK is being stupid, we're coming to the end of a project and there's not enough work for the team. This wouldn't be a problem, except I don't feel right getting on with $PROJECT stuff on work time.

Overall, doing ok.

Comparing C# and JavaScript Syntax

Simple statements

The syntax of simple statements are identical.

for ([initializer]; [test]; [increment]) {
  ...
}

[initializer]
while ([test]) {
  ...
  [increment]
}

if ([test]) {
  ...
} else if ([test]) {
  ...
} else {
  ...
}

Variables

JavaScript is dynamic typed, meaning that any given variable can hold any type of value. C# is static typed, so variables get a type at creation time and can only accept values of the correct type.

Variables in JavaScript are declared with one of three keywords: var, let, const. C# also uses var to declare variables, but they have different meanings.

var in C# tells the compiler to infer the type of the variable from context, e.g. in the statement var x = "string";, x can only be a string type.

var in JavaScript creates a variable that can be used anywhere in a function (the variable definition is hoisted to the start of the enclosing function).

Comparison

Comparison of values is one of the places where JavaScript and C# diverge due to the static/dynamic natures of the languages.

Because JavaScript can only know the type of a value at runtime it relies on type coercion to force values to be the same type. It is therefore safe in JavaScript to compare e.g. a string and a number, because the runtime will coerce the number to a string. However, this leads to unexpected results, so modern JavaScript provides an === operator that does not coerce the values, and so different typed values will not match (e.g., "1" == 1 is true, but "1" === 1 is false).

Functions

Calling a function is mostly the same in both languages: result = function(arg1, arg2). Of course, in C# result must be the correct type (or the code won't compile).

C# won't let you call a function with the wrong number of arguments, or arguments of the wrong type. JavaScript will pass along to the functions any arguments you provide, filling in missing arguments with 'undefined' and adding any extra arguments onto the end of the formal parameters. (You can access extra arguments through the arguments array).

I don't think c# is the right language for an activity pub server. There's to much "fuzzy" - properties that could be a list or a string, or one of a bunch of broadly similar types that aren't really sub-types.

If I had a JavaScript engine I was happy with, that could work, but I don't.

F# feels like it's probably a good fit, and this is probably the right size of project to start with, but I'm not sure. F# feels like a "write only" language (like Scala) in that it's very dense and the compiler picks up much of the slack, so getting a hook into understanding code is tickets.

I'm tempted by C. I can write a basic JSON parser (which knows that it's not going to have to deal with extreme cases), but even so it's the same old C problem (that there's just so much more to type)

Of course, even in c# I don't need to parse JSON into a type right away, although I do still need to cope with the "can be an array or string" (and "object or reference") cases.

I could dereference links at parse time, although I'd need to be careful of depth (and cache timeouts).

I still can't believe that the type tree has two roots (Object and Link).

I need to keep the wire format and the database schema separate in my head so I don't keep running into the "link or object"/"list or string" stuff.

The database can store normalised data, and then separately we can stringify and parse as needed, maybe using intermediate types of it makes life easier.

That should push much more of the complexity to the edge, where it belongs.

New thought: Since the protocol is based on "Actions", do I want to do this as an event sourced thing? Keep current state in memory, write events to disk ("Alice Followed Bob", "Bob Posted a thing", only using ids instead of names) and rebuild state at start time?

Doesn't really solve the "What types do I need" problem, but it makes the database schema easier: an event table with (id, verb, id) triples, and an object table with (id, bob) pairs.

Remote objects (anything I'm not hosting) need to have cache properties along with everything else: last updated, last validated, valid until. Management of that can be wrapped in the Dereference call that can pull from the cache or the network (and might want to do something clever like queue requests for an item while a fetch is in progress).

public readonly struct Entity { public long Id { get; init; } }

public interface ISystem { public void Update(); }

public interface IComponent{ public long Id { get; set; } }

public Dictionary<IComponent, IList> EntityComponents

Linux Kconfig

The point is to give Values to Symbols.

Symbols have:

  • name (string)
  • type (bool, trivalue, string, int, hex, range)
  • default (expression)
  • value (trivalue, string, null)
  • dependencies (expression)
  • reverse dependencies (expression)
  • weak reverse dependencies (expression)
  • visible (expression)
  • help text (multi line string)

Advanced sleep tracker

A page of buttons for activities

A page that shows the data in the approved format, along with a print option (and print css)

Run only on phone, ideally no network after first download

Stone

The stone is old and weathered, still standing proud in the shade of the forest. It pays you no attention as you approach, making no move to attack or retreat.

You reach your hand towards the stone and it remains unflinching, steady as you touch it. For a moment you see time as the stone sees time, the Sun a dull yellow band across a grey sky, the forest reduced to a faint brown and green mist as trees grow, die, and are born again around the stone.

You let your hand drop back to your side and look at the rock. It stands apparently unchanged by the experience. You wonder what it saw in you?

That's me switched from Traefik to Caddy. Not entirely sure why, but then I was never entirely happy with Traefik.

Caddy's config language is less wordy than Traefik (and not yaml!), and while that might mean I've lost some options, they weren't options I was using.

Caddy also installs from an apt repo, rather than as a naked download, so I should get updates in good time.

How to Avoid Tax as a Megacorp

Let's pretend that we own a company that makes widgets. Our widgets sell for £1 each, it costs £0.80 to make a widget, so we're making £0.20 profit per widget.

Clearly, as an honest company, we pay tax on that £0.20.

Our widgets are very high quality, but we're new to the market and don't have name recognition. BigWidget are the brand leaders, and they agree that we can use their name (and follow their quality standards) for, oh, £0.20 per widget.

We agree to the deal, and add the £0.20 licencing cost to our expenses, meaning that we don't make a profit and we stop paying tax.

Why would we agree to that? Well, we're 100% owned by BigWidget and they're based in a jurisdiction that has a zero tax rate.

Havok died today at 20:00.

She hadn't been eating much, sleeping more, and moving less than normal for a few weeks. If you knew Havok, then you'd know that being off her food was a bad sign, so we took her to the vets about a week after we first noticed. The vet examined her, couldn't find any specific problems, and said that Havok was probably stressed from us rearranging the furniture in the lounge.

We took her back to the vet yesterday morning. They examined her and recommended that they keep her for observation and for IV fluids (to help rehydrate her). Early evening the vets called us and recommended moving Havok to the local animal hospital to continue the IV and observation overnight, and to perform an ultrasound in the morning.

We followed their advice and drove Havok to the hospital yesterday evening. This morning the hospital called and told us that Havok had a septic abdomen, and recommended surgery to get more information.

Havok went into surgery and never woke up. She stopped breathing during the procedure and was put on a ventilator.

Cats normally wake up within half an hour of surgery finishing, so the hospital calling at the three and a half hour mark to say she was still asleep and still not breathing on her own made it fairly clear that she was dying.

We asked them to give us/her another hour and a half for something to change. The vets had said that keeping her on ventilation for more than seven hours would start doing more damage, and we wanted to give her as much time as we could without hurting her any more.

(Since she was in a coma, I'm choosing to believe that she was at peace and in no pain. Effectively, to me at least, she died in surgery, and anything after that was an exercise in human pain management.)

The call came and her condition hadn't changed. We drove up to the hospital to say goodbye. We were taken through to her room where she was laying on a table under a blanket with her head and one leg poking out.

She looked fine, other than being unnaturally still. Husband and I stroked her head, called her name, told her it was time for food, but she didn't react to any of it. She didn't flick her ear to stop us from touching it, she didn't push into my hand when I scratched her cheek glands. This wasn't my cat, this was just an empty house that she used to live in. Time to switch it off and stop wasting resources.

The vets assistant disconnected the sensors and removed the tubes that were giving her fluid. The vet gave her a massive dose of anesthetic, and the assistant listened to Havok's heart and confirmed that it had stopped beating. Nothing changed. Havok was just as still as before. I kissed her little head and said goodbye.

Husband and I were both crying. It's not fair! Havok was a fantastic cat and we are good cat parents. We do everything right by our cats, they are fed, they run around outside during the day and sleep safe at night. Havok was a loving, tactile cat. She made the effort to come upstairs and snuggle me a few times in her final weeks (and Gods, if I'd known I would have payed more attention).

Insurance will take care of the bill (something like £6k+ apparently, don't care), and we need to tell the hospital what to do with the body (individual cremation, then probably scatter some ashes in the back yard here and maybe at the coast, but I've got a whole thing about her spirit being attached to the ashes that's causing me worry), and then life goes on.

I hope that, if Havok has a spirt, or a soul, or whatever kind of cosmic energy, she is happy wherever she is now. She liked being stoked. She liked snuggling between husband and me on the sofa. She liked to sleep on the cat tree in the study while I was at work, or sleeping in there. She liked being out in the garden on a summer's day with the rest of the colony playing hunt the sister in the tall grass. She tended to stretch out when she slept, clearly trusting us and her environment. She liked playing with Carnage and she wanted to play more with Storm. She stood up to the dogs who walk past the front garden. She was a good cat, and she will be missed.

Lego wishlist

  • 30694 Space Science Mech £4
  • 60390 Park Tracktor £8
  • 60362 Car Wash £13
  • 60385 Construction Digger £13
  • 60450 Wheel Loader £13
  • 60461 Tractor with Trailer £18
  • 60439 Space Science Lab £35
  • 7992 Container Stacker £40
  • 60365 Apartment Building £55
  • 60446 Modular Galactic Spaceship £70
  • 60422 Seaside Harbour with Cargo Ship £95
  • 60434 Space Bace and Rocket Launchpad £125

Duplicate entire in edit are/were caused by the owner changing, although thinking about it, Im not sure I have found the cause since the save method was setting the owner, so it should have been set for the first save?

Thinking about a web terminal again as somewhere to code during work. Quite tempted to write the server side stuff in C since there's not going to be much of it.

The HTTP side will need to parse requests enough to send static files, but that's only the request line.

Need a select/poll loop for sockets, might as well use sendfile for static files.

I'm thinking about email to blog again, although as a follow up to dealing with a food diary. It's a neat idea in theory, I can send emails from work, my phone, or my desktop/laptop, but the amount of work needed to setup a collation system is similar to (or, probably, more than) that needed to stick up a web form with a database, or add tags to the blog.

The problem I'm trying to solve here is that, as part of a diet, I want to track to food I've eaten. I've got the old calorie count spreadsheet, but it's annoying to add new foods to that, and I'm not actually counting calories this time.

I wrote down yesterday's food in a note on my phone, but I'm not as interested in keeping my phone on me at all times as I was previously.

I'm coming round to the idea that using the blog to log foods isn't a bad idea, although I'll need to collect entries every week for the cult.

I should push the update that fixes edits, move to a database backend, and start thinking about tags.

To remember your current position in the blog, this page must store some data in this browser.

Are you OK with that?