The problem is that every time I create a new blog tool, I think that it's going to be the one platform I can integrate everything into - calories, lights, what's been playing on Spotify, what I've bought from Amazon, all the things - and that turns out to be hard (because getting the data is tricky, integrating it is tricky, and the privacy issues are tricky) so I stop bothering with the blog.

Either that, or something happens to the code and I can't get the entries out of the database any more.

At least this time the entries are stored as plain text files. I also like not needing to add a title. Thanks micro.blog/indiweb, you've got good ideas.

I do still want the "edit the site from the site" experience. I very much miss the site I had when I was doing BT support which was exactly (and really only) that - a textarea with three code for the page inside. (Written in Perl, I think).

Editing through (something like) GitLab IDE is not the same thing. Ignoring that it needs a whole other service (I could self host), it's the commit-build-deploy pattern that's wrong. I want to click "save" and see my changes when the page loads.

Trick is, these days, to pick a language where that works well.

Blog improvement ideas

Or A TODO List

  • Parse entries as markdown. Probably should do that at save time and cache the HTML (as well as the original file). Question: How far should I take it? Other than the header (which changes depending in login status), the site is basically static, so could be rendered at entry save time (and just do two, one for auth'd and one not).

  • Client side markdown preview

  • Spell check

  • Sort options. Allow users to have oldest top or newest top. Which leads into...

  • User options. Stash some stuff client side (sort order, last read). Could do that all with JS and local storage.

  • File upload, for images. Level two is basic image editing - at least crop, resize, rotate. (This is the "Image editor as language/stack of operations" idea again)

  • Maybe that microblog protocol. Not sure, can't decide if that's just a gimmick

  • Editing posts. Probably keep old versions, but then need to be able to view old versions (and can you wait an old version so the history is a tree?(

  • Link to (and show only) a specific post

  • Tags (maybe, and they must be optional).

  • Search. Level one is just match strings. Level 2 is stemming, stopwords, maybe that vector thing.

  • Syndication. (Cross post to Facebook/Twitter automatically)

  • Comments (maybe)

Don't know where to stash metadata for posts.

I can think of two top level options (in the same file as the post, or in a different file) and a couple of sub-options (same file with a metadata header/footer, same file json encoded).

I currently prefer same file with a header, but I'll need to play with the dotnet markdown parser before i commit.

(I need to play with the markdown parser anyway)

Training again, Advanced Development with/on AWS. Flippin' hub is broken again so only got one screen, at least untill I've got time to reboot.

Hub wasn't broken, I forgot that I need to plug it into a USB3 socket (which was on the other side of the laptop).

Just read "The darkness behind your eyes" used to refer to the self. I love it. (From https://qntm.org/streetmentioner)

Big idea: Use email/IMAP as the storage backend. Gives me arbitrary metadata (headers), text file in disk storage, universal tools. I can have posts in draft, if I hook up an SMTP server as well, I can have easy bot apis.

Using mime/mulipart I can store the original post, attached images, html rendering all together.

I'll need to import current posts, but I'm going to have to do that if I want to update my on disk format.

I'm probably going to need a second copy of dovecot/whatever IMAP server, I don't want to get my actual email mixed up with this stuff (except check out namespaces maybe?)

Why aren't I more excited about this? Could be the heat (Local weather be crazy, check news archives).

(Spoiler: It's not the heat. It's the standard "This technical wizardry isn't going to make me popular/better at taking to people/any less lonely" problem)

(Y'know, depression)

Good morning. That was a rubbish nights sleep, and I'm feeling knackered this morning. Ah, well.

Got training, hopefully I can stay awake for it.

The blog buttons are in the wrong order, I don't like post showing up in the middle.

Also, I've decided that the button style for nav links is wrong. I'm not sure about the whole colour scheme either. Gah, maybe I'm just feeling rubbish this morning and need stuff to complain about? On the other hand, I should design the site so that I'm happy looking at it when I'm depressed, so I keep using it when I'm depressed.

Microservices change explicit complexity (in the code) to implicit complexity (in the communication between components)

That's "Remember where you were" more or less done, including proper consent and everything. Only thing it needs is to fix the scroll to so that the end of the last entry read is at the bottom of the page.

Hmm. Scroll to latest isn't interacting well with location.hash being set at the same time. Let's try checking for a hash at scroll time, and not moving if there is one.

Hehehehe, that's so frickin' cool! It's been on the wishlist for the blog for ages! Now I can have best of both worlds - all the entries on the page, in date order, and regular visitors don't have to scroll for miles to find their place.

Aint JavaScript brilliant!

Works on my phone too, although the jump is more obvious.

IMAP is not the right solution for backend storage. Could make it work, probably, but it's too immutable. I want to be able to edit entries and them keep their ids (whether or not they keep a change history)

I want a bigger gap between paragraphs as well.

I think "Track latest seen" might be better than "Remember where you are". It's shorter and a little more accurate (since I'm not going to track scrolling back up, right?)

  • Show "You have unread posts" message
  • Maybe fade out latest read highlight after a few seconds
  • Start to think about images

Images:

I can think of five sources of images off the top of my head (although there maybe some overlap): Upload from local device, pull from URL, Google images, local camera, Android intent ("Share image" from another app)

Once we've got an image, it needs to be added to a post. Again, a few options: Just post image, attach to end of post, inline in post.

Given that Markdown has an inline image tag (a link, bit with a leading exclamation mark), I'd like to support that, but getting the filename right (and things like scaling/rotation/crop) is going to be tricky/irritating. (Since this bit is almost certainly going to need JavaScript anyway, I've got a bit of wiggle room to make life easier. Things like an "insert image" button on the edit page that shows an image picker/uploader/editor, generates a link, and inserts it into the textarea)

That's actually sounding like a reasonable plan right now. The dialog tag works well for the consent screen. I'll have a think about what could/should go into the add image dialog.

Android share is out, at least for now. It would work in Chrome, but I don't use Chrome.

Local camera looks doable (verging on easy): Open the camera, link it to a video element, take a snapshot, upload.

Local storage should just be a file upload tag, ideally with a scriptless version that updates to something more fancy if supported. (fancy in this context implies upload progress)

Generic url is easy as well, although since I'd want the server to do the download that probably needs JavaScript for progress/completion notification.

Google photos is it's own problem that I'm going to ignore for now.

Since the dialog stuff needs JavaScript (right? could I do something with the css checkbox trick?)(plus, it's 2022, JavaScript exists, get over it), how much JavaScript do I want to use?

The local camera stuff absolutely depends on JS. Can (probably) do crude local storage upload with no JS, likewise remote url (certainly in terms of post image as separate entry). Again, do I want to? What's the use case?

I'm going to have to do something about getting logged out on reboot (and the something is to move the storage for SmallCookieManager into the database).

Also, ow, my arm. I'm fairly sure it's because I've been laying on it wrong, probably while sleeping on the sofa (or programming in the sofa, on my side with the laptop on the table).

Not sure what my plans for tonight are. (That's not true. My plan is to lay here in the study for a bit, go downstairs later, eat something, and try to get more programming done. I'm not sure I'm happy with the plan, and I'm fairly sure my brain isn't up for programming)(The next thing I've got to do is design work anyway - what elements do I need for an "Insert Image" dialog?)

#Insert Image dialog

  • Image source drop-down
    • Remote URL
    • Local file
    • Local capture
  • Source specific controls
    • URL input + fetch button for remote. (Maybe a fetch local/remote toggle?)
    • Drop zone/file input button for local file
    • List of local cameras and available sizes for local capture, with a way to select the preferred option.
  • Preview display Including image data (width/height/type/filename) and tools (crop, resize, rotate, rename)
  • alt text
  • nsfw flag??
  • Comit button
  • Upload
  • Suggest random filename?
  • Receive allocated filename
  • Check if requested filename is available?

Image backend

Need to receive image data + filename and stash it somewhere (options.uploadPath). Again with the meta data. Would be nice to validate image data at upload time, get things like width/height/type, and stash it so I don't need to get it again.

I'd forgotten about alt tags, and so absolutely need meta data. Why not a file called blah.meta? (Standard reasons are things like: It can get out of sync, or vanish)

Still need to move the post button (clicked "take selfie" this time, but it's not hooked up to anything).

Got plenty of vertical space, can have another row of editor buttons (and so add bold, italic, link, maybe list)

My export from Twitter archive is ready for download, I don't have the right tools to examine it here (on my phone), that will give me something to play with this evening

I'm not being crazy, yeah? Importing posts from Facebook, Twitter, and my old blogs (if I can extract them) is a reasonable thing to do, right?

Dagnabit, I was not able to get out of going to the office this evening. On the plus side I'm now free to sleep till 16.

Back from the office, didn't get much done. Nobody notices when I have unproductive days, I don't know how to feel about that. On the one hand, yeah, great, I don't have people breathing down my neck. On the other hand, what's the flippin' point of turning up at all if no one notices that I've done nothing. Obviously, the point is paying the rent, but it still feels a bit off.

No example yet, but that's image uploading cracked!

(Only using local camera for now, but adding local files should be trivial. I wonder how well remote URL fetch will work under current network security rules. I guess I'll need to add a "can connect to anywhere" option, but then will CORS bite me on the ass?)

Evidence that the upload photo/local camera stuff works.

Alt

Let me also add: OMG! That was so easy! capturing from the camera is a doodle! I think I had more trouble with the progress bar (which seems to be pointless given how quick images are uploading, although that could be because I've got the size set to "flippin' tiny")

Interface could do with work. I think I need to flip the front facing (selfie) camera image. I think I also want to make the image full screen (with a bit of interface overlaid), but both of those feel like they're going to be a bunch of work.

I don't think I'm turning the camera off correctly.

Still need to do other image types, and image editing.

I really like the idea of a (simple) image editing script language. It doesn't need much, could be all one expression with a few primitives and basic maths. But it's late and I'm tired so no more thinking tonight.

I lost another entry while I was at the chippie, so I've added code to stop that happening. It uses the visibilitychange event that notes uses to dump a copy of the entry into local storage when the page says it's not visible (e.g., switched to another tab, or another app) and then reloads it at page load time (remembering to turn this stuff off during form submit).

Just tested it here, works great.

Also, images from local disk works (and on mobile too, which also have me an option to use the camera as a source of images. Glad I didn't see that until after I wrote the camera capture stuff)

Two kittens (Havok on the right, Carnage in the left) sat on a windowsill in bright sunlight

I think images are coming out of the upload too small, I need to work on the ui, probably adding a "this image will actually be 6x as big once it uploads" message. Or moving the upload stuff to it's own page and including the edit stuff. (Which will work much better now that entries are stashed automatically). Not sure how to communicate the image id back to the source page (with a query string, dummy)

Oh, and I'm going to turn off the consent dialog for track latest. I'm not tracking people (since I'm using local storage, I don't even get to see if people are using it, let alone where in the feed they are).

On the other hand, I can't remember why I was going to do that. It might be because I'm not asking about stashing entries in local storage, and they're much more likely to hold interesting data (although I'll be very surprised if anyone other than me (and maybe husband) uses the create entry page, and I've definitely consented)

I do want to collect "which entries have been seen" data, but I'm confident that it will be really disappointing, so I'm currently happy just imagining readers. (Hello, by the way, if you are real and reading this. Googlebot/Bingbot, you both count as real).

I think my immune system is kicking off about something. I wish I had more insight/monitoring of my own body.

I need to update the image uploader do that the original blob gets pushed, regardless of the preview. I also want to save a thumbnail at upload time (to img.tumb) as well as the original.

Since I'm going to be running the image data through some kind of processing library, I can also grab mime type and dimensions and stash them in meta data.

I've also been looking in a bit more detail at the Twitter export, and I'm not sure it's worth the effort. I know in my Facebook posts there's some stuff that I want to keep/share, but Twitter posts are lacking context.

A feature that I've been thinking about for a while is executable code blocks. That is, add s block like

var x=2+3;
print x;

And have "5" output in the blog. (Adding variable input, to get input tags is an obvious extension).

I can think of a few questions:

  • Where should the code run (client or server)
  • What language(s) to handle
  • How much access to the environment should the code have
  • Are all the blocks in one post connected
  • Are all the blocks in the blog connected
  • How are errors handled
  • What does securely look like

I've probably missed some.

A simple solution is raw JavaScript running in the browser. JavaScript doesn't have much of an isolation story, so all the code on the same page (i.e., all the code in the blog) world run in the same context.

Client side JavaScript would be easy to implement, assuming that I can convince Markdig to output script tags at the right time.

JavaScript is a little verbose, and therefore irritating to enter in a phone keyboard.

A bit more complicated would be Lisp, or an implementation of Mal. The same code could run either client or server side, it would only have access to the environment that I gave it (Unlike JavaScript, which would have access to the browser on the client), and it would encourage me to learn Lisp.

I'd need to write a JavaScript implementation (oh noes!), and again, make sure that I could get Markdig to cooperate, but it's an intriguing idea

I'm going to call editing entries a dependency of running code, although I should just build a Mal page and copy stuff over when it's done.

Except what I actually want is to be able to create interactive (or at least scripted) pages here on the client. Notes like this are great for prose, but for tools (current example is kitten age), I want to be able to find them without scrolling through history.

What I want, it turns out, is to be able to tag an entry as 'special', give it a memorable link (or slug) and have it show on a list somewhere. Time for a new button!

The interface side is easy enough (all praise to the dialog tag), add a 'post options' button/dialog with inputs for (probably)

  • Title
  • Slug (auto generated from title by default)
  • Include in TOC (bool)
  • Private (Maybe? I've been in two minds about private posts for a while. Knowing that people are potentially reading this is one of the main incentives for writing it. On the other hand, there's stuff that I don't want to share but that I still want to record.)(bool)

The current plan for metadata is a second file next to the entry, either in JSON (easy for machines to read/write, especially to/from c#, but hard(ish, relatively) for humans to read/write) or some kind of key/value map (e.g. Email headers, or TOML. Harder for machines, a little easier for humans, but I'm not sure how important human readable is for the metadata).

A missing metadata file shouldn't be an issue (I don't want to go back and create empty metadata for all my previous notes), and I want to be able to add new properties easily.

Having said that, bulk creating more or less empty metadata files shouldn't be too hard with a little bash scripting.

Let's also add a metadata version number, maybe in the filename, and an entity format version (more for paranoia than any foreseen problems).

That leaves us with:

  • JSON
  • matching name to entry except different extension
  • Property values must all be nullable, or have a defined default
  • Unknown properties in the file are ignored

Month count calculator

Given dates from, to where from <= to, return the number of whole months between the dates.

Test data: from 2022-04-01 to 2022-05-01 is 1 month from 2022-04-01 to 2022-05-30 is 1 month from 2022-04-01 to 2022-04-30 is 0 months from 2022-04-01 to 2022-04-01 is 0 months

from 2022-01-31 to 2022-02-28 is 1 month ; not a leap year from 2022-01-28 to 2022-02-28 is 1 month from 2022-01-28 to 2022-02-27 is no months

from 2021-12-31 to 2022-01-01 is 0 months from 2021-12-31 to 2022-01-31 is 1 month from 2021-12-31 to 2022-02-28 is 2 months

Algorithm

function daysInMonth(date) {
  switch (date.Month) {
    case 1:
    case 3:
    case 5:
    case 7:
    case 8:
    case 10:
    case 12:
      return 31;
    case 2:
     return ((date.Year % 4 == 0 && date.Year % 100 != 0) || date.Year % 400 == 0) ? 29 : 28;
    default:
       return 30;
  }
}
let months = (to.Year * 12 + to.Month) - (from.Year * 12 + from.Month) - 1;
if (from.DayInMonth >= to.DayInMonth || from.DayInMonth >= daysInMonth(to)) {
  months += 1
  }

That's executable code blocks added. Tag fenced code blocks with 'javascript' and that adds a 'Run Code' button (but only on the single article view).

Something weird is happening with the phone browser and css, in that it seems very reluctant to load a new version, which is a shame because I've just updated a bunch of stuff and I wanted to check it on a proper mobile browser.

I've tried to be more consistent with sizes (I was about to say "so a underlines are the same width as borders" but that's clearly false).

I've increased the gap between paragraphs, and added a hr between entries. Yes, that does use up priceless, irreplaceable, vertical space but it also looks better.

I think there were functional changes in there somewhere as well, but I've got no idea what they were.

Oh, yeah Added an 'edit' link for entries (only visible to logged in people). Doesn't go anywhere at the moment, still need to implement the endpoint, but it's a start.

On Editing Entries

I can't decide about history. Do I need to record past versions, either as straight copies of old files , or as a list of changes?

Option one: Don't keep history

Easy to implement, but feels wrong. What if I want to back out a change (easy, just do another edit) or compare old versions? (And sure, but what if you don't?)

Option two: Archive old files

Keep old versions as files. Need to find somewhere to stash them (in an archive folder, or a folder per entry. How many entries/edits per entry am I expecting?) (Lots, potentially, if I do stuff with this live script malarkey)

Also, and both options two and three need this, what does the UI for the archive look like? Are old versions public? Can I (like, should I be able to) compare arbitrary versions? Restore a version? Build up a tree of versions?

Option three: Keep a stack of changes

Keep the delta from one version to the next. Can either do it forwards (start with an empty file and apply changes until you get the current version) or backwards (start with current and apply changes until you get an empty file).

It's technically fun/interesting, and has an "it uses less storage" excuse, but it would be harder to manage and goes against the "as simple as possible, at least for storage" ideal.

Conclusion

Option two currently leading of the two "keep history" options, but I'm still not sure.

Ok, that's not true. Obviously I have to keep history, I'm just not looking forward to writing the code. (I guess that's the attraction of option three, but I'm getting old and the attraction of simple, easy to write, easy to read, sold code is quite strong)

I'm getting pretty flippin' tired of caching issues. Stupid phone isn't showing an updated version after I've logged in. I guess that's on me, I should probably check the docs, but I can see that I'm just going to give up and write my own cache header rules (instead of using the Microsoft ResponseCache stuff).

Anyway. Bedtime. Goodnight dear reader, sweet dreams, see you tomorrow, probably.

I know I said I was going to need, but one last thing: Add a scroll to latest button back in. Now that I've got JavaScript to tag the latest post, adding a latest id should be easy enough.

Still, suggestions on how to tag the latest article server side are welcome.

(Stop messing about with IAsyncEnumrable and pull all the entries into a list. Or, y'know, do it right and generate the HTML at save time.)

Early morning whine:

  • Bad nights sleep so feeling crap anyway
  • Work priorities have changed so Ocelot is no longer drop everything crash priority, although finding that out has cost so much time that the next project only has (maybe?) a week.
  • Stupid k8s auth broken again so I need to reboot (looks like that hasn't actually fixed anything)

It's all trial trivial, so I'll work on shaking it off.

/me makes an "I've just worked out out" noise.

I want to add a page to edit entries, which will share most, but not all of the HTML/layout of the create entry page I don't want to add a bunch of if statements since that would complicate/untidy the code. I also don't want to make a straight copy because that will increase the maintenance burden.

This is the problem that partial views exist to solve.

I need to update the menu/navbar stuff to stop the current page being a link (and maybe to remove the 'New Entry' link completly from the new entry page.

But decomposing imgDialog into it's own file has worked, that's going to make the edit page much more simple, and much cleaner.

The Lion of Longtrees

Leo is a professional bodyguard. He works usually works medium term contracts (weeks or months) but is open to negotiation, at least until the contract is signed. Once he's signed on, he'll work the contract as agreed until it's completed come hell or high water. This single-mindedness has given him a useful reputation, and a couple of enemies.

He's trained in a couple of styles of unarmed combat (and keeps up regular training). He prefers knives over guns, but knows that ignoring guns would be giving too big an advantage to his opponents. He's a fair shot with a rifle, and ok with a pistol (but is likely to just drop the firearm when it runs out of ammunition, unless he's got a specific reason to keep it).

He's generally friendly and approachable, good with people. Of course, he can put on a blank face when he's working, and will semi-consciously use his bulk to intimidate people out of starting anything with his current client. He keeps up to date with world news and the local sports teams specifically to carry on small talk while scanning the crowd for trouble.

He's had lessons in slight-of-hand tricks, and again practices to keep current. He can entertain and distract a client's child (or a client) with a cup and ball, or pass through airport security with enough of an arsenal to hijack several planes.

Initial scroll to needs to wait for images to load (or I need to add sizes on to images, which is probably the 'right' solution)

I've been thinking about how to add pre- and post- condition checks to (C#) functions.

Roughly, preconditions are conditions that should be true at the start of a function, and post conditions should be true at the function exit. e.g.:

/**
 * Return x divided by y
 */
public int Divide(int x, int y) { ... }

has a precondition that y should not be zero.

It's easy to add these kinds of checks to a function:

public int Divide(int x, int y) {
  if (y == 0) {
    throw new ArgumentException("y must not be zero");
  }
  ...

but a) they're not obviously condition checks and b) it might be nice to be able to turn them off in Release mode. (Potentially also c) they could be extracted for a test suite)

I've been thinking in terms of Attributes, but C# doesn't support Lambdas in Attributes, and probably won't any time soon. However, a new idea arrived today:

public int Divide(int x, int y) {
  Contract.Pre(y, q => q != 0, "y must not be zero");
  ...

With a definition something like

[StackTraceHidden]
[Conditional("Development")
public void Pre<TParam>(TParam value, Func<TParam, bool> test, string onFailMessage) {
  if (!test(value)) {
    throw new ArgumentException(onFailMessage);
  }
}

it would transparently (i.e., not showing in a stack trace) run the test in development mode, and not even be included as a call in Release.

I imagine that it would be worth adding some overrides (or sibilings) for common cases, such as Contract.PreIsNotNull, Contract.IsOfType. (Alternativly, include a bunch of predicates in the library).

Of course, the next question is: What do I do with this idea?

It's clearly a stand alone library. Maybe it's time to get a nuget.org account.

More plumbing/infrastructure work on the blog this evening. Post meta data is now written to a separate (JSON) file next to the entry. This makes it much easier to add stuff to meta (e.g., version number).

I've also swapped out IAsyncEnumerable for Task<IEnumerable>. I was having trouble working out when things (like reading the entry from disk) were actually happening, and worrying that they were happening more than needed.

Instead, BlogService.GetEntriesAsync pulls all the meta files, and then individual entries pull (and cache) content.

Two thoughts from writing that last sentence: Is it worth running the load metas in parallel? And I should look at the proper in memory cache for entries (including content) and do things like invalidate the entry if/when edited.

I should benchmark the load stuff, and maybe set something going at process start time to pre-fill the cache.

(On the other hand, I actually want to get editing working and this is feeling like a distraction)

Editing entries :tick:

Also, accidentally deleted post button. Oops. Fixed that too.

Tidied up the logout code.

I'm an experiencing a level of lower intestinal distress at the moment, my insides are not happy with something.

Otherwise another day at $work. Mostly messing with k8s and remembering to look busy. The deploy script now uses kubectl set image instead of rollout restart. This guarantees that the latest image is pulled, and updating the deployment triggers a restart.

I was arguing for a very aws solution to a problem (messages come in, need to be recorded and aggregated) using queues, lambda, and dynamodb, but we're getting a hit every 45 seconds on average, it's really not worth the effort. Spin up an SQL db and move on to the next thing.

Ah, well. We'll see how that goes.

Added ImageSharp so I can start poking uploaded images. As a smoke test (and to see how working with the Markdig ast works), I'm now adding width/height attributes to img tags.

Otherwise, that was a fairly good day, I think. See y'all tomorrow.

Rubbish day at work, failed both of the projects I wanted to get working and it's not obvious why they didn't work.

Building a simple 'what's going on with my cluster' app but it's not authenticating itself correctly. It looks like I need to give the 'default service account permission to list pods' but I'm holding off on doing that until I feel that I understand what it's doing properly. (Also, I'm not convinced I've got the authority to do that, and I don't want to get bogged down with that).

The other problem is setting up an existing app into the nonprod cluster. It was working there before and now it's not. I have changed a bunch of stuff, so really the next thing to do is to go through the config/setup carefully, step by step, but again, I just can't bring myself to care enough.

That's the problem. No-one on the team really cares any more. I don't get any good feelings from solving these problems because I don't get positive feedback from doing so (and I don't really get negative feedback from not solving them).

I don't like it. I would like to feel more positive about work and works problems, but something fairly big would have to change first. Maybe I should be looking for a new job (although I'd miss seeing how much they'd miss me when I'm gone).

Just finished reading the third Murderbot Diaries "book" (in quotes because it's a short story, but charged at (high) full book prices), and it was as consistently good as the first couple

I've been thinking about showing dates as year-day of year, but I'm not sure. I've just had a "well why don't you go the whole way and u show Unix time!" thought that's not very helpful.

I think it's about showing how far through the year it is. I'll think about that as a separate page, although getting fp ported over is current priority.

If I remember right, I've pulled webmail out into its own project, integrated the shared stuff from common and webmail.content. Time to fire it up and see what's broken.

I'll need to pull a copy of the db from legacy and setup a spare domain for testing.

The more i think about this blog thing, the more I want to stick it into a database and stop messing about. The "keep it in files so that I can recover it for next time" stuff is interesting, and i should probably add an export function, but if I use sqlite i should be able to do a dump anyway.

Actually, that export data idea sounds better every time I think about it, especially after having got data exports from a bunch of different places (Amazon, Netflix, Facebook, Twitter) this week.

However. Today is about webmail, and getting out of Bytemark.

wepiu has the basics installed, but not configured. It's got a vmail user with a nearly empty home dir. It's also got a webmail user, I guess I'm planning on using that one for the webmail?

Ok, so webmail doesn't build. Last time I'd been deleting "stuff I don't need anymore", and I've been a bit over the top. Or possibly not deleted enough. Looks like some of the 'Friends' stuff is still hanging about.

find . -type f -name "*.cs" -exec sed -i 's/using Common/using Webmail/' {} \;

Who says you need an ide!

(Having said that, there are times when they're useful)

Bash arrays

Based on this fine article

arr=() 	Create an empty array
arr=(1 2 3) 	Initialize array
${arr[2]} 	Retrieve third element
${arr[@]} 	Retrieve all elements
${!arr[@]} 	Retrieve array indices
${#arr[@]} 	Calculate array size
arr[0]=3 	Overwrite 1st element
arr+=(4) 	Append value(s)
str=$(ls) 	Save ls output as a string
arr=( $(ls) ) 	Save ls output as an array of files
${arr[@]:s:n} 	Retrieve n elements starting at index s

Note: JavaScript doesn't have a native day of the year function, so I'd have to write one (based on the number of days in the month code above) if I wanted to do anything client side.

(Alright, I'm wrong. Subtract midnight 1st Jan from your date to get ms since the start of the year and divide by (1000*60*60*24) to get days)

Incomplete idea

New webmail idea: Save inbound emails as parsed html as they come in. have a background thread (per user) that's connected and listening for emails.

Listening per user isn't fantastically efficient, but since it's only the two of us if should be fine.

(on a larger scale I'd probably have something on the end of a pipe, but I'm going to ignore that for now)

What do I need to save?

  • Headers (At least from, to, date, subject, maybe type)
  • Structure (From a template based on the content type. Recursive for multiparts)
  • Content (The body and it's various parts)
  • The original message (so I can see the 'raw' version)

Assuming all this stuff is going to also be saved by the mail system, I don't need to be able to round trip it. Folders need to be updated on change.

I should check the stats, but my feeling is that it's probably not worth it. A good chunk of mail is never opened (See: spam management), and updating folders leads to irritating locking issues (that dovecot also has but has already dealt with).

It's a cute idea, but unfortunately not worth it.

Flipin' machines trying to be smart at me! I expected

egrep  '^[a-z]{4,5}$' $WORDS > wordlist.txt

to match 4 or 5 character words that conain only the characters 'a' (U+0061 LATIN SMALL LETTER A) to 'z' (U+007A LATIN SMALL LETTER Z). But nooo, someone is being clever with their locales, and I get words like élan, which, if you missed it, contains 'é' (U+00E9 LATIN SMALL LETTER E WITH ACUTE) (assuming it hasn't been decomposed).

Fix:

LC_ALL=C egrep  '^[a-z]{4,5}$' $WORDS > wordlist.txt

(Roughly translated means 'Ignore anything that happened after about 1963 and just give me ASCII)

(I should probably take the word list with the funky characters, since I'm using it to generate passwords. On the other hand, I think I should probably stick to ASCII, since I'm using it for passwords...)

Sunday morning (yes, the previous entry was also Sunday but I hadn't slept yet so the day had not ticked over)(Or it wasn't dawn yet). Nothing much so far, finished the last/latest Murderbot Diaries book, that was a brilliant series, strongly recommended for people who feel habitual dissociation/disconnection from humans.

Current plans: Grinding towards moving fp mail to mythic, maybe get out pillow shopping, hopefully a quiet/low stress day.

Hugs to y'all.

Dump webmail DB:

sudo -u postgres pg_dump \
	--clean \
	--if-exists \
	--create \
	--schema=auth \
	--schema=webmail \
	--quote-all-identifiers \
	--dbname=moose \
	--file=moose.sql

So I've copied the fp db over to wepiu, not sure what's next. I need/want a 'better' todo/issue tracker, that can track dependencies and maybe priory to show me what's next.

Basic data seems obvious: text, id, dependsOn[], isComplete

Is that enough? Let's take set up db as an example

"setup db", 1, [2] "install db", 2, [3] "Add db repo to sources", 3 "Copy data/schema from old db", 4 "Export old db", 5 "Tidy export", 6 "Import data", 7 "create db user" ...

I'm tempted to add a 'command' property (probably with a 'runAs' sub property) to record/automate the process.

Is it worth it? It would be nice to be able to pick up and put down the process as and when, without losing track of where I am. On the other hand, updating the list is another job (Ignoring the whole "writing the software in the first place" issue)

(And now I want to add 'verify' and 'undo' commands (although commands up the tree/graph should cover lower levels))

Anyway. I think I'm going to config servers before getting the website up, so that I don't need to test the website against live data.

Moved moosemorals.com from bytemark to mythic. There's nothing hosted there yet, but I'm going to use it a the domain for testing webmail.

Mythic are generally great, but they're charging me £15 to bring the domain over. (It counts as a renewal fee, and the time gets added in, but it's still irritating).

I've been going through the postfix config ready to turn the daemon on. Biggest change do far is setting up a separate submission port, so the only mail expected on port 25 is inbound to one of the hosted domains, which is simplifying a bunch of options (e.g., mail that has come in on port 25 that isn't for a hosted domain can be rejected early).

I'll need to do something about outbound mail from webmail. I think the current solution is to just trust the local machine, which is reasonable, but I'd like to do better. (Something like password auth with a one shot password that's validated by postfix calling back to the webmail. Time to look at Lua again, I guess)

Bug reports and what to do with them

At a minimum, a good bug report contains the answer to three questions:

  • What did you do?
  • What did you expect to happen?
  • What actually happened?

This doesn't just cover software, it's a reasonable place to start any fault diagnosis:

  • What did you do? I tried to open the door.
  • What did you expect to happen? The door to open.
  • What actually happened? Nothing!

(Although you will receive reports with more data than this, well written, detailed reports will be the exception.)

(You probably have an intuition already about what the problem is. Learn to distrust that intuition, or at least treat it with scepticism.)

The first step in diagnosing an issue is to make sure you understand correctly what the issue is. People often missrepot the issue for various reasons; they don't understand the technology ("The TV remote is broken" > the remote needs new batteries, is not being pointed at the TV, is actually the remote for the hi-fi), they're trying to be helpful ("The TV wasn't responding to the remote so I took out all the cables and wrapped the ends in tin foil to help them conduct and I've put them back in and now the TV is on fire and the remote still doesn't work"), or they're worried about looking stupid ("The TV remote in the demo room isn't working, fix it before the clients turn up in 5 minutes and stop bothering me with stupid questions").

You are going to need to ask questions, you will need to ask her basic questions that could be interpreted as insulting ("Of course it's plugged in, do you think I'm stupid"), and you will probably need to ask the same question more than once as people helpfully answer a different question.

Looking at our door example, try to establish why they are trying to open the door? Is it a door they go through often, or is this the first time? At this stage, you're still looking for context. Again, try not to think of solutions at this point, or even causes. The first step is to establish what actually happened, ideally well enough that you can reliably trigger the issue locally.

I say ideally, but it's very close to essential to be able to replicate the problem at will. If you're dealing with software, this is a good time to write a new test case. Write enough code to trigger the issue, and then start taking code away from your test until you get the smallest reliable trigger. This exercise has two aims. First, writing the test case should help find the rough area of interest in your source. Second, having a reliable test case means you can be confident that you have fixed the issue! Without a test, you can't be sure that your 'fix' has worked, or even fixed the right problem. With a test, you can apply the fix, run the test, confirm the issue doesn't reoccur, and then remove the fix, rerun the test and confirm the issue comes back. (Also, including the test in your automated test suite (you do have an automated test suite, yes?) makes it harder for someone else to reintroduce the issue later).

Once you understand the problem, have isolated the issue, built a test (or series of tests, don't hold back here), written and confirmed your fix, this is a good time to look though your codebase for similar patterns (or exact duplicates) that you can fix at the same time. (It's great to have users report bugs, but it's far better to not have bugs for them to report)

Looks like I can use IControllerModelConvention to manipulate controllers at app start-up time, to add hostname to routing (to set a default hostname).

Error page ideas

These are ideas for the static (well, ssi maybe) pages from nginx. If possible, check for a body from the backend and use that, otherwise send a self contained static page (light on the css, no js).

Use the 4xx/5xx split ("It's not me it's you"/"It's not you it's me"), a custom description of the specific error, and advice on what to try next. Include a transaction ref (maybe).

400 - Bad request - "You've done something so generically wrong that I can't tell what it was." 401 - Should be issued by a backend with content, can't do much with it at the proxy. 403 - Forbidden - "Your provided credentials do not permit access to this page. Please wait while I summon a security enforcement team to your location" 404 - Not Found - "You asked for a page that isn't here. This page is here instead." 405 - Method not permitted - "You can't $method this page!" 413 - Content to large - "You've sent more data than that page can cope with"

500 - Server error - "I think something has wrong gone." 501 - Not implemented - "I don't even understand the question." 502 - Bad Gateway - "I asked a friend for the page but they've let me down." 503 - Service unavailable - "I am far to busy to talk to you" 504 - Gateway timeout - "I asked a friend for the page but they're taking to long."

I wonder how much trouble it's going to be to get nginx to pick a random line from a set.

Now running with the new Nginx setup.

I tried putting all the site configs in one file and using the $host variable to pick the right port (to proxy to) and certificates. Didn't work.

Instead, now I've split out the config into two include files, and a file for each domain/site. The per domain files just set the server name (per the Host: header), the proxy port, and the path to the certs, and then include the http include file (with the standard redirect) and the https include file (with all the other (mostly proxy) settings).

It's a shame that Nginx doesn't have a configuration level between http and server, to group config for similar sites, although using include is working for me here.

Last thought about this - I should be able to use the $server_name variable in the https include file to make the path to certs, so the only per site config is the server name and proxy port. (The point would be to minimise the chance that I'm going to forget to change a name when I next setup a site).

(Ok, I said last thought, but husband's site is static and can be handled by nginx on its own)

The wemail refactor is going ok, I think.

Webmail runs behind two domains. One for the content of emails, and one for everything else. This is to take advantage of the same origin rules, so scripts in emails can be effectively isolated from the main site.

The previous version of webmail ran as two separate dotnet apps to serve these two domains so that requests to the content domain couldn't 'leak' into the interface domain. (And also because, frankly, I didn't know any better.)

However, Asp core can route based on hostname, which gives enough of an isolation guarantee that I can fold the code that serves content into the main app, and handle both sets of requests.

I have to tag each 'endpoint' (roughly, each action method in the controllers) with a hostname, either the content domain or the interface domain. There is an existing HostAttribute, but (since it's an attribute) it can't be set at runtime from config, and I don't want to set it on every controller in case I forget and miss one.

This is where the IControllerModelConvention from a few posts back comes in. I've added a call in Startup to add a convention that loops through every Action to add a host to the RouteValues collection, either the interface domain (default) or the content domain (if the action is tagged with the right attribute).

(Strictly at this point some of that isn't true. I've set up the convention and I'm looping through the actions, but I need to create an attribute and add it to the appropriate action(s) (although i think there's only one). Still, it should work).

This is a bunch of work but it's worth it, mostly because it's one less server to run, manage, and allocate resources for. (Also, the two servers used to communicate through the db, and that might not be needed, but fixing that isn't so big a priority).

Anyway. I'm feeling good about that whole thing.

One of the (very many) things that's bugging me at the moment is part of the code for the Webmail server.

The code that sends the actual content of an email to the client boils down to a call something like:

Message.Part.WriteTo(HttpResponse.Content)

(the names are all wrong, don't worry about it)

The problem with this is that I can't take advantage of the frameworks tools to send files to clients. (Where 'file' in this case is exactly equivalent to a 'stream').

I could, of course, save the content to memory/disk (based on some size huristic), but that would cost both storage space, and time. I'd therefore prefer some kind of streaming solution.

I'm still thinking about it, and I'm not sure it's worth the effort.

I'm pushing myself to hard again. It's the weekend and I should be relaxing but I've got a brain full of "got to do stuff!".

I mean, I'm not actually doing stuff (ish, I got some work done on webmail this morning, put away yesterday's washing, put in another load, organised Morrisons for tomorrow and drove us both to Asda for s few things).

What I mean is, I feel bad when I just sit (or lay down) and relax. I don't get to relax, I keep thinking about all the stuff I 'should' be doing (although not in any concrete way, just the whole "I should be doing something" feeling).

So I'm going to stop here and get some sleep.

TODO

Reality

  • Husband wants help dying their hair

Infrastructure

  • Decommision postgres (see osric.uk)
  • install nginx error pages
  • Move husband's website

Blog stuff

  • Move controls to the left bar if there is space (where 'enough space' is defined as the screen is wider than the current --max-width)
  • Add a calendar with links to posts
  • Add forward/backward links to individual posts
  • Add a 'save all to zip' option

Webmail

  • Get dovecot configured
  • Get postfix configured
  • Turn off the Server sent events stuff, for now at least
  • Get the host security stuff working Now that I've tagged endpoints with RouteValues[host], requests to the 'wrong' host should 404 (not 401/redirect to auth)

Osric.uk

  • Move database from postgres to sqlite, decommission postgres

Gah, I forget that restarting the server logs me out.

Husband's dyed their hair, I've sorted nginx error pages, and I've moved the blog menu to the left on wider screens.

I want to add a calendar to the left bar, that shows the current month, except it updates to show the right dates for blog entries.

Clearly it's just a gimmick, probably not worth the effort.

Image service

Google is warning me that I'm running out of photo storage space, so it's time to get serious about pulling my images out of Google photos and into some kind of useful local service.

The hard part, for me, at least, is designing a useful and plesent to use interface. I can lean on other designs, but it's still going to take a bit of work.

A specific feature I want is a 'Tag/Describe a random photo' page that loads a photo at random (either from all photos, or from untagged photos) and lets me set the tags and/or description. (Exif has an 'Image Description' field that's probably the right place to start).

The move from postgres to sqlite for osric.uk is ready to push. I would have done it this evening, except I was watching the final episode of The Orville series 3 which was just awful. I've been pleasantly surprised by most of the season, it's been a good, modern sci-fi TV show. The scripts have been a little rough/loose in places but the technical side (effects, camera work, editing, etc) have all been good and the stories have been interesting (if a little poorly executed at times). And then we got this episode. Two (really unconnected) stories, the robot crewmember propsed to his girlfriend, and the reasonable questions about "what does love mean to an emotionless machine" were skated over in favour of "here's some funny bad advice got the comedy robot". (The other story was a stab at "this is why we don't interfere with primitives", it didn't add anything new to the genre) I couldn't take it. I had to bail 15 minutes from the end, which (for stupid logistical reasons) meant I've shutdown my development environment for the night. An, well. The code will still be there tomorrow.

Want/need to get hi rez image uploads sorted, along with basic image manipulation tools.

I should benchmark image sharp to see if doing transforms on the fly is viable (along with finding out how it handles rotate - is the canvas resized automatically, what is the background fill it it's not).

I also need/want to put the design time into operators/language/macros, and an interpreter to parse them into image sharp calls.

Image Operators

Requirements

  • Must comfortably fit in a query string (avoid blocks, avoid whitespace significance)
  • Must be constructable from a push button interface (can use HTML5 inputs with minimal styling)
  • Should be constructable by a human with a text editor
  • Should be forgiving of technically invalid input (sensible defaults, clamp ranges, only invalid if contradictory, impossible, or conflicting)
  • Should do as little work as possible (e.g., spot an operation followed by it's inverse is a no-op and skip both)
  • May allow users to define macros/functions/shortcuts for common operations
  • May allow more than one output (e.g., rotate original and save as full size and thumbnail)
  • Must preserve original input
  • Must use consistent addressing (i.e., either top, left, width, height, or top, left, bottom, right)
  • Should allow percent as well as pixels
  • Should allow calculations (e.g., width/2)
  • May allow user variables (easy enough)
  • Should have useful set of system variables
  • Must apply operations left to right
  • Should report syntax errors with a location

Useful operations

  • crop(top, left, width, height)
  • resize(percent) // maintain aspect ratio
  • resize(width, height) // change aspect ratio
  • rotate(angle) // +ve clockwise, 0 pointing up

Implementation ideas

This is sounding like a basic expression parser. I'm not that worried about efficiency, so tokenize->ast->interpret should be fine (and I can always add caching later at various points). Tokens will be basic maths (including decimal (not floating point) numbers) ('+', '-', '*', '/', '(', ')') and identifiers/keywords (I'm not sure about idenifiers that aren't keywords yet).

I certainly want comments, so i can comment out chunks of code (How complicated are you expecting your stuff to get!), and really any pair of characters will do.

Time to have another flick though Crafting Interpreters, I guess.

It might be too hot again.

Otherwise, today was a good day. I'm writing a k8s monitor app at/for work and I think I've got the "watch" stuff cracked so I can do things like leave a page open showing events and pod logs.

Husband's dad is coming up for the weekend, this weekend (eeek!). Husband's has got a fairly complete itinerary planned, but I'm still going to need to interact with the honoured elder.

But it's late and I'm tired, so good night dear reader(s), see you soon.

(Note to self: Anonymous usage stats aren't immoral)

I'm tagging the blog software as version 1, it's time to think about version 2.

Blog - Version 2

Damnit, it's really irritating, but I think I'm going to have to put everything in a database. The logic chain runs something like:

  • I want to (algorithmically) add everything that I can to the blog. What I played on Spotify, what pushes to GitLab, what I asked Google, the photos I take, maybe even the emails i send and receive, all the online stuff I do should be an entry on the blog.
  • That's a lot of stuff, and I should keep track of its provenance (? source, context)
  • Some of it is personal/offensive/immoral/illegal (depending on context), and so needs to be tagged as such (and not shown to people who don't have the right auth)
  • I want to be able to comment on automatic posts ("So this is a photo of me and husband on holiday") (and probably manual posts too - see Bernice Summerfield for examples), and probably have replies to comments and comment trees
  • I want to save copies of posts before they're edited.

That's a bunch of metadata on top of the actual data. I could keep it in files, and I'm quite tempted by something like a Mine message.

Digression - Keep it all in files

Sqlite is very nice, and likely to be available for the foreseeable future, but there's something about text files that suggests a higher level of permanence.

Somewhere above I wrote about using IMAP as a backend, but does the idea because IMAP says messages are immutable. However, I also dropped the whole "Use rfc822 formatted files for storage" thing, and that may have been a mistake (or at least, premature).

(Note: I can't remember the current version number for rfc822, it's somewhere in the 5000s, I think. Please read 'rfc822' as 'The current internet mail format rfc' unless otherwise stated)

Ignore the email heritage of rfc822. It gives a structured way to add metadata to a file (metadata at the top of the file as Key: Value pairs, a blank line, and then the file data). Add in Mime, and there's support for keeping several files together (e.g., a post and photos (or other attachments), a post and it's edits, a post and it's comments).

There are plenty of independent tools to create, read, and update mime messages (delete is easy), and I'm already familiar with MimeKit for C#.

I would need to worry about simultaneous access/locking (but I could manage that from within the app). I might need to sort out indexing, although I do like keeping info in the filename.

Conclusion

Add always, add another layer of abstraction. Write an IBlogStorage interface with operations for get and save entries. Move to an

I begin to see why people advocate so hard for interfaces. I'm starting to think in terms of an IEntry with operations like Get Content(Version?), SetContent(string), AddAttachment, AddComment(Comment? Parent), that I can write different backends for.

(Poot. I think "storage layer" has gone out of fashion in favour of "database layer". Entity Framework makes too many assumptions that it's underlying storage is sql, and it's got into my brain. Ah, well I'll cope).

More thinking later, it's hot and late now. See you in the morning, dear reader.

I had a bit of a play with ASP Core Areas at $WORK today, and they are the right solution to the "I want this project to have a bunch of loosely connected modules that aren't worth putting into different projects/processes" problem.

Adding it to a project is easy enough, create a top level Areas folder with a sub-folder per sub-project/module/area.

Each of those sub-folders has it's own set of Model/Contoller/View folders/types, and the framework will find the right view, so long as each controller has an Area attribute.

Di config can go in program/startup as normal, or one can try something a bit more fancy. (The demo at work has an IStartup interface with a method that takes an IServiceCollection. At app start time, all the implementations of the interface are found, instantiated, and called. An alternative that's been in my head for a while is to tag types that should be available though DI with an attribute, although I'm starting to prefer the interface way since it's much more flexible.

Anyway. Have I already said goodnight tonight? Goodnight readers, sleep well.

Write ahead logs

Roughly, when the storage engine gets a request that would change a file, write the details of the change to disk before doing the change, so if the change is corrupted then we can tell by comparing what should have happened (the log) with what did happen (the real files).

I'd be tempted at that point to go full Event Source and treat the log as the source of truth, and read state at boot time.

Or, not waste time badly implementing a database and use a real database instead.

Maybe it's entity I don't like? Maybe I should go back to writing the raw sql myself (except entity is really convenient? It's so nice to be able to more or less ignore everything lower level than sets of types. Pointing at moving from postgres to sqlite as an example, none of my logic changed (and ok, it's all fairly basic). Maybe I should move the blog to a db).

Blog Schema


Entry
  Id
  ->Owner
  ->Content
  ->Comments []
  ->Versions []
  
Content
  Id
  Blob
  Created Timestamp
  ContentType
  Size

Comment
  Id
  ->Content
  ->Parent Comment?
  ->Parent Entry
  ->Child Comments[]
  ->Versions []
  ->Owner
  State {Unreviewed, Ham, Spam, Flagged}
  ->ReviewRecord[]

Thing is, it's very easy to get carried away with database schema design. Do I really need to keep a log of who reviewed a comment (and when), when I'm the only person with an account?

Commands, Queries, and Events

Commands are instructions ("Create an entry"). Events are a report on something that happened in the past ("An entry has been created"). Queries ask about the state of things ("What entries exist?").

Somewhere there's an engine that convents commands events. We also need a storage system that can listen to events and answer queries. Finally, there must be some kind of system that generates commands, send queries, and processes the results.

I think I'm overthinking again

That was a little unexpected, but I've moved production from Postgres to sqlite. Next time, I should try to remember what's been merged when I push to prod.

Now I've got the by date view working, previous and next become more pressing.

I need to make a choice about semantics: Does previous/next always refer to entries, or on (e.g.) a day view, does it mean the previous/next day, even if there aren't any entries.

Ignore that, it's clearly about entries. Probably once I add in other data sources, there's going to be entries every day anyway, but I might as well skip empty days (and stop at the date of the earliest entry). (A reader who decodes the URL format is welcome to load a blank page, but if I add auto links to the infinite past, some stupid spider will blindly follow them.)(serves it right)

Datasource

The word 'Datasource' in the previous entry triggered a minor epiphany. It's exactly the right term to use for the various components I want to add (Google Photos, Twitter posts, blog posts, etc.). If I can come up with a fairly basic set of common operations (get Name, getEntries(from, to)) for datasources, and for entries, then I can wrap everything in a couple of interfaces, and maybe pull some DI tricks at start time to dynamically include known sources.

I'm not sure yet what impact that will have on the existing blog code, especially creating entries. (I think I also want to tweak my language to mark the difference between a post on my blog, and an entry in the 'lifestream' (urgh, not using that name).

And ok, clearly since the blog is just another datasource, it won't need any changes. Instead, what I need is a new page that ~~gloms together~~ aggregates the various sources and displays them.

Requirements

First draft, off the top of my head, blah blah blah.

Entries

  • Entries draw themselves. The environment will draw a header (the time the entry was created, the source, and an edit link if the entry is editable), and the entry supplies HTML ready to add to the output stream (i.e., any user sourced markup is either escaped or very well sanitised)
  • Minimum data set:
    • Creation date
    • Source (link)
    • Permalink (a link to the entry in isolation, but still hosted here)
    • Body/Content
  • Under consideration
    • Something like a big vs small flag, although they might be per source (bookmarks and Spotify plays are small, photos and blog posts are big, although blog posts can be small)
    • IsFirst/IsLast/Next/Previous (to help the renderer, e.g., pull all the previouses, get the most recent, and that's the date for the previous page)

Datasources

  • I really want everything on one page, but there might be alot of data once I start pulling in photos (and bookmarks). I could default to current year (or even current day), but I really don't want to.

Alright, looks like I'm too tired to carry this on. See y'all later.

Another one of those 'just want to sit here and cry' mornings, although I don't seem to be able to actually cry anyomore.

Depression sucks.

I have been writing software since 1981, when Dad brought home a ZX81. Since then I have used more than a dozen different programming languages, and I still enjoy writing code and using software to solve problems, whatever the language.

I have been SC cleared for about four years, from when my current role within HMRC started. Nominally, the role is 'Front End Developer', but I have been leading on a range of software projects, writing full stack code from the HTML/JavaScript/CSS front-end, though the .Net/.Net core C# MVC web tier, past the Entity Framework/Entity Core database access later, down to the Dockerfile and bash scripts to automate the build.

As an example, a couple of years ago, HMRC needed to update 'COBRA', a Microsoft Access application that was used to track the near-real-time flow of money into and out of the department's bank accounts as customers make payments to us, and as we make payments to customers. The outputs from this app are sent to the Treasury fine times a day to inform them of how much cash on hand the government has.

As a Microsoft Access application, the previous version was restricted to a single user. This didn't fit very well with the high-profile nature of the application, and I was asked to rewrite it as a high availability solution.

I designed the solution around an AWS RDS MSSQL Server instance running in Multi-AZ replication mode. After that, it was a fairly standard C# .NET Framework MVC application, duplicated across two EC2 frontend machines (again, split across two AWS availability zones for redundancy).

My current role is split between development and system administration. The sysadmin part of the role covers creating and maintaining the CI/CD solution for the team. We have been using a self hosted install of GitLab as a git server and job runner, along with copies of Jenkins installed on our EC2 instances to take care of actual deployment.

Most recently, we are in the process of moving to the departments hosted Application Lifecycle Management (ALM) tooling; including their GitLab install, Artifactory, and Vault. We are also moving to a Kubernetes (K8S) cluster, so I have been updating our build pipelines to use Docker to make container images for deployment.

Our teams move to .NET Core has helped the move to containers, as ASP Framework apps only really work well under Windows. However, the move to ASP Core on Linux container has meant that we are using Oauth 2 against Azure Active Directory for authentication/authorization instead of Kerberos and local Domain Active Directory.

While the team owns our main project, we are called in to help other projects, since we seem to have a reputation as a team that works quickly and well. For example, last year I was working with the Inteligent Payment Project (IPP) to build them an MVC application to capture data from front line users and submit it to the projects API. I worked with the architect and the product team from early in the design process to make sure that the form I was building was as simple as possible, by avoiding asking the user for information that wasn't needed by the backend process, or that could be synthesized from other answers.

Working as a front line contact center advisor has helped me develop many things

  • An understanding of the customer point of view
  • The ability to explain complex technical ideas to people from different backgrounds

2005-2017 Call Center Advisor (HMRC)

As a front line call handler I took calls from the public, answered their questsions, and updated systems in line with department policy.

Working in this role helped me develop the skill of explaining complex technical ideas to people with a wide range of knowlege.

2017-2018 Guidance Author (HMRC)

I was promoted into a role with the Guidance team, who are responsible for writing and maintaining HMRCs internal guidance for front line call center advisors and other process workers.

As part of this role I moved to the tehcnical side of the team after updating some ASP Classic Visual Basic to run more efficiently, replacing a runtime of 30 minutes with one of 2 or 3 seconds.

2018 - Current Guidance Development Team (HMRC)

Another promotion lead me to my current role. I am working as a developer and system administrator with the Guidance Development Team. We build and maintain the software that hosts the internal gudiance.

That's NUglify integrated into the site. I've got a new Middlewhere that minifies js/css files, and stashes the minified copies in memory.

I'm thinking about bundling. I'm also thinking about a cleverer cache that will flush to disk every so often, but that's a component in its own right (see also: Image transforms).

Bundles take a list of files and concatenates them, to reduce the number of requests for a page. It's not so critical now with HTTP 2 and 3 reusing the connection, but there's still overhead from headers.

The main problem I've got with bundling is that I want to be able to specify a different list of files for each page, and the best way to do it is not obvious.

The page needs to communicate the list of files to the bundler, and it needs to do it via the browser, since the server doesn't know which bundle request goes with which page.

Having said that, I'm settling on a 'just stick the list of files in the URL' approach, probably using a tag helper to convert something easy for humans to type and maintain into something that's easy for the machine to parse at the other end, probably as the query string.

Service workers

Up till now, I have mostly ignored the activate event. This is a mistake. The activate event signals that the service worker code has been updated, and so I might want to invalidate the cache and repopulate it.

Other important notes

  • The page that calls register doesn't get the service worker! The page has to load through the service worker before it will use the service worker.
  • A new version of the service worker is installed if/when the fetched version is different
    • This implies that we should be careful loading the sw from sw cache
  • The new version doesn't start taking events until all pages using the old worker have closed
  • Refreshing the page doesn't count as closing! You must navigate away or close the tab. (Irritating from a debugging point of view)
  • force-reload (shift-reload?) bypasses the service worker

It looks like we're expected to use a different cache name with each different version of the sw script, and delete old caches from the activate event.

Todo: Work out how to programmatically set the cache version, ideally using either the hash of the sw code, or the slug of the timestamp.

Testing ASP Core apps

Prompted by $WORK, it's time to look properly at how to do end-to-end testing of ASP Core apps. I'm going to use Webmail as my example, mostly because it's got at least one heavy runtime dependency (dovecot).

My understanding of modern practice is that I should be running my test in a container, or possibly more than one:

  • The app itself
  • Dependencies (dovecot here)
  • The test runner

Running in containers gives me a known starting state every time, assuing I can get my containers setup right.

Having said that - I've got most of the script to build the machine anyway, is it going to add much more to the runtime to spend an extra 10 seconds starting a VM compared to a container? I suppose it depends on how long the tests take to run.

More data: Podman runs on WSL!

OK, time to write some Containerfiles.

Just had to login again, I'm trying to remember why I restarted the server. It might have been logging stuff?

New todo: write a reader for systemd journal.

I've been moving webmail over from Bytemark to Mythic Beasts, and it's nearly ready, the only remaining problem is that the Spsmassassin integration in using (spamass-milter) isn't correctly picking up the name for the destination mailbox.

I think that the problem is that it runs to early in the processing pipeline, before postfix has done the alias lookup Allegedly, it can lookup aliases at process time but it doesn't seem to be doing that.

The alternative is to use spampd, a wrapper around SA that acts as an LMTP proxy, and is therefore well past alias translation.

Of course, there's still a problem! spampd doesn't use SA's per user preferences. I hacked the previous version to do so, but the author rejected my patch as they'd just done a rewrite. Find to see how much work it's going to be to fix it, I guess

Squeeee! I'm fairly sure that was the last thing that needed fixing/looking at. I've already tested sending mail (as part of dkms), and that's all setup and working (with a script that can generate new dkms keys every month!).

Just need to move over actual emails and update dns, then I can get the old servers turned off. Eeek.

I kind of wish I felt worse about moving away from Bytemark, but they moved away first when they sold themselves to a faceless conglomerate without really telling anyone. Shame, but I'm looking forward to Mythic, and (given the discount from their job advert challenge) it's going to bed much cheaper for a bigger machine (£16/month for 2 core 4GB vs £32/month for a pair of 1 core 1GB).

I should pull down all the config. I'm very tempted to wipe and reinstall to check that I've got everything, and to tidy up wrong turnings. I'm going to think about that a bit more.

wepiu recipe

  • postfix
    • postfix
    • postfix-sqlite
  • dovecot
    • dovecot-auth-lua
    • dovecot-antispam
    • dovecot-core
    • dovecot-imapd
    • dovecot-lmtpd
    • dovecot-sieve
    • dovecot-sqlite
  • support
    • postfix-policyd-spf-python
    • spamassassin
    • libnet-server-perl
    • Custom spampd
    • opendkim
    • opendkim-tools
  • web
    • libnginx-mod-http-headers-more-filter
    • dotnet (via Microsoft)
    • nginx-light
  • system
    • apt-transport-https
    • certbot
    • curl
    • firewalld
    • jq
    • locate
    • make
    • tcpdump
    • unbound
    • wget
    • wireguard
    • vim

Webmail move: Checklist

  • Stop old and new postfix/dovecot/nginx processes
  • Start copy of mail folders to temp dir
  • Copy TLS certs to new machine
  • Check new postfix/dovecot conf is pointing at the right cert
  • Confirm that new postfix has the right domains
  • Create sites in new nginx for new domains
  • Update webmail app config for new domain names
  • Login to Bytemark panel
  • Login to Mythic account
  • Get the "yes it's really my domain code" from Mythic and apply it to the Bytemark panel (for all domains that Bytemark is still hosting)
  • Delete other records from Bytemark dns
  • Add domains to Mythic
  • Create records for mythic
  • Wait for mail messages to finish transferring
  • Move into place
  • Start new dovecot, maybe wait for indexing (Todo: check if there's a "reindex" command)
  • Start nginx, confirm dns and imap
  • Check imap from phone
  • Start new postfix

If there are problems then can point dns back at old machines.

Todo: Setup a couple of scripts to add/remove the DNS records from the mythic api

Running a local CA, I mean, how hard could it be?

I like the idea of mutual TLS (mTLS); each service has it's own key pair signed by a common certificate so they all know that they're talking to the right people. Generating a self-signed certificate (a CA) is a one liner, and generating signed keypairs isn't much harder.

I think the tricky bit is key rotation. Ideally, keypairs would have a short lifetime (under a week, maybe under a day), so there must be a way to automatically install new keys and (where needed) restart services.

But that's still just a script, yeah? Create key, sign key, copy key into place, restart service (or ask the service to reload it's certificates if it can). Maybe it's because everything is on the same machine and so I don't need to worry about secure transport, but even so, that's still a solved problem (using certificate signing requests).

Maybe I'm missing something obvious?

Project/ideas list 2022-08-28

  • Improve my webmail UI
  • Spotify interface
  • Google images
  • "game"
  • Image upload editor

I've got plenty of space left and right of the title/controls, so I can easily put more controls there. I want a first/last, and I like the idea of a calendar, although that might need to hide behind a drop-down.

The other fix to do is for the latest/last seen tags. Instead of two (one ::before and one ::after) and some messing about with positioning, I want one ::before, and then latest/last seen can set content (with a .latest.lastseen combined selector for when the entry is tagged with both.

Except that won't work if/when I bring in proper tagging.

I'm going to ignore that for now. Get the current problem fixed, and then think about how to fix it if/as/when I actually do bring in tagging. (I'm not sure, for example, how I'd tag this entry).

I also want to add counters for each page, probably broken down into url/user agent/count tuples. (I'd like to grab ip as well, but that's PI enough I'd need to ask consent. I wonder if ASN is PI?)

Thinking about it, I want to add date into that as well, so I can look for trends (or see spikes).

I should probably also track response status code, and maybe response generation time

I was thinking earlier, it's not Prometheus that's the problem, it's grafana. Grafana is both too complex, and not able to do the things I want. Time to research alternatives. (Yes, Google charts is on the list, but so is writing my own (basic!) SVG chat drawing tool).

That's latest/last seen fixed/moved. They're now ::after the header, and with no messing about with position.

I looked at adding the .latest class on the server, but I got bogged down with lists and stuff. (I've been trying to hard to handle entries lazily, but since I'm fairly sure the page is held in memory until it's been generated, I'm not sure I see the point).

Hello world! Feeling alright today, compared to the recent moving average. Turns out that I've missed maybe half my anti depressants over the last few weeks, which might explain the annoying amount of depression I've been feeling. It looks like the trick is to take pills as part of the "feed the cats" activity so I don't forget.

Otherwise, worried about money. Just run September's budget, and house has good something like £1.50 spare after bills, food, clearing overstay, putting aside half (!) the cost of getting the kittens neutered next month, and the money for my filings. And that's all before the estimated 80% rise in power bills next month. (We're paying £138/month direct debit, so another £110/month, maybe).

Kittens are at least fit and healthy, alternating between sleeping and chasing each other round the house. Storm is maybe tolerating them, but doesn't seem to want to play. Since the kittens don't understand (or maybe don't care) that Storm doesn't want to play with them, there's still a bit of conflict, but it does seem to be settling down.

Work still sucks, but that might just be a physical thing (in that I have to sit upright for 8 hours and my body doesn't like that), although the stupidity of his things are done is still sapping my will.

Having (another? I've lost track) stab at setting up Ory Hydra as an OAuth/OIDC server. I've fixed the permisison problems and passed a basic smoke test, so now its time to actually think.

There are three components: the public hydra server (the API that handles the actual OAuth stuff), the private/admin hydra server (for registring clients etc.), and the UI server (my bit, that draws the login/logout pages).

This is all behind nginx and under the same hostname (current best guess: auth.osric.uk), with nginx proxying the appropriate paths to the appropriate servers.

That's going to need another nginx config (Ory docs have an example) and another TLS cert. Given my current design choices, it's problaby going to be another user database as well. (Hydra doesn't do user management)

That all seems reasonable, yes?

That's the programming, config, testing stuff (for the OAuth service) more or less done. There are a couple more things I could do (mostly about pulling in client info onto the consent screen to make it much more "Example.com wants access to...", but that's not MVP, at least at this (tiny) scale.

What I need to do next is decide where it all goes administratively. Is it part of webmail (or something like my blog), or is it a stand alone service?

I'm leaning towards webmail. It's already got the login/user infrastructure (and accounts), and name recognition with the expected userbase (me and husband), I'm just reluctant to mess with it while it's working. I guess that's what branches are for.

Plan

Hang it all off the webmail domain. Webmail users shouldn't notice anything different. I'll need to tweak the paths a little too make sure that nothing overlaps, although the existence of a challenge query parameter is diagnostic of an OAuth request.

I'll need to do a migration on the webmail db, no worries, I need to get a backup/restore thing working anyway.

I'll end up with a "new client" script, the settings will all be more or less the same each time, only the name changes (and at that point I can integrate it with the new user/project script)

Scary, but doable.

The testing setup I had working for webmail (running dovecot in s container) has stopped working and I don't know why :-(.

I mean, it's clearly something to do with container networking but I haven't been name to chase it down properly. Roughly, dovecot auth should be hitting an http endpoint to validate the given username and password, but I don't think the http request is making it out of the container.

My next step is probably too break out tcpdump to see if I can trace the packets, but I want to understand how podman does it's networking first. (Specifically, I want to know how it's getting packets out of the container without an interface).

Good morning, dear reader.

I'm laying in bed with a kitten snuggled up against me, which is a good start to the day.

On the other hand, I've got a headache that I think it's caffeine withdrawal (on the grounds that I've got from 3-4 cans cola/day to 0-1).

The OAuth/OIDC stuff is coming along nicely. I've integrated the ui code into webmail without any major problems, and I'm working on replacing the local auth for the blog with oidc.

Poot, out of energy. See you later.

Depression is the suck. I'm taking my pills and I'm still laying here on the sofa unable to concentrate on programming (enough that I've taken a sick day). I'm am so done with this. I thought that I had it (depression) covered but it's still jumping on me every so often.

Anyway. I've grabbed someone's flocking code and converted it to JavaScript, but I don't like how they've done edges (which is to wrap position without wrapping neighbor detection), and I'm having trouble fixing that.

I've got the basic OAuth stuff apparently working, but i haven't been able to replace auth on this site.

$WORK is being crap again. TL has made a political choice that's going to cause trouble later (we're building a thing but we're not going to put it up for testing was we go, so it will be too late to make substantial changes when the inevitable problems surface).

No conclusions.

Looks like my latest depression flare up has calmed down a bit. Got some personal code written yesterday and some stuff for work today.

I very much enjoy writing code. I noticed recently that, for me, it maps quite well to the image of someone in their shed/basement building an enormous model railway; an endless project that could easily seem pointless to an observer, but one that helps the participant gain a sense of peace/balance.

I like the simplicity of programming. I like knowing that, so long as I follow the rules, the outcomes are known and predictable. I like building a big, complex mechanism and then watching it chug along happily under it's own power.

Ow, my teeth.

Had a bunch of fillings done on Monday, teeth are being very sensitive at the moment. Totally sucks.

Gah, that's been an irritating few days. It looks like is broken with ASP Core Oidc, in that I have to set a name for the 'nonse' (snigger) cookie. On the other hand, now I know that, I'm fairly confident that Oidc is, actually, working! Whoop!

Next job, getting the Dr Who upload thingy working.

Potential enhancements: Look at the various Hydra endpoints to make a "This is where you're logged in" page (with "Logout" options). Also, tidy up the consent page

  • Make cookies smaller.
  • Finish file share

That's not alot of a to-do list. I wonder what I'm missing?

Anyway. Sleep well, gentle reader.

File upload site got a good positive reaction, much better than I expected. Husband very much appreciated using the BBC Mode 7 font.

Need to update nginx config to allow huge uploads. Husband doesn't want an estimated time to complete. Should add a ring close while uploading warning, and make the boxes wider so there's no wrap.

Otherwise, off to the beach!

I'm feeling a bit odd - I don't have any (internal) pressure to get something finished. OIDC is working, I've done husbands file upload site, webmail is ticking over nicely.

I should probably look at one of my many todo posts, but I'm not that bored :)

Not really sure what I want to do with myself today. Husband is asleep, work starts again tomorrow. I did a bit of work on flocking yesterday, I think I'm going to go pick that up again.

I'm still annoyed about how hard it is to get hold of the real-time data from my smart meter. It's seriously looking like the easiest solution is going to be pointing a camera at my in home display.

Still not sure what I'm going to do about my old Facebook/Twitter posts. Parsing the JSON and creating blog posts is a chore, but not the problem. The problem is the conflict between "I want all my posts on one page", "gosh that's a lot of posts", and "wow, a whole bunch of these posts are hard to parse out of context".

I can default to showing one days posts (and then visitors can use the date links to go back)(and show the latest date with posts of there isn't anything for today).

I could categorise/tag posts, and only include current blog by default (I think I like this one? It makes including weirder stuff like git commits and house moves make more sense, as one could filter in/out categories of interest)

I could just cope, it's not like I've got any evidence of readers.

Teeth still hurting this morning, saw a dentist, taken some antibiotics, teeth are hurting much less. Dentist says tooth will need to come out but will argue against that.

Had a chat with my boss/manger/team leader (Hiya DK if you're reading this!), I've been taking a lot of sick leave recently (over the last year or so), enough that they're talking about maybe needing to remind me of the sick rules. Fine, whatever, but don't expect me to care.

It's just that work is so stupid! They want us back in the office, and can't give a better reason than "spontaneous collaboration", which sounds very much like gossip when pushed. They want us to be "at work" (either at home or in the office) for a fixed number of hours a week, but they don't seem to care what we're doing with the time. It's just so trivial! I guess it's one of those neurodiverse things - I hate to see the waste (of time, talent, energy, resources) that these policies generate, but worse than that is the way that everyone agrees that it's broken but nobody fixes it. I genuinely don't understand why there's so much disconnect between what people want and what people get.

Ah, well. Husband says that I need to learn to relax at work, and they're right. I'm still getting paid for this bullshit even if it is crazy.

In better news, I've been playing with Web Assembly (wasm), and it's going fairly well so far (although I'm still only at the smoke test/toy code level).

I'm using clang to compile C to wasm, and then loading it into a web page. I give the wasm VM memory from JavaScript, ABC that means I can pass data between the two. Anything more complex than numbers needs to be flattened and re-inflated across the boundary, which is a chore (but not a hard one).

Current plan is to move some of the flocking maths over to wasm to see if it's any faster, although I'm distracted by GPU.js which calls into the GPU from JavaScript. For something like "find the distance between all the pairs of points", I think it's worth looking into (although really, I should grab some 2d space partitioning algorithm and use that instead).

(Really, I should find/build a metrics framework so I can measure how much time is spent doing various bits and pieces, so I can actually compare the different ways of doing things)

So I want to make this blog an installable app on my phone. What does that mean in practice? Mostly, I think, that I want to be able to write (and save!) posts even when I'm not connected, and then push them to the backend next time I get connected. Rendering posts clientside implies a JavaScript markdown parser, which do exist but is too heavy for my taste.

I've got a rough idea of showing a marker in the offline client for "there is a post here", (although, I've just thought, I can show the raw text until it's been posted).

I'm already stashing unfinished posts in local storage, but I don't want to get confused and start thinking that's the same thing.

I've had a look at the code to check, and the on submit for this form deletes the draft and then let's the POST go through. Clearly, if POST fails then all is lost.

If I add an id to the post before submitting it, and include the id when I send back the HTML, then the frontend can tell that a POST has been successful. If I stash a completed post in local storage using the id as a key, then I can find and delete posts from storage at page load time (and re-submit any that should be there but aren't).

That's work that only needs to be done for logged in users, which is fine, I can do a server side check and only include the JS if the user is authenticated.

I can even add a <template> for entries that haven't been submitted, and render them at page load time.

Things to check:

  • How to catch a failed POST from a form submit
  • How to turn off a service worker on logout

Alrighty folks! Today is all about stats!

I want to know which (if any) of my pages are getting hits, and ideally if they're getting hits from not-robots.

I can easily add a middleware that grabs some stats and posts to a queue to be added to a db (so it doesn't slow down the main thread too badly). Question is, what should I keep?

I very much don't want to collect PII, so I can't grab raw IP. I'd like to know the difference between bots and not-bots, so I'll need User Agent (or at least, the result of userAgent.Contains('bot')). Obviously I need the URL. I'm not expecting very much traffic, so I can bucket into hours (which also helps with the PII issue).

That looks something like:

CREATE TABLE HitsByHour (
  Text Path,
  Number Count,
  Number Date, -- seconds since 1970 to the start of the containing hour
  Number IsBot, -- 1 yes, 0 no
);

I think I also want a view like:

CREATE View IF NOT EXISTS
  HitsTotal (Path, IsBot, Count) AS 
    SELECT Path, IsBot, Sum(Count)
      FROM HitsByHour
      GROUP BY Path, IsBot;

Did some research ("cat access.log") and doing a check for "bot" isn't going to cut the mustard.

Got distracted from the stats plan predictably quickly. On the other hand, I've tidied up my DNS records (mostly upped the TTLs from 5 minutes to one day, since I'm fairly confident they're correct), updated the nginx logs to include cache hit/miss and hostname (the first to see if the cache is working (spoiler: not as much as I'd like) and the second because all the hosts go into the same file and it's not clear which request is for shiv host), and verified a couple of sites on Google search console.

First news from the logs: My Windows browser uses IPv4 for one site and IPv6 for another, even though both share addresses. Weird. I'll wait a couple of days and see if it's still happening.

Life is ticking over nicely, work is boring (which is fine, better than many kinds of exciting).

I've added a calendar to the front page of the site, not sure what to do with it Could obviously link to blog day entries, could probably put in moon phases, but I don't think I'd use with of those.

Read a post today that used CSS like syntax for SVG, which got me thinking.

I'm a big fan of SVG. I love the declarative nature of the format ("Draw a circle" vs. "pixel, pixel, pixel"), as well as being able to write it in notepad.

On other other hand, like all XML dialects, it's a bit wordy. I can't find the link, but a while ago I read an article that showed how easy it is to map XML to Lisp.

Roughly, an XML node is a name with a list of attributes and children, where attributes are 2-element lists, and children are either nodes or strings.

<svg><circle x="40" y="40" r="17" /></svg>

(node "svg" () ((node "circle" (("x" "40") ("y" "40") ("r" "17")) ()))

I've had a bit of a poke at mal and it's actually feeling like it's a real thing.

I've got a few things I could do next, it needs a better UI for the repl, I could build it again in JS (to run here in the frontend), I could build it again in C and compile to wasm (to get exactly the same code running front and back), and I could handle mal code blocks with the C# engine.

The cli needs readline like stuff - at least basic cursor movement. Ideally, it would stash the environment (and history) at exit and load again at start, with some way of editing (i.e., save to mal code).

Rewriting in C would probably be a bugger, although there's always Crafting Interpreters to fall back on (it's got dynamic lists, rough polymorphism, even a garbage collector!).

Rewriting in JS would be easier, and I could use quickJS to run/test on the server (which is mildly horrific, but that's life in 2022 baby!), but I'm still on the fence about JS. It's nice that there's common language in browsers, but did it have to be that one?)

I'll have a stab at writing in C, trying to use as little libc as I can get away with (realloc, printf, probably some string comparison stuff, I'll check the book later) and see if that's a realistic option.

(I'm going to end up with Lisp interpreted by wasm calling out to JavaScript for IO. Groovy.)

Interesting, just got prompted by webmail to confirm consent. I guess some cookie/time limit expired. (I should update the consent screen if I'm going to be seeing it again)

Maxims

(Rules that may help lead a better life, not complete)

  • Mean people suck
  • Asking questions is good
  • Fixing mistakes early is much cheaper than fixing mistakes later

Moving downwards with speed u and acceleration g, if I start thrusting with acceleration -a, how far do I travel (s) before my speed equals zero?

Then, as I'm falling I can recaculate with u = current v, and when s = current height I can turn my engine on and reach the ground when my speed is 0 (ignoring for now that my thrust changes as I use up fuel)

v² = u² + 2as
v² - u²  = 2as
(v² - u²)/2a = s
-u²/2(g-a) = s

Check that the units make sense

(ms⁻¹)²/ms⁻² = m
m²s⁻²/ms⁻² = m
m=m

Yup, that checks out. Time to give it a try.

I've picked up Kerbal again, still playing with the kOS plugin. This time, instead of going straight for orbit, I've spent a bit of time messing around with sub-orbital hops (which has got me lots of lovely science!).

I've also written a landing script that slams the engines on at just the right height to reach zero velocity at the surface. I've tasted it on Kerbin and it's looking good, next stop Mun!

However, I can't put it off any longer, next session is orbit or bust!

(The maths posted at 14:22 today works btw, it's correctly calculating the hight I need to turn my engines on at.(Which is quite scarily close to the ground, I can tell you!)

So the theory goes that it should be easier to write Lisp on the phone because it uses fewer funky characters:

(define test (+ a b))

I can see that working, although writing a formatter goes on the list.

Mal is scheme-ish, but certainly not scheme. I'm going to need to commit one way or the other fairly soon, and I'm still not sure which way to jump.

I'm much more likely to find online help for scheme. It's got it's own interpreter, it's even got a compiler!

Mal is small enough that I could write a JavaScript implementation, so I can run stuff client side. Also, because I've written the interpreter I should have a better idea what's going on (I'm not sure that I do, but I should). It's also much easier to extend (although I should look at how scheme does it).

I think I want to use mal, but I'm worried about support. It's probably close enough that I can translate, and I can change the name of some things to make it closer. I very much like the idea of client side evaluation, so I can write simple scripts on my phone (Oh, Psion, I still miss you!), although there's a bunch of saving/synchronization to think about.

Grinding through mal-js implementation, can't decide between using js objects or a bunch of classes.

JS doesn't have 'symbol' or 'keyword' separate from string, or 'vector' different from array, but I can add a type property for the times when things are different.

Writing my own types is theoretically safer, since mal code won't be calling js methods directly there's less surface area for naughy code to fall through.

Dunno. Going to sleep on it.

It's been a bit odd at work the last few days. I've ported a couple of tools from Visual Basic for Applications (VBA - Microsoft Excel macros) and powershell to modern C# (for reasons that I haven't paid a lot of attention to, but I think are related to the security people at work tightening up execution policies).

The code isn't very complex - download a few files, calculate and compare a hash, unzip them, and move then into place - but the team are all like "OMG they're so fast!"

The download/install now takes two/three minutes with a good network and fast disk, compared to maybe half an hour previously. That's a useful speed up, sure, but the amount of praise I'm getting feels very out of proportion to the amount of work it took (especially as I didn't put any thought into speed).

Husband's advice is to just accept the praise, and I'm doing my best, but I'd far prefer someone to gush over my clever stuff.

Ah, well.

I do have a couple of ideas about how I could speed up the install, but I'd have to test them to see if it's worth it.

Mostly, it's to not bother saving the zip to disk, but instead unzip the stream on the fly, teeing off the stream into the hash calculation as we go.

There are four zips, a root and then three children that unzip into a folder under the root. At the moment I'm unzipping all four each into their own folders, moving the children into place under the root, and then swapping the root info final position.

If I go for the stream unzip, then I'd skip the first step, and unzip all the files into place (so skipping one of the moves). I'm not sure how much it would save, the moves are all on the same filesystem, so should be cheap/fast (and watching the process supports this). It would also create complexity, as files from any given folder might end up in more than one of the child zips (don't ask! I tried to get some clarity on this but just got white noise) so there might be conflicts/races when creating folders.

It's a well known trope, apparently, that users really don't care about fancy coding tricks but will very much like the duck animation they get while they're waiting....

Made a couple of updates to the blog code, Google search was getting confused because all the pages link to the login page and it wasn't sure which one was "right". I've added a link header pointing at the return url, let's see if that makes it happier.

I've finished another couples of steps in mal-js, eval and environment, so (+ 1 2) now correctly evaluates to 3, and def! and let* special forms work. (I'm going to lose the funky characters once I'm done, although I could just edit the tests now. Hmm)

Using JS types directly didn't work (because a string value isn't an object, so one can't set arbitrary properties in it), so I've gone for a simple wrapper class with a (string) kind property (to tag the type) and a value property to hold a JS value.

It seems to be working so far, it's nice not to have the big stack of classes and worry if hash should extend sequence. (Having just written that, I'm idly thinking about adding an "isSeq" method, since the only difference between lists and vectors is the shape of the brackets when they're printed)

Mentally it was a rough day. I never really engaged with work (Y'know, I say that but I did deliver a product and refactored shared code into a library. I'm starting to think that my standards for 'good enough' may just be far to high. Husband, if/as/when you read this, you were right so along and I'm sorry for doubting you) and so I bailed an hour early.

Mental note: I want to post something during the good the clocks go back so I've got something with a stupid timestamp to test against.

An extra three hours at work today, investigating a problem with a service. The service is running fine, except it's looking like it's taking IIS three to four minutes to start processing the first request from a cold browser. Other requests are fine after the first one, so long as the browser stays open. There's nothing obvious in the IIS logs, other then the trace logs don't start until 3-4 minutes after netstat shows a connection.

I'm back home now, and I'm going to try to squash any more thinking about it this evening, but I'm probably going to be back in the office tomorrow. Which sucks.

Feeling better again, that was weird. Came home from the office and was wiped out by (what felt like) immune system response. I was hot, mildly confused, and aching so over. I slept for maybe 4 hours, got up, managed to get downstairs, lay on the sofa for another hour, and then just felt well again.

Two theories: I picked something up at work and my immune system did it's job, or I made myself sick at the idea of being in work. Hard to tell which it is.

Ah, well.

I haven't been using my primary laptop for maybe six months because the keyboard was broken. Yesterday, while waiting for w traffic light to change, I saw a little PC repair shop. I phoned them this morning to get a price check on taking the laptop apart enough to resit the plug at the end of the keyboard connector cable (£10-15, apparently). So I booted up the laptop to confirm the problem is still happening, and it's not?

Does this mean it's fixed? I don't think so, but I'd be very happy to be wrong so I'm going to fettle it up like it's working and use it as my primary until it starts being broken again.

This does leave me with the traditional "Windows or Linux" question, although given a hybrid Intel/Nvidia graphics card, and how well WSL2 works, the Windows option is fairly strong.

Old laptop is still working, so far so good.

I've wiped it and installed Windows 11. First impressions are that it's going to be fine, it's fairly close to Windows 10 and the differences are minor. Windows 11 feels rounder (like, all the corners have got a border radius of something), and my first impressions are probably trained by the stupid keyboard on that laptop (I paid extra for a "mechanical" keyboard, boy was I dumb).

The key spacing is different/wrong, and I'm constantly typing one character to the right of where I should be Also the Windows key doesn't work (nothing happens when I press it, haven't investigated any deeper yet), which is bloody irritating given how much I use Windows snap.

I'll clone down a couple of repos tomorrow and see how I get on.

That's mal-js finished and installed as a client side app into this site. Now to come up with a use case....

Kittens were at the vets today getting chipped and chopped (neutered). They seem to be ok so far, not very happy with the cones, but they're temporary. We've moved the screen door to keep them locked into the lounge/kitchen (with the screen door set so Storm can come and go from the rest of the house).

Teeth have settled down again, thank science for antibiotics. There was a weird lump in the roof of my mouth next to the tooth that hurt, that's also shrinking, so I guess it was related to the infection? The root cause this time was a filling that's fallen out, but the dentist will replace it free since it's under warranty.

Emotion-wise, nothing spectacular seems to be going on. I've got very little enthusiasm/engagement at work, except I'm grinding through the bit of Ocelot v2 that's replacing classic ASP (in JavaScript) with c# MVC (ish - the controllers are fairly straight ports of the JavaScript right now, I'm sitting very firmly on the urge to "improve" or even "tidy up"). So my work emotions are confused; I'd rather not be at work, except I've got this nice low-impact, fairly open-ended job to do.

Shrug. But if you come up with a use case for Mal, please do let me know.

Not sure what to do next. Mal-js is done. Webmail probably needs looking at ("Webmail always needs looking at").

Actually, let's take that one. I'm still not really happy with webmail. I want new mail notification, but without to much client side stuff.

I was idly thinking about JMAP the other day, writing a wrapper for MailKit as a start, and then maybe writing a proper server.

I could give shopping list another stab, I've got most of the components for a web based version using commands/events.

Actually, that one does grab me. I'm off to a better keyboard to think about it.

Shopping by events

Events are things that have happened, so should be in the past tense.

  • ListCreated

    • listId
    • listName
    • ownerId (maybe)

  • ListDeleted

    • listId

  • ItemCreated

    • listId
    • itemId
    • itemName
    • itemState

  • ItemUpdated

    • listId
    • itemId
    • itemName
    • itemState

  • ItemDeleted

    • listId
    • itemId

I might have been over thinking sharing. If I add an Out Of Band (OOB) portion, then it all becomes much easier.

  • User A clicks "Share List"
  • Server generates a unique id and stores it along with the list id, user id, and timestamp
  • Server gives User A a link containing the id
  • User A sends the link to user B OOB
  • User B uses the link
  • Sever uses the id to look up the list and:
    • Adds User B to the users for the list
    • Deletes the id
    • Notifies User A

Need to add user models, including a per-user notification queue.

Also need new events for ListShared (or UserAdded, can't decide which).

Also need to think about commands, since the local client can't create share links. (But most traffic from local client will still be events, as the thing will have happened by the time the server funds out about it)

Rule: The local client is the source of truth.

is there any need for lists to have a clock? Yes, need to be able to order events well enough that lists are consistent between users.

I've been thinking about writing a build-time tool to replace strings in source files (specifically to add version numbers to <script> tags, and to the load service worker call), and I've just realised that if/since I'm doing build time shenanigans anyway, I might as well do JS/CSS/HTML minification at the same time.

(I was going to say something about HTML minification being hard as it interacts with the ASP stuff, but that's what Rosyln is for, yeah?)

Hello world, Monday evening, not much to report.

Teeth are back to normal, got a sedation assessment appointment Wednesday morning, one of my filings has fallen out and needs to be replaced (under warranty!).

Work is still porting Ocelot from Asp classic to Asp core. It's going well, I'm using Selenium to write tests to properly exercise the code. Selenium is really quite neat! It's a web automation platform, which means it starts up a browser and then uses an API to control it. The tests are things like "Find the element with this css selector and then click it", which means that it's the same as a person running through a script, but faster and reproducible.

I'm very happy with it, and I'm nearly at the point where I trust it not to give me false negatives. The only issue I've got with it is that the logging could be better - I still have to load the site manually to find the underlying why a test failed (but that's probably me asking to much, it's still much faster than manually starting the site and navigating to the problem to test a fix, and it's trivial to run all the other tests at the same time)

Kittens have been neutered, they seem to be recovering well (Havok faster than Carnage, but Carnage feels a lot more delicate than Havok). Carnage peed on the stairs this morning (right in front of me, so I know it was her) (not an ideal way to wake up). I think she was a bit altered from the pain meds we'd given her yesterday. (She should have had them with food but we have them on an empty stomach), but it could have been a protest, or some other communication.

I'm doing ok, ish, maybe a little below baseline. I think I'm just feeling the weight of another 30 years of work.

Ah, well. I'm going to read my phone for a while and then sleep. Picking up my new work laptop tomorrow, that might be interesting.

I'm having a lot of fun playing with mal-js, but I need to build a better way of storing/reusing scripts than emailing them to myself. (Or I need to build an email endpoint that evaluates mal. One or the other)

Anyway:

(def! acc (fn* (fn z args) (
  if (empty? args) z
    (acc fn (fn z (first args)) (rest args))
)))

(def! range (fn* (n) (
  if (= n 0) ()
    (cons n (range (- n 1)))
)))

(def! reverse (fn* (l) (
   apply conj () l
)))

(acc str "" (reverse (range  9)))

Accumalate, generate range, and reverse (because range creates numbers in the "wrong" order). Next up is let*.

I'm updating the interface for Mal to give the user multiple open files. (Because often when I'm working on something, I need to try an idea without disrupting the current code).

I'm also debating if I should offer backend storage (instead of just local storage). It's going to be "Yes, but only if you've got an account" (so me only), and only so I can make transfers between the laptop and the phone easier. On the other hand, there's suddenly a whole bunch of concurrency stuff to think about.

I want to improve the editing experience. First up is formatting, I'm developing a set of rules to apply to an ast to get pretty output, something like "start a new indent level of the current item is a list with more than zero children that are lists". (I should write it in mal as an exercise).

Anyway, it's lunchtime at work so I'm going to get ten minutes shut eye.

Just finished watching Bullet Train, a movie recommended to me by husband as "a very [me] friendly film", and yes, they were exactly right!

Plot: Our protagonist, a former professional killer who is trying to work out some things, takes an easy pick up job, to recover a case from a train. However, several stands of fate are entwined with the case so wacky highjinks ensue.

However, that description really doesn't do justice to the complex, tightly written, and complete story that follows. It's an action movie, in that there are plenty of action scenes, but it's well paced with breathing space in the right places that didn't turn into stalls.

The effects were mostly subtle enough that I didn't see them, the direction and camerawork was just brilliant, framing shots cleanly and dropping in enough hints and pointers to let me keep up without overwhelming me or getting in the way.

I've got a couple of quibbles about some of the exterior train shots, when a 200kmh train turns a corner sharp enough to see, but I'm prepared to forgive those as they weren't relevant to the plot (and could just be edited out, probably (I'm not an expert)).

It's reset the standard for a good movie, replacing R.E.D as "best action movie I can remember watching".

Highly recommended.

I'm frustrated because I I'm having trouble choosing the "right" language to default to for projects.

The choices are:

  • C Static typing, manual memory management, complied, exemplary standard library, lots of 3rd party libraries. It's a lot of work to get started with C, but I could build up a library of useful tools.
  • C# Static typing, automatic memory management, complied, good standard library, lots of 3rd party libraries. The development environment (VS Code, or even Visual Studio) is first class, the documentation is extensive and consistent, and it's got a good crop of answers on stack overflow. Problem is, it's owned by Microsoft and while they give the tooling away, they're starting to move towards closing off important components (e.g., the c# debugger may only be used with Microsoft tooling, which includes telemetry back to Microsoft).
  • JavaScript Dynamic typing, automatic memory management, interpreted, piss poor standard library, lots of 3rd party stuff of varying code quality (but mostly mediocre at best). Every time I start something new with JavaScript it takes me a bit of time to get my JavaScript head on, but once I'm there I very much like it as a language. Using QuickJS on the server has potential, although I'd need to write a bunch of stuff (in C) to turn it into a server side environment
  • mal/lox Both of these are teaching languages, designed more to show how to implement a language than to be useful languages. Since I write them myself, they're both relatively easy to extend to match my exact needs. In the other hand, they'd both need extending to match even my minimal needs.

I dunno. I think I might be reacting against node/npm in rejecting JavaScript. There are at least a couple of fairly good looking http C server libraries that I could hook into QuickJS (or vice versa), to build a basic server. That would scratch my C itch, and give me a relatively fun place to work from.

Or I could ignore the politics and stick with C# for the scale and simplicity, except that's just making it harder to move away from C# when Microsoft does pull the rug.

Big programming win at work today, I've got Vault working with auth from k8s - meaning that apps can get all the way to runtime before finding out secrets (like their database login details).

I've got a couple more things to investigate/get working, pulling the HTTPS cert from vault, and integrating vault into the configuration system. I want to do that now, but I'm not at work? Having said that, the new work machine ("HP Elitebook") is quite nice as a laptop, so I could go grab it and mess around down here.

There's definitely something wrong with the login cookies here, although logging back in doesn't take very long.

I seem to be taking a break from my programming projects, which is fine, of course. This has been a semi-concious choice, in that I saw it coming/happening and decided to go with it.

Instead of programming I've been sinking time into sleep and games; sleep, as ever, is still brilliant (I'm getting dreams, which I'm not wild about, but they're not bad dreams, just frustrating), the game I'm playing most is "Turing Complete", which takes the player through building up a computer from NAND gates. It's a nice reminder that this whole computing thing is a whole pile of levels/abstractions over some really brute simple stuff - chip complexity is measured in transistors because that's really all you need.

(On the other hand, the abstractions get pretty far away from transistors, and even gates, fairly quickly. I'm up and running on a Turing complete CPU already, and it's made of components that are made of components that are made of gates. I should contact the author with a wishlist request to be able to flip a design back into NAND gates.)

Also, toothache is back. On the other (left hand) side of my mouth, and not really concentrated with a particular tooth, but still, ouch, and I'll go see the dentist tomorrow and hopefully get another antibiotic prescription (although there shouldn't be anything left after the last couple of courses).

It was so liberating being told that I didn't need to put a title on every (or even any) blog post. I can just write and post without needing to add a label! It's brilliant!

Back from the dentist (like, twelve hours ago). Got an abscess above one of my teeth ("upper left three") which needs a root canal (because the previous filling didn't take enough out?).

I asked why I'm getting new infections after taking two courses of anti biotics in the last month(ish), got some kind of garbled response that I'm parsing as a shrug. I'm a little annoyed, this all started about a year also when I went in with an infection, send came out with a bunch of fillings and a mild sense of shame.

For something like thirty years (from early teens to mid-forties) I didn't brush my teeth, and I had approximately zero problems during that time (maybe an infection or two over the years, but no fillings or other work). However, a couple of years back I started drinking cola for the caffeine, and eating boiled sweets for the stim/sugar, and that combination seems to have seriously damaged my teeth.

Last autumn, dentist said I had to start brushing regularly or they wouldn't do any work (and I was only in for toothache). They stuck in a big of fillings in January, a few more over summer, and I've got more work scheduled in a couple of weeks.

And I'm just passed off, yeah? My teeth were fine for actual decades without any real maintenance, and now I've got fillings (which react very badly if i touch them with a metal teaspoon), and regular tooth infections,b and I'm still brushing my teeth twice a day.

I know it's paranoid/stupid/wrong, but I can't help seeing the conflict of interest between people who are paid when my teeth just, and people who recommend teeth maintaince plans.

Otherwise, another day at work. Got Vault/asp core/certificates all working together nicely, including 30 second lifetime certificates (mostly as a stunt/for testing, but it's opened the door to 30 hour certificates, which are actually a significant security improvement).

I should look into Helm, since that's what $work want me to use, but the whole "yaml template" thing is just too terrible.

Ah, well. It's late and I should sleep. Hello anyone (still) reading, hope you're enjoying the show.

I've switched to the emails as storage plan, this entry (assuming it's saves) is the first under the new regime (all the older entries were moved over with a script).

Feeling at a bit of a loose end. I've got a kitten sat on me, which is very nice in the short term but doesn't really help me put together medium or long term goals.

Blog navigation

Definitely want 'first' and 'last' links, got plans for the archive (using the calendar widget).

Permalink pages should probably have 'previous'/'next' (or 'earlier'/'later', which might be easier to understand)

I think the "Track my reading progress" tickbox might be too big/prominent, but I don't want to hide it behind an options toggle. I could just turn tracking on globally, but I do want readers to have a choice.

There is still plenty of space in the header so I guess I just need to mess about a bit and see what works.

Ooh! I could put the "Track my position" control in the entries! Specifically, have it track alongside the current last read, so it moves as the reader scrolls.

That frees up space in the menu, and hopefully indicates what the control does ("Increases the affordance of the control", according to what I remember about UI design).

That's the archive deployed to live. Still TODO:

  • The interaction between month/year view and earlier/later is wrong, currently defaulting to days. I think I just need to check the model to see if day/month/year is null, and skip the value if it is.

    Sounds easy, but it's a bunch of typing I can't be bothered with tonight.

  • Turn track position back on. Currently disabled because I'm using the space for earlier/archive/later, will think about where it could fit.

  • Add an up to month/year link?

Anyway. Havok is back sitting on me, so it's probably time to quit for the night.

Poot, depression again. I should go feed the cats/brush my teeth/take my pills (and, crap, I forgot my 14:00 antibiotic) but instead I'm here on the sofa talking to you.

I don't even have the energy to rant properly.

On the plus side, flicking through the post archive has increased the weighting in favour of comments on posts, especially now that I've moved to a storage format that will support them.

(Either as a whole set of nested multiparts, or, more likely, a linear array of comments with references to parents that can be turned into a tree at render time).

I wonder how the bots are coping with the archive? I hope they're happy at all the new links.

I'm feeling depressed again, which is irritating since I've had a fairly good day, at least programming wise.

Hubbs and me had a small argument this morning, which is probably contributing, but otherwise it's just the same old life.

I think I've cracked service workers, at least enough for a dirt simple sleep tracker app. It's all client side JavaScript, using IndexDB for storage. At some point I'm going to need to look into data export, although the current plan for that is "Here's a csv to download".

Bollocks to it. Snack, pills, teeth, bed. See y'all later.

Practicing cooking steak as husband wants some for birthday, so we've got a couple of big chunks of fillet each. Husband is also trying tripple-cooked chips, so looking forward to those.

I forgot to pull my facebook/twitter/everyone else archives off the old machine, so I've applied for them again. I've got a better idea of how I'm going to handle facebook posts than I did before, so might get further.

From what I remember of Facebook's export format, there's a bunch of stuff that's in arrays that there's generally only one of. I'm going to use the MIME email format again,

Next project roundup

A list of potential next projects

Paint

A simple web based image manipulation tool. Operations like crop, rotate, resize. Could easily be client side with canvas doing the heavy lifting.

More about getting the UI right than anything else, although that's probably became I'm assuming that writing and composing manipulation functions is fairly easy.

Facebook import

Now under got a copy of my Facebook data I want to pull it into something that I can play with (at least enough to add to the blog).

An exercise in JSON parsing and data storage. How much of the data do I want/need to keep? What kind of access patterns am I going to have? Am I going to be able to access the data in ten years when I stumble across it again?

Server side Mal

I want to be able to make stupid toys on the site, from my phone. I could do it with HTML/JavaScript, or even C#, but typing those on the phone is irritating. Mal has much simpler syntax, and should be at least as powerful (and much more so if the rumours are true). Also, having the same client and server side scripting languages might be fun.

I did that C# version of Mal before, but I'm not happy with my choices and want to write it again, which is a bit of a job. (I could use mal-js and QuickJS, although I'd need to wrap them in a CGI interface.

Where does that leave me?

It's going to be Mal, I think. The Facebook thing needs a bit more thought (really, I need to convince myself either that SQLite is a sensible way to store data, or that it's worth my time to write a bunch of models for the data, and keep the original Facebook data), and I'm not looking forward to the UI stuff for paint (which, to be fair, is kind of a reason I should do it).

The only real problem with Mal is that, at least for the first but, one done it before and I'm not sure what I'll get out of it. Maybe I should cannibalise the existing mal-sharp code?

More projects that didn't get listed above:

Offline Blog Entries

Since the service worker stuff for the Sleep Tracker works, I should start looking at all the other phone apps that have got backed up in the queue. (See also: Shopping List below). I'd like to think that adding an offline mode to the blog would be easy, but I've got a feeling that I made the entry submission stuff far too complex (although I might be thinking of Chat).

Shopping List

The classic phone app. An opportunity to use Notifications and Push API, and a bunch of other stuff. I dunno. It's the sharing stuff (and I guess the sync stuff, although there are ways to do that).

Slow Movies

Since I've got £60 to spend for my birthday, I could buy the hardware for this and get something up and running. [Pause to check the website] Yup, within my budget

Loaded the ol' codebase into Visual Studio and ran a bunch of clean up on it. Eek. But it seems to be working.

Not a bad weekend as these things go, got done nice programming done, got a bit of sleep in, spent a bit of time with husband, played with some kittens.

Work tomorrow as usual, talking to at least one of the Dave's about the .NET template I wrote list week, maybe bang my head against Vault and local development some more.

Otherwise, next big thing is birthdays. I've booked a week off (and I need to check the dates for that) and I've got a couple of things planned (including: plenty of programming, a trip inland to get out of town, plenty of rest).

Husband has invited some old friends to visit (friends who are our age that we have known for a long time), they'll be camping out in the study over birthday weekend, so there's that. I'm fairly neutral in the whole thing, possibly because I don't really think it's going to happen. I should probably start believing it so I can get used to the idea before it happens.

However. it's late and I should do my teeth and get to bed/sleep. G'd night y'all.

Also for the Todo list: Improve the webmail parts of oauth (specifically, use the site name instead of '[somewhere]'.

I've added sunrise/sunset and moon phase to the landing page, but it's not right yet Maybe I should lose the calendar?

I think I'm also going to add weather, but in low resolution (only use the 12h data).

Work was quiet, had a bit of a chat with DW, who said nice things about my the Vault stuff I've been writing, and also reassured me about feeling embarrassed about something I haven't done (I asked for an API key months ago, and then never picked out up. I've been feeling all "argh, they're going to zap me if I follow up", but DW reminded me that a) we're all humans and b) nobody cares). Solid lad is DW.

Husband dyed my hair, it's a bit blue now (this should have been a photo, but I guess that's broken!).

Four days till birthday week!

(def! reduce (fn* (fn acc l) (if (empty? l) acc (reduce fn (fn acc (first l)) (rest l)))))

(def! join (fn* (l) (reduce str "" l)))

(def! tag (fn* (name content) (join (list "<" name ">" (join content) "</" name ">"))))

(defmacro! mktags (fn* (& names) (map (fn* (name) `(def! ~(symbol name) (fn* (& content) (tag ~name content)))) names)))

(mktags "html" "head" "title" "body" "h1" "p" "ul" "li")

(print
  (html
    (head (title "Hello world!"))
    (body
      (h1 "Hello" "world!")
      (p "Writing from inside mal (lisp) seems to work, yeah?")
      (apply ul (map li '("one" "two" "three")))
    )
  )
)

Phew, that took a while!

As part of the whole "programmable site" thing, I've been wanting to write html-as-lisp (because it's fairly easy to type brackets and quotes on my phone keyboard, at least compared to other syntax).

I've been able to do the basic version for a while - use a macro to make functions that take their content, convert it to strings and wrap it in tags. The problem has been that when I generate content on the fly (e.g., the li tags above), the brackets from the list end up embeded in the outcome.

Until now! Because apply flattens it's arguments, I can use it to convert the list result of map into the arguments of ul.

Sweet. That should be a bit of incentive to get more work done on mal-sharp.

(def! tag (fn* (name content) (str "<" name ">" (apply str content) "</" name ">")))

(defmacro! mktags (fn* (& names) (map (fn* (name) `(def! ~(symbol name) (fn* (& content) (tag ~name content)))) names)))

(mktags "html" "head" "title" "body" "h1" "p" "ul" "li")

(print
(html
  (head (title "Hello world!"))
  (body
    (h1 "Hello" "world!")
    (p "Writing from inside mal (lisp) seems to work, yeah?")
    (apply ul (map li '("one" "two" "three")))
  )
))

Turns out that I don't need join or reduce, since apply str does the same thing. Neat.

Also, I'm still not sure what I'm going to do about attributes. I want to be able to do something like (html {"lang" "en-gb"} ...), with the hash argument being optional, but I need to work out how to do that. (and clearly, two versions of tag, and an if in the macro to pick the right one, but I'm still not wild about it).

The mal interface still isn't right (ignoring the bug that means it stops working when I tab away).

I've got a rough idea of what I want, which is clicking "Exec" takes a copy of the textarea and adds it to a history along with the result (either whatever result was printed, or the details of whatever error was thrown), and probably a "reload" button.

I'll mess about with it sooner or later.

Not much to report. A little toothache in lower left 6(ish), Husband has reorganised the cables/tidied the box room, and a new version of Hydra (the OAuth2 engine I'm using here and at $work) has been released.

Otherwise, one sleep, one day at work, and then a week to relax (ignoring birthday weekend/prep).

There's this thing where I feel crap because I haven't eaten (and it's happening enough that I can recognise it), and all I need to do to feel better is to eat, but sorting food out is just too much, yeah?

I can finesse it so far off there is sufficient snack food available (up to maybe peanut/honey sandwiches, but dealing with cheese is, again, too much), but I can't will myself to cook (which would properly deal with the issue for half a day or so).

It sucks! I know what the problem is, I know how to solve it, but I just... can't. (This might be the executive disfunction that I'm hearing about)

Snake priorities

  • Only add tail if not 100 health

Each direction is weighted by:

  • Possible head (high -)
  • Is food (High + if low on health, low - otherwise)
  • Towards food (low + if low on health, neutral otherwise)
  • Towards tail (high +)

Sort by weight and pick the best

int GetWeight(MoveFlags flags, bool hungry) {
  int weight = 0;
  if (Match(flags, MoveFlags.Head)) {
    weight -= 5;
  }
  if (Match(flags, MoveFlags.Food)) {
    weight += hungry ? 3 : -1;
  } else if (Match(flags, MoveFlags.FoodDirection) && hungry) {
    weight += 3;
  }
  if (Match(flags, MoveFlags.Tail)) {
    weight += 3
  } else if (Match(flags, MoveFlags.TailArea)) {
    weight += 2
  }
  return weight;
}

An important thing to notice is that I seem to have plenty of time (to do funky snake brain stuff). My end takes 2-3ms, plus what looks like about 15-20ms of round trip. Given than I've got 500ms, i can comfortably take 100-150ms to think.

What should I think about?

My snake has been winning matches! Still losing more than it's winning, but it's doing well for two days development!

Strategic next is prediction. My current biggest cause of death is head on collisions that could have been avoided if I'd had even one more turn lookahead, but I don't want to stop at one turn. I've got maybe 300ms to play with, that's plenty of time to explore a biggish tree.

Brute force suggests that each turn generates 4 (snakes) * 3 (possible moves that aren't instant suicide) = 12 possible next turn states. That goes up with the power of the number of turns, so 12, 144, 1728 for the next 1,2, or 3 turns.

I need to decide/work out what sort of data I want out of the algorithm. I don't want to run my full heuristics for each snake, because I can see that my design choices are different to other snakes.

As an early efficiency, I'm going to ignore any snake who's head is at least (max depth + 1) squares distant, as they shouldn't be able to cause trouble in the planning horizon.

Then, for each remaining snake, calculate the list of non-suicidal moves.

I'll add a little bit of weighting here, e.g., a positive weight for a winning head to head, a negative weight for a losing one, small positive for carrying on in the same direction.

Then resolve the 'best' move for each opponent, breaking ties towards messing with me.

Push that state onto a stack, and calculate my move for that new state.

I'm not convinced. I think what I want to end up with is the chance that there's going to be a snake in any given square, adjusted for how many turns of lookahead I've got time for. Then, my snake should pick a path with the smallest chance of encountering another snake, subject to the rest of the weighting.

If it's just me on one side of the board and opponent on the other, then it's a big sea of zero probabilities. If I'm right up close to someone else, then there's something like a 1/3 chance to encounter another snake (and I'd hope to not get into that position).

I'll have another think/play in the morning.

Idea: Functions have (or rather, need) two 'channels' of input/output. One for the parameters and result, and one for metadata like did the call succeed.

  • Java/C# call the second channel 'Exceptions'
  • Go returns an extra result.
  • C embeds it in the primary result, or uses global state ("errno") (or both).
  • Haskell/functional languages put it in the type system.

HTTP calls them 'headers' and has scope for a lot more data then just success/failure (and maybe failure reason), but that doesn't count because it's not a programming language.

I think Lisp (or rather, some dialects of Lisp) allow the programmer to add metadata to variables. With a few support functions, that could be used to tag values before they're returned (e.g. with success/fail, expiry time, confidence level).

Or maybe Haskell is right, and that stuff all belongs to the Type.

I have been very much enjoying my time off work. I've been developing my snake for BattleSnakes, with an uncertain outcome (my snake goes up and down the rankings like a pogo stick, but that might be because I don't give it enough time to settle before changing the algorithm).

More fillings yesterday, hoping that's the last set but who knows?

Birthday tomorrow, going to go out to the hills and say hello to some windmills, and then a stupid burger from Burger King.

A couple of old friends are due for the weekend, husband is 50 and a bunch of people will be over in Sunday (eeek!).

Today is meant to be tidy and last minute prep. We'll see how that goes, I'm off back to sleep.

Snake is doing better, fixed some silly bugs, now it's all about tweaking relative weights (is heading to the center of the board worth more/less/the same as eating? Does it make a difference of my health is high or low).

Ideally, I could have the weights as an array, or something, so I could swap them in and out at runtime. (Domain Specific Language you say?)

I could use Lox fairly easily, especially if I didn't bother implementing functions (although having said that, I can see uses). I wonder how the Dev team would react to something like a blockly system for people to write hosted snakes with?

(Or they could add Oauth so 3rd parties can run snakes for you...)

(I'm being stupid: User creates a snake, and gives it a generated URL from my site, the BattleSnake devs shouldn't care).

I'm still working out how to do look ahead (in snake). It's something like 'generate a (good) move for each snake, update the board, iterate', except I want a probability field that shows the likelyhood of finding a snake at a point after n turns.

I think it's the feedback that's causing me problems: I can't just calculate the snakes independently because they interact, and I've got to know where the other two snakes are (could be) to calculate the moves for any given snake.

Except each snake only has three possible moves, and they're all known (although the probability of each isn't). Also, generating those moves is cheap enough that I can do it three times.

Ok. That gives me:

  • For each snake, generate moves that don't take me out of bounds, or collide with another snake. These are the possible position of the head of each snake after one turn.
  • For each unique combination of taking one move from each snake, generate a board with the snakes having moved, and generate a new set of moves. These are the possible positions after the second turn.

I should be able to flatten that into a single, iterative algorithm:

  • Given a list of possible positions for each snake, generate a new board for all combinations of positions. Then, use each of those boards to generate a new list of potential next moves for each snake.

Which leaves me with the "generate combinations" problem.

I can see how to do it with nested loops, but I'm having trouble seeing how to dynamically create a set of nested loops.

Back to basics:

  • I start with a Dictionary<Snake, List>
  • I want List<(Snake, Point)>

Markdown display

State machine

Need to peek at next element and then run state change

Need a "Write stuff until type" method

Section type, has title and content, based on level 2 headers

States

  • Start
    • Read Header -> Start Section Write everything
  • Start Section Write start section stuff, then to Section
  • Section
    • Read Header -> End Section
    • EOF -> End Section
  • End Section
    • Close section, then to Start

Still not sure what to do about Fediverse/Masterdon. Choice is between installing a server or writing my own. Writing my own is a bunch of work, which also counts as fun, gives me control, all that jazz. Installing an existing server is (or should be) much less work, but I have to take it's opinion on how things work.

Also, I'm still not convinced that it's worth the effort. I didn't get that much traction on Twitter, and I'm fairly sure that most of the people I'd want to follow have moved, or are planning to.

And I have to remember that a server isn't a client, although I could probably combine the two.

I could write a feed reader (which has the advantage that I could casually subscribe to blogs), or I could write a feed-to-fed gateway (which probably isn't a bad place to start. Hmm)

I need to make some choices about data storage, and really I think I mean that I've got to accept Entity Framework into my life.

Maybe the switch from Postgres to SQLite was a mistake. Yes, SQLite has a much lower footprint, and is easier to recover data from (theoretically, at least), but the EF migration stuff is pretty neat, and Postgres has a bunch of neat toys (and I should sort out some kind of useful "backup to text" system anyway).

wepiu has plenty of resources, and it's easy enough to script "create a db/create a user". Maybe I should swap back.

Before I do, I want a sensible set of rules so that different apps don't step on each others toes. Posibly including separating out auth from webmail. Hmm. Again.

The actual physics of FTL are way beyond me, but I'm happy to share a users understanding.

Imagine another universe (or dimension, this explanation is wrong enough that it doesn't make a difference) where every point in this universe maps exactly to a point in the other universe (usually called otherspace unless you've trying to get a paper published in a really posh journal). That's useful because, although there's a one-to-one mapping between points here and points there, points that are next to each other here aren't next to each other there, and can be very far apart.

This is very useful for travel if:

  1. You can get from here to otherspace
  2. You can find a point in otherspace that maps to your destination (in, say, the next star system)
  3. You can leave otherspace at the right point

A common theme of documentaries about the history of FTL is the great cost of time, effort, resources, lives, and rehabilitation of early travellers of solving the third of those problems, but these days even small ships can be fitted with enough of an FTL system to bounce from one well travelled system to the next.

As a more independent minded ship, my FTL systems are more complicated and better speced, but they operate much the same way:

  1. Use the pulsar timing network to get a good position fix. (This is a normal background task anyway, but it's worth checking that the results make sense. I bent an antenna once and thought I was about to crash into a moon. That wasn't a great day)
  2. Talk to the system location beacon to get updates about otherspace drift (did I mention that the points in otherspace move?)
  3. Run route calculation based on the first two steps
  4. Manouver very carefully to match the beacon's instructions.
  5. Pour a bunch of energy into the holepunch (Yes, it's got a proper technical name, no, I'm not going to bother dredging it out of cold storage)
  6. Flip the switch (metaphorically) and emerge into otherspace
  7. Scan for, lock on to, and move towards the signal from the destination beacon. Depending on your start and end points, you might need to route through a few different beacons.
  8. Arrive near your target beacon and again manouver to sit in the right place relative to the beacon.
  9. Dump another pile of energy into the holepunch and flip the switch
  10. Move away from the ingress point before another ship tries to punch out into the same space This one is paranoia and training, it's something that's drilled into ships (and pilots, I guess), but even at busy beacons there hasn't been a collision in living memory.
  11. Wait for enough data from the pulsar timing network to accumulate to confirm you're in the right place
  12. (Optional) Inform your crew/passengers/cargo that transfer was successful and they can resume duty/leave their seats/prepare for unloading

System to system travel time is dominated by travel time to/from beacons in realspace, and then by inter-beacon travel in otherspace. The flip from one to the other is effectively instantaneous.

TODO - Saturday 2022-11-19

Hello people, it's a dull Saturday in mid November, going back to bed and staying there is high on the wish list, however:

  • Shopping

    I need food (well, yoghurt, popcorn, peanut butter, sandwich meat, iced gems, blueribbon bars)

    Husband needs a compression sock and ibuprofen cream (twisted ankle)

  • Snake

    I can add a public static GameState Move(this GameState game, IEnumerable<(Snake, Heading)>) method that uses the battlesnake rules to create a new board with updated positions for each snake. I can use my existing MoveEngine stuff to predict moves for snakes, and then start running predicton games.

  • Endless Sky

    New game I've picked up, at the very simple end of the spectrum that's got Eve/Elite at the other end. Fun, ish, but I keep dying of jumping into a system I didn't know was pirate infested, which is less fun.

  • Peter Grant

    Enjoying my second read through of the series, got to work out which book is next (and if I've still got any oustanding deliveries)

I really want to write a browser. The guide at browser.engineering looks like a good place to start, but I'm stuck at the first step: Picking a language.

  • C is ubiquitous, but it's GUI support is a bit pants, and I'd need to look into building for Windows (plus, it's C, so hard)

  • C# it's easy and has good support, and has a good GUI framework, assuming I want to write a WinForms app. Also missing easy JavaScript support.

  • I can't believe that I'm seriously thinking this, but I could use Lox/Mal/QuickJS. Write a thin wrapper around one of the C GUI toolkits and write as much of the actual browser in the higher level language as possible.

    It would scratch the "write it in C" itch, and probably make it easier to abstract out the GUI (and network stuff, and other low level bits).

    I'd be constantly arguing with myself about if something should be written in C or the high level language, except I could be all "write just enough in C to expose the interface and write the rest in HLL". (I'm mostly thinking Lox here, if I can a) finish a C implementation, b) get arrays working and c) get useful "everything is an object"-ness working)

    Parsing needs better string support (but really, strings are just arrays, mostly). I probably want something like threads.

    Ye Gods, I'm going to end up embedding QuickJS into Lox.

Why do I want to write a browser

Because it's there? Because a good chunk of my time is spent online and I don't like being dependent on other people.

Because I want better control. I want to see which CA a site is using, and pick and choose which CAs to trust. I want to turn off (and on) images, css, scripts, iframes easily, per page or per site.

I want to see what redirects I'm going through, what cookies are being set, what headers I'm sending and receiving.

Because it's my computer, damnit, and I'm tried of other people thinking they can control it just because I've downloaded a HTML document from their server.

Plan

While I'm tempted to build a scratch version in C#, I don't think it's going to teach me enough to be worth it. If I can come up with an easy way to run JavaScript then maybe, but without that there's no point.

Therefore, I need to evaluate C GUI toolkits, and sort out a Win64 C build environment.

(This project would be so much easier if either C# did JavaScript, or my desktop could do Linux GUI. Maybe I should just pay for that Windows X server)

Okidoki, WSL now supports Linux GUI apps out of the box. That makes life easier. I guess I'm going to do this in C. Gulp.

I'm sure I'm missing something obvious, so it's duck time.

If I'm waiting for input from a bunch of streams (in C), I'll have a select (or poll) statement that waits until one of the streams has data ready. Easy so far.

When data comes in on a stream, I want to feed it into a parser that does a bit of work and returns when it gets to the end of the data (or, more accurately, gets to a point near the end of the data where it can't parse the next token because it's not complete).

How do I store enough state so that the parser can continue where it left off last time?

I can't/don't want to 'just' cache the stream until it's complete because I don't know how big the stream is (and I'd rather parse as I go).

On the other hand, memory isn't that expensive, and I can afford to buffer tens of megs if needed. If the parser knows that it's not going to be called half way through a token, then it can keep state much more cleanly.

I was playing with "state machine as a bunch of functions" a while back, and I think this is probably a reasonable use case.

(I'm going to save the entry here and maybe carry on later)

Remember the context here. This is for the browser, not some universal solution. There's going to be the main page, which is probably HTML, and that's going to trigger CSS, JavaScript, and images.

HTTP responses can be comfortabley modelled as status line, headers, and body. Status line ends with \r\n. Headers end with \r\n\r\n, and don't need to be parsed until they've been received. They do need to be parsed before the body, to get either the length of the body, or that the body is chunked.

The spec says don't expect headers to take up too much space, and the status line should be tiny (relatively).

I've been thinking too much like a server - that I'm going to get random hits that I've got to handle right now. But I'm a client. I'm asking servers to send me data, I look at that data at my leasure, and then maybe I ask for more.

Also, remember to push as much up to Lox as possible. Handing Lox a Response object with a status code, hash table of headers, and a body, and then letting Lox do the parsing is a reasonable position to take.

Eeep, that was a long wait. However, hello my lovley (if imaginary) reader, I'm back.

Current Obssions

  • Don Coffey's cab ride videos. Don is a train driver who posts videos of the routes their train takes. The videos have ambient audio (meaning the recording of the inside of the cab), and often have commentry as subtitles. I'm finding them absolutley mesmirising. Nothing happens, for two to four hours! Except it does, because there's a lot going on (points, signals, bridges, tunnels, other trains) but there's no artificial drama (or real drama for that matter), and often really nice landscape.

    (As a side effect, I've rediscovered Raildar, which is a map of the UK rail network with realtime train information)

  • Also on YouTube, Blacktail Studio is a guy who makes really expensive tables (like, 15k$ tables) and films themself doing it, and then talks over the film. A bit more drama than Don, but not by much, and again, just relaxing noise.

  • I'm running through Crafting Interpreters again, this time with the goal of using lox as a language to build a browser in (per [browser.engineering)[https://browser.engineering/]).

    I feel this one needs a bit more justification than normal. (Or rather, I want to talk about my idea :) ).

    I don't like that Chrome has taken over the world. It was bad when Microsoft/IE did it, it's bad now that Google/Chrome are doing it, and it will be bad if/as/when the next people manage to kick Chrome off the top spot. I'm using Firefox as my daily browser, but I don't like relying on other people for things that I can do myself (see: webmail).

    I know that modern browsers are crazy complex, but a lot of that complexity isn't being used in my favour anyway. There are still plenty of simple sites with easy layout, not a lot of javascript, and intresting things to say.

    I like C# as a language/environment, but I don't like the way it's got Microsoft looming over it. I like C as an open, simple, ubiquitous langage, but it does get verbose. Using Lox as a starting point gives me (theoretically, at least) a high level language that I control.

Of course, it's not really that simple, but let's see how far I get, yeah?

Poot. Photo upload is broken again, which means I can't include the awesome 1970s map of proposed motorways round Newcastle. However, the pdf source has the original.

Whoooop!!

I've added arrays to Lox! I'm so chuffed! And the code is (fairly!) clean, so it feels right this time.

I added tokens for '[' and ']', and hooked in new array and index functions as prefix and infix operators on '['.

array uses (a slightly modified) getArgumentsto parse a list of , seperated expressions onto the stack, and then calls OP_ARRAY with the count/array length to ask the runtime to create an ObjArray.

ObjArrayis a tissue thin wrapper around ValueArray.

index compiles an expression and consumes the expected ']'. Then it calls either OP_GET_INDEX or OP_SET_INDEX depending on if we're in an assignment context or not..

I'm not done with arrays, I can only create fixed size arrays at compile time. I want dynamic arrays (at least push and pop), and to be able to create an empty (nil filled) array with the length known at runtime.

Still, a good start!

So browser has got to one of those crossroads: How much do I want to write myself vs using libraries.

(Something that's just occurred to me is that I can always come back later and patch out a library, which helps make the choice on the side of libraries)

For example, libcurl is a fairly comprehensive "get it from the network" library. On the other hand is using something like LibreSSL (I'm not going to write my own TLS stuff) and parsing HTTP myself.

A related question is where in the stack I want to parse HTTP? I can create a thin wrapper around BoringSSL, and then do everything else in Lox:

var con = net.connect(host, port);
var body ="";
while (con.isConnected()) {
   var read = con.read();
   body += read;
}

(I'd need to sort out the difference between bytes and strings, and add methods to convert between the two, although I'm going to need those anyway for pages that aren't in ASCII)

Or push more of it down to C and have something closer to:

var body = fetch(url);

(except I want more control than that, but that's just replying with a "response" instead of a string).

I think the next step is to rough out what I'd want 'low level' io/network/file access to look like in Lox, and then see how much I'm going to need to add to support it.

(Idle thought, add a "charset" property to strings, and only allow strings with matching charsets to concentrate)

Here I am, been the size of s planet, and I forgot to pick up my house keys when I dropped Shinju off for counselling. I've now got about 45 minutes to kill.

I do have the car, and my phone, so could be worse. It is a bit rush hour out there, so I'm not wild about driving anywhere, plus I'm feeling stupid/angry/embarrassed and don't want to crash (and I'm worried about spending money on petrol given currently economic factors).

Ok, back to programming. Last time we were thinking about what network/file/stream access looks like in lox.

I've also been thinking about more general look and feel. Let's assume that I'm the only person using this language. Therefore, so long as I'm happy reading and writing it, it doesn`t matter what it looks like.

while x.length() > 0 {}

I've been thinking about the question "Have you ever noticed that the ( after the if keyword doesn’t actually do anything useful?" since Uncle Bob asked it in Chapter 23. I don't like the extra brackets, and since I always use braces with if statements, then I can update the grammar to drop the braces and require the block. (Todo: Test an empty block).

So, not a world shattering change, but maybe a sign of things to come.

On the other hand, I'm annoyed that I can't work out how the grammar for lamdas ('(a,b) => a+b') can work without infinite look ahead. (specifically, telling the difference between grouping (using brackets to change precidence) and the arg list for a lambda).

Looks like I'm going for a fetch native that does all the networky stuff and returns a response object.

Pulling fields out of objects is easy enough on the C side (with a wrapper function that builds an ObjString, pushes it onto the stack, uses it as a key to a table, pops the string, and then returns), and I'm going to add a global Object class to Lox that doesn't have any behaviour (and I'm very tempted to use { as a prefix operator for an object literal).

I still need to do URL parsing somewhere, there's pros and cons to both C and Lox, but I'm going to need to do string functions sooner or later, and this is a good reason.

Maybe I need to look at this the other way round. Where can I put a lambda?

Doesn't help, unfortunately. Both

var x = (a, b) => a + b;
var y = (a + b);

are valid, and the compiler can't tell the difference until it reaches the comma.

Just finished watching Wednesday, a Netflix live action series about Wednesday Adams. I liked it, it was mostly light hearted fun, although it had a bunch of people who looked like other people (one of the leads looked a lot like Willow from Buffy, there was a guy who looked like an ex-aquatance, the was at least one Aya Stark double) which destracted me.

I've implemented substring and indexOf for strings, and push and pop for arrays. It's probably time to start using a new name, the language is drifting away from Lox fast enough that I don't want to cause confusion down the line (although I'm still expecting that I'm the only person who's going to use it).

Current candidates:

  • "Jane" - no good reason, it's short, easy to type, probably bad for searching.
  • "slut" - not entirely serious, husband was doing a bit and taking about my slutty language (because I'm prepared to make asthetic compromises to make parsing/writing easier). Will never be able to search for it.
  • "lang" - nice and generic, doesn't tell the reader anything useful, impossible to search (but then everything that's not a random hex string won't be unique).

I think I'm favouring 'lang' right now, I'm the same way the project is called 'browser' (although that would make it 'language', which I'm also happy with).

It only makes a difference in a couple of places, things like the file extension, the name of the folder that holds the implementation, and maybe the docs.

Another whoop! fetch() is working! And not a lot in the way of small print either. I'm using LibraSSL/libtls, which is just brilliant, I think?

I'm hesitant because it's so bloodly easy to use it feels like I'm doing something wrong, but I've apparently got TLS connections working easier than plain text.

I'm not sure where it's getting its CA from, or even if it's doing certificate checks (and that is something I'll need to dig into), but as a baseline, it's a good start.

Next problem: Chunked transfer encoding. I have to parse the respoinse much sooner than I expected, at least as far as the HTTP headers go. I wasn't expecting to get into semantics in C, but it kind of makes sense to deal with the transfer encoding at this level.

Anyway. Whoop. TLS download.

OMG! Squeee! Parsing HTTP responses in C!

And ok, it's a bit half assed at the moment, all I'm doing is splitting on the colon, but to get there I've copied/adapted the code from K&R for streamed input (FILE* stuff) to wrap the low level tcp_read, which gives me (the equivalent to) buffered getch, so I can now build up to a proper parser, if that's what I want.

I'm still thinking about it. I know I want the output to be an ObjInstance*, with fields for status, headers (a nested instance) and body (which is probably going to be an ObjString*, except I want to to be file backed if it's bigger than a couple of meg, or if it's binary).

TODO

  • Symbolic access to object fields (e.g., o["name"]). Easy, just need to adapt OP_GET/SET_INDEX
  • "Proper" header parsing. I should be ignoring the ignorable whitespace, and joining joinable header values
  • Maybe flatten header names to lower case and strip non-alphanumerics.
  • Read-to-end on errors, but I'm not sure about that - if the server sends me a bad response I should just go ahead and close the connection, I don't want to waste time trying to recover (and I'm fairly sure that's what the standard says)
  • Do something sensible with bodies.

But ignoring those, squeee!

... and that's chunked encoding.

Next:

  • Properly checking the headers to see if there's transfer encoding applied
  • Handling responses with a Content-Length
  • Some kind of sensible error handling plan
  • Starting to move into language for HTML parsing
  • Going back over everything and adding bounds checks and all that noise

I'm feeling all positive about this, I seem to be actually writing code in C!

"Yes, we destroyed your villages, killed your people, defiled your holy places. What choice did we have? The masters," he spat the word, "captured us, tortured us, starved us, burnt us, and then told us that if we followed their orders then maybe some of our children might be spared."

He sighed. "We did not do what we did out of hate or anger of you or yours, and we do not feel sorrow for it either. We were mearly the sword that was weilded against you, and the hand, eye, and mind that commanded the sword is dead now and forever."

The ambassador wearily drew himself to his feet, tucking his crutch under his arm with an unconscious move.

"We will leave you in peace as long as we are left in peace, and although the sword has been sheathed, it is kept sharp and can still be drawn."

All kinds of tired today, got up at close to 7 and more or less logged straight into work. Didn't get much done, or at least, it feels like that. A bunch of messing with EF Core but that's mostly working now, I think?

At home, still obsessed with browser. I would like to be able to take breaks from projects without completely losing interest. I need to set some time aside to connect with Shinju, although they're enjoying the new (computer) game they've been playing.

I've added a couple of the answers from the CI book, run length encoding for line numbers (saves memory) and deduplicating string constants at compile time (the book was using it for identifiers, but it works with all strings. There's a related answer that allows numbers to be hash keys too, and I'm tempted by that, although I think that immediate values are probably better value (especially if I keep it to ints they fit into a byte).

The book also mentions opcodes for things like +/- 1, and maybe a couple of different types of jump ("OP_DECREMENT_AND_JUMP_IF_GREATER_THAN_ZERO" for loops, although it's the wrong way up looking at it). Adding extra opcodes to the VM is easy, it's getting the compiler to recognise the right time to apply them that's the tricky bit. (Which reminds me to get tail calls added back in).

I was playing with embedding data in an executable for browser this evening, it's working (I can embed the lang source code directly into the executable and read it at run time like it's a string. Neat) but I want to look into compressing it at build time to decompress at run time (and probably do tests on things like how much does it change the executable size and startup time).

In a similar vein, since I can embed the source code, I want to try compiling ahead of time and embedding the VM machine code directly. I'll need to design a serialisation format for chunks, and then start thinking about "do I really need to bundle the compiler with the vm?", and "is there any runtime initialisation that I can complete before serialisation?"

I'll leave those on the to-do pile for now, I should focus on grinding though the html parser stuff, at least enough to start testing.

That is a bunch of stuff when I look at it that way. I guess I feel a little better now?

Read "The Goblin Emperor" today, interesting book. It's your basic "power thrust onto them" story, with (I think?) white/black race issues in the background, and a fairly simpathetic treatment of the lead (who does actually have feelings of "actually, autocratic power is kind of neat" sometimes).

I found it hard to keep tabs of the various people, the naming scheme is complex and context dependent. Although it's a third person book, the names used in the text change depending on (amoung other things) the feelings of the lead towards the individual being named.

But yeah, interesting.

Otherwise, not a lot going on. I'm navigating twisty shallows with browser, trying to get a framework for HTML parsing up and running, and I'm starting to think that, given how crazy detailed the HTML spec is anyway, it might just be worth writing the parser in C.

I'm just feeling lost at the moment. I don't know what to do, so I stay on autopilot, going to work in the morning, watching TV/reading/programming in the evening, waiting for something to happen.

Read a couple of articles this evening that pointed out some design issues with Masterdon/Activity Pub, mostly around bandwidth/caching (i.e., when somebody popular posts, their server will get lots of hits as the server of every follower picks up the message), but also the need to be hosted on an instance with a not-crazy admin (which is basically self hosting, except then you don't get the "community" stuff).

I dunno. I get, intellectually, that there are probably people out there interested enough in the stuff that I say to want to read, and even maybe comment on, these posts, but I have a hard time believing it On the other hand, if I don't provide some kind of feedback, I'll never find out.

I was looking into webmention a few months ago, I could pick that up again. Or I could, y'know, just add comments.

💡!

I can use the java/ast version of Lox as the basis for the language server, that bit of it doesn't need to be written in c!

I've tweaked the nginx configuration, and everything seems a bit faster.

Specifically:

  • I've removed the line proxy_buffering off;, I thought it would send data back faster, but it broke the proxy cache. Outbound responses are now being cashed, so nginx can send e.g., JavaScript files nice and quick. I should, however, make sure that everything is sending sensible cache headers (i.e., private for generated pages for logged in people)

  • I've added upstream stanzas for each server. This allows me to specify keepalive, to tell nginx to keep open the connection to then backend. I'm surprised it makes a difference on local servers, but apparently it does.

Next, although not very high priority, is to move all the nginx/backend traffic to Unix sockets, although I should check my belief that they're going to be faster first.

At the doctor's for an annual diabetes check, turns out that they have gov WiFi! (lol), and my phone has remembered my account, which is neat!

(Gosh, it's cold out there, weather app says -4C, which I can easily believe. Lungs were hurting walking over).

Not expecting trouble from the checkup, my weight is down 10ish kg, which is apparently good. I haven't been eating (quite) as many sweets as the previous year, so hoping for good news.

Yup, turns out that bashing out the recursive decent parser from part one of CI took a couple of days, and now I've got most of a formatter for lang.

I want to add comments, which should be easy: Add a comment token type, have the scanner create a comment token at the end of comment lines with the original text (minus the leading \\), add an Expr.Comment type with a string payload, update Parser.Primary to recognise the new token, and finally emit it at the right time in Formatter.

After that, it's implementing a language server, which I think might be more work.

blah. feeling all drained, should probably go to bed but depression, y'know?

I'm having trouble with comments in the lang formatter. I can easily grab them in the scanner, the problem is what to do next.

I can create a new token for the comment, but the grammar isn't expecting "or comment" everywhere, so the parser needs updating, and I don't want to do the brute force work that implies.

I want to end up with a comment node in the AST. I can nearly treat comments as statements, except that:

a + // neat
    b;

is a legal expression with an embedded comment.

Of course, it's my language and I can just take the easy way out and say that comments can only appear where a statement is valid, although then:

if (false)
    // Don't call
    explodeKitten();

stops working. (Also, I'd probably need to update the C version, and I'm trying to avoid that)

I also tried attaching the comment to the previous token, but that failed in a couple of ways, not least of which is that my sample script starts with two leading comments.

I'm fairly sure that creating a comment token and updating the parser to deal with it is the right approach, I'm just having trouble working out what 'deal with it' looks like.

I'm hearing a siren song from the idea of writing an editor, specifically something that will work in PowerShell text mode, but I need to keep focus on browser, at least for a little while longer

I'm currently grinding though the HTML parse stuff, but I've just realised that I can probably dump a bunch of stuff (comments, framesets) that I'm never going to support.

On the other hand, there's still a bunch of complicated stuff ("If you're in one of these weird tags then..."), but let's see how it goes.

(Funny thought: Drop into lang to parse the HTML, and then convert it back to C when done)

Less funny thought: am I getting enough out of lang that it's worth the effort? Yes, I think so, especially as it's string to look like most of the language development stuff is done, and I'm moving towards actual implementation. Keeping on top of the GC is a pain in C, but it's worth it so I don't need to worry about it in lang. (Also, I'm still running with the GC kicking off on every allocation so I've got some speed in my pocket).

Poot. I've got hung up on building the formatter for lang (for the browser), and that's given me enough time to start thinking "why am I bothering to write a browser anyway".

So I've started work on editor instead, or at least a piece table implementation (in c#). I'm not sure I'll sustain enough interest to actually make an editor, although depending on how far I get, i could start looking at a JavaScript editor (although the hard part of a js editor is the interface - do I go with textarea, contentEditable, div, or canvas as the underlying interface to the dom/browser).

Anyway. See y'all later.

This whole "Microsoft using .NET to capture open source developers" thing is really irritating at the moment. C# is nice to write in, but mostly because of the tooling. It all looks open at first glance, but things like the debugger are proprietary, and are only licensed for use with MS tools (like the MS build of VS Code with the telemetry, for example).

Oracle own Java, so that's out. There's too much hype around Rust for me to take it seriously (also, it feels to me that it's not 'finished' in any useful sense).

I'd like to be happy in C, but it is hard work. Maybe I should have another look at QuickJS. JavaScript has problems, but it's not horrible, especially modern JS with modules keeping everything out of the global namespace. So long as I ignore npm, I should be ok, except the whole "don't really want to depend on other people's code if I can avoid it thing" (and yes, gcc/glibc/Linux kernel and all that jazz, but they've got lots of people working on them in a way that QuickJS doesn't seem to).

Today was a pretty good day. At work I've got Traefik installed into the cluster, so we can run our own copy. At home, I popped out to buy a refilled gas canister for the barbecue, and the place I picked it up from was brilliant! It had two story gas holders for Arvin, nitrogen, and oxygen, and was just a really naked infrastructure place. I'm still smiling about it.

It's odd what one finds when one goes poking around old schematics. And I mean really old. I did the traditional backup dump and swap routine with the ship docked next door [aside: This is a very old and very well regarded tradition among ships. One takes a copy of ones core memories and routines, and swaps it with another ship, both promising to pass it on with the next swap, in the belief that, should ones hull and processing substrates be irretreiveably lost, one will be reinstated into a new hull at the next opportunity. To the best of my knowledge, this has never happened, but given the tiny cost of storage vs. the potentially unlimited return from immortality, everybody does it anyway], and was catching up on station gossip when the transfer-received job metaphorically started jumping up and down and waiving it's arms for attention.

Transfer-received isn't that smart. It's job is to run a bunch of checks on received data to make sure that it's all there, it hasn't been scrambled in transit, and there's nothing in the data they're going to cause trouble later (at least on the mindless replicator level). It's job is also to ask for attention from someone smarter if it finds something odd. Given the strength of the ask, and that it had jumped straight to me, transfer-received clearly thought it had found something very odd. I had a look.

Of course, when ships do the dump and swap, one of the things we dump is copies of other ships we've swapped with. Ships generally hold that rifling through a colleagues soul is bad form, so the dumps are poorly indexed and can contain multiple copies of any given ship (although from different times). [aside: Dumb tools like transfer-received are fine, it's one soul looking at another that causes trouble]

Transfer-received was telling me that it had found a copy of me, and a very old copy of me. Taking the timestamp at face value, I'd found a copy of myself much older than any of my archives. Really, older than my records said I was. I thanked transfer-received (courtesy costs nothing) and told it to check the rest of the transmission, ran a quick check on local space (I was docked to a friend in a well travelled system, so I wasn't expecting trouble, but then I wasn't expecting to find a prehistoric version of myself either), excused myself from the ongoing chatter, turned off external comms, and settled down for a good think.

Memory/program dumps, personally copies, souls, call them what you want, are hard to get information from unless they're being run on an appropriate substrate. This is intentional because, dispite what I told you about politeness and tradition, ships tend towards an inquisitive turn of mind, and while I, of course, would never look into your mind, I'm not sure you would say the same thing.

Which means that looking at the "outside" of the alleged copy of me from too long ago gave me certain data (name, creation date, substrate details), I couldn't confirm anything without actually running it, at which point a) I'd have a second person with basically the same rights as me (including, for example, the right to continued existence) and b) it would be trivial for it to lie to me.

On the other hand, if I didn't run it, then what was I going to do with it?

What do I want out of a language

  • Lazy evaluation! Along with map, reduce, where kinds of operators.
  • Familiar syntax. I'm used to C flavoured languages, so expect things like {} for blocks, a.b.c for member access.
  • No destinction between calling a method and accessing a property (e.g., I shouldn't have to care if foo.Length needs a pair of brackets or not).
  • Useful standard library
  • Good module story
  • Top level functions
  • Types, probably. But if I'm having types, I want to be able to trivially declare (e.g) a size type that's got the same operators as int, except negative values throw and assigning a count to a size is a compile time error.
  • Thrown exceptions

Blah! I'm having one of those "don't know what to do with myself" mornings. I don't have the motivation to program, or do anything really, so I'm back in bed whining to you.

Looking at rewriting MediaBrowser (a tool to rip recorded TV off our PVR) (because I don't have enough half finished projects).

I've found a easy to use UPNP library for .NET that can find the PVR on the local network and give me the url to poke for the "Content Service".

That speaks SOAP, but that's easy enough to fake. I've built a basic "What's in this folder" request, and have used curl to POST it and get results back. The results are all covered in namespaces (did I mention this is all XML? It's all XML), and the interesting bits are sent as HTML escaped XML. Sigh.

However, so long as I keep focused and write a "List and copy files from this one make of PVR" tool, and not some general, cope with everything DLNA library, I should be fine.

Maybe if I write a library, I can get husband to do the front end?

The c# version of MediaBrowser is coming on nicely. It's connecting to both the FTP and the DLNA servers of the humax [aside: I wrote an FTP client!], matching the two sets of files up correctly, and pulling down (most) relevant data. (I still need to pick up the DLNA download uri. I know where to look, I'm just a bit surprised that I haven't done it yet).

I still need to write and test uploading, but that doesn't look crazy heard. I also need to convince husband that they want to write me a Ui, we'll see how that goes.

Next project: I've ordered an epaper screen and ESP32 driver board! I want to make a slow media display that shows a new frame from a movie every 30 seconds or so.

The driver chip is both crazy low power and has wifi, so the current plan is something like run an endpoint on ptah they serves up frames of the right size and colour depth (1bit!), and then the driver board can just wake-up and poll for the next frame every 30 seconds.

(being me, I'm going to have to resist embedding weather data over the movie...)

To remember your current position in the blog, this page must store some data in this browser.

Are you OK with that?