The problem is that every time I create a new blog tool, I think that it's going to be the one platform I can integrate everything into - calories, lights, what's been playing on Spotify, what I've bought from Amazon, all the things - and that turns out to be hard (because getting the data is tricky, integrating it is tricky, and the privacy issues are tricky) so I stop bothering with the blog.

Either that, or something happens to the code and I can't get the entries out of the database any more.

At least this time the entries are stored as plain text files. I also like not needing to add a title. Thanks micro.blog/indiweb, you've got good ideas.

I do still want the "edit the site from the site" experience. I very much miss the site I had when I was doing BT support which was exactly (and really only) that - a textarea with three code for the page inside. (Written in Perl, I think).

Editing through (something like) GitLab IDE is not the same thing. Ignoring that it needs a whole other service (I could self host), it's the commit-build-deploy pattern that's wrong. I want to click "save" and see my changes when the page loads.

Trick is, these days, to pick a language where that works well.

Blog improvement ideas

Or A TODO List

  • Parse entries as markdown. Probably should do that at save time and cache the HTML (as well as the original file). Question: How far should I take it? Other than the header (which changes depending in login status), the site is basically static, so could be rendered at entry save time (and just do two, one for auth'd and one not).

  • Client side markdown preview

  • Spell check

  • Sort options. Allow users to have oldest top or newest top. Which leads into...

  • User options. Stash some stuff client side (sort order, last read). Could do that all with JS and local storage.

  • File upload, for images. Level two is basic image editing - at least crop, resize, rotate. (This is the "Image editor as language/stack of operations" idea again)

  • Maybe that microblog protocol. Not sure, can't decide if that's just a gimmick

  • Editing posts. Probably keep old versions, but then need to be able to view old versions (and can you wait an old version so the history is a tree?(

  • Link to (and show only) a specific post

  • Tags (maybe, and they must be optional).

  • Search. Level one is just match strings. Level 2 is stemming, stopwords, maybe that vector thing.

  • Syndication. (Cross post to Facebook/Twitter automatically)

  • Comments (maybe)

Don't know where to stash metadata for posts.

I can think of two top level options (in the same file as the post, or in a different file) and a couple of sub-options (same file with a metadata header/footer, same file json encoded).

I currently prefer same file with a header, but I'll need to play with the dotnet markdown parser before i commit.

(I need to play with the markdown parser anyway)

Training again, Advanced Development with/on AWS. Flippin' hub is broken again so only got one screen, at least untill I've got time to reboot.

Hub wasn't broken, I forgot that I need to plug it into a USB3 socket (which was on the other side of the laptop).

Just read "The darkness behind your eyes" used to refer to the self. I love it. (From https://qntm.org/streetmentioner)

Big idea: Use email/IMAP as the storage backend. Gives me arbitrary metadata (headers), text file in disk storage, universal tools. I can have posts in draft, if I hook up an SMTP server as well, I can have easy bot apis.

Using mime/mulipart I can store the original post, attached images, html rendering all together.

I'll need to import current posts, but I'm going to have to do that if I want to update my on disk format.

I'm probably going to need a second copy of dovecot/whatever IMAP server, I don't want to get my actual email mixed up with this stuff (except check out namespaces maybe?)

Why aren't I more excited about this? Could be the heat (Local weather be crazy, check news archives).

(Spoiler: It's not the heat. It's the standard "This technical wizardry isn't going to make me popular/better at taking to people/any less lonely" problem)

(Y'know, depression)

Good morning. That was a rubbish nights sleep, and I'm feeling knackered this morning. Ah, well.

Got training, hopefully I can stay awake for it.

The blog buttons are in the wrong order, I don't like post showing up in the middle.

Also, I've decided that the button style for nav links is wrong. I'm not sure about the whole colour scheme either. Gah, maybe I'm just feeling rubbish this morning and need stuff to complain about? On the other hand, I should design the site so that I'm happy looking at it when I'm depressed, so I keep using it when I'm depressed.

Microservices change explicit complexity (in the code) to implicit complexity (in the communication between components)

That's "Remember where you were" more or less done, including proper consent and everything. Only thing it needs is to fix the scroll to so that the end of the last entry read is at the bottom of the page.

Hmm. Scroll to latest isn't interacting well with location.hash being set at the same time. Let's try checking for a hash at scroll time, and not moving if there is one.

Hehehehe, that's so frickin' cool! It's been on the wishlist for the blog for ages! Now I can have best of both worlds - all the entries on the page, in date order, and regular visitors don't have to scroll for miles to find their place.

Aint JavaScript brilliant!

Works on my phone too, although the jump is more obvious.

IMAP is not the right solution for backend storage. Could make it work, probably, but it's too immutable. I want to be able to edit entries and them keep their ids (whether or not they keep a change history)

I want a bigger gap between paragraphs as well.

I think "Track latest seen" might be better than "Remember where you are". It's shorter and a little more accurate (since I'm not going to track scrolling back up, right?)

  • Show "You have unread posts" message
  • Maybe fade out latest read highlight after a few seconds
  • Start to think about images

Images:

I can think of five sources of images off the top of my head (although there maybe some overlap): Upload from local device, pull from URL, Google images, local camera, Android intent ("Share image" from another app)

Once we've got an image, it needs to be added to a post. Again, a few options: Just post image, attach to end of post, inline in post.

Given that Markdown has an inline image tag (a link, bit with a leading exclamation mark), I'd like to support that, but getting the filename right (and things like scaling/rotation/crop) is going to be tricky/irritating. (Since this bit is almost certainly going to need JavaScript anyway, I've got a bit of wiggle room to make life easier. Things like an "insert image" button on the edit page that shows an image picker/uploader/editor, generates a link, and inserts it into the textarea)

That's actually sounding like a reasonable plan right now. The dialog tag works well for the consent screen. I'll have a think about what could/should go into the add image dialog.

Android share is out, at least for now. It would work in Chrome, but I don't use Chrome.

Local camera looks doable (verging on easy): Open the camera, link it to a video element, take a snapshot, upload.

Local storage should just be a file upload tag, ideally with a scriptless version that updates to something more fancy if supported. (fancy in this context implies upload progress)

Generic url is easy as well, although since I'd want the server to do the download that probably needs JavaScript for progress/completion notification.

Google photos is it's own problem that I'm going to ignore for now.

Since the dialog stuff needs JavaScript (right? could I do something with the css checkbox trick?)(plus, it's 2022, JavaScript exists, get over it), how much JavaScript do I want to use?

The local camera stuff absolutely depends on JS. Can (probably) do crude local storage upload with no JS, likewise remote url (certainly in terms of post image as separate entry). Again, do I want to? What's the use case?

I'm going to have to do something about getting logged out on reboot (and the something is to move the storage for SmallCookieManager into the database).

Also, ow, my arm. I'm fairly sure it's because I've been laying on it wrong, probably while sleeping on the sofa (or programming in the sofa, on my side with the laptop on the table).

Not sure what my plans for tonight are. (That's not true. My plan is to lay here in the study for a bit, go downstairs later, eat something, and try to get more programming done. I'm not sure I'm happy with the plan, and I'm fairly sure my brain isn't up for programming)(The next thing I've got to do is design work anyway - what elements do I need for an "Insert Image" dialog?)

#Insert Image dialog

  • Image source drop-down
    • Remote URL
    • Local file
    • Local capture
  • Source specific controls
    • URL input + fetch button for remote. (Maybe a fetch local/remote toggle?)
    • Drop zone/file input button for local file
    • List of local cameras and available sizes for local capture, with a way to select the preferred option.
  • Preview display Including image data (width/height/type/filename) and tools (crop, resize, rotate, rename)
  • alt text
  • nsfw flag??
  • Comit button
  • Upload
  • Suggest random filename?
  • Receive allocated filename
  • Check if requested filename is available?

Image backend

Need to receive image data + filename and stash it somewhere (options.uploadPath). Again with the meta data. Would be nice to validate image data at upload time, get things like width/height/type, and stash it so I don't need to get it again.

I'd forgotten about alt tags, and so absolutely need meta data. Why not a file called blah.meta? (Standard reasons are things like: It can get out of sync, or vanish)

Still need to move the post button (clicked "take selfie" this time, but it's not hooked up to anything).

Got plenty of vertical space, can have another row of editor buttons (and so add bold, italic, link, maybe list)

My export from Twitter archive is ready for download, I don't have the right tools to examine it here (on my phone), that will give me something to play with this evening

I'm not being crazy, yeah? Importing posts from Facebook, Twitter, and my old blogs (if I can extract them) is a reasonable thing to do, right?

Dagnabit, I was not able to get out of going to the office this evening. On the plus side I'm now free to sleep till 16.

Back from the office, didn't get much done. Nobody notices when I have unproductive days, I don't know how to feel about that. On the one hand, yeah, great, I don't have people breathing down my neck. On the other hand, what's the flippin' point of turning up at all if no one notices that I've done nothing. Obviously, the point is paying the rent, but it still feels a bit off.

No example yet, but that's image uploading cracked!

(Only using local camera for now, but adding local files should be trivial. I wonder how well remote URL fetch will work under current network security rules. I guess I'll need to add a "can connect to anywhere" option, but then will CORS bite me on the ass?)

Evidence that the upload photo/local camera stuff works.

Alt

Let me also add: OMG! That was so easy! capturing from the camera is a doodle! I think I had more trouble with the progress bar (which seems to be pointless given how quick images are uploading, although that could be because I've got the size set to "flippin' tiny")

Interface could do with work. I think I need to flip the front facing (selfie) camera image. I think I also want to make the image full screen (with a bit of interface overlaid), but both of those feel like they're going to be a bunch of work.

I don't think I'm turning the camera off correctly.

Still need to do other image types, and image editing.

I really like the idea of a (simple) image editing script language. It doesn't need much, could be all one expression with a few primitives and basic maths. But it's late and I'm tired so no more thinking tonight.

I lost another entry while I was at the chippie, so I've added code to stop that happening. It uses the visibilitychange event that notes uses to dump a copy of the entry into local storage when the page says it's not visible (e.g., switched to another tab, or another app) and then reloads it at page load time (remembering to turn this stuff off during form submit).

Just tested it here, works great.

Also, images from local disk works (and on mobile too, which also have me an option to use the camera as a source of images. Glad I didn't see that until after I wrote the camera capture stuff)

Two kittens (Havok on the right, Carnage in the left) sat on a windowsill in bright sunlight

I think images are coming out of the upload too small, I need to work on the ui, probably adding a "this image will actually be 6x as big once it uploads" message. Or moving the upload stuff to it's own page and including the edit stuff. (Which will work much better now that entries are stashed automatically). Not sure how to communicate the image id back to the source page (with a query string, dummy)

Oh, and I'm going to turn off the consent dialog for track latest. I'm not tracking people (since I'm using local storage, I don't even get to see if people are using it, let alone where in the feed they are).

On the other hand, I can't remember why I was going to do that. It might be because I'm not asking about stashing entries in local storage, and they're much more likely to hold interesting data (although I'll be very surprised if anyone other than me (and maybe husband) uses the create entry page, and I've definitely consented)

I do want to collect "which entries have been seen" data, but I'm confident that it will be really disappointing, so I'm currently happy just imagining readers. (Hello, by the way, if you are real and reading this. Googlebot/Bingbot, you both count as real).

I think my immune system is kicking off about something. I wish I had more insight/monitoring of my own body.

I need to update the image uploader do that the original blob gets pushed, regardless of the preview. I also want to save a thumbnail at upload time (to img.tumb) as well as the original.

Since I'm going to be running the image data through some kind of processing library, I can also grab mime type and dimensions and stash them in meta data.

I've also been looking in a bit more detail at the Twitter export, and I'm not sure it's worth the effort. I know in my Facebook posts there's some stuff that I want to keep/share, but Twitter posts are lacking context.

A feature that I've been thinking about for a while is executable code blocks. That is, add s block like

var x=2+3;
print x;

And have "5" output in the blog. (Adding variable input, to get input tags is an obvious extension).

I can think of a few questions:

  • Where should the code run (client or server)
  • What language(s) to handle
  • How much access to the environment should the code have
  • Are all the blocks in one post connected
  • Are all the blocks in the blog connected
  • How are errors handled
  • What does securely look like

I've probably missed some.

A simple solution is raw JavaScript running in the browser. JavaScript doesn't have much of an isolation story, so all the code on the same page (i.e., all the code in the blog) world run in the same context.

Client side JavaScript would be easy to implement, assuming that I can convince Markdig to output script tags at the right time.

JavaScript is a little verbose, and therefore irritating to enter in a phone keyboard.

A bit more complicated would be Lisp, or an implementation of Mal. The same code could run either client or server side, it would only have access to the environment that I gave it (Unlike JavaScript, which would have access to the browser on the client), and it would encourage me to learn Lisp.

I'd need to write a JavaScript implementation (oh noes!), and again, make sure that I could get Markdig to cooperate, but it's an intriguing idea

I'm going to call editing entries a dependency of running code, although I should just build a Mal page and copy stuff over when it's done.

Except what I actually want is to be able to create interactive (or at least scripted) pages here on the client. Notes like this are great for prose, but for tools (current example is kitten age), I want to be able to find them without scrolling through history.

What I want, it turns out, is to be able to tag an entry as 'special', give it a memorable link (or slug) and have it show on a list somewhere. Time for a new button!

The interface side is easy enough (all praise to the dialog tag), add a 'post options' button/dialog with inputs for (probably)

  • Title
  • Slug (auto generated from title by default)
  • Include in TOC (bool)
  • Private (Maybe? I've been in two minds about private posts for a while. Knowing that people are potentially reading this is one of the main incentives for writing it. On the other hand, there's stuff that I don't want to share but that I still want to record.)(bool)

The current plan for metadata is a second file next to the entry, either in JSON (easy for machines to read/write, especially to/from c#, but hard(ish, relatively) for humans to read/write) or some kind of key/value map (e.g. Email headers, or TOML. Harder for machines, a little easier for humans, but I'm not sure how important human readable is for the metadata).

A missing metadata file shouldn't be an issue (I don't want to go back and create empty metadata for all my previous notes), and I want to be able to add new properties easily.

Having said that, bulk creating more or less empty metadata files shouldn't be too hard with a little bash scripting.

Let's also add a metadata version number, maybe in the filename, and an entity format version (more for paranoia than any foreseen problems).

That leaves us with:

  • JSON
  • matching name to entry except different extension
  • Property values must all be nullable, or have a defined default
  • Unknown properties in the file are ignored

Month count calculator

Given dates from, to where from <= to, return the number of whole months between the dates.

Test data: from 2022-04-01 to 2022-05-01 is 1 month from 2022-04-01 to 2022-05-30 is 1 month from 2022-04-01 to 2022-04-30 is 0 months from 2022-04-01 to 2022-04-01 is 0 months

from 2022-01-31 to 2022-02-28 is 1 month ; not a leap year from 2022-01-28 to 2022-02-28 is 1 month from 2022-01-28 to 2022-02-27 is no months

from 2021-12-31 to 2022-01-01 is 0 months from 2021-12-31 to 2022-01-31 is 1 month from 2021-12-31 to 2022-02-28 is 2 months

Algorithm

function daysInMonth(date) {
  switch (date.Month) {
    case 1:
    case 3:
    case 5:
    case 7:
    case 8:
    case 10:
    case 12:
      return 31;
    case 2:
     return ((date.Year % 4 == 0 && date.Year % 100 != 0) || date.Year % 400 == 0) ? 29 : 28;
    default:
       return 30;
  }
}
let months = (to.Year * 12 + to.Month) - (from.Year * 12 + from.Month) - 1;
if (from.DayInMonth >= to.DayInMonth || from.DayInMonth >= daysInMonth(to)) {
  months += 1
  }

That's executable code blocks added. Tag fenced code blocks with 'javascript' and that adds a 'Run Code' button (but only on the single article view).

Something weird is happening with the phone browser and css, in that it seems very reluctant to load a new version, which is a shame because I've just updated a bunch of stuff and I wanted to check it on a proper mobile browser.

I've tried to be more consistent with sizes (I was about to say "so a underlines are the same width as borders" but that's clearly false).

I've increased the gap between paragraphs, and added a hr between entries. Yes, that does use up priceless, irreplaceable, vertical space but it also looks better.

I think there were functional changes in there somewhere as well, but I've got no idea what they were.

Oh, yeah Added an 'edit' link for entries (only visible to logged in people). Doesn't go anywhere at the moment, still need to implement the endpoint, but it's a start.

On Editing Entries

I can't decide about history. Do I need to record past versions, either as straight copies of old files , or as a list of changes?

Option one: Don't keep history

Easy to implement, but feels wrong. What if I want to back out a change (easy, just do another edit) or compare old versions? (And sure, but what if you don't?)

Option two: Archive old files

Keep old versions as files. Need to find somewhere to stash them (in an archive folder, or a folder per entry. How many entries/edits per entry am I expecting?) (Lots, potentially, if I do stuff with this live script malarkey)

Also, and both options two and three need this, what does the UI for the archive look like? Are old versions public? Can I (like, should I be able to) compare arbitrary versions? Restore a version? Build up a tree of versions?

Option three: Keep a stack of changes

Keep the delta from one version to the next. Can either do it forwards (start with an empty file and apply changes until you get the current version) or backwards (start with current and apply changes until you get an empty file).

It's technically fun/interesting, and has an "it uses less storage" excuse, but it would be harder to manage and goes against the "as simple as possible, at least for storage" ideal.

Conclusion

Option two currently leading of the two "keep history" options, but I'm still not sure.

Ok, that's not true. Obviously I have to keep history, I'm just not looking forward to writing the code. (I guess that's the attraction of option three, but I'm getting old and the attraction of simple, easy to write, easy to read, sold code is quite strong)

I'm getting pretty flippin' tired of caching issues. Stupid phone isn't showing an updated version after I've logged in. I guess that's on me, I should probably check the docs, but I can see that I'm just going to give up and write my own cache header rules (instead of using the Microsoft ResponseCache stuff).

Anyway. Bedtime. Goodnight dear reader, sweet dreams, see you tomorrow, probably.

I know I said I was going to need, but one last thing: Add a scroll to latest button back in. Now that I've got JavaScript to tag the latest post, adding a latest id should be easy enough.

Still, suggestions on how to tag the latest article server side are welcome.

(Stop messing about with IAsyncEnumrable and pull all the entries into a list. Or, y'know, do it right and generate the HTML at save time.)

Early morning whine:

  • Bad nights sleep so feeling crap anyway
  • Work priorities have changed so Ocelot is no longer drop everything crash priority, although finding that out has cost so much time that the next project only has (maybe?) a week.
  • Stupid k8s auth broken again so I need to reboot (looks like that hasn't actually fixed anything)

It's all trial trivial, so I'll work on shaking it off.

/me makes an "I've just worked out out" noise.

I want to add a page to edit entries, which will share most, but not all of the HTML/layout of the create entry page I don't want to add a bunch of if statements since that would complicate/untidy the code. I also don't want to make a straight copy because that will increase the maintenance burden.

This is the problem that partial views exist to solve.

I need to update the menu/navbar stuff to stop the current page being a link (and maybe to remove the 'New Entry' link completly from the new entry page.

But decomposing imgDialog into it's own file has worked, that's going to make the edit page much more simple, and much cleaner.

The Lion of Longtrees

Leo is a professional bodyguard. He works usually works medium term contracts (weeks or months) but is open to negotiation, at least until the contract is signed. Once he's signed on, he'll work the contract as agreed until it's completed come hell or high water. This single-mindedness has given him a useful reputation, and a couple of enemies.

He's trained in a couple of styles of unarmed combat (and keeps up regular training). He prefers knives over guns, but knows that ignoring guns would be giving too big an advantage to his opponents. He's a fair shot with a rifle, and ok with a pistol (but is likely to just drop the firearm when it runs out of ammunition, unless he's got a specific reason to keep it).

He's generally friendly and approachable, good with people. Of course, he can put on a blank face when he's working, and will semi-consciously use his bulk to intimidate people out of starting anything with his current client. He keeps up to date with world news and the local sports teams specifically to carry on small talk while scanning the crowd for trouble.

He's had lessons in slight-of-hand tricks, and again practices to keep current. He can entertain and distract a client's child (or a client) with a cup and ball, or pass through airport security with enough of an arsenal to hijack several planes.

Initial scroll to needs to wait for images to load (or I need to add sizes on to images, which is probably the 'right' solution)

I've been thinking about how to add pre- and post- condition checks to (C#) functions.

Roughly, preconditions are conditions that should be true at the start of a function, and post conditions should be true at the function exit. e.g.:

/**
 * Return x divided by y
 */
public int Divide(int x, int y) { ... }

has a precondition that y should not be zero.

It's easy to add these kinds of checks to a function:

public int Divide(int x, int y) {
  if (y == 0) {
    throw new ArgumentException("y must not be zero");
  }
  ...

but a) they're not obviously condition checks and b) it might be nice to be able to turn them off in Release mode. (Potentially also c) they could be extracted for a test suite)

I've been thinking in terms of Attributes, but C# doesn't support Lambdas in Attributes, and probably won't any time soon. However, a new idea arrived today:

public int Divide(int x, int y) {
  Contract.Pre(y, q => q != 0, "y must not be zero");
  ...

With a definition something like

[StackTraceHidden]
[Conditional("Development")
public void Pre<TParam>(TParam value, Func<TParam, bool> test, string onFailMessage) {
  if (!test(value)) {
    throw new ArgumentException(onFailMessage);
  }
}

it would transparently (i.e., not showing in a stack trace) run the test in development mode, and not even be included as a call in Release.

I imagine that it would be worth adding some overrides (or sibilings) for common cases, such as Contract.PreIsNotNull, Contract.IsOfType. (Alternativly, include a bunch of predicates in the library).

Of course, the next question is: What do I do with this idea?

It's clearly a stand alone library. Maybe it's time to get a nuget.org account.

More plumbing/infrastructure work on the blog this evening. Post meta data is now written to a separate (JSON) file next to the entry. This makes it much easier to add stuff to meta (e.g., version number).

I've also swapped out IAsyncEnumerable for Task<IEnumerable>. I was having trouble working out when things (like reading the entry from disk) were actually happening, and worrying that they were happening more than needed.

Instead, BlogService.GetEntriesAsync pulls all the meta files, and then individual entries pull (and cache) content.

Two thoughts from writing that last sentence: Is it worth running the load metas in parallel? And I should look at the proper in memory cache for entries (including content) and do things like invalidate the entry if/when edited.

I should benchmark the load stuff, and maybe set something going at process start time to pre-fill the cache.

(On the other hand, I actually want to get editing working and this is feeling like a distraction)

Editing entries :tick:

Also, accidentally deleted post button. Oops. Fixed that too.

Tidied up the logout code.

I'm an experiencing a level of lower intestinal distress at the moment, my insides are not happy with something.

Otherwise another day at $work. Mostly messing with k8s and remembering to look busy. The deploy script now uses kubectl set image instead of rollout restart. This guarantees that the latest image is pulled, and updating the deployment triggers a restart.

I was arguing for a very aws solution to a problem (messages come in, need to be recorded and aggregated) using queues, lambda, and dynamodb, but we're getting a hit every 45 seconds on average, it's really not worth the effort. Spin up an SQL db and move on to the next thing.

Ah, well. We'll see how that goes.

Added ImageSharp so I can start poking uploaded images. As a smoke test (and to see how working with the Markdig ast works), I'm now adding width/height attributes to img tags.

Otherwise, that was a fairly good day, I think. See y'all tomorrow.

Rubbish day at work, failed both of the projects I wanted to get working and it's not obvious why they didn't work.

Building a simple 'what's going on with my cluster' app but it's not authenticating itself correctly. It looks like I need to give the 'default service account permission to list pods' but I'm holding off on doing that until I feel that I understand what it's doing properly. (Also, I'm not convinced I've got the authority to do that, and I don't want to get bogged down with that).

The other problem is setting up an existing app into the nonprod cluster. It was working there before and now it's not. I have changed a bunch of stuff, so really the next thing to do is to go through the config/setup carefully, step by step, but again, I just can't bring myself to care enough.

That's the problem. No-one on the team really cares any more. I don't get any good feelings from solving these problems because I don't get positive feedback from doing so (and I don't really get negative feedback from not solving them).

I don't like it. I would like to feel more positive about work and works problems, but something fairly big would have to change first. Maybe I should be looking for a new job (although I'd miss seeing how much they'd miss me when I'm gone).

Just finished reading the third Murderbot Diaries "book" (in quotes because it's a short story, but charged at (high) full book prices), and it was as consistently good as the first couple

I've been thinking about showing dates as year-day of year, but I'm not sure. I've just had a "well why don't you go the whole way and u show Unix time!" thought that's not very helpful.

I think it's about showing how far through the year it is. I'll think about that as a separate page, although getting fp ported over is current priority.

If I remember right, I've pulled webmail out into its own project, integrated the shared stuff from common and webmail.content. Time to fire it up and see what's broken.

I'll need to pull a copy of the db from legacy and setup a spare domain for testing.

The more i think about this blog thing, the more I want to stick it into a database and stop messing about. The "keep it in files so that I can recover it for next time" stuff is interesting, and i should probably add an export function, but if I use sqlite i should be able to do a dump anyway.

Actually, that export data idea sounds better every time I think about it, especially after having got data exports from a bunch of different places (Amazon, Netflix, Facebook, Twitter) this week.

However. Today is about webmail, and getting out of Bytemark.

wepiu has the basics installed, but not configured. It's got a vmail user with a nearly empty home dir. It's also got a webmail user, I guess I'm planning on using that one for the webmail?

Ok, so webmail doesn't build. Last time I'd been deleting "stuff I don't need anymore", and I've been a bit over the top. Or possibly not deleted enough. Looks like some of the 'Friends' stuff is still hanging about.

find . -type f -name "*.cs" -exec sed -i 's/using Common/using Webmail/' {} \;

Who says you need an ide!

(Having said that, there are times when they're useful)

Bash arrays

Based on this fine article

arr=() 	Create an empty array
arr=(1 2 3) 	Initialize array
${arr[2]} 	Retrieve third element
${arr[@]} 	Retrieve all elements
${!arr[@]} 	Retrieve array indices
${#arr[@]} 	Calculate array size
arr[0]=3 	Overwrite 1st element
arr+=(4) 	Append value(s)
str=$(ls) 	Save ls output as a string
arr=( $(ls) ) 	Save ls output as an array of files
${arr[@]:s:n} 	Retrieve n elements starting at index s

Note: JavaScript doesn't have a native day of the year function, so I'd have to write one (based on the number of days in the month code above) if I wanted to do anything client side.

(Alright, I'm wrong. Subtract midnight 1st Jan from your date to get ms since the start of the year and divide by (1000*60*60*24) to get days)

Incomplete idea

New webmail idea: Save inbound emails as parsed html as they come in. have a background thread (per user) that's connected and listening for emails.

Listening per user isn't fantastically efficient, but since it's only the two of us if should be fine.

(on a larger scale I'd probably have something on the end of a pipe, but I'm going to ignore that for now)

What do I need to save?

  • Headers (At least from, to, date, subject, maybe type)
  • Structure (From a template based on the content type. Recursive for multiparts)
  • Content (The body and it's various parts)
  • The original message (so I can see the 'raw' version)

Assuming all this stuff is going to also be saved by the mail system, I don't need to be able to round trip it. Folders need to be updated on change.

I should check the stats, but my feeling is that it's probably not worth it. A good chunk of mail is never opened (See: spam management), and updating folders leads to irritating locking issues (that dovecot also has but has already dealt with).

It's a cute idea, but unfortunately not worth it.

Flipin' machines trying to be smart at me! I expected

egrep  '^[a-z]{4,5}$' $WORDS > wordlist.txt

to match 4 or 5 character words that conain only the characters 'a' (U+0061 LATIN SMALL LETTER A) to 'z' (U+007A LATIN SMALL LETTER Z). But nooo, someone is being clever with their locales, and I get words like élan, which, if you missed it, contains 'é' (U+00E9 LATIN SMALL LETTER E WITH ACUTE) (assuming it hasn't been decomposed).

Fix:

LC_ALL=C egrep  '^[a-z]{4,5}$' $WORDS > wordlist.txt

(Roughly translated means 'Ignore anything that happened after about 1963 and just give me ASCII)

(I should probably take the word list with the funky characters, since I'm using it to generate passwords. On the other hand, I think I should probably stick to ASCII, since I'm using it for passwords...)

Sunday morning (yes, the previous entry was also Sunday but I hadn't slept yet so the day had not ticked over)(Or it wasn't dawn yet). Nothing much so far, finished the last/latest Murderbot Diaries book, that was a brilliant series, strongly recommended for people who feel habitual dissociation/disconnection from humans.

Current plans: Grinding towards moving fp mail to mythic, maybe get out pillow shopping, hopefully a quiet/low stress day.

Hugs to y'all.

Dump webmail DB:

sudo -u postgres pg_dump \
	--clean \
	--if-exists \
	--create \
	--schema=auth \
	--schema=webmail \
	--quote-all-identifiers \
	--dbname=moose \
	--file=moose.sql

So I've copied the fp db over to wepiu, not sure what's next. I need/want a 'better' todo/issue tracker, that can track dependencies and maybe priory to show me what's next.

Basic data seems obvious: text, id, dependsOn[], isComplete

Is that enough? Let's take set up db as an example

"setup db", 1, [2] "install db", 2, [3] "Add db repo to sources", 3 "Copy data/schema from old db", 4 "Export old db", 5 "Tidy export", 6 "Import data", 7 "create db user" ...

I'm tempted to add a 'command' property (probably with a 'runAs' sub property) to record/automate the process.

Is it worth it? It would be nice to be able to pick up and put down the process as and when, without losing track of where I am. On the other hand, updating the list is another job (Ignoring the whole "writing the software in the first place" issue)

(And now I want to add 'verify' and 'undo' commands (although commands up the tree/graph should cover lower levels))

Anyway. I think I'm going to config servers before getting the website up, so that I don't need to test the website against live data.

Moved moosemorals.com from bytemark to mythic. There's nothing hosted there yet, but I'm going to use it a the domain for testing webmail.

Mythic are generally great, but they're charging me £15 to bring the domain over. (It counts as a renewal fee, and the time gets added in, but it's still irritating).

I've been going through the postfix config ready to turn the daemon on. Biggest change do far is setting up a separate submission port, so the only mail expected on port 25 is inbound to one of the hosted domains, which is simplifying a bunch of options (e.g., mail that has come in on port 25 that isn't for a hosted domain can be rejected early).

I'll need to do something about outbound mail from webmail. I think the current solution is to just trust the local machine, which is reasonable, but I'd like to do better. (Something like password auth with a one shot password that's validated by postfix calling back to the webmail. Time to look at Lua again, I guess)

To remember your current position in the blog, this page must store some data in this browser.

Are you OK with that?