What's it all about, Alfie?

Well, what a nice development... Today I got an email from Comcast that they are now are removing the 1TB/month transfer limit and allowing accounts like mine to have unlimited transfer per month. This started a little earlier in the lockdown, but I guess they saw it was popular, and so they let it stick.

I, for one, am very glad that I don't have to pay extra for the "Unlimited Transfer" any more. 🙂

UPDATE: I got an email from Comcast about this, and it turns out that they are not quite giving it away... what they are doing is "bundling" the XFi router rental and the Unlimited Bandwidth at roughly a $25/mo savings to me. Well... it's better than nothing, and it's not like I'm going to be getting rid of either anytime soon... so OK. I'll take the $25/mo. It's pizza money. 🙂

As part of the onboarding process at The Shop, today was getting set up with the development tools to run all the Docker containers for an Airplane Mode development set-up on my laptop. Nice... I have always liked that mode - as it limits damage that can be done, and at the same time, allows everyone to develop without stepping on each other's toes.

I've used MongoDB before - back at PEAK6 with the MMD (Magic Middle Dude), and there was a lot ot like about it back in 2011, but they certainly have been making changes since then, and it was fun to get back to working with it - but the first thing was to get a decent command-line client for mongo - and thankfully, Homebrew was there to help!

The steps are pretty easy - given that I didn't want to run mongo on my laptop - Docker was here for all that... I just needed the client:

  $ brew tap mongodb/brew
  $ brew install mongodb-community-shell

and then it's ready to use as mongo. Could not be simpler!

Running it against the Dockerized replica set, that was part of the set-up, wasn't bad:

  $ mongo --host rsetname/host1:27017,host2:27017,host3:27017 \
          --username dev-guy  --password dev-local \
          --authenticationDatabase dev-local

This is just the example of the replica set called rsetname running on the three hosts called host1, host2, and host3 - all on the default mongo port of 27017. The username, password, and authentication database are all simple examples, and this is all easily made into an alias that makes it even easier to start.

I'm looking forward to working with mongoDB again... it's been a while.

Today is the first day at a new Shop, and I have to say... I'm really looking forward to getting back to creating again. Writing code... running it - seeing the results... this is what really fuels my soul. It's what I have always loved about working with electronics and computers. You don't have to wait for the pain to dry... or the adhesive to set... it's there - it's an expression of what's in your mind - and it can be run as soon as you can type it in. 🙂

I learned a lot at the last Shop, but this new opportunity was just too good to pass up. So I didn't. I'm excited for all that this means, and it'll unfold as it should in the days to dome.

One of the Finance Shops I was at happened to have a Post-Trade Management system, and it was built in Java that had all kinds of interesting capabilities to aggregate and plot different levels of aggregations and filtering on the positions, and folks really seemed to like it. One of the key components to that system was a UI toolkit from a consulting shop that was entirely driven on XML configuration files.

I can remember adding a few lines of XML, and adding a very complex dialog box to the app, and thinking - There is no way that's a 1:1 mapping of the config! And I was right. These were heavily built library modules, and adding a few lines really added entire subsystems and connections for those UI elements.

I've also worked at a Dot-Com Shop where the complete build of a machine was described in a single YAML file. This includes all the user accounts, all the software packages, all the configuration for the software packages, and all sitting in a YAML file.

I'm currently looking at a CI/CD Pipeline that is completely specified by a YAML file. This includes shell commands, with options, and variable expansion, and while it's understandable why the developers of each of these systems chose XML, or YAML, as their configuration files - there are loads of parsers that are solid and reliable, and the files can mix in simple data structures, and in those data structures, you can add shell commands and can then do variable expansion... so it makes sense.

What concerns me is that so many developers seem to feel that these configuration files are not nearly as important as the code backing them. That it's easy, safe, and simple to change the configuration and try it again... and maybe for some things it is... but chances are, if your configuration is really your code - then you need to treat it as such.

For example, it is very common to have multiple layers of configuration files. Maybe there's a company level, and overlaid on that is a project level, and overlaid on that is an environment level. These probably get parsed, and then merged one on another to allow certain things to be set at one level, and others at another, and the sum total of all the config data is what's used.

What could go wrong?

Well... imagine if one component of the config is a list of strings that have to be processed in a specific order - let's say top to bottom, as they appear in the file. But then another of the layered config files has another list - with the same name - maybe it's a typo, maybe it's not. How are these merged?

Does the latter stacked file overwrite the lower level? That's one way, but then it's hard to make sure that those lower-level commands are being run/parsed properly. Could lead to duplication in the upper-level files, and that's not really the point of the stacking, is it?

What if you simply append the upper entries to the lower entries? That could be just as bad because the writer of the lower ones may be making assumptions about the state left after the processing of the upper file.

In short - having configuration files store data structures, is fine - and it's useful... but having it include what amounts to executable code means that it needs to be treated like code. And can you imagine writing a function that's layered from multiple files, and then executed at runtime? The difficulty in tracking down errors would more than offset any gains in reuse.

So if you want to have layered configuration files - Great! Just leave it to data that's easily flattened, and tested... but if you're going to have it include executable code - make it simpler - a single layer. You'll be glad you did. 🙂

It's always good to be kind to the "Next Guy" who has to pick up your code. Writing clean code, and making sure it's readable and understandable is very important, but there's really far more to it than that, and I was reminded about that in Slack this morning, and I wanted to write it down so that I wouldn't forget about it.

When a project gets so large that the tests take more than, say 15 min, to run, often times, a developer will disable the tests because it really gets in the way of making progress. This makes perfect sense. But then you get into a problem that the more you do this, the more likely it is that someone will check in code that fails on one of these disabled tests, and then you have a broken build.

Sure, some will say "That's OK... that's what the CI/CD Pipeline is for." which could be true, until the tests are long enough that multiple people are submitting pull requests and they don't want to wait for the full set of tests to run, and so they move on. But then there's a failure, and another team member has more changes that need to go in, and their tests passed, and so the first developer has to let the other dev go through, and then update their pull request based on the new code, and that's frustrating to them.

So now, developers don't want to add tests, because that just makes the situation worse, and now there's less coverage on the code, and less reliability on the system overall, and it just starts to slide downhill. But there's more... if they are waiting too long for tests, they could be skipping writing good docs or commit messages because they know they have a long wait ahead of them, and they aren't thrilled about that.

Take away good tests, complete tests, and good docs, and now we're really starting to erode the quality of the code - simply because things take too long to be comfortable, and it's perfectly understandable. But we each need to push through that initial response, and work to make the tests faster, and the documentation better, and make sure that the "Next Guy" who works in this code gets the best possible codebase they can have.